Pulling and pushing behavior identification method based on improved TSM video classification algorithm
By improving the TSM video classification algorithm and combining CME and IAU-Net modules, the problem of traditional methods struggling to identify multiple-person pulling behavior in complex scenes is solved, achieving higher detection accuracy and lower false positive rate.
Patent Information
- Application Number
- PCT/CN2024/100915
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-09-28
- Filing Date
- 2024-06-24
- Publication Date
- 2026-02-19
AI Technical Summary
Traditional deep learning algorithms and abnormal behavior recognition methods based on skeleton sequences have difficulty accurately identifying pulling behavior between multiple people in complex scenarios, and there are problems of false positives and false negatives.
An improved TSM video classification algorithm is adopted, which extracts motion features and captures contextual relationships by adding a channel-level motion feature coding module (CME) and an interactive aggregation update module (IAU-Net) to the TSM network, thereby identifying pulling behavior.
It improves the accuracy of pulling behavior detection in complex scenarios, reduces false detection and false negative rates, and enhances the model's recognition capabilities in multi-person and occluded scenarios.
Smart Images
Figure CN2024100915_19022026_PF_FP_ABST
Abstract
Description
Method for recognizing pulling behavior based on improved TSM video classification algorithm
[0001] The present application claims priority from the Chinese patent application No. 202311272743X filed on September 28, 2023, and entitled "Method for recognizing pulling behavior based on improved TSM video classification algorithm", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD
[0002] The present application belongs to the field of novel artificial intelligence, more specifically, it relates to a method for recognizing pulling behavior based on an improved TSM video classification algorithm. BACKGROUND
[0003] Currently, traditional deep learning object detection algorithms are combined with object tracking algorithms to determine whether a person is exhibiting abnormal behavior. The effectiveness of this method is highly dependent on the accuracy of the object detection algorithm used to detect the person. If the object detection algorithm performs poorly in complex scenes, occlusions, lighting changes, etc., it will affect the accuracy of abnormal behavior detection.
[0004] Traditional methods are difficult to accurately determine abnormal behavior among multiple people. This is because these methods mainly focus on the actions and behaviors of a single target, while in a multi-person scene, the interactions and coordinated actions between different people increase the complexity of judgment. At the same time, traditional object detection and tracking algorithms may not perform well in complex and crowded scenes. It is difficult to handle complex abnormal behaviors, and some abnormal behaviors may not only be based on the movement or action of a single person, but may involve collaboration or complex action sequences between multiple people. Traditional methods may be difficult to effectively capture and recognize this type of abnormal behavior.
[0005] Due to the above reasons, this method is highly dependent on the accuracy of the target detection model, and it is difficult to determine complex abnormal behaviors, which reduces the accuracy of abnormal behavior recognition and causes problems such as misjudgment and missed judgment of abnormal behavior.
[0006] The skeleton sequence-based abnormal behavior recognition method used in the prior art is common in the field of human motion analysis and behavior recognition. This method can change at different angles and postures of the human body, which can affect the representation ability and recognition performance of the model. This can make it difficult for the model to accurately capture and understand variable human actions in complex real-world scenarios. At the same time, this method may have poor generalization ability in new environments or unseen scenarios. Since these methods mainly learn specific skeleton patterns from training data, the recognition performance can decrease when encountering different environmental factors from the training data. And this method usually only focuses on the motion trajectory of the human body, ignoring the surrounding environment and context information. This can make it difficult to understand and distinguish between normal and abnormal behavior in some cases, as the environment and context can affect the interpretation and judgment of behavior.
[0007] Chinese patent document (CN115346266A) discloses a method for extracting key nodes of human behavior in skeleton-based monitoring video. This application has certain generality in the field of behavior detection and can be used as a module of the behavior detection framework. The patent takes fighting and falling behavior as an example to demonstrate the method of extracting key nodes of behavior. In view of the irrationality of the existing behavior detection framework in extracting video frames at a fixed frame rate, this application proposes to dynamically adjust the position of extracting video frames and the frame rate according to the motion information of the key points of the human skeleton, and more video frames are extracted for behavior nodes with faster motion. Adding a key node extraction module for human behavior based on the skeleton in the existing behavior detection framework can more accurately determine the spatiotemporal boundary of a specific behavior and achieve better behavior detection results. For this abnormal behavior recognition method, a target detection algorithm in the field of deep learning computer vision is used. From the process, it can be seen that this method uses frame extraction technology for video processing and performs target detection on the extracted image frames to determine the fighting behavior according to the results of target detection. In the main steps of this method, a Harris3D spatiotemporal interest point detector is used to detect areas in the image that have significant changes in both spatial and temporal domains. The number of Harris3D spatiotemporal interest points in each candidate key frame in a video sequence is counted, and the top 50% of candidate frames with the most interest points are selected as key frames.
[0008] Due to the above reasons, the skeleton sequence-based abnormal behavior recognition method may be limited by posture changes, labeling difficulty, poor generalization ability, and other issues in some cases. These problems can lead to poor generalization ability of the model, resulting in false positives and false negatives of this type of algorithm in diverse scenarios.
[0009] With the traditional TSM detection algorithm, the TSM module is embedded in the network of ResNet50 to determine whether the personnel in the video has abnormal behavior. The basic network of this method is based on ResNet50, and the TSM module is added after the convolution layer of the basic network to improve the accuracy of image classification. However, this method cannot accurately obtain the motion features of the target in the video and the relationship between the context, so this method cannot quickly and accurately determine whether there is an abnormal behavior in the video when performing abnormal behavior detection, and false positives and false negatives may occur in various scenes. The model generalization ability of this method is low, and it cannot be applied to complex detection scenes.
[0010] Therefore, it is necessary to propose a method for identifying pulling behavior based on an improved TSM video classification algorithm, which can accurately identify all pulling behaviors in complex scenes without misidentification and missed identification, improving the accuracy of identification.
[0011] SUMMARY
[0012] Therefore, it is necessary to propose a method for identifying pulling behavior based on an improved TSM video classification algorithm, which can accurately identify all pulling behaviors in complex scenes without misidentification and missed identification, improving the accuracy of identification.
[0013] The method for identifying pulling behavior based on the improved TSM video classification algorithm specifically comprises the following steps:
[0014] S1: generating a pulling detection dataset, classifying the content of the dataset, and dividing it into a training set and a validation set;
[0015] S2: improving the TSM network, adding a channel-level motion feature encoding module (CME module) after obtaining the feature map in the TSM network, and adding an interactive aggregation update module (IAU-Net module) after the second two-dimensional convolution module;
[0016] S3: inputting the pulling detection dataset generated in step S1 into the improved TSM algorithm in step S2 to obtain the algorithm model weight of the improved TSM training, and obtaining a pulling behavior detection model;
[0017] S4: image data acquisition, obtaining each frame of image data;
[0018] S5: inputting each frame of image data obtained in step S4 into the pulling behavior detection model trained in step S3 for inference to obtain an inference result;
[0019] S6: analyzing the reasoning result in the step S5, judging whether the pulling behavior occurs, if the pulling behavior occurs, saving the pulling behavior, and judging whether the alarm needs to be given.
[0020] With the technical solution, the channel-level motion feature coding module and the interactive fusion update IAU module are added in the TSM video classification model to detect and judge the pulling behavior, the TSM video classification algorithm is used to detect the pulling behavior, the channel-level motion feature coding module and the interactive fusion update IAU module are added in the TSM, the problems of lacking context dependency between different frames and strong dependency on samples and scenes in the traditional deep learning algorithm and the abnormal behavior detection algorithm based on the skeleton sequence are avoided. When there are more than two people in the monitoring area, the continuous frames are extracted from the video stream and input into the improved TSM model to detect and judge the pulling behavior, and if the pulling behavior occurs, the pulling behavior is pre-alerted. That is, the video classification model is used in combination with the channel-level motion feature coding module and the interactive fusion update IAU module, so that the pulling behavior detection model can better obtain the context relationship of the target in different frames, the pulling behavior detection model can effectively detect and judge the pulling behavior between people, the accuracy of the model detection is improved, and the problems of difficult abnormal behavior detection for multiple people and various scenes, high false positive rate and high false negative rate in the traditional abnormal behavior recognition mode are solved.
[0021] Preferably, the pulling detection data set in the step S1 is in a video format, a video simulating pulling and non-pulling behaviors in different scenes is generated to generate a pulling data set, and different labels are made according to the content of the video. The video with pulling behavior is labeled as a class, and the video without pulling behavior is labeled as a class. The video data set with pulling behavior is labeled as pulling behavior, and the data set without pulling behavior is labeled as normal behavior.
[0022] Preferably, the channel-level motion feature coding module is added in the step S2 to extract motion features and excite motion channels. The specific steps of the channel-level motion feature coding module to extract motion features are as follows:
[0023] S21: first, the input video frame is subjected to a 1x1 2D spatial convolution to reduce the feature channel dimension, assuming that the scaling factor is r, the input dimension is [N, T, C, H, W], and after the convolution, the output dimension of the feature map is [N, T, C / r, H, W];
[0024] S22: the obtained feature map is split into each frame along the time dimension T, then the adjacent frames are differentiated, and the image feature X t+1 of the latter frame is subtracted from the image feature Xt Due to the motion of the foreground target in the video, the position of the target changes between two frames, and direct difference may cause feature confusion; therefore, in order to eliminate feature confusion, the spatial feature of the next frame is first transformed by 3x3 2D convolution, and then the obtained transformed feature is subtracted from the feature of the previous frame image, so as to obtain the motion representation between the adjacent two frames, assuming that the motion feature from the t-th frame to the t+1-th frame is M t , then the formula is: M t =Conv 3×3 (X t+1 )-X t , M t ∈R N×C / r×H×W ;
[0025] Where X t represents the t-th frame image feature, X t+1 is the t+1-th frame image feature, Conv 3×3 (·) represents 3x3 2D spatial convolution; R represents a real number;
[0026] S23: The obtained motion features between adjacent frames are spliced to obtain the motion feature representation of the input video as M, that is: M=Concat(M1, M2, …, M T ), M∈R N×T×C / r×H×W ;
[0027] Where Concat(·) represents the connection of motion features; M1, M2, …, M T all refer to motion features.
[0028] S24: Since the input feature X has undergone convolution channel dimension reduction, the dimension of the obtained video motion feature M is [N, T, C / r, H, W]; the obtained video motion feature M is input into two parallel branches, that is, a motion encoding branch and a motion channel excitation branch; so as to obtain the motion feature according to the motion encoding branch and the motion channel excitation branch.
[0029] Preferably, in the motion encoding branch in step S24, a 1x1 2D convolution is designed to encode the motion feature M, and the channel dimension is restored from C / r to C, so as to ensure that the feature information will not be lost, and facilitate fusion with the input feature; assuming that the feature obtained after motion encoding is M encode , then: M encode =Conv 1×1 (M), M encode ∈R N×T×C×H×W ;
[0030] Where Conv 1×1(·) represents a 2D convolution operation of 1x1; the motion encoding branch obtains motion features between encoded adjacent frames.
[0031] Preferably, because the feature channel contains the motion features of the foreground target and the background features, the attention mechanism is used in the step S24 to learn the attention weight of the channel of the motion feature after splicing, to excite the motion feature channel and inhibit the background feature channel; specifically, considering that the channel feature weight is irrelevant to the spatial position, the motion feature is first globally averaged and pooled in space to obtain the spatial-independent motion feature, denoted as M pooling , i.e. the formula is: M pooling = Pooling(M), M pooling ∈R N×T×C / r×1× 1;
[0032] wherein Pooling(·) represents a spatial pooling operation, and M pooling has a dimension of [N, T, C / r, W, H];
[0033] M pooling is then convolved and dimensioned, and input to a sigmoid activation function to obtain an attention weight matrix denoted as A, i.e. the formula is: A = sigmoid(Conv 1×1 (M pooling )), A ∈R N×T×C×1×1 ;
[0034] wherein sigmoid(·) represents a sigmoid activation function, and Conv 1×1 (·) represents a 2D convolution operation of 1x1; the motion encoding feature M encode is multiplied by the obtained channel attention weight A to excite the motion feature channel, while a residual connection is added to keep the original input feature from being lost, i.e. the formula is: Y = X + A·M encosde , Y ∈R N×T×C×H×W ;
[0035] wherein X is the input feature, and Y is the final output feature of the module; and X and Y have the same dimension, i.e. the channel-level motion feature encoding module (CME module) does not change the output dimension of the input feature.
[0036] Preferably, the step S2 of adding an interaction-aggregation-update module (IAU-Net module) after the second two-dimensional convolution module comprises an interaction module, an aggregation module and an update module, the interaction module is used to pull the interaction between the node and its adjacent nodes in the image data by the behavior detection model; the aggregation module is used to aggregate the information propagated from the adjacent nodes to the center node by the behavior detection model; the update module is used to update the features or representations of the center node after the aggregated information is calculated. The interaction-aggregation-update IAU module can better capture the context dependency relationship of the same target between different frames, and improve the accuracy of abnormal behavior judgment. The interaction-aggregation-update IAU module: Interaction-Aggregation-Update (Interaction-Aggregation-Update) module is a concept used to describe the general process or pipeline in Graph Neural Networks (GNNs). This module is particularly suitable for modeling the relationships and dependencies between entities in graph-structured data. Graph neural networks have become popular in tasks such as node classification, link prediction, and graph classification. The "Interaction-Aggregation-Update" process enables these networks to effectively extract information from the structure of the graph and make predictions or classifications based on that information. Interaction: In the interaction phase, the model processes the interaction between a node and its adjacent nodes in the graph. The nodes in the graph can represent entities, objects, users, etc., while the edges represent the relationships or connections between these entities. In this phase, information flows from adjacent nodes to the center node through the edges of the graph; Aggregation: After the interaction phase, the model aggregates the information propagated from the adjacent nodes to the center node. Aggregation can involve various mathematical operations, such as summation, averaging, or more complex aggregation functions. The goal is to combine the information from different neighbors in a meaningful way to capture relevant contextual information for the center node; Update: Once the aggregated information is calculated, it is used to update the features or representations of the center node. This update step usually involves some form of transformation or neural network operation that integrates the aggregated information with the node's existing features. The updated features become the new representation of the node, combining its original features with the contextual information from the neighbors.
[0037] By bypassing the strong dependence of the traditional deep learning algorithm and the skeleton sequence algorithm on samples and the problem that the unimproved TSM detection algorithm model has low generalization ability and cannot be applied to complex scenes, the improved TSM video classification algorithm is used to determine the pulling behavior of the monitoring video, and the dependence on abnormal behavior samples is reduced. At the same time, the idea of detecting moving targets by using the frame difference method is used, a channel-level motion feature encoding module CME (Channel-wise Motion Encoding) is added to the original TSM network to extract motion features and stimulate motion channels, and the background channels are suppressed, so that more effective short-time motion features are obtained, and the accuracy of the pulling behavior recognition is improved; in order to better obtain the abnormal behavior features of the personnel in the continuous frames, the IAU (Interaction-Aggregation-Update) module is introduced, which can better capture the context dependence relationship of the same target between different frames and improve the accuracy of the abnormal behavior determination.
[0038] Preferably, the step S5 first performs frame extraction processing on the video stream, specifically, according to the image data obtained in the step S4, one image frame is saved every 10 image data, and when the number of saved image frames reaches 5, the pulling behavior detection model obtained in the step S3 is input for inference.
[0039] Preferably, the specific steps of the step S6 are as follows:
[0040] S61: input the image frame extracted in the step S5 into the pulling behavior detection model trained in the step S3 to obtain the inference result, determine whether the pulling abnormal behavior appears in the video stream, if yes, go to step S61; if no, return to step S4.
[0041] S62: if the pulling abnormal behavior appears in the video, the video is saved for later inspection, and an alarm information is sent for the pulling abnormal behavior. If the pulling abnormal behavior does not appear, no alarm information is sent.
[0042] Compared with the prior art, the present application has the beneficial effects that:
[0043] (1) The video format data set is used as the data set for training, and the TSM video classification algorithm is used to detect the pulling behavior; the TSM is used to train the pulling behavior detection model, and the TSM uses the video format data set as the data set for training, which reduces the difficulty of data set marking; in the aspect of marking, the manpower is greatly saved, and the training efficiency of the model is improved;
[0044] (2) using the improved TSM network for model training, adding a CME module and an IAU-Net module in the original TSM network, so as to better obtain the relationship between the motion characteristics of the target and the context and improve the detection accuracy of the model. BRIEF DESCRIPTION OF DRAWINGS
[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor on the basis of these drawings.
[0046] Fig. 1 is a flowchart of the method for recognizing the pulling behavior based on the improved TSM video classification algorithm of the present application;
[0047] Fig. 2 is a diagram of the channel-level motion feature coding CME module added in step S2 in the method for recognizing the pulling behavior based on the improved TSM video classification algorithm of the present application;
[0048] Fig. 3 is a diagram of the interactive aggregation update IAU-Net module added in step S2 in the method for recognizing the pulling behavior based on the improved TSM video classification algorithm of the present application;
[0049] Fig. 4 is a diagram of the improved TSM module in the method for recognizing the pulling behavior based on the improved TSM video classification algorithm of the present application;
[0050] Fig. 5 is a diagram of the network structure of the traditional TSM video classification algorithm in the method for recognizing the pulling behavior based on the improved TSM video classification algorithm of the present application;
[0051] Fig. 6 is a diagram of the accuracy comparison between the traditional TSM video classification algorithm and the improved TSM video classification algorithm in the method for recognizing the pulling behavior based on the improved TSM video classification algorithm of the present application. DETAILED DESCRIPTION
[0052] The embodiments of the present application will be described in detail below with reference to the drawings, and the following embodiments are only used to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the protection scope of the present application.
[0053] The well-known knowledge in the technical solution of the present application is as follows: TSM video classification algorithm.
[0054] Specifically, the video action classification network is TSM (Temporal Shift Module for Efficient Video Understanding), which is a general and effective time shift module (TSM) that promotes information exchange between adjacent frames by shifting part of the channels along the time dimension. At the same time, it can be inserted into a 2D convolutional neural network to realize zero-computation and zero-parameter time modeling, combining the efficiency of 2D convolution with the high performance of 3D convolution.
[0055] TSM improves the efficiency of video understanding by processing the temporal features of video frames. TSM is widely used in many video analysis and recognition tasks, such as face recognition, motion recognition, etc. The core idea of TSM is to divide video frames into different time periods and then analyze the features of each time period independently, thereby improving the accuracy of video analysis.
[0056] For abnormal behavior video recognition, the time conversion module TSM deep learning model can effectively describe the information characteristics of the domain, and this model inserts the TSM module into the ResNet50 network. This module improves the network's video understanding ability through time shifting, and it moves part of the channels in the time dimension, thus promoting information exchange between adjacent frames. By inserting it into a 2D convolutional neural network, it can model the time axis information with zero computation.
[0057] TSM video classification model details:
[0058] Motivation and principle: In the task of video classification, understanding the information in the time dimension is very important for correct classification. However, the standard convolutional neural network (CNN) architecture often fails to fully capture temporal information when processing videos. The motivation of TSM is to introduce a shift operation in the time dimension of the feature map to enhance the model's ability to perceive temporal changes.
[0059] Shift operation: The key part of TSM is the shift operation. It actually rearranges the input by shifting it on different channels, so that the model can observe the input at different time steps. The advantage of this is that the model can perceive actions and changes in the video sequence from different time perspectives, thereby improving classification performance.
[0060] Network architecture: TSM can be combined with different network architectures, and it is usually embedded in a certain position of these architectures to enhance the perception of the time dimension. TSM is usually introduced after some convolutional layers in the network to ensure that enough feature extraction is performed before the shift operation.
[0061] Training Strategy: During training, the TSM module introduces displacement operations, but it is important to control the complexity of the model to prevent overfitting. Generally, TSM increases the perceptual ability of the model with a small computational cost and parameter increase.
[0062] Effects and Advantages: The introduction of TSM technology can significantly improve the performance of video classification tasks. By enhancing the model's understanding of time series, TSM can better capture actions, changes, and key information in videos, thereby improving classification accuracy.
[0063] Embodiment: As shown in FIG. 1, the method for recognizing pulling behavior based on the improved TSM video classification algorithm specifically includes the following steps:
[0064] S1: Generate a pulling detection dataset and classify the content of the dataset, then divide it into a training set and a validation set;
[0065] The pulling detection dataset in step S1 is in video format. A large number of videos related to abnormal behavior of pulling are collected, and videos simulating pulling and non-pulling behavior in different scenes are generated to generate a pulling dataset. According to the content of the video, different labels are given, and the video with pulling behavior is labeled as one class, and the video without pulling behavior is labeled as one class. Among them, the video dataset with pulling behavior is labeled as pulling behavior, and the dataset without pulling behavior is labeled as normal behavior. Then it is divided into a training set and a validation set.
[0066] S2: Improve the TSM network, add a channel-level motion feature encoding module (CME module) after obtaining the feature map in the TSM network, and add an interactive aggregation update module (IAU-Net module) after the second two-dimensional convolution module;
[0067] As shown in FIGS. 2-4, the channel-level motion feature encoding module in step S2 is used to extract motion features and stimulate motion channels. The specific steps of the channel-level motion feature encoding module for extracting motion features are as follows:
[0068] S21: First, reduce the feature channel dimension of the input video frame through a 1x1 2D spatial convolution. Assuming that the scaling factor is r, the input dimension is [N, T, C, H, W], and after convolution, the output dimension is [N, T, C / r, H, W] feature map;
[0069] S22: Split the obtained feature map into individual frames along the time dimension T, then difference adjacent frames, and use the feature X t+1 of the latter frame image to subtract the feature X tDue to the motion of the foreground target in the video, the position of the target changes between two frames, and direct difference may cause feature confusion; therefore, in order to eliminate feature confusion, the spatial feature transformation is first performed on the next frame by using a 3x3 2D convolution, and then the obtained transformed feature is subtracted from the feature of the previous frame image, so as to obtain the motion representation between the adjacent two frames, assuming that the motion feature from the t-th frame to the t+1-th frame is M t , then the formula is: M t = Conv 3×3 (X t+1 )-X t , M t ∈R N×C / r×H×W ;
[0070] wherein, X t represents the t-th frame image feature, X t+1 is the t+1-th frame image feature, Conv 3×3 (·) represents a 3x3 2D spatial convolution; R represents a real number;
[0071] S23: The obtained motion features between adjacent frames are spliced to obtain the motion feature representation of the input video as M, that is: M = Concat (M1, M2, …, M T ), M ∈ R N×T×C / r×H×W ;
[0072] wherein, Concat(·) represents the connection of the motion features; M1, M2, …, M T all refer to motion features;
[0073] S24: Since the input feature X has undergone convolution channel dimension reduction, the dimension of the obtained video motion feature M is [N, T, C / r, H, W]; the obtained video motion feature M is input into two parallel branches, that is, a motion encoding branch and a motion channel excitation branch; so as to obtain the motion feature according to the motion encoding branch and the motion channel excitation branch;
[0074] In the step S24, in the motion encoding branch, a 1x1 2D convolution is designed to encode the motion feature M, and the channel dimension is restored from C / r to C, so as to ensure that the feature information will not be lost, and facilitate fusion with the input feature; assuming that the feature obtained after motion encoding is M encode , then: M encode = Conv 1×1 (M), M encode ∈R N×T×C×H×W ;
[0075] wherein, Conv 1×1 (·) represents a 1x1 2D convolution operation; the motion encoding branch obtains the motion feature between the adjacent frames after encoding;
[0076] Because the feature channel contains the motion feature of the foreground target and the background feature, the attention mechanism is used in step S24 to learn the attention weight of the channel of the motion feature after splicing, to stimulate the motion feature channel and inhibit the background feature channel. Specifically, considering that the channel feature weight is irrelevant to the spatial position, the motion feature is first globally averaged and pooled in space to obtain the spatial-independent motion feature, denoted as M pooling , i.e. the formula is: M pooling = Pooling(M), M pooling ∈R N×T×C / r×1×1 ;
[0077] wherein Pooling(·) represents a spatial pooling operation, and M pooling has a dimension of [N, T, C / r, W, H];
[0078] M pooling is then upgraded in dimension through convolution and input to a sigmoid activation function to obtain an attention weight matrix denoted as A, i.e. the formula is: A = sigmoid(Conv 1×1 (M pooling )), A ∈R N×T×C×1×1 ;
[0079] wherein sigmoid(·) represents a sigmoid activation function, and Conv 1×1 (·) represents a 1×1 2D convolution operation; A, the obtained channel attention weight, is multiplied by the motion encoding feature M encode to stimulate the motion feature channel therein, while a residual connection is added to keep the original input feature from being lost, i.e. the formula is: Y = X + A·M encosde , Y ∈R N×T×C×H×W ;
[0080] wherein X is the input feature, and Y is the final output feature of the module; and X and Y have the same dimension, i.e. the channel-level motion feature encoding module (CME module) does not change the output dimension of the input feature;
[0081] In step S2, an Interaction-Aggregation-Update (IAU-Net) module is added after the second 2D convolutional module. This module includes an interaction module, an aggregation module, and an update module. The interaction module is used to manipulate the behavior detection model to process the interactions between nodes and their neighboring nodes in the image data. The aggregation module is used to manipulate the behavior detection model to aggregate information propagated from neighboring nodes to the central node. The update module is used to update the features or representation of the central node after calculating the aggregated information. The Interaction-Aggregation-Update (IAU) module can better capture the contextual dependencies of the same target across different frames, improving the accuracy of abnormal behavior detection. The Interaction-Aggregation-Update (IAU) module is a concept used to describe the general flow or pipeline in Graph Neural Networks (GNNs). This module is particularly suitable for modeling the relationships and dependencies between entities in graph-structured data. Graph Neural Networks have become popular in tasks such as node classification, link prediction, and graph classification. The "Interaction-Aggregation-Update" flow enables these networks to effectively extract information from the structure of the graph and make predictions or classifications based on that information. Interaction: In the interaction phase, the model processes the interactions between nodes in the graph and their neighboring nodes. Nodes in the graph can represent entities, objects, users, etc., while edges represent relationships or connections between these entities. During this phase, information flows from neighboring nodes to the central node through the graph's edges. Aggregation: After the interaction phase, the model aggregates the information propagated from neighboring nodes to the central node. Aggregation can involve various mathematical operations, such as summation, averaging, or more complex aggregation functions. The goal is to combine information from different neighbors in a meaningful way to capture relevant contextual information about the central node. Update: Once the aggregated information is computed, it is used to update the features or representation of the central node. This update step typically involves some form of transformation or neural network operation that integrates the aggregated information with the node's existing features. The updated features become the node's new representation, incorporating its original features and contextual information from its neighbors.
[0082] S3: Input the pulling detection dataset generated in step S1 into the improved TSM algorithm in step S2 to obtain the weights of the improved TSM training algorithm model and obtain the pulling behavior detection model.
[0083] S4: Image data acquisition, acquiring each frame of image data through video or RTSP stream;
[0084] S5: Input the image data of each frame obtained in step S4 into the pulling behavior detection model trained in step S3 for inference and obtain the inference result.
[0085] The step S5 first performs frame extraction processing on the video stream, specifically, according to the image data obtained in step S4, one frame of image is saved every 10 frames of image data, and when the number of saved image frames reaches 5, the pulling behavior detection model obtained in step S3 is input for inference;
[0086] S6: analyze the inference result in step S5 to determine whether a pulling behavior occurs, if so, save the pulling behavior and determine whether to issue an alarm;
[0087] The specific steps of step S6 are:
[0088] S61: input the image frame extracted in step S5 into the inference result obtained by the pulling behavior detection model trained in step S3 to determine whether a pulling abnormal behavior occurs in the video stream, if so, go to step S61; if not, return to step S4.
[0089] S62: if a pulling abnormal behavior occurs in the video, save this segment of video for later inspection, and issue an alarm information for the pulling abnormal behavior. If no pulling abnormal behavior occurs, no alarm information is issued.
[0090] As shown in FIG. 5, the traditional TSM video classification algorithm generally uses ResNet50 as the basic network structure of the model, and this kind of algorithm improves the accuracy of image classification by embedding the TSM module into the convolution layer of the ResNet50 network. The classification effect in some scenes is ideal, but this method cannot accurately obtain the motion features of the target and the relationship between the context in the video, which leads to the fact that this method cannot accurately determine whether an abnormal behavior occurs in the video in a complex scene, and even may cause false detection and missed detection. Therefore, the generalization ability of this algorithm is low, and it cannot be applied to complex scenes. In order to solve the problem of low applicability of the traditional TSM video classification algorithm in complex scenes, the improved TSM video classification algorithm is proposed in the present application, in order to better obtain the motion features of the target and the relationship between the context to improve the accuracy of the model in identifying abnormal behaviors, the CME module and the IAU-Net module are proposed, and the CME module is added after the feature map in the original TSM network, so that the model can better obtain the motion features of the target. In order to better obtain the relationship between the context of the target, we put the IAU-Net module into the second layer of convolution in the TSM network, and the improved TSM network structure is shown in FIG. 4.
[0091] Compared with the traditional TSM video classification algorithm, the improved TSM video classification algorithm can be better applied to complex scenes, and the accuracy of the recognition of the pulling behavior in the complex scenes is higher than that of the traditional TSM video classification algorithm. The recognition effect of the pulling behavior in the multi-person scene and the occlusion scene is better than that of the traditional TSM video classification algorithm. The accuracy comparison of the traditional TSM video classification algorithm and the improved TSM video classification algorithm is shown in FIG. 6. Ten pulling behavior videos in complex scenes are taken as test. As can be seen from FIG. 6, the improved TSM video classification algorithm in the present application can accurately recognize all the pulling behaviors in complex scenes, and there is no misrecognition and missed recognition. However, the traditional TSM video classification algorithm has misjudgment and missed judgment in the recognition of the pulling behavior in the complex scene, and the accuracy of the pulling behavior detection is low. The improved TSM video classification algorithm can better obtain the relationship between the motion features of the target and its context, so that the algorithm can be better applied to the recognition of the pulling behavior in the complex scene, and the accuracy of the algorithm in the pulling behavior detection is improved.
[0092] For those skilled in the art, the specific embodiments are only exemplary descriptions of the present application, and it is obvious that the specific implementation of the present application is not limited by the above method. Any non-essential improvement or direct application of the concept and technical solution of the present application to other occasions is within the protection scope of the present application.
Claims
1. A method for recognizing a pulling behavior based on an improved TSM video classification algorithm, characterized in that, Specifically comprising the following steps: S1: generating a pulling detection dataset, and classifying the content of the dataset, and then dividing it into a training set and a validation set; S2: improving the TSM network, adding a channel-level motion feature encoding module after obtaining the feature map in the TSM network, and adding an interaction aggregation update module after the second two-dimensional convolution module; S3: inputting the pulling detection dataset generated in step S1 into the improved TSM algorithm in step S2 to obtain the algorithm model weight of the improved TSM training, and obtaining a pulling behavior detection model; S4: image data acquisition, obtaining each frame of image data; S5: inputting each frame of image data obtained in step S4 into the pulling behavior detection model trained in step S3 for inference to obtain an inference result; S6: analyzing the inference result in step S5 to determine whether a pulling behavior occurs, and if so, saving the pulling behavior and determining whether to alarm.
2. The method of claim 1, wherein the method of identifying a pulling behavior based on an improved TSM video classification algorithm is characterized by, The pulling detection dataset in step S1 is in video format, simulating the generation of a pulling dataset by shooting pulling and non-pulling behaviors in different scenes, and marking different videos according to their content, marking the videos with pulling behaviors as one class and the videos without pulling behaviors as another class.
3. The method of claim 1, wherein the method of identifying a pulling behavior based on an improved TSM video classification algorithm is characterized by, The channel-level motion feature encoding module in step S2 is used to extract motion features and stimulate motion channels. The specific steps of the channel-level motion feature encoding module for extracting motion features are as follows: S21: First, reduce the feature channel dimension of the input video frame by a 1x1 2D spatial convolution. Assuming that the scaling factor is r, the input dimension is [N, T, C, H, W], and after convolution, the output dimension is [N, T, C / r, H, W] feature map; S22: split the obtained feature map along the time dimension T into individual each frame, then difference between adjacent frames, i.e. use the feature X t+1 of the later frame minus the feature X t of the former frame, then first do spatial feature transformation on the later frame with 3x3 2D convolution, then use the obtained transformed feature to do difference with the feature of the former frame The difference between the two frames is obtained, so as to obtain the motion representation between the two adjacent frames. Assuming that the motion representation from the tth frame to the t+1th frame is M t The formula is: M t = Conv 3×3 (X t+1 )- X t , M t ∈ R N×C / r×H×W ; where X t represents the t-th frame image feature, X t+1 is the t+1-th frame image feature, Conv 3×3 (·) represents a 3x3 2D spatial convolution; R represents a real number; S23: splice the obtained inter-frame motion features to obtain the motion feature representation M of the input video, i.e. M = Concat (M1, M2..., M T ), M e R N×T×C / r×H×W ; where Concat(·) denotes concatenation of motion features; M1, M2, ···, M T all refer to motion features; S24: Since the input feature X has been convolved and the channel dimension has been reduced, the dimension of the obtained video motion feature M is [N, T, C / r, H, W]. The obtained video motion feature M is input into two parallel branches, i.e. the motion encoding branch and the motion channel excitation branch. Thus, the motion feature is obtained according to the motion encoding branch and the motion channel excitation branch.
4. The method for recognizing pulling behavior based on the improved TSM video classification algorithm according to claim 3, characterized in that, In step S24, a 1x1 2D convolution is designed in the motion encoding branch to encode the motion feature M and restore the channel dimension from C / r to C, ensuring that the feature information is not lost and facilitating fusion with the input feature; Assume that the feature obtained after motion coding is M encode Then, we have: M encode = Conv 1×1 (M), M encode ∈ R N×T×C×H×W ; where Conv 1×1 (·) denotes a 1 x 1 2D convolution operation; the motion encoding branch gets the motion features between the encoded neighboring frames.
5. The method for recognizing pulling behavior based on the improved TSM video classification algorithm according to claim 4, characterized in that, In step S24, an attention mechanism is used to learn the attention weight of the channel of the spliced motion feature, to excite the motion feature channel and suppress the background feature channel. Specifically, the motion features are first subjected to spatial global average pooling to obtain spatially independent motion features, denoted as M. pooling That is, the formula is: M pooling =Pooling(M), M pooling ∈R N×T×C / r×1×1 ; where Pooling(·) denotes a spatial pooling operation, resulting in M pooling with dimensions [N, T, C / r, W, H]; M pooling After convolution dimensionality, and input to the sigmoid activation function, get the attention weight matrix is recorded as A, that is, the formula is: A = sigmoid(Conv 1×1 (M poolingg )), A ∈ R N×T×C×1×1 ; where sigmoid(·) denotes a sigmoid activation function, Conv 1×1 (·) denotes a 1 × 1 2D convolution operation, and the obtained channel attention weight A is multiplied with the motion encoding feature M encode to excite the motion feature channels therein while adding a residual connection to keep the original input features from being lost, i.e., the formula is: Y = X + A - M encosde , Y e R N×T×C×H×W ; Wherein, X is the input feature, Y is the final output feature of the module; and X and Y have the same dimension, i.e. the channel-level motion feature encoding module does not change the output dimension of the input feature.
6. The method of claim 4, wherein the improved TSM video classification algorithm-based recognition of a pulling behavior is characterized by, The interactive aggregation update module added after the second two-dimensional convolution module in the step S2 comprises an interaction module, an aggregation module and an update module, the interaction module is used for pulling behavior detection model to process the interaction between a node and its adjacent nodes in image data, and the aggregation module is used for pulling behavior detection model to aggregate the information propagated from the adjacent nodes to the center node. The update module is used for updating the features or representation of the center node after the aggregated information is calculated.
7. The method of claim 4, wherein the method is based on an improved TSM video classification algorithm. The step S5 first performs frame extraction processing on the video stream, specifically, according to each frame of image data obtained in the step S4, one frame of image is saved every 10 frames of image data, and when the number of saved image frames reaches 5 frames, the pulling behavior detection model obtained in the step S3 is input for inference.
8. The method of claim 7, wherein the method is based on an improved TSM video classification algorithm. The specific steps of the step S6 are as follows: S61: according to the inference result of the image frame extracted in the step S5 and input into the pulling behavior detection model trained in the step S3, it is judged whether the abnormal behavior of pulling appears in the video stream, if yes, the step S61 is turned to; if no, the step S4 is returned; S62: if the abnormal behavior of pulling appears in the video, the video is saved, and an alarm information is sent for the abnormal behavior of pulling.