An action recognition method for workshop intelligent video surveillance

Abstract

This invention provides a method for motion recognition in intelligent workshop video surveillance. The motion recognition network provided by the invention primarily consists of two parts: a local spatiotemporal feature extraction module within a segment, which extracts local spatiotemporal features within each time segment and captures subtle changes between actions; and a channel timing sensitivity module, which determines the sensitivity of different channels to timing information and captures global temporal features between time segments. These two modules are embedded in a two-dimensional convolutional neural network to construct a motion recognition model. The motion recognition network designed by the invention enables end-to-end learning and demonstrates excellent motion recognition accuracy. A system for identifying abnormal motion in a workshop was constructed based on this method.

Description

Technical Field

[0001] The present invention relates to an action recognition method for workshop intelligent video monitoring, and belongs to the technical field of computer vision. Background Art

[0002] Safety is always a top priority in a company's production process. However, workshop workers may engage in irregular or unusual behavior, necessitating video surveillance to identify these behaviors and prevent dangerous situations. However, current video surveillance relies primarily on manual observation, requiring dedicated personnel to monitor the situation 24 / 7, making it extremely easy to overlook unusual events. Therefore, applying motion recognition to intelligent workshop video surveillance is crucial for improving the real-time and accuracy of video surveillance and ensuring safe production in production workshops.

[0003] In recent years, convolutional neural networks have been widely used in action recognition. The general process involves first dividing the video into multiple segments, randomly extracting a frame from each segment for segment feature extraction, and then performing global temporal modeling to obtain the entire video features for action classification. However, these methods still have certain problems. First, the segment feature extraction ignores the local motion characteristics within each time segment, which can reduce recognition effectiveness for fine-grained videos. Second, most methods fail to consider the sensitivity of feature channels to temporal information during global temporal modeling, which is also detrimental to improving action recognition performance. Summary of the Invention

[0004] The present invention aims to provide a method for motion recognition based on intelligent video surveillance in workshops, which improves the recognition effect of videos with higher granularity and improves the motion recognition effect.

[0005] To achieve the above-mentioned purpose, the present invention is implemented through the following technical solutions:

[0006] Step 1: Collect surveillance video information and divide it into T video frames in the time domain; and divide it into a test set and a training set;

[0007] Step 2: Build an action recognition model and construct a residual block based on Resnet. The specific structure is as follows:

[0008] The first layer of the action recognition model is a local spatiotemporal feature extraction module within a segment, which extracts local spatiotemporal features from the input video frames. A pooling layer is connected after the local spatiotemporal feature extraction module to downsample the features. The pooling layer is connected after the constructed residual block to extract spatiotemporal features from the input. The residual block adds a 1D Conv with a convolution kernel of 3 before each 1×1 convolution in the Bottleneck convolutional network, and a channel timing sensitive module is added after the 1D Conv. The residual block is connected after the fully connected layer for result recognition.

[0009] Step 3: Input the training set into the constructed action recognition model, train the model, and obtain the trained model;

[0010] Step 3-1: Obtain the local spatiotemporal features of each time segment through the intra-segment local spatiotemporal feature extraction module to capture subtle changes between actions;

[0011] Step 3-2: Aggregate global temporal information through the channel timing information sensitive module, while highlighting the motion sensitive channel, enhancing useful features, and de-emphasizing useless features to obtain global spatiotemporal features;

[0012] Step 4: Input the test set into the trained action recognition model and output the action recognition results.

[0013] Preferably, there are 50 residual blocks.

[0014] Preferably, the specific steps of extracting the local spatiotemporal features within the segment are as follows:

[0015] Step 3-1-1: Input the captured video frame into the first branch of the intra-segment local spatiotemporal feature extraction module for feature extraction to obtain the original spatial feature X1 with a dimension of [64, H, W], where H and W represent the height and width of the spatial feature X1;

[0016] Step 3-1-2: Input the selected video frames in each video segment into the second branch of the local spatiotemporal feature extraction module to obtain local motion features, while enhancing useful features and de-emphasizing useless features;

[0017] Step 3-1-3: Combine the original spatial feature X1 with the local motion feature Adding in the spatial domain, we can obtain the local spatiotemporal features of each segment:

[0018] Preferably, the specific steps for obtaining local motion features, enhancing useful features, and weakening useless features are as follows:

[0019] The video frame is differentiated by RGB, and then the RGB differences are merged in the channel dimension to obtain the merged feature D with a dimension of [12, H, W]. Feature D is extracted through a 2D convolution to obtain a feature X2 with a dimension of [64, H, W]. Global average pooling is performed in the spatial domain to scale the feature to 1×1, and the channel-level global feature F is obtained with a dimension of [64, 1, 1]. The specific formula is as follows:

[0020]

[0021] Among them, i and j represent the index of column coordinates and row coordinates in the feature X2 space dimension respectively.

[0022] Then, the importance of each channel is learned through two levels of 1×1 convolution. The first 1×1 convolution reduces the dimension and compresses the channel dimension according to the ratio r to fully capture the relationship between channels. The second 1×1 convolution restores the dimension, and then a Sigmoid operation is performed to obtain the weight s corresponding to each channel. The specific formula is as follows:

[0023] s=σ(W2*δ(W1*F))

[0024] Among them, W1 is the first 1×1 convolution, δ is the Relu activation function, W2 is the second 1×1 convolution, and σ represents the Sigmoid activation function;

[0025] Multiply feature X2 by the corresponding weight to get The dimension is [64,H,W].

[0026] Preferably, the specific steps of extracting the global spatiotemporal features are as follows:

[0027] Step 3-2-1: Perform 1D convolution on the input feature map in the time domain to extract the timing information and obtain the feature X with dimensions [T, C, H, W] 1D , where T represents the number of time periods and C represents the number of feature channels;

[0028] Step 3-2-2: Set feature X 1D The input channel timing information sensitive module obtains the sensitivity weight of each channel in the time domain;

[0029] Step 3-2-3: Set feature X 1D With weight s 1D Multiply and then add feature X 1D Add together to get the output of the timing information sensitive module:

[0030] F MS =s 1D ·X 1D +X 1D .

[0031] Preferably, the specific steps for obtaining the sensitivity weight are as follows:

[0032] The global spatial information feature F with the dimension [T, C, 1, 1] is obtained by average pooling in the spatial domain 1×1 :

[0033]

[0034] The global spatial information feature F 1×1 The dimension is reshaped to [1, C, T], and temporal features are extracted through a 1D convolution K1 with a convolution kernel of 3;

[0035] After convolution, the feature dimension is reshaped to [T, C, 1, 1] and sent to Sigmoid to obtain the sensitivity weight s 1D .

[0036] A motion recognition system for intelligent video surveillance in workshops, including an embedded artificial intelligence processing module, a USB camera, an alarm, a display device, and a data storage module;

[0037] The embedded artificial intelligence processing module can run the trained action recognition model to form an action recognition processor;

[0038] The USB camera collects real-time video data and adjusts it to a fixed resolution, which is then fed into a motion recognition processor to detect whether the worker in the video has any abnormal movements.

[0039] The alarm module alarms after receiving the alarm signal;

[0040] The display device receives the recognition result of the embedded artificial intelligence processing module and displays it;

[0041] The data storage module stores identification data.

[0042] The advantages of the present invention are: the present invention strengthens the local motion features in each time period, improves the recognition effect for videos with higher granularity; and at the same time considers the sensitivity of the feature channel to timing information, thereby improving the action recognition effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0044] Figure 1 This is a schematic diagram of the overall network framework structure of the present invention.

[0045] Figure 2This is a schematic diagram of the intra-segment local spatiotemporal feature extraction module (LTEM) provided by the present invention.

[0046] Figure 3 FIG. 1 is a schematic diagram of a channel timing sensitivity module (CTM) provided by the present invention. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] Step 1: Collect surveillance video information and divide it into T video frames in the time domain; and divide it into a test set and a training set;

[0049] Specifically, for the abnormal action dataset in production workshops, nine abnormal actions were included: using a mobile phone while moving, approaching dangerous equipment, climbing over railings, climbing machinery, smoking, throwing objects from a height, falling and injuring, eating, and not wearing a helmet. 2,700 videos were collected, with 10 volunteers shooting 30 videos for each category. Each video was 2 seconds long, with a frame rate of 30 fps.

[0050] Step 2: Build an action recognition model and construct a residual block based on Resnet. The specific structure is as follows:

[0051] The first layer of the action recognition model is a local spatiotemporal feature extraction module within a segment, which extracts local spatiotemporal features from the input video frames. A pooling layer is connected after the local spatiotemporal feature extraction module to downsample the features. The pooling layer is connected after the constructed residual block, which has 18 residual blocks and extracts spatiotemporal features from the input. The residual block adds a 1D Conv with a convolution kernel of 3 before each 1×1 convolution in the Bottleneck convolutional network, and a channel timing sensitive module is added after the 1D Conv. The residual block is connected after the fully connected layer for result recognition.

[0052] Step 3: Input the training set into the constructed action recognition model, train the model, and obtain the trained model;

[0053] Step 3-1: Obtain the local spatiotemporal features of each time segment through the intra-segment local spatiotemporal feature extraction module to capture subtle changes between actions;

[0054] The specific steps of extracting local spatiotemporal features within the segment are as follows:

[0055] Step 3-1-1: Input the collected video frame into the first branch of the intra-segment local spatiotemporal feature extraction module for feature extraction to obtain the original spatial feature X1 with a dimension of [64, H, W], where H represents the height of the spatial feature X1 and W represents the width of the spatial feature X1;

[0056] Step 3-1-2: Input the selected video frames in each video segment into the second branch of the local spatiotemporal feature extraction module to obtain local motion features, while enhancing useful features and de-emphasizing useless features;

[0057] Step 3-1-3: Combine the original spatial feature X1 with the local motion feature Adding in the spatial domain, we can obtain the local spatiotemporal features of each segment:

[0058] Preferably, the specific steps for obtaining local motion features, enhancing useful features, and weakening useless features are as follows:

[0059] The video frame is differentiated by RGB, and then the RGB differences are merged in the channel dimension to obtain the merged feature D with a dimension of [12, H, W]. Feature D is extracted through a 2D convolution to obtain a feature X2 with a dimension of [64, H, W]. Global average pooling is performed in the spatial domain to scale the feature to 1×1, and the channel-level global feature F is obtained with a dimension of [64, 1, 1]. The specific formula is as follows:

[0060]

[0061] Among them, i and j represent the index of column coordinates and row coordinates in the feature X2 space dimension respectively.

[0062] Then, the importance of each channel is learned through two levels of 1×1 convolution. The first 1×1 convolution reduces the dimension and compresses the channel dimension according to the ratio r to fully capture the relationship between channels. The second 1×1 convolution restores the dimension, and then a sigmoid operation is performed to obtain the weight s corresponding to each channel. The specific formula is as follows:

[0063] s=σ(W2*δ(W1*F))

[0064] Among them, W1 is the first 1×1 convolution, δ is the Relu activation function, W2 is the second 1×1 convolution, and σ represents the Sigmoid activation function;

[0065] Multiply feature X2 by the corresponding weight to get The dimension is [64,H,W].

[0066] Step 3-2: Aggregate global temporal information through the channel timing information sensitive module, while highlighting the motion sensitive channel, enhancing useful features, and de-emphasizing useless features to obtain global spatiotemporal features;

[0067] The specific steps of extracting the global spatiotemporal features are as follows:

[0068] Step 3-2-1: Perform 1D convolution on the input feature map in the time domain to extract the timing information and obtain the feature X with dimensions [T, C, H, W] 1D , where T represents the number of time periods and C represents the number of feature channels;

[0069] Step 3-2-2: Set feature X 1D The input channel timing information sensitive module obtains the sensitivity weight of each channel in the time domain;

[0070] Step 3-2-3: Set feature X 1D With weight s 1D Multiply and then add feature X 1D Add together to get the output of the timing information sensitive module:

[0071] F MS =s 1D ·X 1D +X 1D .

[0072] Preferably, the specific steps for obtaining the sensitivity weight are as follows:

[0073] The global spatial information feature F with the dimension [T, C, 1, 1] is obtained by average pooling in the spatial domain 1×1 :

[0074]

[0075] The global spatial information feature F 1×1 The dimension is reshaped to [1, C, T], and temporal features are extracted through a 1D convolution K1 with a convolution kernel of 3;

[0076] After convolution, the feature dimension is reshaped to [T, C, 1, 1] and sent to Sigmoid to obtain the sensitivity weight s 1D .

[0077] Step 4: Input the test set into the trained action recognition model and output the action recognition results.

[0078] Example 2

[0079] A motion recognition system for intelligent video surveillance in workshops, including an embedded artificial intelligence processing module, a USB camera, an alarm, a display device, and a data storage module;

[0080] The embedded artificial intelligence processing module can run the trained action recognition model to form an action recognition processor;

[0081] The USB camera collects real-time video data and adjusts it to a fixed resolution, which is then fed into a motion recognition processor to detect whether the worker in the video has any abnormal movements.

[0082] The alarm module alarms after receiving the alarm signal;

[0083] The display device receives the recognition result of the embedded artificial intelligence processing module and displays it;

[0084] The data storage module stores identification data.

[0085] Example 3

[0086] Experiments were conducted on the public dataset Something-Something V1 and compared with advanced methods to verify the effectiveness of our proposed method.

[0087]

[0088]

[0089] The action recognition methods in the table can be divided into two categories. The first category consists of methods based on 3D convolutional neural networks, including I3D, ECO, SATA, and GST. These methods have large network models, resulting in high FLOPs. In contrast, our proposed method achieves superior performance while maintaining a low FLOP count. Specifically, when sampling 8 frames as input, our method's accuracy improves by 5% compared to SAST and by 2.3% compared to GST. The second category consists of methods based on 2D convolutional neural networks, including TSN, TSM, TRN, and TEI. Our proposed method also outperforms these methods. When sampling 8 frames as input, the FLOP count increases slightly to 34G, but our accuracy improves by 3.7% compared to TSM and 1.9% compared to TEI. Among all the above methods, our proposed method maintains a relatively low FLOP count while achieving the highest recognition accuracy, demonstrating the effectiveness of our proposed method.

[0090] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A motion recognition method for workshop intelligent video surveillance, characterized in that: The following steps are involved: Step 1: Collect surveillance video information and divide it into T video frames in the time domain; and divide it into a test set and a training set; Step 2: Build an action recognition model and construct a residual block based on Resnet. The specific structure is as follows: The first layer of the action recognition model is a segment-local spatiotemporal feature extraction module, which extracts segment-local spatiotemporal features from the input video frames. A pooling layer is connected after the segment-local spatiotemporal feature extraction module to downsample the features. The pooling layer is connected after the constructed residual block to extract spatiotemporal features from the input. The residual block adds a 1D Conv with a convolution kernel of 3 before each 1×1 convolution in the Bottleneck convolutional network, and a channel timing sensitive module is added after the 1D Conv. The residual block is then connected to a fully connected layer for result recognition; Step 3: Input the training set into the constructed action recognition model, train the model, and obtain the trained model; Step 3-1: Obtain the local spatiotemporal features of each time segment through the intra-segment local spatiotemporal feature extraction module to capture subtle changes between actions; The specific steps of extracting local spatiotemporal features within the segment are as follows: Step 3-1-1: Input the collected video frame into the first branch of the local spatiotemporal feature extraction module to extract features and obtain the original spatial features. , the dimension is ,in and Representing spatial features height and width; Step 3-1-2: Input the selected video frames in each video segment into the second branch of the local spatiotemporal feature extraction module to obtain local motion features, while enhancing useful features and de-emphasizing useless features; Step 3-1-3: Original spatial features Local motion features Adding in the spatial domain, we can obtain the local spatiotemporal features of each segment: ; Step 3-2: Aggregate global temporal information through the channel timing information sensitive module, while highlighting the motion sensitive channel, enhancing useful features, and de-emphasizing useless features to obtain global spatiotemporal features; The specific steps of extracting the global spatiotemporal features are as follows: Step 3-2-1: Perform 1D convolution on the input feature map in the time domain to extract the timing information and obtain the dimension Features , where T represents the number of time periods and C represents the number of feature channels; Step 3-2-2: Add features The input channel timing information sensitive module obtains the sensitivity weight of each channel in the time domain; Step 3-2-3: Add features With weight Multiply and then add the feature Add together to get the output of the timing information sensitive module: ； Step 4: Input the test set into the trained action recognition model and output the action recognition results.

2. The method for motion recognition for workshop intelligent video surveillance according to claim 1, characterized in that: There are 50 residual blocks.

3. The method for motion recognition for workshop intelligent video surveillance according to claim 1, characterized in that: The specific steps to obtain local motion features, enhance useful features, and weaken useless features are as follows: Perform RGB difference on the video frame, and then merge the RGB difference in the channel dimension to obtain the merged features , the dimension is [12,H,W]; the feature Extract features through a 2D convolution to obtain features of dimension [64, H, W] , perform global average pooling in the spatial domain to scale the features to 1×1 and obtain channel-level global features , the dimension is [64,1,1], the specific formula is as follows: in, and Represents characteristics The indices of the column and row coordinates in the spatial dimension; Then, the importance of each channel is learned through two levels of 1×1 convolution. The first 1×1 convolution reduces the dimension and compresses the channel dimension according to the ratio r to fully capture the relationship between channels. The second 1×1 convolution restores the dimension, and then a Sigmoid operation is performed to obtain the weight corresponding to each channel. , the specific formula is as follows: in, is the first 1×1 convolution, is the Relu activation function, is the second 1×1 convolution, Represents the Sigmoid activation function; The features Multiply by the corresponding weight to get , The dimension is [64,H,W].

4. The method for motion recognition for workshop intelligent video surveillance according to claim 1, characterized in that: The specific steps to obtain the sensitivity weight are as follows: The global spatial information feature with the dimension [T, C, 1, 1] is obtained by average pooling in the spatial domain : The global spatial information features The dimension is reshaped to [1, C, T] and a 1D convolution with a convolution kernel of 3 is performed. Perform time feature extraction; After convolution, the feature dimension is reshaped to [T, C, 1, 1] and sent to Sigmoid to obtain the sensitivity weight. .

5. A motion recognition system for workshop intelligent video surveillance, characterized in that: The system can execute the motion recognition method for workshop intelligent video surveillance according to any one of claims 1 to 4, including an embedded artificial intelligence processing module, a USB camera, an alarm, a display device, and a data storage module; The embedded artificial intelligence processing module can run the trained action recognition model to form an action recognition processor; The USB camera collects real-time video data and adjusts it to a fixed resolution, which is then fed into a motion recognition processor to detect whether the worker in the video has any abnormal movements. The alarm module alarms after receiving the alarm signal; The display device receives the recognition result of the embedded artificial intelligence processing module and displays it; The data storage module stores identification data.

Images