A method and system for identifying abnormal human behavior at power grid construction sites

Through the dual-channel three-dimensional convolutional neural network and feature fusion network, the problem of poor real-time performance in the detection of abnormal behaviors of humans on the power grid infrastructure is solved, real-time and accurate abnormal behavior recognition is achieved, and the safety of on-site operators of the power grid infrastructure is ensured.

CN115311740BActive Publication Date: 2024-12-06STATE GRID JIANGSU ELECTRIC POWER CO LTD SUZHOU BRANCH +2
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210885390.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-07-26
Publication Date
2024-12-06
Estimated Expiration
2042-07-26

AI Technical Summary

Technical Problem

The prior art has poor real-time performance in the detection of abnormal behaviors of humans on the power grid infrastructure site, and cannot fully utilize the time series information between videos, resulting in the inability to detect abnormal behaviors in real time and prevent them in advance.

Method used

The dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network are adopted to extract the spatial and temporal features of videos through the spatial and streaming three-dimensional convolutional neural network and the time-stream three-dimensional convolutional neural network respectively, and perform end-to-end fusion, combining with real-time video Data for behavioral identification.

Benefits of technology

Real-time detection of abnormal behaviors of humans on the power grid infrastructure site has been realized, the accuracy and accuracy of detection have been improved, and abnormal behaviors can be prevented in a timely manner.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115311740B_ABST
    Figure CN115311740B_ABST
Patent Text Reader

Abstract

A method and system for identifying abnormal human behavior at a power grid infrastructure site. The identification method comprises: building a dual-channel three-dimensional convolutional neural network and a dual-stream feature fusion network, training the network using a historical staff operation video data set collected by a power grid infrastructure site operation monitoring device, and obtaining a power grid infrastructure site abnormal human behavior recognition model; collecting power grid infrastructure site staff operation video data from the power grid infrastructure site operation monitoring device in real time, obtaining the latest 16 frames of RGB images of the real-time video and calculating the optical flow map of the 16 frames of RGB images; inputting the latest video stream and the optical flow map into the power grid infrastructure site abnormal human behavior recognition model, and obtaining a behavior classification result output; judging according to the classification loss result, if no abnormality occurs, returning to continue to execute the power grid infrastructure site abnormal human behavior detection task; if an abnormality occurs, performing alarm processing, and then continuing to execute the power grid infrastructure site abnormal human behavior detection task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of power system safety monitoring, and in particular relates to a method and system for identifying abnormal human behavior at a power grid infrastructure site. Background Art

[0002] In recent years, it has become particularly urgent to apply artificial intelligence to the identification of abnormal behavior at infrastructure construction sites. Abnormal behavior can be divided into single-person abnormal behavior, interactive abnormal behavior, and group abnormal behavior. Its definition depends on the application scenario in real life and is subject to a certain degree of subjective influence. At present, in the field of abnormal human behavior detection at power grid infrastructure sites, the most common approach is through daily manual inspections or manual supervision through monitoring equipment.

[0003] Power grid infrastructure construction has the characteristics of diverse types, large scale and large number of construction workers. It is impossible to grasp the status of all construction workers in real time by relying solely on manual inspection. The supervision of on-site workers of power grid infrastructure by video surveillance equipment is also relatively limited. Many abnormal conditions are discovered in the post-event stage, which does not achieve the original intention of preventing abnormal behaviors of on-site workers of power grid infrastructure. Therefore, it can be seen that the research on the detection method of abnormal human behavior at the power grid infrastructure site has important theoretical significance and practical value.

[0004] At present, there are relatively few studies on abnormal human behavior detection technology at power grid infrastructure sites, but experts and scholars at home and abroad have proposed some fruitful implementation solutions based on artificial intelligence theory for related difficult problems. Behavior recognition based on deep learning has also been studied in many aspects at home and abroad. A two-stream convolutional neural network has been proposed. This network can independently use RGB picture frames and optical flow images between consecutive frames to obtain static features and dynamic features. Finally, the two branches are fused in the fully connected layer, and the category with the highest score after fusion is used as the category of the current behavior. A dual-resolution convolutional neural network has been proposed. One channel of this network inputs the original resolution picture frame, and the other channel inputs the low-resolution picture frame. After independently training the two models, they are fused in the fully connected layer, and the fused features are used in subsequent behavior recognition. By improving the original two-dimensional convolutional neural network, the researchers proposed a three-dimensional convolutional neural network to better describe the temporal information of the video.

[0005] Although the above-mentioned deep learning method based on convolutional neural network has a high detection accuracy, the algorithm accuracy is largely limited by the data size of abnormal data; and the overall algorithm detection cannot be carried out in real time, which means that it is impossible to detect the abnormal behavior of workers on the power grid infrastructure site in real time and cannot achieve prior prevention.

[0006] In view of this, this paper makes full use of the work data of personnel at the power grid infrastructure site, based on the dual-channel feature fusion method, and uses a three-dimensional convolutional neural network to solve the problem of poor real-time performance of traditional models and inability to fully utilize the timing information between video frames, thereby scientifically and effectively realizing the task of human anomaly detection at the power grid infrastructure site. Summary of the invention

[0007] In order to solve the deficiencies in the prior art, the purpose of the present invention is to provide a method and system for identifying abnormal human behavior at a power grid infrastructure construction site, so as to solve the problem that the traditional model has poor real-time performance and cannot fully utilize the timing information between video frames, thereby scientifically and effectively realizing the task of human anomaly detection at the power grid infrastructure construction site.

[0008] The present invention adopts the following technical solution: A method for identifying abnormal human behavior at a power grid infrastructure site, characterized in that it comprises the following steps:

[0009] Step 1: Build a dual-channel three-dimensional convolutional neural network and a dual-stream feature fusion network, connect three fully connected layers to the dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network, and use the historical staff operation video data set collected by the power grid infrastructure site operation monitoring device to train the dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network to obtain a human abnormal behavior recognition model at the power grid infrastructure site;

[0010] Step 2: Collecting the operation video data of the power grid infrastructure construction site workers in real time from the power grid infrastructure construction site operation monitoring device to obtain the real-time video stream and optical flow map;

[0011] Step 3, input the latest video stream and optical flow map into the abnormal human behavior recognition model at the power grid infrastructure construction site to obtain the behavior classification result output;

[0012] Step 4, judging based on the classification result of step 3, if no abnormality occurs, return to continue to execute the task of abnormal human behavior detection at the power grid infrastructure construction site; if an abnormality occurs, perform alarm processing, and then continue to execute the task of abnormal human behavior detection at the power grid infrastructure construction site.

[0013] Preferably,

[0014] In step 1, the dual-channel three-dimensional convolutional neural network includes: a spatial stream three-dimensional convolutional neural network and a temporal stream three-dimensional convolutional neural network. The spatial stream three-dimensional convolutional neural network is used to extract spatial features related to the behavior, and the temporal stream three-dimensional convolutional neural network is used to extract temporal features.

[0015] In step 1, the spatial stream three-dimensional convolutional neural network and the temporal stream three-dimensional convolutional neural network share network parameters, which include: network weights and bias values.

[0016] In step 1, the dual-channel 3D convolutional neural network includes: 8 3D convolutional layers and 5 3D pooling layers, and a dual-stream feature fusion network is built, including: 1 splicing layer, 1 3D convolutional layer and 1 3D pooling layer.

[0017] In step 2, the video stream is an RGB video stream, 16 frames of video stream are obtained in real time, and the optical flow map is obtained by calculating the 16 frames of video stream.

[0018] Preferably, step 1 includes: establishing a spatial stream three-dimensional convolutional neural network model, whose input is X RGB That is, the RGB video frame, the output feature is recorded as the output feature X′ RGB ;

[0019] Establish a time stream 3D convolutional network model, whose input is X FLOW That is, the optical flow map of the RGB image, output time flow feature X′ FLOW ; Specifically include:

[0020] The input of the spatial stream 3D convolutional neural network is

[0021] in,

[0022] X RGB 16 consecutive RGB video frames representing the input spatial stream of the 3D convolutional neural network,

[0023] Represents a set of several consecutive 16-frame RGB video frames.

[0024] D 1 Indicates the number of input video frame channels,

[0025] T represents the number of input video frames,

[0026] W 1 and H 1 Respectively represent the width and height of the video frame;

[0027] The input of the temporal stream 3D convolutional network is

[0028] in,

[0029] X FLOW Optical flow graph representing the x- and y-axis optical flow extraction directions of the input temporal stream 3D convolutional neural network,

[0030] Represents a set of optical flow maps containing several x- and y-axis optical flow extraction directions,

[0031] D 2 Indicates the number of input video frame channels,

[0032] B represents the number of directions of optical flow extraction,

[0033] W 2 and H 2 Respectively represent the width and height of the optical flow map;

[0034] The output characteristics of the spatial stream 3D convolutional network are The characteristics of the output of the time stream 3D convolutional network are

[0035] in,

[0036] D 1 ′ represents the number of video frame channels output by the spatial stream 3D convolutional network,

[0037] T′ represents the number of video frames output by the spatial stream 3D convolutional network,

[0038] W 1 ′ represents the width of the video frame output by the spatial stream 3D convolutional network,

[0039] H 1 ′ represents the height of the video frame output by the spatial stream 3D convolutional network,

[0040] D 2 ′ represents the number of video frame channels output by the temporal stream 3D convolutional network,

[0041] B′ represents the number of optical flow extraction directions output by the temporal stream 3D convolutional network,

[0042] W 2 ′ represents the width of the optical flow map output by the time stream 3D convolutional network,

[0043] H 2 ′ represents the height of the optical flow map output by the time stream 3D convolutional network,

[0044] D 1 ′=D 2 ′, T′=B′, W 1 =W 2 ′,H 1 =H 2 ′.

[0045] Step 1.2, fuse the output features of the time stream 3D convolutional network and the output features of the spatial stream 3D convolutional network; first, X′ RGB and X′ FLOW Perform splicing in the channel dimension, and then use a three-dimensional convolution layer and a three-dimensional pooling layer to perform time feature fusion on the spliced ​​results to obtain a fused feature set Z after time feature fusion; specifically including:

[0046] Perform spatial and temporal fusion on the features output in step 1.1.

[0047] First, perform spatial fusion and transform X′ RGB and X′ FLOW Perform splicing in the channel dimension:

[0048] Z cat =concat(X′ RGB ,X′ FLOW )

[0049] in,

[0050] Z cat The spatial fusion feature set after the spatial features output by the spatial stream neural network and the temporal features output by the temporal stream network are concatenated is:

[0051] in,

[0052] Represents the set of fused features after the two-stream feature concatenation and fusion.

[0053] 2D′×U×W′×H′ represents the fusion feature dimension after the two-stream feature concatenation and fusion.

[0054] 2D′=D 1 ′+D 2 ′, U=T′=B′, W′=W 1 ′=W 2 ′, H′=H 1 ′=H 2 ′;

[0055] Next, we use a 3D convolution layer and a 3D pooling layer to transform Z cat Perform time feature fusion, and obtain the fusion feature set after time feature fusion:

[0056] in,

[0057] Z represents the fusion features after dimensionality reduction.

[0058] D″×U′×W″×H″ represents the dimension of the fusion feature after dimensionality reduction processing, D″=D′, U′=U, W″=W′, H″=H′.

[0059] Step 1.3, input the fused feature set Z into the three fully connected layers, reduce the feature dimension, and finally output the predicted probability of the i-th behavior in the last fully connected layer; specifically, it includes:

[0060] The fusion result after step 1.2 is input into 3 layers of fully connected layers to reduce the feature dimension and finally output in the last fully connected layer.

[0061] in,

[0062] N represents the number of abnormal human behavior categories at the power grid infrastructure construction site,

[0063] Represents the probability of the occurrence of the i-th behavior predicted by the fusion network output.

[0064] Step 1.4: The spatial flow feature X′ output by the dual-channel three-dimensional convolutional neural network in step 1.1 RGB and the time flow feature X′ FLOW Input the three fully connected layers respectively, and obtain the probability of the occurrence of the ith behavior predicted by the spatial stream network output and the probability of the occurrence of the ith behavior predicted by the temporal stream network output; specifically, including:

[0065] The spatial flow feature X′ output by the dual-channel three-dimensional convolutional neural network in step 1.1 RGB and the time flow feature X′ FLOW Input each of the three fully connected layers to reduce the feature dimension and finally output it in the last fully connected layer. and

[0066] in, Represents the probability of the occurrence of the i-th behavior predicted by the output of the spatial stream network.

[0067] Represents the probability of the occurrence of the i-th behavior predicted by the output of the time stream network.

[0068] Step 1.5, using the historical operator monitoring video data set collected by the on-site video monitoring device of the power grid infrastructure, manually annotate to obtain the video sample X and the annotated true value Y, input the video sample X into the network training built in steps 1.1 to 1.4, and obtain the probability of the behavior predicted by the corresponding output of the spatial flow network output, the probability of the behavior predicted by the temporal flow network output, and the probability of the predicted behavior output based on the fusion feature output, and use the cross entropy loss calculation method to determine the degree of fitting of the video sample X, thereby determining the spatial flow three-dimensional convolutional neural network, the temporal flow three-dimensional convolutional neural network, and the cross entropy loss influencing factor of the fusion of spatial flow features and temporal flow features.

[0069] The cross entropy loss calculation formula is as follows:

[0070] in,

[0071] λ 1 represents the cross entropy loss influencing factor of the spatial flow three-dimensional convolutional neural network,

[0072] λ 2 represents the cross entropy loss influencing factor of the time stream 3D convolutional neural network,

[0073] λ 3 Represents the cross entropy loss influencing factor of the fusion of spatial stream features and temporal stream features,

[0074] i∈[1,N], represents one of the categories of abnormal human behavior at the power grid construction site,

[0075] Determine λ through the training process 1 ,λ 2 ,λ 3 Specific value.

[0076] A human abnormal behavior recognition system at a power grid infrastructure construction site includes: a model building module, a data acquisition module, a classification and recognition module, and a result output module, wherein:

[0077] The model building module is used to build a dual-channel three-dimensional convolutional neural network and a dual-stream feature fusion network. Three fully connected layers are connected to the dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network respectively. The network is trained using the historical staff operation video data set collected by the power grid infrastructure site operation monitoring device to obtain a human abnormal behavior recognition model at the power grid infrastructure site.

[0078] The data acquisition module is used to collect the operation video data of the power grid infrastructure site workers in real time from the power grid infrastructure site operation monitoring device, and obtain the real-time video stream and optical flow map;

[0079] The classification and recognition module is used to input the latest video stream and optical flow map into the abnormal human behavior recognition model at the power grid infrastructure construction site to obtain the behavior classification result output;

[0080] The result output module is used to judge the classification loss result of the classification recognition module. If no abnormality occurs, it returns to continue to execute the task of abnormal human behavior detection at the power grid infrastructure construction site; if an abnormality occurs, it performs alarm processing and then continues to execute the task of abnormal human behavior detection at the power grid infrastructure construction site.

[0081] The beneficial effect of the present invention is that, compared with the prior art,

[0082] (1) The present invention adopts a method for identifying abnormal human behavior at a power grid construction site based on a dual-channel three-dimensional convolutional neural network, which realizes the real-time detection task of abnormal human behavior at a power grid construction site while ensuring accuracy;

[0083] (2) Different from the prior art method of using only two-dimensional convolutional networks to construct feature extractors, the present invention proposes a feature extractor based on three-dimensional convolutional neural networks, and extracts spatial features and temporal features of videos respectively through a dual-channel approach, achieving end-to-end fusion, and extracting temporal features between video sequences in the temporal dimension. On the basis of taking into account both real-time and accuracy, the problem that the model cannot well learn long-term information between video frames in human body abnormality data at power grid infrastructure construction sites is solved;

[0084] (3) The present invention is different from the prior art in that it adopts the decision-level fusion method. In the network model training process, the present invention performs feature fusion of the spatial network and the temporal network instead of the decision score fusion. The spatial stream feature and the temporal stream feature are fused in the 5b convolution layer. The feature fusion method can obtain higher-dimensional information, which is conducive to improving the accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 This is a flow chart of a method for identifying abnormal human behavior at a power grid infrastructure construction site proposed by the present invention;

[0086] Figure 2 It is a schematic diagram of the dual-channel three-dimensional convolutional neural network structure constructed by the present invention. DETAILED DESCRIPTION

[0087] The present application is further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present application.

[0088] Example 1.

[0089] like Figure 1 As shown, a method for identifying abnormal human behavior at a power grid infrastructure construction site includes the following steps:

[0090] Step 1, build a dual-channel three-dimensional convolutional neural network and a fusion network, and respectively connect three fully connected layers after the dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network. The dual-channel three-dimensional convolutional neural network includes: a spatial stream three-dimensional convolutional neural network and a temporal stream three-dimensional convolutional neural network. The dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network are trained using the historical staff operation video data collected by the power grid infrastructure site operation monitoring device to obtain a human abnormal behavior recognition model at the power grid infrastructure site;

[0091] Preferably, in this embodiment, the spatial stream three-dimensional convolutional neural network and the temporal stream three-dimensional convolutional neural network share network parameters, and the network parameters include: network weights and bias values.

[0092] Step 1.1, build a dual-channel three-dimensional convolutional neural network, such as Figure 2Shown are the spatial stream 3D convolutional neural network and the temporal stream 3D convolutional neural network. Both networks are composed of 8 3D convolutional layers and 5 3D maximum pooling layers, which are used to extract the spatial features and temporal features of the staff's work video data respectively.

[0093] Establish a spatial stream 3D convolutional neural network model with input X RGB That is, the RGB video frame, the output feature is recorded as the output feature X′ RGB ;

[0094] Establish a time stream 3D convolutional network model, whose input is X FLOW That is, the optical flow map of the RGB image, output time flow feature X′ FLOW ;

[0095] The input of the spatial stream 3D convolutional neural network is

[0096] in,

[0097] X RGB 16 consecutive RGB video frames representing the input spatial stream of the 3D convolutional neural network,

[0098] Represents a set of several consecutive 16-frame RGB video frames.

[0099] D 1 Indicates the number of input video frame channels,

[0100] T represents the number of input video frames,

[0101] W 1 and H 1 Represents the width and height of the video frame respectively.

[0102] The input of the temporal stream 3D convolutional network is

[0103] in,

[0104] X FLOW Optical flow graph representing the x- and y-axis optical flow extraction directions of the input temporal stream 3D convolutional neural network,

[0105] Represents a set of optical flow maps containing several x- and y-axis optical flow extraction directions,

[0106] D 2 Indicates the number of input video frame channels,

[0107] B represents the number of directions of optical flow extraction,

[0108] W 2 and H 2 Represent the width and height of the optical flow map respectively.

[0109] The output features of the spatial stream 3D convolutional network are The output features of the time stream 3D convolutional network are

[0110] in:

[0111] D 1 ′ represents the number of video frame channels output by the spatial stream 3D convolutional network,

[0112] T′ represents the number of video frames output by the spatial stream 3D convolutional network,

[0113] W 1 ′ represents the width of the video frame output by the spatial stream 3D convolutional network,

[0114] H 1 ′ represents the height of the video frame output by the spatial stream 3D convolutional network,

[0115] D 2 ′ represents the number of video frame channels output by the temporal stream 3D convolutional network,

[0116] B′ represents the number of optical flow extraction directions output by the temporal stream 3D convolutional network,

[0117] W 2 ′ represents the width of the optical flow map output by the time stream 3D convolutional network,

[0118] H 2 ′ represents the height of the optical flow map output by the time stream 3D convolutional network,

[0119] ″

[0120] D 1 ′=D 2 ′, T′=B′, W 1 =W 2 ′,H 1 =H 2 ′;

[0121] The purpose of this step is to extract spatial flow features and temporal flow features of the same dimension for the dual-channel neural network.

[0122] Step 1.2, fuse the output features of the time stream 3D convolutional network and the output features of the spatial stream 3D convolutional network; first, X′ RGB and X′ FLOW Perform splicing in the channel dimension, and then use a three-dimensional convolution layer and a three-dimensional pooling layer to perform time feature fusion on the spliced ​​results to obtain a fused feature set Z after time feature fusion;

[0123] Construct a two-stream feature fusion network, such as Figure 2As shown, the fusion network is used to fuse the spatial features X′ output by the spatial stream 3D convolutional neural network RGB And the temporal features X′ output by the temporal stream 3D convolutional neural network FLOW .

[0124] In this embodiment, preferably, the dual-stream feature fusion network includes: 1 concatenation layer, 1 3D convolution layer and 1 3D pooling layer.

[0125] First, perform spatial fusion and transform X′ RGB and X′ FLOW Perform splicing in the channel dimension:

[0126] Z cat =concat(X′ RGB ,X′ FLOW )

[0127] Z cat The spatial fusion feature set after the spatial features output by the spatial stream neural network and the temporal features output by the temporal stream network are concatenated is:

[0128] in,

[0129] Represents the set of fused features after the two-stream feature concatenation and fusion.

[0130] 2D′×U×W′×H′ represents the fusion feature dimension after the two-stream feature concatenation and fusion.

[0131] 2D′=D 1 ′+D 2 ′, U=T′=B′, W′=W 1 ′=W 2 ′, H′=H 1 ′=H 2 ′;

[0132] Next, we use a 3D convolution layer and a 3D pooling layer to transform Z cat Perform time feature fusion, and obtain the fusion feature set after time feature fusion:

[0133] in,

[0134] Z represents the fusion features after dimensionality reduction.

[0135] D″×U′×W″×H″ represents the dimension of the fusion feature after dimensionality reduction.

[0136] D″=D′, U′=U, W″=W′, H″=H′.

[0137] By fusing spatial and temporal features, higher-dimensional features can be extracted, which is conducive to improving the accuracy of the algorithm.

[0138] Step 1.3: Input the fused feature set Z into three fully connected layers to reduce the feature dimension, and finally output the predicted probability of the occurrence of the i-th behavior in the last fully connected layer; input the fused result in step 1.2 into three fully connected layers to reduce the feature dimension, and finally output the last fully connected layer

[0139] in,

[0140] N represents the number of abnormal human behavior categories at the power grid infrastructure construction site,

[0141] Represents the probability of the occurrence of the i-th behavior predicted by the fusion network output.

[0142] Step 1.4: The spatial flow feature X′ output by the dual-channel three-dimensional convolutional neural network in step 1.1 RGB and the time flow feature X′ FLOW Input the three fully connected layers respectively, and obtain the probability of the occurrence of the ith behavior predicted by the spatial flow network output and the probability of the occurrence of the ith behavior predicted by the temporal flow network output; specifically, the spatial flow feature X′ output by the dual-channel three-dimensional convolutional neural network in step 1.1 RGB and the time flow feature X′ FLOW Input each of the three fully connected layers to reduce the feature dimension and finally output it in the last fully connected layer. and

[0143] in,

[0144] N is the number of abnormal human behavior categories at the power grid construction site,

[0145] represents the probability of the occurrence of the ith behavior predicted by the output of the spatial stream network,

[0146] Represents the probability of the occurrence of the i-th behavior predicted by the output of the time stream network.

[0147] Step 1.5, use the historical operator monitoring video data set collected by the power grid infrastructure on-site operation video monitoring device to perform manual annotation to obtain video sample X and the annotation true value Y, input the video sample X into the network training built in steps 1.1 to 1.4, and obtain the probability of the behavior predicted by the corresponding output of the spatial flow network output, the probability of the behavior predicted by the temporal flow network output, and the probability of the predicted behavior based on the output of the fusion feature. The cross entropy loss calculation method is used to determine the degree of fitting of the video sample X, thereby determining the spatial flow three-dimensional convolutional neural network, the temporal flow three-dimensional convolutional neural network, and the cross entropy loss influencing factor of the fusion of spatial flow features and temporal flow features.

[0148] The historical operator monitoring video dataset collected by the on-site video monitoring device of the power grid infrastructure is manually labeled to obtain the video sample X and the annotation true value Y = {y 1 ,y 2 ,…,y N}, input the video sample X into the network training built in the above step, and get the corresponding output and The cross entropy loss algorithm is used to calculate the output and The degree of fitting of the video sample X makes it as close as possible to the true value Y:

[0149]

[0150] in,

[0151] λ 1 represents the cross entropy loss influencing factor of the spatial flow three-dimensional convolutional neural network,

[0152] λ 2 represents the cross entropy loss influencing factor of the time stream 3D convolutional neural network,

[0153] λ 3 Represents the cross entropy loss influencing factor of the fusion of spatial stream features and temporal stream features,

[0154] i∈[1,N], represents one of the categories of abnormal human behavior at the power grid infrastructure construction site.

[0155] Determine λ through the training process 1 , 2 , 3 Specific value.

[0156] By calculating the loss function and using the back-propagation algorithm to update the network parameters, the final loss function converges to a very small value, thus completing the algorithm training and generating a model for identifying abnormal human behavior at the power grid infrastructure site.

[0157] Step 2: Collecting the video data of the power grid construction site operation in real time from the power grid construction site operation monitoring device, obtaining the real-time video stream and optical flow map, and forming the input X of the abnormal human behavior recognition model at the power grid construction site;

[0158] Preferably, in this embodiment, the video stream is an RGB video stream, 16 frames of video stream are acquired in real time, and the optical flow map is obtained by calculating the 16 frames of video stream.

[0159] Step 3, input the data into the abnormal human behavior recognition model at the power grid infrastructure construction site generated in the above algorithm training phase, and obtain the behavior classification result output;

[0160] Step 4: Based on the behavior classification result obtained by the abnormal human behavior recognition model at the power grid infrastructure construction site in step 3, if the model outputs an abnormal behavior category, an alarm is issued, and then the abnormal human behavior detection task at the power grid infrastructure construction site is continued; if the model outputs a normal behavior category, the task of detecting abnormal human behavior at the power grid infrastructure construction site is returned to continue.

[0161] Example 2.

[0162] A human abnormal behavior recognition system at a power grid infrastructure construction site includes: a model building module, a data acquisition module, a classification and recognition module, and a result output module, wherein:

[0163] The model building module is used to build a dual-channel three-dimensional convolutional neural network and a dual-stream feature fusion network. The network is trained using the historical staff operation video data set collected by the power grid infrastructure site operation monitoring device to obtain a human abnormal behavior recognition model at the power grid infrastructure site.

[0164] The data acquisition module is used to collect the operation video data of the power grid infrastructure site workers in real time from the power grid infrastructure site operation monitoring device, obtain the latest 16 frames of RGB images of the real-time video and calculate the optical flow map of these 16 frames of RGB images;

[0165] The classification and recognition module is used to input the latest video stream and optical flow map into the abnormal human behavior recognition model at the power grid infrastructure construction site to obtain the behavior classification result output;

[0166] The result output module is used to judge the classification loss result of the classification recognition module. If no abnormality occurs, it returns to continue to execute the task of abnormal human behavior detection at the power grid infrastructure construction site; if an abnormality occurs, it performs alarm processing and then continues to execute the task of abnormal human behavior detection at the power grid infrastructure construction site.

[0167] The beneficial effect of the present invention is that, compared with the prior art,

[0168] (1) The present invention adopts a method for identifying abnormal human behavior at a power grid construction site based on a dual-channel three-dimensional convolutional neural network, which realizes the real-time detection task of abnormal human behavior at a power grid construction site while ensuring accuracy;

[0169] (2) Different from the prior art method of using only two-dimensional convolutional networks to construct feature extractors, the present invention proposes a feature extractor based on three-dimensional convolutional neural networks, and extracts spatial features and temporal features of videos respectively through a dual-channel approach, achieving end-to-end fusion, and extracting temporal features between video sequences in the temporal dimension. On the basis of taking into account both real-time and accuracy, the problem that the model cannot well learn long-term information between video frames in human body abnormality data at power grid infrastructure construction sites is solved;

[0170] (3) The present invention is different from the prior art in that it adopts the decision-level fusion method. In the network model training process, the present invention performs feature fusion of the spatial network and the temporal network instead of the decision score fusion. The spatial stream feature and the temporal stream feature are fused in the 5b convolution layer. The feature fusion method can obtain higher-dimensional information, which is conducive to improving the accuracy.

[0171] The applicant of the present invention has made a detailed explanation and description of the implementation examples of the present invention in conjunction with the drawings in the specification. However, those skilled in the art should understand that the above implementation examples are only preferred implementation schemes of the present invention, and the detailed description is only to help readers better understand the spirit of the present invention, but not to limit the scope of protection of the present invention. On the contrary, any improvements or modifications based on the inventive spirit of the present invention should fall within the scope of protection of the present invention.

Claims

1. A method for identifying abnormal human behavior at a power grid construction site, characterized in that: The following steps are included: Step 1, build a dual-channel three-dimensional convolutional neural network and a dual-stream feature fusion network, the dual-channel three-dimensional convolutional neural network includes: a spatial stream three-dimensional convolutional neural network and a temporal stream three-dimensional convolutional neural network; input the video frame into the spatial stream three-dimensional convolutional neural network, and output the spatial feature; input the image optical flow map into the temporal stream three-dimensional convolutional neural network, and output the temporal feature; fuse the spatial feature and the temporal feature to obtain a fusion feature set Z, input the fusion feature set Z into three layers of fully connected layers, and output the predicted probability of the occurrence of the i-th behavior in the last layer of fully connected layers; use the historical staff operation video data set collected by the power grid infrastructure site operation monitoring device to train the dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network to obtain a power grid infrastructure site abnormal human behavior recognition model, the recognition model includes using the cross entropy loss calculation method to determine the spatial stream three-dimensional convolutional neural network, the temporal stream three-dimensional convolutional neural network, and the cross entropy loss influencing factor of the fusion of spatial stream features and temporal stream features, as shown in the following formula: Among them, y i represents the i-th value in the labeled true value Y, represents the probability of the occurrence of the ith behavior predicted by the output of the spatial stream 3D convolutional neural network, represents the probability of the occurrence of the ith behavior predicted by the output of the time stream 3D convolutional neural network, represents the probability of the occurrence of the i-th behavior predicted by the output of the dual-stream feature fusion network; the spatial stream features and temporal stream features output by the dual-channel three-dimensional convolutional neural network are respectively input into their respective three-layer fully connected layers, and the probability of the occurrence of the i-th behavior predicted by the output of the spatial stream three-dimensional convolutional neural network and the probability of the occurrence of the i-th behavior predicted by the output of the temporal stream three-dimensional convolutional neural network are respectively obtained; λ1 represents the cross-entropy loss influence factor of the spatial stream three-dimensional convolutional neural network, λ2 represents the cross-entropy loss influence factor of the temporal stream three-dimensional convolutional neural network, λ3 represents the cross-entropy loss influence factor of the fusion of spatial stream features and temporal stream features, i∈[1,N], represents one of the abnormal human behavior categories at the power grid infrastructure site, N represents the number of abnormal human behavior categories at the power grid infrastructure site, and the specific values ​​of λ1, λ2, and λ3 are determined through the training process; Step 2: Collecting the operation video data of the power grid infrastructure construction site workers in real time from the power grid infrastructure construction site operation monitoring device to obtain the real-time video stream and optical flow map; Step 3, input the video stream and the optical flow map into the abnormal human behavior recognition model at the power grid infrastructure construction site to obtain the behavior classification result output; Step 4: Based on the behavior classification result obtained by the abnormal human behavior recognition model at the power grid infrastructure construction site in step 3, if the model outputs an abnormal behavior category, an alarm is issued, and then the abnormal human behavior detection task at the power grid infrastructure construction site is continued; if the model outputs a normal behavior category, the task of detecting abnormal human behavior at the power grid infrastructure construction site is returned to continue.

2. A method for identifying abnormal human behavior at a power grid infrastructure site according to claim 1, characterized in that: In step 2, the video stream is an RGB video stream, 16 frames of video stream are obtained in real time, and the optical flow map is obtained by calculating the 16 frames of video stream.

3. A method for identifying abnormal human behavior at a power grid infrastructure construction site according to claim 1, characterized in that: In step 1, the spatial stream three-dimensional convolutional neural network and the temporal stream three-dimensional convolutional neural network share network parameters, which include: network weights and bias values.

4. A method for identifying abnormal human behavior at a power grid infrastructure construction site according to claim 3, characterized in that: In step 1, the dual-channel 3D convolutional neural network includes: 8 3D convolutional layers and 5 3D pooling layers.

5. A method for identifying abnormal human behavior at a power grid infrastructure construction site according to claim 4, characterized in that: The two-stream feature fusion network includes: 1 concatenation layer, 1 3D convolution layer and 1 3D pooling layer.

6. A method for identifying abnormal human behavior at a power grid infrastructure construction site according to claim 5, characterized in that: Step 1 includes: Step 1.1: Establish a spatial stream 3D convolutional neural network model with input X RGB That is, the RGB video frame, the output feature is recorded as the output feature X' RGB ; Establish a time stream 3D convolutional neural network model with input X FLOW That is, the optical flow map of the RGB image, output time flow feature X' FLOW ; Step 1.2, fuse the output features of the time stream 3D convolutional neural network and the output features of the space stream 3D convolutional neural network; first, X' RGB and X' FLOW Perform splicing in the channel dimension, and then use a three-dimensional convolution layer and a three-dimensional pooling layer to perform time feature fusion on the spliced ​​results to obtain a fused feature set Z after time feature fusion; Step 1.3, input the fused feature set Z into three fully connected layers, reduce the feature dimension, and finally output the predicted probability of the occurrence of the i-th behavior in the last fully connected layer; Step 1.4: The spatial flow feature X' output by the dual-channel three-dimensional convolutional neural network in step 1.1 RGB and the time flow characteristic X' FLOW Input the three fully connected layers respectively, and obtain the probability of the occurrence of the i-th behavior predicted by the output of the spatial stream three-dimensional convolutional neural network and the probability of the occurrence of the i-th behavior predicted by the output of the temporal stream three-dimensional convolutional neural network respectively; Step 1.5, use the historical operator monitoring video data set collected by the power grid infrastructure on-site operation video monitoring device to perform manual annotation to obtain video sample X and the annotation true value Y, input the video sample X into the network training built in steps 1.1 to 1.4, and obtain the probability of the behavior predicted by the output of the spatial stream three-dimensional convolutional neural network, the probability of the behavior predicted by the output of the temporal stream three-dimensional convolutional neural network, and the probability of the predicted behavior based on the output of the fusion feature. The cross-entropy loss calculation method is used to determine the degree of fitting of the video sample X, thereby determining the cross-entropy loss influencing factors of the spatial stream three-dimensional convolutional neural network, the temporal stream three-dimensional convolutional neural network, and the fusion of spatial stream features and temporal stream features.

7. A method for identifying abnormal human behavior at a power grid infrastructure construction site according to claim 6, characterized in that: In step 1.1, it specifically includes: The input of the spatial stream 3D convolutional neural network is in, X RGB 16 consecutive RGB video frames representing the input spatial stream of the 3D convolutional neural network, Represents a set of several consecutive 16-frame RGB video frames. D1 represents the number of input video frame channels. T represents the number of input video frames, W1 and H1 represent the width and height of the video frame respectively; The input of the temporal stream 3D convolutional neural network is in, X FLOW Optical flow graph representing the x- and y-axis optical flow extraction directions of the input temporal stream 3D convolutional neural network, Represents a set of optical flow maps containing several x- and y-axis optical flow extraction directions, D2 represents the number of input video frame channels, B represents the number of directions of optical flow extraction, W2 and H2 represent the width and height of the optical flow map respectively; The characteristics of the spatial stream 3D convolutional neural network output are The characteristics of the output of the time stream 3D convolutional neural network are in, D1′ represents the number of video frame channels output by the spatial stream 3D convolutional neural network, T′ represents the number of video frames output by the spatial stream 3D convolutional neural network, W1′ represents the width of the video frame output by the spatial stream 3D convolutional neural network, H1′ represents the height of the video frame output by the spatial stream 3D convolutional neural network, D2′ represents the number of video frame channels output by the temporal stream 3D convolutional neural network, B′ represents the number of optical flow extraction directions output by the temporal stream 3D convolutional neural network, W2′ represents the width of the optical flow map output by the temporal 3D convolutional neural network. H2′ represents the height of the optical flow map output by the temporal 3D convolutional neural network. D1′=D2′, T′=B′, W1′=W2′, H1′=H2′.

8. A method for identifying abnormal human behavior at a power grid infrastructure site according to claim 6 or 7, characterized in that: In step 1.2, it specifically includes: first, perform spatial fusion, transform X′ RGB and X′ FLOW Perform splicing in the channel dimension: Z cat =concat(X′ RGB ,X′ FLOW ) in, Z cat It represents the spatial fusion feature set after the spatial features output by the spatial stream 3D convolutional neural network and the temporal features output by the temporal stream 3D convolutional neural network are spliced ​​together. in, Represents the set of fused features after the two-stream feature concatenation and fusion. 2D′×U×W′×H′ represents the fusion feature dimension after the two-stream feature concatenation and fusion. 2D′=D1′+D2′, U=T′=B′, W′=W1′=W2′, H′=H1′=H2′; Next, we use a 3D convolution layer and a 3D pooling layer to transform Z cat Perform time feature fusion, and obtain the fusion feature set after time feature fusion: in, Z represents the fusion features after dimensionality reduction. D″×U′×W″×H″ represents the dimension of the fusion feature after dimensionality reduction. D″=D′, U′=U, W″=W′, H″=H′.

9. A system for identifying abnormal human behavior at a power grid infrastructure site using the method according to any one of claims 1 to 8, wherein the system comprises: Model building module, data collection module, classification and recognition module and result output module; characterized in that, The model building module is used to build a dual-channel three-dimensional convolutional neural network and a dual-stream feature fusion network. Three fully connected layers are connected to the dual-channel three-dimensional convolutional neural network and the dual-stream feature fusion network respectively. The network is trained using the historical staff operation video data set collected by the power grid infrastructure site operation monitoring device to obtain a human abnormal behavior recognition model at the power grid infrastructure site. The data acquisition module is used to collect the operation video data of the power grid infrastructure site workers in real time from the power grid infrastructure site operation monitoring device, and obtain the real-time video stream and optical flow map; The classification and recognition module is used to input the latest video stream and optical flow map into the abnormal human behavior recognition model at the power grid infrastructure construction site to obtain the behavior classification result output; The result output module is used to judge the classification loss result of the classification recognition module. If no abnormality occurs, it returns to continue to execute the task of abnormal human behavior detection at the power grid infrastructure construction site; if an abnormality occurs, it performs alarm processing and then continues to execute the task of abnormal human behavior detection at the power grid infrastructure construction site.