Abnormal behavior recognition method, system, device and medium enhanced by skeleton joint points
Through the space-time convolution network enhanced based on skeleton nodes, the problems of insufficient expression ability and high computational complexity of skeleton behavior recognition in the dynamic environment in the prior art are solved, and efficient abnormal behavior recognition in complex contexts is achieved, improving the Identification accuracy and recall rate.
Patent Information
- Application Number
- CN202210568679.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-05-24
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-05-24
AI Technical Summary
The existing skeleton-based behavior recognition method has poor results in dynamic environments and complex backgrounds. The expression ability of graph convolutional networks is limited, the spatiotemporal feature extraction is insufficient, and the computational complexity is high.
The space-time convolution network based on skeleton joint node enhancement is adopted to input the continuous space-time convolution module through multi-frame skeleton sequences to perform joint enhancement. Combined with the space-enhanced shift module and the time-shift module, the local and timing characteristics of the joint are learned. , use the global average pooling layer and classifier for exception behavior recognition.
Effectively detect abnormal behaviors in the human body, improve the recognition accuracy and recall rate, achieve high performance with less calculation amount, and highlight the characteristic information of important nodes.
Smart Images

Figure CN114782998B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to an abnormal behavior recognition method based on a skeleton joint point enhanced spatiotemporal convolutional network. Background Art
[0002] Abnormal human behavior detection is one of the research hotspots in the field of human behavior recognition. It has important applications in smart cities, intelligent monitoring, and human-computer interaction. In the field of medical monitoring, through abnormal behavior recognition and detection technology, real-time monitoring of unattended patients or elderly people can be achieved, judging whether the target has fallen or other accidents and calling the police in time to ensure that they receive treatment and help as soon as possible; in the field of traffic supervision, relevant departments can use abnormal human behavior recognition and detection technology to monitor abnormal conditions inside and outside the car, including driver behavior, to reduce the risk of accidents; in the field of public safety, this technology can be used to detect abnormal conditions of people in public places and judge fighting and other violations of social security. Classic video behavior recognition algorithms mostly use RGB video as input, but when there is a lot of background dynamic interference, unstable lighting, and severe noise in the RGB video, the effect of traditional behavior recognition algorithms will be affected. In recent years, skeleton-based behavior recognition has received more and more attention due to its strong adaptability to dynamic environments and complex backgrounds.
[0003] The current mainstream method for skeleton-based action recognition is the graph convolutional network, which applies the graph convolutional network to the spatiotemporal modeling of skeleton sequences and designs an adjacency matrix to represent the physical structure of the human body. "Yan Sijie, Xiong Yuanjun, Lin Dahua. Skeleton Action Recognition Based on Spatiotemporal Graph Convolutional Network [C] / / AAAI Conference on Artificial Intelligence, 2018." proposed a spatiotemporal graph convolutional network, applied the graph convolutional network to the spatiotemporal modeling of skeleton sequences, and designed an adjacency matrix to represent the physical structure of the human body, so that the network has better expression ability and higher performance. However, the skeleton graph used by the network is heuristically predefined, and the expression ability is limited by the spatial rules of the graph convolutional neural network, and the spatiotemporal feature extraction is insufficient. "Shi Lei, Zhang Yifan, Cheng Jian, Lu Hanqing. Skeleton Action Recognition Based on Two-Stream Adaptive Graph Convolutional Network [C] / / IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019." combines the information of joint points and bone lengths, and realizes the adaptability of the adjacency matrix by calculating the correlation of different joints, but there are problems such as too large number of parameters and high computational complexity. "Cheng Ke, Zhang Yifan, He Xiangyu. Skeleton Action Recognition Based on Shifted Graph Convolutional Network [C] / / IEEE / CVF Computer Vision and Pattern Recognition (CVPR), 2020." The shifted convolution is combined with GCN and applied to the action recognition task. The lightweight shift operation provides a flexible receptive field for the spatial graph and the temporal graph, which greatly reduces the computational complexity. However, the network cannot effectively focus on the key information in the skeleton sequence and does not fully extract spatiotemporal features. Summary of the invention
[0004] In view of the defects in the prior art, the purpose of the present invention is to provide an abnormal behavior recognition method based on skeleton joint point enhanced spatiotemporal convolutional network.
[0005] According to one aspect of the present invention, a method for identifying abnormal behavior based on a skeleton joint point enhanced spatiotemporal convolutional network is provided, comprising:
[0006] After preprocessing the dataset, a multi-frame skeleton sequence is generated;
[0007] The multi-frame skeleton sequence is input into a number of consecutive spatiotemporal convolution modules to perform joint enhancement and output temporal features;
[0008] The time series features are input into the global average pooling layer and the classifier to obtain the abnormal behavior recognition result.
[0009] Preferably, the acquiring of a multi-frame skeleton sequence comprises:
[0010] Normalized skeleton joint data:
[0011] For a T-frame video, the center joint point of the first person appearing in the first frame is used as the origin of the coordinate system, and the coordinates of each joint point in the first frame are subtracted from the coordinates of the center joint point; the displacement of the joint in the time dimension is obtained by using the difference between the joint point coordinates of other frames and the first frame;
[0012] Constructing a spatiotemporal topology map:
[0013] Extract a multi-frame skeleton sequence as input through OpenPose, where the skeleton sequence consists of a spatiotemporal graph of N joint points and T frame skeleton sequences;
[0014] In a skeleton graph with N joint points, each joint point is represented by a position coordinate vector X = (x, y, z), where x and y are the horizontal and vertical coordinates of the joint point, and z is the confidence of the joint point;
[0015] In the single-frame spatial dimension, different joint points in the same frame are connected into edges according to the human body structure to form a skeleton graph;
[0016] In the multi-frame time dimension, the same joint point between two adjacent frames is connected into an edge to form a timing graph of T frames.
[0017] Preferably, the number of the spatiotemporal convolution modules is 9;
[0018] The input of each spatiotemporal convolution module is C in ×N×T, the output is C out ×N×T,C in is the feature of the input joint point (number of channels), C out is the feature of the output joint point (number of channels), N is the number of joint points, and T is the total number of frames; the input feature of each layer is the output feature of the previous layer;
[0019] The number of input channels, output channels, and stride lengths of the 9-layer spatiotemporal convolution module are:
[0020] The number of input channels of spatiotemporal convolution block B1 is 3, the number of output channels is 64, and the stride is 1;
[0021] The number of input channels of spatiotemporal convolution block B2 is 64, the number of output channels is 64, and the step size is 1;
[0022] The number of input channels of spatiotemporal convolution block B3 is 64, the number of output channels is 64, and the stride is 1;
[0023] The number of input channels of spatiotemporal convolution block B4 is 64, the number of output channels is 128, and the stride is 2;
[0024] The number of input channels of spatiotemporal convolution block B5 is 128, the number of output channels is 128, and the stride is 1;
[0025] The number of input channels of spatiotemporal convolution block B6 is 128, the number of output channels is 128, and the stride is 1;
[0026] The number of input channels of spatiotemporal convolution block B7 is 128, the number of output channels is 256, and the stride is 2;
[0027] The number of input channels of spatiotemporal convolution block B8 is 256, the number of output channels is 256, and the stride is 1;
[0028] The number of input channels of spatiotemporal convolution block B9 is 256, the number of output channels is 256, and the stride is 1;
[0029] The spatiotemporal feature vector output by the 9-layer spatiotemporal convolution module is C out ×N×T, where C out is the number of output channels (256).
[0030] Preferably, each of the spatiotemporal convolution modules comprises: a spatial enhancement shift module and a temporal shift module; wherein,
[0031] The spatial enhancement shift module includes: spatial shift and joint point enhancement;
[0032] The spatial shift operation in the spatial enhanced shift learns the local features of the joints in a single frame;
[0033] After calculating the similarity weight coefficients between joints, different weights are obtained for each joint point;
[0034] The weight of each joint is extracted using a two-layer perceptron, and then an intermediate feature vector containing the weight coefficient of each joint is generated;
[0035] The features after the spatial shift operation are added to the intermediate features to achieve joint point enhancement;
[0036] The features output by the spatial enhancement shift module are added to the input features and input into the time shift module to learn the temporal features.
[0037] Preferably, the process of spatial shifting includes:
[0038] The single-frame skeleton sequence feature is Where N is the number of joints, C in is the number of input channels, the translation distance of the i-th channel is i modN, the nodes are connected and then translated, each node obtains the information of other nodes in the same frame through feature translation, and then realizes the information exchange between channels through 1×1 convolution;
[0039] Preferably, the process of joint point enhancement includes:
[0040] Calculate the similarity between each joint through Gaussian function;
[0041] The joint similarity weight coefficient is generated after normalization using the Softmax function and input into the global average pooling layer;
[0042] Combine two fully connected layers with the tanh activation function and the sigmoid activation function to extract the joint weight distribution;
[0043] The intermediate feature vector is obtained by weighting the joints in the skeleton sequence through the joint weights. The normalized intermediate feature is added to the input feature to obtain the output feature. The output feature of the spatial enhancement shift module can be expressed as
[0044] Preferably, the time shift module adopts a time-series shift convolution operation, including:
[0045] The input features of the spatiotemporal convolution module and the output features of the spatial enhancement shift module are added as the input features of the temporal shift module, expressed as
[0046] Connect consecutive frames in the time dimension, and shift them so that each frame obtains information from its adjacent frames;
[0047] Each channel has a learnable parameter S i To learn the degree of network displacement;
[0048] After the shift convolution operation, the 1×1 convolution is combined to learn the time series features. The feature vector formed after superimposing the time series features is
[0049] According to a second aspect of the present invention, there is provided an abnormal behavior recognition system based on a skeleton joint point enhanced spatiotemporal convolutional network, comprising:
[0050] Preprocessing module, the original data set is preprocessed to generate a multi-frame skeleton sequence;
[0051] Spatiotemporal convolution module, the multi-frame skeleton sequence is input into several consecutive spatiotemporal convolution modules to perform joint enhancement and output temporal features;
[0052] The identification module inputs the time series features into the global average pooling layer and the classifier to obtain abnormal behavior identification results.
[0053] According to a third aspect of the present invention, an electronic device is provided, the electronic device comprising a processor and a memory, the memory being used to store execution instructions, and the processor being used to execute the execution instructions stored in the memory to implement the methods described in the above embodiments. The processor includes but is not limited to a computer, a server, a network device, a microprocessor, etc.
[0054] According to a fourth aspect of the present invention, a computer-readable storage medium is provided. The computer program related to the abnormal behavior identification system of the embodiment of the present invention can be stored in the storage medium, wherein the steps in any of the above embodiments can be executed when the computer program is running. The above storage medium can be a random access memory (RAM), a read-only memory (ROM), a flash memory, an optical disk, a magnetic disk, a solid-state disk or other memory, or it can be an electronic device, such as a mobile device, a computer, a tablet computer, etc.
[0055] Compared with the prior art, the present invention has the following beneficial effects:
[0056] In an embodiment of the present invention, a method for identifying abnormal behavior based on a skeleton joint point enhanced spatiotemporal convolutional network assigns different degrees of weight to each joint point through a spatial enhanced shift module to highlight the characteristic information of important joint points. The joint point enhanced spatiotemporal convolutional model trained in this embodiment can effectively detect abnormal human behavior and issue an alarm in time. This method achieves a high accuracy and recall rate with less computational effort. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0058] Figure 1 A flowchart of a joint point enhanced shifted spatiotemporal convolutional network according to an embodiment of the present invention;
[0059] Figure 2 It is a flow chart of an abnormal behavior recognition method based on skeleton joint point enhanced spatiotemporal convolutional network according to an embodiment of the present invention;
[0060] Figure 3 A schematic diagram of a skeleton space-time topological structure according to an embodiment of the present invention;
[0061] Figure 4 Schematic diagram of the structure of a spatiotemporal convolution module according to an embodiment of the present invention;
[0062] Figure 5 It is a structural schematic diagram of a spatial enhancement shift module according to an embodiment of the present invention;
[0063] Figure 6 A schematic diagram of a process of abnormal behavior detection according to an embodiment of the present invention;
[0064] Figure 7 A schematic diagram of visualization of abnormal behavior recognition results according to an embodiment of the present invention. DETAILED DESCRIPTION
[0065] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.
[0066] The server of the hardware operating environment involved in the embodiment of the present invention includes a processor, a communication bus, a network interface, a user interface, a memory, etc. The communication bus is used to realize the connection and communication between the components, the user interface is used to receive the user's input instructions, the network interface is used to communicate data with the external network, and the server calls the abnormal behavior recognition program stored in the memory through the processor.
[0067] In the server, the processor includes but is not limited to a computer, a server, a network device, etc., the user interface may be a display screen, an input unit (keyboard), a standard wired interface and a wireless interface, etc., the network interface may be a wired interface and a wireless interface, and the memory may be an optical disk, a magnetic disk, a solid-state disk, etc. When the memory is used as a computer storage medium, it may include an operating system, a network communication module, a user interface module, and an abnormal behavior recognition program.
[0068] like Figure 1 FIG. 1 is a framework diagram of an abnormal behavior recognition method based on a skeleton joint point enhanced spatiotemporal convolutional network according to an embodiment of the present invention, comprising:
[0069] S1, multi-frame skeleton sequence generated after dataset preprocessing;
[0070] S2, inputs the multi-frame skeleton sequence obtained in S1 into several consecutive spatiotemporal convolution modules, performs joint enhancement and outputs temporal features;
[0071] S3, inputs the time series features obtained in S2 into the global average pooling layer and classifier to obtain the abnormal behavior recognition results.
[0072] In a preferred embodiment of the present invention, S1 is implemented to obtain a multi-frame skeleton sequence, which specifically includes:
[0073] S101, normalizing skeleton joint point data: for a video of T frames, the center joint point of the first person appearing in the first frame is used as the origin of the coordinate system, and the coordinates of each joint point in the first frame are subtracted from the coordinates of the center joint point; the displacement of the joint in the time dimension is obtained by using the difference between the joint point coordinates of other frames and the first frame;
[0074] S102, constructing a spatiotemporal topological graph: extracting a skeleton atlas as input through OpenPose, the skeleton atlas consists of a spatiotemporal graph of N joints and a T-frame skeleton sequence; in a skeleton graph with N joints, each joint is represented as a position coordinate vector X=(x, y, z), where x and y are the horizontal and vertical coordinates of the joint, and z is the confidence of the joint; connecting different joints into edges according to the human body structure to form a skeleton graph; connecting the same joint between two adjacent frames to form a T-frame timing graph, such as Figure 3 shown.
[0075] In a preferred embodiment of the present invention, the multi-frame skeleton sequence obtained in S1 is subjected to S2. In this embodiment, 9 spatiotemporal convolution modules are included, and the input features of each layer of spatiotemporal convolution modules are the output features of the previous layer.
[0076] Each spatiotemporal convolution module includes a spatial enhancement shift module and a temporal shift module. The spatial enhancement shift module includes spatial shift and joint point enhancement respectively. The processing process is:
[0077] S201, learning local features of single-frame joints by spatial shift operation in spatial enhancement shift module;
[0078] S202, calculating the similarity weight coefficients between joints and generating an intermediate feature vector;
[0079] S203, extracting joint weight distribution in the skeleton sequence using a two-layer perceptron;
[0080] S204, finally, the features after the shift operation are added to the intermediate features to achieve joint point enhancement; (the features after the shift operation are the features before calculating the joint weights, and the intermediate features are the features after calculating the joint weights).
[0081] S205, the features output by the spatial enhancement shift module are added to the input features and input into the time shift module to learn the time series features, and the time series features are output.
[0082] like Figure 2 FIG. 1 is a flow chart of an abnormal behavior recognition method based on skeleton joint point enhanced spatiotemporal convolutional network according to an embodiment of the present invention. The process of the abnormal behavior recognition method includes:
[0083] S11, a self-made daily behavior dataset, extracts human skeleton sequences using the OpenPose method;
[0084] S12, skeleton-based time series dynamic modeling, construction of spatiotemporal topological graph;
[0085] S13, skeleton data is input into the joint point enhanced spatiotemporal convolutional network for training;
[0086] S14, using the Softmax classifier to classify the output features into behavior categories and obtain corresponding behavior labels;
[0087] S15, obtaining surveillance video and inputting it into the trained model for recognition to determine whether the target person falls.
[0088] In one embodiment of the present invention, S11 specifically includes: there are seven types of data sets, among which walking, sitting down, standing up, undressing, dressing and throwing are taken from the public behavior recognition data set Northwest UCLA, and each action is performed one to six times by ten subjects. The falling behavior data is taken from the public data set Le2i, which contains four scene videos of home, office, coffee room, and lecture room. The data set has a total of 1029 video sequences. According to the division strategy of deep learning, 70% are selected as training sets and 30% as validation sets. The first 300 frames of each video are captured. If there are less than 300 frames, they are filled in a loop;
[0089] In a preferred embodiment, S11 further includes:
[0090] (1) Use ffmpeg to convert the video format;
[0091] The resolution of all videos in the original dataset was adjusted to 320×240 using the ffmpeg algorithm, and the frames per second (FPS) was set to 30.
[0092] (2) The pose estimation algorithm extracts the skeleton coordinate information in the video;
[0093] The pose estimation algorithm extracts the position information of the skeleton joints in the video, including: using the OpenPose algorithm to extract the position feature information (2D coordinates) of the 18 skeleton joints in each frame of the dataset. When there are multiple people in the video, two people are selected based on the average joint confidence.
[0094] (3) Normalized skeleton joint data;
[0095] Normalizing the skeleton joint data, including: for a T-frame video, taking the center joint of the first person appearing in the first frame as the origin of the coordinate system, subtracting the coordinate of the center joint from the coordinate of each joint in the first frame; using the difference between the joint coordinates of other frames and the first frame to obtain the displacement of the joint in the time dimension;
[0096] In another embodiment of the present invention, after S11 extracts the human skeleton sequence, S12 is performed to dynamically model the skeleton time series and construct a spatiotemporal topological map.
[0097] The multi-frame skeleton sequence extracted as input by OpenPose consists of a spatiotemporal graph of N nodes and T-frame bone sequences. In the skeleton graph with N joint points, each joint point can be represented as a position coordinate vector X = (x, y, z), where x and y are the horizontal and vertical coordinates of the joint point, and z is the confidence of the joint point. Then the feature vector set of N joints can be expressed as:
[0098] V={X i |i=1,2,...,N}
[0099] Among them, X i Represents the position coordinate vector of the i-th joint point. The connection relationship of the joint points in a single-frame skeleton can be expressed as:
[0100] E s ={X ti X tj |i,j=1,2,...,N}
[0101] Among them, X ti represents the position coordinates of the i-th joint point at time t, X tj Represents the position coordinates of the j-th joint point at time t.
[0102] In space, the skeleton graph is composed of N joints and the connection relationship between the joints. According to the human body structure, the skeleton graph is represented as G s =(V,E s ). The timing graph is composed of T frames with the same skeleton graph, so the same joint point between two adjacent frames is connected, and the connection relationship in the time dimension can be expressed as:
[0103] E F ={X ti X (t+1)i |t=1,2,...,T}
[0104] Among them, X ti represents the position coordinates of the i-th joint point at time t, X (t+1)i represents the position coordinates of the i-th joint point at time t+1. s and E F The edge set E that constitutes the timing graph is merged, so the human skeleton sequence of T frames can be expressed as G = (V, E).
[0105] The schematic diagram of the skeleton space-time topology is as follows Figure 3 As shown, in the same frame (space), each joint point is connected into edges as bone length according to the human body structure to form a spatial skeleton diagram. In multiple frames (time), the same joint point in two adjacent frames is connected to form a timing diagram.
[0106] In another embodiment of the present invention, based on the multi-frame skeleton sequence obtained in S11 and S12, S13 is performed. In this embodiment, the joint point enhanced spatiotemporal convolutional network refers to Figure 1 As shown, it includes: normalization layer, 9-layer spatiotemporal convolution module, global average pooling layer, and fully connected layer.
[0107] Each spatiotemporal convolution module includes: a spatial enhancement shift module and a temporal shift module.
[0108] The skeleton spatiotemporal graph obtained after S11 and S12 preprocessing can be expressed as N×C×T×V×M. Among them, N represents the number of videos, C represents the number of features contained in the joint points (position coordinates and confidence), T represents the number of video frames, V represents the number of human joints, and M represents the number of human skeletons in a single frame. The obtained joint point coordinate data is normalized in the range of [-1,1] according to the size of the video, so the skeleton data is converted into a three-dimensional tensor C in ×N×T is input into the first spatiotemporal convolution module, where N is the number of joint points and C in are the features x, y and z of the input joint points (i.e. the number of channels is 3), and T is the number of video frames.
[0109] like Figure 4 As shown, the spatial enhancement shift module further includes: the first spatial shift, 1×1 convolution, the second spatial shift, Gaussian function + Softmax, global average pooling layer, fully connected layer, tanh activation function, fully connected layer, Sigmoid activation function, normalization, and RELU activation function.
[0110] The spatial enhancement shift module further comprises:
[0111] (1) Spatial displacement;
[0112] The spatial shift operation learns the local features of single-frame joints: the single-frame skeleton sequence feature is Where N is the number of joints, C in is the number of input channels, the translation distance of the i-th channel is i modN, and the nodes are connected and then shifted and convolved. For node v, its adjacent node set is Where n represents the number of adjacent nodes of v. Divide the channel of node v into n+1 partitions evenly. The first partition retains the characteristics of v, and the others are divided into n+1 partitions. Shift in. Perform a shift operation on each node of F, the formula is as follows:
[0113]
[0114] in, || indicates connection by channel. In the shift feature, each node obtains information from its receptive field, and then combines with 1×1 convolution to realize information exchange between channels.
[0115] That is, after the above-mentioned spatial shift, skeleton feature information in the spatial dimension is extracted from each frame of the multi-frame skeleton sequence.
[0116] (2) Joint enhancement;
[0117] like Figure 5 As shown, first use the Gaussian function to calculate the similarity between joints in the same frame to determine the connection strength between two joint points. The calculation formula is as follows:
[0118]
[0119] Among them, k i and k j denote the original features of node i and node j respectively, θ and φ are two Gaussian functions, θ(k i )=W θ k i ,φ(k j )=W φ k j , W θ and W φ is the weight value corresponding to node i and node j, and Ψ represents the transposition operation. Then the joint similarity weight coefficient is generated after normalization using the softmax function:
[0120] α ij =softmax(Score(k i k j ))
[0121] Among them, α ij Represents the joint similarity between node i and node j, based on the weight coefficient α ij Weighted sum of joint information After concatenating with the original features, the intermediate feature K is generated i '=[k i :k i '], where k i is the original feature, k i ' is the weighted sum of joint information;
[0122] The two-layer perceptron (s, r) is combined with tanh and sigmoid nonlinear operations to extract the joint weight distribution and generate the joint weight matrix. The calculation formula is:
[0123] V = sigmoid(W s tanh(W r K'+br )+b s )
[0124] Among them, K' is the intermediate feature generated by adding the weighted sum of the joint features and the original features, and W r and b r is the weight and bias of the activation function tanh, W s and b s is the weight and bias of the activation function sigmoid. Based on the joint weight v i Weight the joints in the skeleton sequence and input feature k i and features v with weight distribution i k i After summing, we get the output features Achieved enhanced attention on joint points;
[0125] The output features of the spatial enhancement shift module can be expressed as
[0126] The output of the spatial enhancement shift module learning spatial features is added to the input features of the spatiotemporal convolution module and input into the time shift module to learn temporal features. The time shift module adopts a temporal shift convolution operation, including: the first time shift operation, 1×1 convolution, the second time shift operation, normalization, and RELU activation function;
[0127] Time shift operation: The input features of the spatiotemporal convolution module and the output features of the spatial enhancement shift module are added as the input features of the time shift module, which can be expressed as Connect consecutive frames in the time dimension and divide the channel into 2u+1 partitions on average. The time shift distance of each partition is -u, -u+1, ..., 0, ..., u-1, u. The shifted channel is truncated and directly filled with 0. After the shift operation, each frame obtains information from its adjacent frames. Since different layers require different temporal receptive fields, each channel has a learnable time shift parameter S i To learn the degree of network shift, if S i If it is an integer, the gradient cannot be transferred, so S i The restriction of is relaxed to real numbers. The time shift is calculated by linear interpolation method:
[0128]
[0129] Where λ = S i -|S i | is due to the time-shift parameter S i After the real number conversion, the residual is generated, which needs to be compensated by interpolation. After the shift convolution operation, the 1×1 convolution is combined to learn the time series features. The feature vector formed by superimposing the time series features is
[0130] In another embodiment of the present invention, the spatiotemporal feature vector learned by the joint point enhanced spatiotemporal convolutional network based on S13 is sequentially passed through the global average pooling layer and the fully connected layer, and S14 is implemented and input into the Softmax classifier for classification.
[0131] The training set is input into the joint point enhanced spatiotemporal convolutional network for training to obtain the joint point enhanced spatiotemporal convolutional network recognition model. The training process further includes:
[0132] The stochastic gradient descent optimization strategy was used to adjust the network parameters during iterative training. The network parameters further included: a total of 100 iterations, an initial learning rate of 0.1, which was divided by 10 at 40, 60, and 80 iterations, and a weight decay coefficient of 1×10 -4 , the training batch size is set to 32;
[0133] like Figure 6 The schematic diagram of the process of implementing abnormal behavior detection in S15 according to an embodiment of the present invention is shown, and the abnormal behavior identification process further includes:
[0134] During the abnormal behavior detection process, relevant practices are implemented under the Pytorch deep learning framework, and frameworks such as PyCharm, CUDA, OpenCV, and Caffee are used to compile and run computer programs for abnormal behavior identification.
[0135] In a preferred embodiment, S15 further includes:
[0136] Acquiring surveillance videos includes but is not limited to: acquiring action video samples by photographing the target object with a camera; and uploading action video data locally.
[0137] The acquired video is used as the input of the trained joint point enhanced spatiotemporal convolutional network recognition model to obtain the classification result and determine whether there is abnormal behavior.
[0138] The present invention also provides a specific application embodiment, such as Figure 7 The visual schematic diagram of abnormal behavior recognition according to an embodiment of the present invention is shown, and the overall abnormal behavior recognition process includes:
[0139] Using Python libraries, OpenPose open source tools, and OpenCV software under the deep learning Pytorch framework to visualize abnormal behavior detection, including:
[0140] Use Python's scikit-video library to process the input video. Call the skvideo.io.vread() function to load the video data into memory as a separate ndarry, so that the video can be loaded frame by frame. Call the skvideo.io.write() function to write the ndarry as a video file, and adjust the relevant video parameters through the skvideo.io.FFmpegWriter() function. In order to ensure the timeliness of the system and that data will not be lost during the input and output process, set the video frame input and output buffer queues. The abnormal behavior recognition model can process 30 frames of data per second, the input video has a maximum of 300 frames, and the frame rate is 30fps. Therefore, the input buffer queue is set to store a maximum of 30 frames of data, and the output buffer queue can store a maximum of 10 frames of data.
[0141] The continuous skeleton data frame sequence of human body movement is obtained from the WrapperPython interface provided by OpenPose. After running the program, the monitoring video frame data is input into the Python interface provided by OpenPose to obtain the coordinate information of the human body joint points in the video frame (including 18 joint points), including: using the upper left corner vertex position of each frame as the coordinate origin to establish a plane rectangular coordinate system, and using numbers to indicate the positions corresponding to the human body joint points: 0-nose, 1-neck, 2-left shoulder, 3-right elbow, 4-right hand, 5-left shoulder, 6-left elbow, 7-left hand, 8-right hip, 9-right knee, 10-right ankle, 11-left hip, 12-left knee, 13-left ankle, 14-left eye, 15-right eye, 16-left ear, 17-right ear. The specific position of each joint is represented by x = (x, y, z): x and y are the horizontal and vertical coordinates of the joint point, and z is the confidence of the joint point. According to the human body structure, adjacent joints are connected to form a complete skeleton sequence. The specific connection rules are as follows: (4, 3), (3, 2), (7, 6), (6, 5), (13, 12), (12, 11), (10, 9), (9, 8), (11, 5), (8, 2), (5, 1), (2, 1), (0, 1), (15, 0), (14, 0), (17, 15), (16, 14). For example, (4, 3) means connecting the right hand corresponding to 4 and the right elbow corresponding to 3. By analogy, all adjacent joints are connected to form a complete human skeleton.
[0142] The multi-frame skeleton data is integrated, the joint coordinate data is parsed, and the features are extracted and input into the trained joint-enhanced spatiotemporal convolutional network model. The corresponding behavior category labels are matched according to the scores to determine whether the target person in the video has abnormal movements. The character's movements and judgment results are visualized through the OpenCV drawing module. If an abnormal situation (such as falling) occurs, an alarm is issued in time and the final monitoring results are saved locally.
[0143] Based on the same inventive concept, the present invention also provides an abnormal behavior recognition system based on skeleton joint point enhanced spatiotemporal convolutional network, comprising:
[0144] Preprocessing module, the original data set is preprocessed to generate a multi-frame skeleton sequence;
[0145] Spatiotemporal convolution module: 300 frames of skeleton sequence are input into several consecutive spatiotemporal convolution modules to perform joint enhancement and output temporal features;
[0146] Recognition module, the recognition module inputs the time series features into the global average pooling layer and the classifier to obtain the score of each label, matches the corresponding category label according to the score and determines whether the action is abnormal.
[0147] Based on the same inventive concept, the present invention also provides an electronic device, the electronic device includes a processor and a memory, the memory is used to store an abnormal behavior recognition program, and the processor is used to execute the abnormal behavior recognition program stored in the memory to implement the software function modules described in the above embodiments. The processor includes but is not limited to a computer, a server, a microprocessor, etc.
[0148] Based on the same inventive concept, the present invention also provides a computer-readable storage medium. The storage medium can store a computer program related to the abnormal behavior identification system of the embodiment of the present invention, wherein the computer program can execute the steps in any of the above embodiments when it is running. The above storage medium can be a random access memory (RAM), a read-only memory (ROM), a flash memory, an optical disk, a magnetic disk, a solid-state disk or other memory, or it can be an electronic device, such as a mobile device, a computer, a tablet computer, etc.
[0149] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various modifications or variations within the scope of the claims, which does not affect the essence of the present invention. The above preferred features can be used in any combination without conflicting with each other.
Claims
1. A method for abnormal behavior recognition based on skeleton joint point enhanced spatiotemporal convolutional network, characterized in that: include: After preprocessing the dataset, a multi-frame skeleton sequence is generated; The multi-frame skeleton sequence is input into a number of consecutive spatiotemporal convolution modules to perform joint enhancement and output temporal features; Inputting the time series features into the global average pooling layer and the classifier to obtain abnormal behavior recognition results; Each of the spatiotemporal convolution modules includes: a spatial enhancement shift module and a temporal shift module, Wherein, the space enhancement shift module includes: space shift and joint point enhancement; The spatial shift operation learns the local features of the joints of a single frame; after calculating the similarity weight coefficients between the joints of the same frame, different weights for each joint point are obtained; the weights of each joint point are extracted using a two-layer perceptron, and then an intermediate feature vector containing the weight coefficients of each joint is generated; The joint point enhancement operation adds the feature after the spatial shift operation to the intermediate feature to achieve joint point enhancement; The features output by the spatial enhancement shift module are added to the input features and input into the time shift module to learn the time series features; The process of spatial displacement includes: The single-frame skeleton sequence feature is Where N is the number of joints, C in is the input channel size, the translation distance of the i-th channel is i mod N, the joints are connected and then translated, each joint obtains the feature information of other joints in the same frame through feature translation, and then the information exchange between channels is realized through 1×1 convolution; The process of joint point enhancement includes: Calculate the similarity between each joint through Gaussian function; The joint similarity weight coefficient generated after normalization using the Softmax function is input into the global average pooling layer; Combine two fully connected layers with the tanh activation function and the sigmoid activation function to extract the joint weight distribution; The intermediate feature vector is obtained by weighting the joints in the skeleton sequence through the joint weights. The normalized intermediate feature is added to the input feature to obtain the output feature. The output feature of the spatial enhancement shift module can be expressed as The time shift module adopts a time-series shift convolution operation, including: The input features of the spatiotemporal convolution module and the output features of the spatial enhancement shift module are added as the input features of the temporal shift module, expressed as Connect consecutive frames in the time dimension, and shift the convolution operation so that each frame obtains information from its adjacent frames; Each channel has a learnable parameter S i To learn the degree of network displacement; The shift convolution operation is combined with the 1×1 convolution to learn the time series features. Finally, BN normalization is performed after the RELU activation function. The feature vector formed after superimposing the time series features is 2. According to claim 1, the abnormal behavior recognition method based on skeleton joint point enhanced spatiotemporal convolutional network is characterized in that: The data set is preprocessed to generate a multi-frame skeleton sequence, including: normalizing skeleton joint point data and constructing a spatiotemporal topological map; The normalized skeleton joint point data includes: For a T-frame video, the center joint point of the first person appearing in the first frame is used as the origin of the coordinate system, and the coordinates of each joint point in the first frame are subtracted from the coordinates of the center joint point; the displacement of the joint point in the time dimension is obtained by using the difference between the joint point coordinates of other frames and the first frame; The constructing of the spatiotemporal topological graph includes: Extract a multi-frame skeleton sequence as input through OpenPose, where the skeleton sequence consists of a spatiotemporal graph of N joint points and T frame skeleton sequences; In a skeleton graph with N joint points, each joint point is represented by a position coordinate vector X = (x, y, z), where x and y are the horizontal and vertical coordinates of the joint point, and z is the confidence of the joint point; In the spatial dimension (single frame), different joint points in the same frame are connected into edges to form a skeleton graph according to the human body structure; In the time dimension (multi-frame), the same joint points between adjacent frames are connected into edges to form a timing graph of T frames.
3. According to claim 1, the abnormal behavior recognition method based on skeleton joint point enhanced spatiotemporal convolutional network is characterized in that: The number of the spatiotemporal convolution modules is 9; The input of each spatiotemporal convolution module is C in ×N×T, the output is C out ×N×T,C in is the feature of the input joint point (number of channels), C out is the feature of the output joint point (number of channels), N is the number of joint points, and T is the total number of frames; the input feature of each layer is the output feature of the previous layer; The number of input channels, output channels, and stride lengths of the 9-layer spatiotemporal convolution module are: The number of input channels of spatiotemporal convolution block B1 is 3, the number of output channels is 64, and the stride is 1; The number of input channels of spatiotemporal convolution block B2 is 64, the number of output channels is 64, and the step size is 1; The number of input channels of spatiotemporal convolution block B3 is 64, the number of output channels is 64, and the stride is 1; The number of input channels of spatiotemporal convolution block B4 is 64, the number of output channels is 128, and the stride is 2; The number of input channels of spatiotemporal convolution block B5 is 128, the number of output channels is 128, and the stride is 1; The number of input channels of spatiotemporal convolution block B6 is 128, the number of output channels is 128, and the stride is 1; The number of input channels of spatiotemporal convolution block B7 is 128, the number of output channels is 256, and the stride is 2; The number of input channels of spatiotemporal convolution block B8 is 256, the number of output channels is 256, and the stride is 1; The number of input channels of spatiotemporal convolution block B9 is 256, the number of output channels is 256, and the stride is 1; The spatiotemporal feature vector output by the 9-layer spatiotemporal convolution module is C out ×N×T, where C out is the number of output channels (256).
4. An abnormal behavior recognition system based on skeleton joint point enhanced spatiotemporal convolutional network, characterized in that: include: Preprocessing module, the original data set is preprocessed to generate a multi-frame skeleton sequence; Spatiotemporal convolution module, the multi-frame skeleton sequence is input into several consecutive spatiotemporal convolution modules to perform joint enhancement and output temporal features; A recognition module, wherein the recognition module inputs the time series features into a global average pooling layer and a classifier to obtain an abnormal behavior recognition result; Each of the spatiotemporal convolution modules includes: a spatial enhancement shift module and a temporal shift module, Wherein, the space enhancement shift module includes: space shift and joint point enhancement; The spatial shift operation learns the local features of the joints of a single frame; after calculating the similarity weight coefficients between the joints of the same frame, different weights for each joint point are obtained; the weights of each joint point are extracted using a two-layer perceptron, and then an intermediate feature vector containing the weight coefficients of each joint is generated; The joint point enhancement operation adds the feature after the spatial shift operation to the intermediate feature to achieve joint point enhancement; The features output by the spatial enhancement shift module are added to the input features and input into the time shift module to learn the time series features; The process of spatial shifting includes: The single-frame skeleton sequence feature is Where N is the number of joints, C in is the input channel size, the translation distance of the i-th channel is i mod N, the joints are connected and then translated, each joint obtains the feature information of other joints in the same frame through feature translation, and then the information exchange between channels is realized through 1×1 convolution; The process of joint point enhancement includes: Calculate the similarity between each joint through Gaussian function; The joint similarity weight coefficient generated after normalization using the Softmax function is input into the global average pooling layer; Combine two fully connected layers with the tanh activation function and the sigmoid activation function to extract the joint weight distribution; The intermediate feature vector is obtained by weighting the joints in the skeleton sequence through the joint weights. The normalized intermediate feature is added to the input feature to obtain the output feature. The output feature of the spatial enhancement shift module can be expressed as The time shift module adopts a time-series shift convolution operation, including: The input features of the spatiotemporal convolution module and the output features of the spatial enhancement shift module are added as the input features of the temporal shift module, expressed as Connect consecutive frames in the time dimension, and shift the convolution operation so that each frame obtains information from its adjacent frames; Each channel has a learnable parameter S i To learn the degree of network displacement; The shift convolution operation is combined with the 1×1 convolution to learn the time series features. Finally, BN normalization is performed after the RELU activation function. The feature vector formed after superimposing the time series features is 5. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement any one of the following methods: - the method according to any one of claims 1 to 3, or, - The system according to claim 4.
6. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded by the processor and executes any one of the following methods: - the method according to any one of claims 1 to 3, or, - The system according to claim 4.
Citation Information
Patent Citations
Method for constructing human body behavior recognition model based on graph convolution network
CN111652124A
Human body behavior recognition method based on multi-scale attention map convolutional network
CN113343901A
Human body action recognition method, human body action recognition system, and device
WO2022000420A1
