A multi-stage abnormal human motion detection method based on residual network

Abstract

The present invention relates to a multi-stage abnormal human motion detection method based on a residual network, the steps of which are: cutting a video clip to be detected into video instances of equal length; using a target detection network model to obtain a human target bounding box, position and size appearing in each monitoring video instance; using an action recognition network model to calculate the category and confidence of the human motion in the bounding box in each monitoring video instance according to the human bounding box information; using an anomaly score learning model to give an anomaly score for each monitoring video instance and perform weighted fusion. The present invention can quickly obtain abnormal motion information in a monitoring video, and designs a target detection network model required for detecting a human bounding box, an action recognition network model for analyzing human motion, and an anomaly score learning model for predicting anomaly scores. The present invention realizes abnormal human motion detection in common monitoring video scenarios, the method is simple, the false alarm rate is low, and it has certain practical value.

Description

Technical Field

[0001] The present invention relates to a multi-stage abnormal human motion detection method based on residual network, specifically a human motion recognition and classification method based on residual network. Some of its technologies involve relevant theories in the fields of machine learning, deep learning, computer vision and anomaly detection. It is mainly used for abnormal motion detection in various real surveillance video scenes, especially in the fields of smart home and smart security. Background Art

[0002] Driven by the rapid development of society, people's safety awareness is increasing year by year, and frequent public safety issues have also received more and more attention from the public. Security projects based on video recording systems and monitoring systems have developed rapidly. A large number of monitoring video products are now used in traffic arteries, key buildings and public places with dense and frequent flow of people such as campuses in every city. At the same time, the application of security monitoring systems in homes is also an important part of modern life. Whether in the field of smart life assistants in every family or in the field of smart security monitoring in public places, many key information can be extracted from monitoring videos. However, these key information requires a lot of time and energy for the relevant staff to find out, and in the actual situation of various occasions today, the staff often have to face a large amount of monitoring data, and the efficiency and accuracy of manual analysis of abnormal action information in the video cannot reach the ideal level. Therefore, realizing the intelligence of the video monitoring system, allowing the computer to complete the task of intelligent detection of abnormal actions, quickly extracting key information and passing it to the staff, can improve the detection efficiency and accuracy of the video monitoring system, achieve the real-time detection of abnormal actions, effectively protect people's public safety and normal activities, and reduce the incidence of public safety problems.

[0003] Traditional video surveillance systems are limited to the function of video recording, but complex tasks such as real-time detection and analysis of abnormal events still require human operation. However, relying solely on manual methods to handle abnormal behavior events is difficult to meet the needs of current monitoring systems, and intelligent video surveillance technology has become one of the best ways to handle abnormal behavior events in real time and quickly.

[0004] The abnormal action recognition task lacks sufficient fully labeled learning samples because the frequency of abnormal events is much lower than that of normal events. In addition, the categories of abnormal actions are diverse, and it is impossible to predict all types of abnormal events in advance.

[0005] The abnormal action recognition task can be learned using weakly supervised training, and the model is trained through coarse-grained labels, that is, video-level labels, and cannot be completed through fine-grained labels. This application adopts a multi-stage human abnormal action detection method based on residual networks, using multiple network models to calculate the human target bounding box and position and size, the category and confidence of the human action, and the abnormal score of the surveillance video instance, and then perform weighted fusion. Summary of the invention

[0006] The present invention overcomes some of the shortcomings of the prior art and provides a multi-stage human abnormal motion detection method based on a residual network. It adopts a target detection network model, a human motion recognition model and an abnormal score learning model, and can quickly obtain abnormal motion information in surveillance videos. It also designs a target detection network model required for detecting human bounding boxes, a motion recognition network model for analyzing human motions, and an abnormal score learning model for predicting abnormal scores. It classifies and identifies abnormal human motions such as falls and beatings in surveillance videos, and has good real-time detection speed and accuracy.

[0007] The technical solution of the present invention is a multi-stage abnormal human motion detection method based on a residual network, which is suitable for classifying and identifying abnormal human motions (such as falling, beating, etc.) in surveillance videos, and the steps include:

[0008] (1) Divide the video clip to be detected into m video instances of equal length;

[0009] (2) According to the video instances segmented in step (1), a target detection network model based on a residual network is used to obtain the position and size of the human target bounding box appearing in each surveillance video instance;

[0010] (3) Based on the position and size of the human target bounding box in step (2), a human action recognition model based on a two-stream spatiotemporal residual network is used to calculate the category and confidence of the human action in the human target bounding box in each surveillance video instance;

[0011] (4) Based on the category and confidence of the human body movements in step (3), an anomaly score learning model is used to give an anomaly score for each surveillance video instance, and weighted fusion is performed, so that abnormal human body movements such as falling and beating can be classified and identified.

[0012] The step (1) is specifically implemented as follows:

[0013] (11) If the applicable dataset is a training and validation dataset, then according to the abnormal event annotations in the dataset, the video clips containing abnormal events are divided into abnormal video instance packages, expressed as Video clips that do not contain abnormal events are divided into normal video instance packages, represented as

[0014] (12) If the applicable dataset is a test dataset, the video clip to be detected is divided into m video instances of equal length;

[0015] (13) Crop or scale the video instance resolution obtained in step (11) or (12) to 228×228.

[0016] The step (2) is specifically implemented as follows:

[0017] (21) Using Faster R-CNN as the target detection network model, the video instance obtained in step (13) is used as input. The target detection network model uses the residual network as the skeleton network. The main body conv2_x, conv3_x, conv4_x and conv5_x that complete the feature extraction part in the residual network are composed of several groups of residual module structures. The three convolutional layers are connected with the residual to form a group of residual modules, among which conv2_x is composed of 3 residual modules, conv3_x is composed of 4 residual modules, conv4_x is composed of 23 residual modules, and conv5_x is composed of 3 residual modules. After the conv5_x network layer, an average sampling layer, a fully connected layer and a Softmax layer are added;

[0018] (22) Combine the feature pyramid network as a candidate region generator, add a convolution layer with a convolution kernel size of 1×1 to reduce the dimension of the feature map of the upper layer, and upsample the feature map of the side convolution layer to obtain a feature map of the same size. Then, perform element-by-element superposition on the convolution feature maps of the two convolution layers after processing. The result is used as the input of the next layer of the side convolution layer. After convolution, dimensionality reduction and sampling operations, the feature maps of conv2_x, conv3_x, conv4_x and conv5_x are superimposed element by element, and different predictions of each layer are output according to different feature maps.

[0019] (23) Approximate joint training is used for the candidate region network in Faster R-CNN and Fast R-CNN. During each stochastic gradient descent iteration, the candidate regions generated during the training of Fast R-CNN are fixed and pre-computed. The backward pass combines the training loss from the candidate region network and the training loss of Fast R-CNN in a shared layer.

[0020] (24) The bounding box of the human target is calculated parametrically, expressed as t x =(xx a ) / w a , t y =(yy a ) / ha , t w =log(w / w a ), t h =log(h / h a ), Where x, y, w, and h represent the horizontal axis coordinate, vertical axis coordinate, and width and height of the predicted bounding box center point, respectively. a ,y a , w a ,h a Represents the horizontal axis coordinate, vertical axis coordinate, width and height of the center point of the anchor box bounding box, x * ,y * , w * ,h * Represents the horizontal axis coordinate, vertical axis coordinate, width and height of the center point of the real bounding box;

[0021] (25) The optimization goal of the target detection network model is to reduce the error in predicting the bounding box and category of human targets and minimize the multi-task training objective function Where L c (p,u)=-logp u , L r (p,u)=R(pu), R(x) is the smooth L1 loss function, i represents the subscript index of the anchor box, p i represents the probability that the i-th anchor box contains the target object, is the sample label used for learning, t i is a vector representing a set of parameterized coordinates of the bounding box of the human target. is a set of parameterized coordinates of the anchor boxes with positive sample labels, L c is the logarithmic classification loss for predicting whether a human target exists, and λ is the loss function in the adjustment The output of the target detection network model is the position and size of the human target bounding box in the video instance.

[0022] The step (3) is specifically implemented as follows:

[0023] (31) A dual-stream spatiotemporal network is used as a human action recognition model. The position and size of the human target bounding box are used as input. The human action recognition model uses two residual networks of the temporal stream and the spatial stream as the skeleton network. For a set of input data samples {x1, x2, x3, ..., x m}, using β and γ as translation parameters and scaling parameters, batch normalization calculates the transformed y i , the mean of each batch calculated along the channel is Calculate the variance of each batch along the channel calculate Complete scaling and translation transformations

[0024] (32) The first layer of filters in the temporal stream are adjusted through three RGB filter channels to operate on horizontal and vertical optical flow stacks, each stack containing 10 video frames, and cross-layer residual connections are used in the temporal stream and spatial stream residual networks, respectively. and cross-stream residual connections Deployed before the residual module from the time stream to the spatial stream, where f is the ReLU activation function, X l is the input matrix of the lth layer, Y l is the output matrix of the lth layer, g(·,·) is a nonlinear residual mapping function composed of the convolution filter weights W l express, is the input of the l-th layer spatial stream, is the input of the l-th layer of time stream, is the connection weight matrix of the lth layer residual neurons of the spatial stream, is the output of the l-th layer spatial stream;

[0025] (33) A residual network with a frame rate of α and a resolution of size C is used as a spatial stream network to determine the static area in the video. At the same time, a residual network with a frame rate of 8α and a resolution of size C / 8 is used as a temporal stream network to determine the dynamic area in the video. The spatial stream network is called the slow channel, and the temporal stream network is called the fast channel. The slow channel network uses the time span as a hyperparameter to control the number of video frames spanned per second. In a data set with a video frame rate of 30, about 2 frames of features are extracted in 1 second, and the fast channel network extracts 15 frames of features in 1 second. The data of the fast channel network is transmitted to the slow channel network through the side connection. The network model performs dimensionality transformation on the data transmitted from the fast channel network to the side connection, and then integrates it into the slow channel network. When performing the dimensionality transformation of the data samples, the fast channel network performs three-dimensional convolution every certain number of frames. In the last layer of the fast channel network and the slow channel network, the human action recognition network model performs global average sampling and reduces the dimension of the data samples. The results are then fused and passed to the fully connected classification layer, and the confidence score of each action is given by Softmax.

[0026] (34) The optimization goal of the human action recognition model is to reduce the error of human action category prediction and minimize the cross entropy loss function of the Softmax classifier. where x i is the action confidence score of the ith human action category of the output vector of the Softmax classifier, y iis the i-th element in the corresponding true action label vector.

[0027] The step (4) is specifically implemented as follows:

[0028] (41) A multi-layer perceptron is used as the anomaly score learning model, with the category and confidence score of human actions as input, and a multi-instance learning method is used. The anomaly score obtained for each video instance in the training and prediction instance bags, where It is an abnormal video instance package. It is an abnormal video instance package. are the corresponding abnormal video instances and normal video instances, is the category and confidence score of the human action in the video instance. h is a multi-layer perceptron model consisting of three fully connected layers. The input layer has 512 neurons, the hidden layer has 32 neurons, and the final output layer has 1 neuron. It predicts the abnormal score of a given video instance, and the output value ranges from 0 to 1.

[0029] (42) The optimization goal of the anomaly score learning model is to maximize the distance between samples of the two categories between abnormal video instance packages and normal video instance packages so that the multi-layer perceptron model can classify them, using a hinge-based ranking loss. Considering that in real situations, the probability and frequency of normal events are much greater than abnormal events, a sparse constraint is added to the loss function Make some dangerous and violent abnormal human actions such as falling and beating, which occur less frequently than other normal events, have higher abnormality scores, where ∈ is a hyperparameter for adjusting the weight of hinge ranking loss and sparse constraints;

[0030] (43) Jointly train the target detection network model, human action recognition model, and anomaly score learning model to optimize Accumulate the loss function and balance the weights, λ od ,λ af ,λ fil are regularization hyperparameters that control each loss function.

[0031] The advantages of the present invention compared with the prior art are:

[0032] (1) The model of the present invention extracts rich human target bounding box features and human motion features, and can effectively learn the abnormal motion characteristics of the human body from the training samples. Based on the existing technology of weakly supervised multi-instance learning model to detect whether an abnormal event is included, the model of the present invention can also give the abnormal motion category and confidence score of the human body;

[0033] (2) By segmenting the video instances and combining the features extracted by the model, it is possible to quickly learn abnormal patterns and detect the spatiotemporal locations of abnormal actions in the surveillance video. Based on the existing technology of detecting abnormal behavior time series information using weakly supervised multi-instance learning models, the model described in the present invention also has the ability to quickly locate the spatial location of abnormal behaviors;

[0034] (3) The model of the present invention can be applied to surveillance videos with a resolution lower than 228×228 and unclear images. In the experimental results of the surveillance video data set, the area under the ROC curve (AUC) for abnormal actions reached 77.43%. As a reference for the performance index of model classification, the recognition ability of the model of the present invention can achieve better classification performance than the existing technologies such as time series regularity learning model (50.60%), high frame rate abnormal event detection model (65.51%) and weakly supervised multiple instance learning model (75.41%). BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A flow chart for realizing the method of the present invention;

[0036] Figure 2 Schematic diagram of the internal architecture of the two-stream residual network;

[0037] Figure 3 This is a diagram of the Faster R-CNN architecture;

[0038] Figure 4 Schematic diagram of the composite network model for target detection;

[0039] Figure 5 It is a schematic diagram of the dual-stream network model architecture;

[0040] Figure 6 Schematic diagram for video instance division;

[0041] Figure 7 Schematic diagram of the multi-layer perceptron architecture. DETAILED DESCRIPTION

[0042] The present invention will be described in detail below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, the method of the present invention specifically comprises the following steps:

[0044] (1) Divide the video clip to be detected into m video instances of equal length;

[0045] (2) According to the video instances segmented in step (1), a target detection network model based on a residual network is used to obtain the position and size of the human target bounding box appearing in each surveillance video instance;

[0046] (3) Based on the position and size of the human target bounding box in step (2), a human action recognition model based on a two-stream spatiotemporal residual network is used to calculate the category and confidence of the human action in the human target bounding box in each surveillance video instance;

[0047] (4) Based on the category and confidence of the human body movements in step (3), an anomaly score learning model is used to give an anomaly score for each surveillance video instance, and weighted fusion is performed, so that abnormal human body movements such as falling and beating can be classified and identified.

[0048] The basic implementation process is as follows:

[0049] (1) Using the UCF-Crime dataset, based on the abnormal event labels of the video clips in the input dataset, the video clips containing abnormal labels are divided into abnormal video instance packages, and the video clips without abnormal events are divided into normal video instance packages, such as Figure 6 As shown;

[0050] (2) Use the FFMPEG audio and video library to crop or scale the video instance resolution to 228×228;

[0051] (3) Faster R-CNN is used as the target detection network model. The target detection network model uses a 101-layer residual network as the skeleton network. The main body conv2_x, conv3_x, conv4_x and conv5_x that complete the feature extraction part in the residual network are composed of several groups of residual module structures. The three convolutional layers are connected with the residual to form a group of residual modules, among which conv2_x is composed of 3 residual modules, conv3_x is composed of 4 residual modules, conv4_x is composed of 23 residual modules, and conv5_x is composed of 3 residual modules. After the conv5_x network layer, an average sampling layer, a fully connected layer and a Softmax layer are added. The architecture is as follows: Figure 3 As shown;

[0052] (4) Combined with the feature pyramid network as a candidate region generator, the architecture of Faster R-CNN combined with the feature pyramid network is as follows Figure 4As shown in the figure, upsampling operations are added layer by layer in the feature pyramid network, and the network layers of Faster R-CNN and the feature pyramid network are horizontally connected. Faster R-CNN reduces the dimension of the feature map of the upper layer by adding a convolution layer with a convolution kernel size of 1×1. At the same time, the feature map of the side convolution layer in the feature pyramid network is upsampled to obtain feature maps of the same size, and then the convolution feature maps of the two convolution layers are element-by-element superimposed. The result is used as the input of the next layer of the side convolution layer. Faster R-CNN performs element-by-element superposition on the feature maps of conv2_x, conv3_x, conv4_x and conv5_x after convolution, dimensionality reduction and sampling operations. The feature pyramid network outputs different predictions for each layer based on different feature maps.

[0053] (5) Approximate joint training is used for the candidate region network in Faster R-CNN and Fast R-CNN. During each stochastic gradient descent iteration, the candidate regions generated during the training of Fast R-CNN are fixed and pre-computed. The backward pass combines the training loss from the candidate region network and the training loss from Fast R-CNN in a shared layer.

[0054] (6) The bounding box of the human target is calculated parametrically, expressed as t x =(xx a ) / w a , t y =(yy a ) / h a , t w =log(w / w a ), t h =log(h / h a ), Where x, y, w, and h represent the horizontal axis coordinate, vertical axis coordinate, and width and height of the predicted bounding box center point, respectively. a ,y a , w a ,h a Represents the horizontal axis coordinate, vertical axis coordinate, width and height of the center point of the anchor box bounding box, x * ,y * , w * ,h * Represents the horizontal axis coordinate, vertical axis coordinate, width and height of the center point of the real bounding box;

[0055] (7) The optimization goal of the target detection network model is to reduce the error in predicting the bounding box and category of human targets and minimize the multi-task training objective function Where L c (p,u)=-logp u , L r (p,u)=R(pu), R(x) is the smooth L1 loss function, i represents the subscript index of the anchor box, p i represents the probability that the i-th anchor box contains the target object, is the sample label used for learning, t i is a vector representing a set of parameterized coordinates of the bounding box of the human target. is a set of parameterized coordinates of the anchor boxes with positive sample labels, L c is the logarithmic classification loss for predicting whether a human target exists, and λ is the loss function in the adjustment The output of the target detection network model is the position and size of the bounding box of the human target in the video instance;

[0056] (8) A two-stream spatiotemporal network is used as a human action recognition model. The position and size of the human target bounding box are used as input. The human action recognition model uses two residual networks of the temporal stream and the spatial stream as the skeleton network. The internal architecture of the residual network is as follows: Figure 2 As shown in the figure, the input layer stride of the temporal network is 2, the number of channels of the conv1 convolution layer is 8, the input layer stride of the spatial network is 16, and the number of channels is 64. Both network streams add a sampling layer after the conv1 convolution layer. The feature map size of the conv2, conv3, conv4, and conv5 convolution layers of the temporal network is 1 / 8 of the corresponding convolution layers of the spatial network. In the dual-stream spatiotemporal network, for a set of input data samples {x1, x2, x3, ..., x m}, using β and γ as translation parameters and scaling parameters, batch normalization calculates the transformed y i , the mean of each batch calculated along the channel is Calculate the variance of each batch along the channel calculate Complete scaling and translation transformations

[0057] (9) The first layer of filters in the temporal stream are adjusted through three RGB filter channels to operate on horizontal and vertical optical flow stacks, each stack containing 10 video frames. Cross-layer residual connections are used in the temporal stream and spatial stream residual networks, respectively. and cross-stream residual connections Deployed before the residual module from the time stream to the spatial stream, where f is the ReLU activation function, X l is the input matrix of the lth layer, Y lis the output matrix of the lth layer, g(·,·) is a nonlinear residual mapping function composed of the convolution filter weights W l express, is the input of the l-th layer spatial stream, is the input of the l-th layer of time stream, is the connection weight matrix of the lth layer residual neurons of the spatial stream, is the output of the l-th layer spatial stream;

[0058] (10) A residual network with a frame rate of α and a resolution of size C is used as a spatial stream network to determine the static area in the video. At the same time, a residual network with a frame rate of 8α and a resolution of size C / 8 is used as a temporal stream network to determine the dynamic area in the video. The spatial stream network is called the slow channel and the temporal stream network is called the fast channel. The slow channel network uses the time span as a hyperparameter to control the number of video frames spanned per second. In a dataset with a video frame rate of 30, about 2 frames of features are extracted in 1 second, and the fast channel network extracts 15 frames of features in 1 second. The data of the fast channel network is transmitted to the slow channel network through the side connection. The network model transforms the dimension of the data transmitted from the fast channel network to the side connection, and then integrates it into the slow channel network. When performing the dimensional transformation of the data samples, the fast channel network performs three-dimensional convolution every certain number of frames. In the last layer of the fast channel network and the slow channel network, the human action recognition network model performs global average sampling and reduces the dimension of the data samples. The results are then fused and passed to the fully connected classification layer. The confidence score of each action is given by Sofimax. The architecture of the two-stream spatiotemporal residual network is as follows: Figure 5 As shown;

[0059] (11) In the learning process of the human action recognition model, the goal is to reduce the error of human action category prediction and minimize the cross entropy loss function of the Softmax classifier. where x i is the action confidence score of the i-th human action category of the output vector of the Sofimax classifier, y i is the i-th element in the corresponding true action label vector;

[0060] (12) Use a multi-layer perceptron as the anomaly score learning model, take the category and confidence score of human action as input, and use a multi-instance learning method The anomaly score obtained for each video instance in the training and prediction instance bags, where It is an abnormal video instance package. It is an abnormal video instance package. are the corresponding abnormal video instances and normal video instances, is the category and confidence score of the human action in the video instance, h is a multi-layer perceptron model, and the architecture is as follows Figure 7 As shown in the figure, it consists of three fully connected layers, the input layer has 512 neurons, the hidden layer has 32 neurons, and the final output layer has 1 neuron, which predicts the anomaly score of a given video instance, and the output value ranges between 0 and 1;

[0061] (13) The optimization goal of the anomaly score learning model is to maximize the distance between samples of the two categories between abnormal video instance packages and normal video instance packages so that the multi-layer perceptron model can classify them, using a hinge-based ranking loss. Considering that in real situations, the probability and frequency of normal events are much greater than abnormal events, a sparse constraint is added to the loss function Make some abnormal human actions such as falling and beating, which occur less frequently than other normal events, have higher abnormality scores, where ∈ is a hyperparameter for adjusting the weight of hinge sorting loss and sparse constraints;

[0062] (14) Jointly train the target detection network model, human action recognition model, and anomaly score learning model to optimize Accumulate the loss function and balance the weights, λ od ,λ af ,λ fil are regularization hyperparameters that control each loss function.

[0063] The above descriptions are only some specific implementation methods of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person familiar with the art within the technical scope disclosed in the present invention should be covered within the protection scope of the present invention.

Claims

1. A multi-stage human abnormal motion detection method based on residual network, characterized in that the steps include: (1) Divide the video clip to be detected into video instances of equal length to obtain segmented video instances; (2) Based on the video instances segmented in step (1), a residual network-based target detection network model is used to obtain the position and size of the human target bounding box appearing in each surveillance video instance; (3) Based on the position and size of the human target bounding box in step (2), a human action recognition model based on a two-stream spatiotemporal residual network is used to calculate the category and confidence of the human action in the human target bounding box in each surveillance video instance; (4) Based on the category and confidence of the human body action in step (3), an anomaly score learning model is used to give an anomaly score for each surveillance video instance, and weighted fusion is performed to classify and identify abnormal human body actions; The step (3) is specifically implemented as follows: (31) A dual-stream spatiotemporal network is used as a human action recognition model. The position and size of the human target bounding box are used as input. The human action recognition model uses two residual networks, the temporal stream and the spatial stream, as the skeleton network for a set of input data samples. ,pass As translation parameters and scaling parameters, batch normalization calculates the transformed , the mean of each batch calculated along the channel is , calculate the variance of each batch along the channel ,calculate , completing the scaling and translation transformation ; (32) The first layer of filters in the temporal stream are adjusted through three RGB filter channels to operate on horizontal and vertical optical flow stacks, each stack containing 10 video frames, and cross-layer residual connections are used in the temporal stream and spatial stream residual networks, respectively. and cross-stream residual connections , deployed before the residual module from the time stream to the spatial stream, where is the ReLU activation function, For the The input matrix of the layer, For the The output matrix of the layer, is a nonlinear residual mapping function composed of the convolution filter weights express, It is The input of layer space flow, It is Input of layer time stream, is the spatial flow The connection weight matrix of the residual neurons in the layer, For the Output of layer space stream; (33) Use a frame rate of , size is The residual network of the resolution of is used as the spatial stream network to determine the static area in the video, and the frame rate is , size is The residual network with a resolution of is used as a temporal stream network to determine the dynamic area in the video. The spatial stream network is called the slow channel, and the temporal stream network is called the fast channel. The slow channel network uses the time span as a hyperparameter to control the number of video frames spanned per second. In the data set with a video frame rate of 30, about 2 frames of features are extracted in 1 second, and the fast channel network extracts 15 frames of features in 1 second. The data of the fast channel network is transmitted to the slow channel network through the side connection. The network model transforms the dimension of the data transmitted to the side connection of the fast channel network, and then integrates it into the slow channel network. When performing the dimensional transformation of the data samples, the fast channel network performs three-dimensional convolution every certain number of frames. In the last layer of the fast channel network and the slow channel network, the human action recognition network model performs global average sampling and reduces the dimension of the data samples. Then the results are fused and passed to the fully connected classification layer, and the confidence score of each action is given by Softmax. (34) The optimization goal of the human action recognition model is to reduce the error of human action category prediction and minimize the cross entropy loss function of the Softmax classifier. ,in is the output vector of the Softmax classifier Action confidence scores for individual human action categories, is the first elements.

2. The multi-stage human abnormal motion detection method based on residual network according to claim 1 is characterized in that: The step (1) is specifically implemented as follows: (11) If the applicable dataset is a training and validation dataset, then according to the abnormal event annotations in the dataset, the video clips containing abnormal events are divided into abnormal video instance packages, expressed as , video clips that do not contain abnormal events are divided into normal video instance packages, expressed as ; (12) If the applicable dataset is a test dataset, the video clips to be tested are divided into video instances of equal length; (13) Crop or scale the video instance resolution obtained in step (11) or (12) to .

3. The multi-stage human abnormal motion detection method based on residual network according to claim 1 is characterized in that: The step (2) is specifically implemented as follows: (21) Faster R-CNN is used as the target detection network model, with video instances as input. The target detection network model uses a residual network as the skeleton network. The main body conv2_x, conv3_x, conv4_x and conv5_x in the residual network that completes the feature extraction part are composed of several groups of residual module structures. The three convolutional layers are connected with the residual to form a group of residual modules, among which conv2_x is composed of 3 residual modules, conv3_x is composed of 4 residual modules, conv4_x is composed of 23 residual modules, and conv5_x is composed of 3 residual modules. An average sampling layer, a fully connected layer and a Softmax layer are added after the conv5_x network layer. (22) Combine the feature pyramid network as a candidate region generator to add a convolution kernel size of The convolution layer is used to reduce the dimension of the feature map of the upper layer, and the feature map of the side convolution layer is upsampled to obtain the feature map of the same size. The convolution feature maps of the two convolution layers are then element-by-element superimposed. The result is used as the input of the next layer of the side convolution layer. After convolution, dimensionality reduction and sampling operations, the feature maps of conv2_x, conv3_x, conv4_x and conv5_x are superimposed element by element, and different predictions of each layer are output according to different feature maps. (23) Approximate joint training is used for the candidate region network in Faster R-CNN and Fast R-CNN. During each stochastic gradient descent iteration, the candidate regions generated during the training of Fast R-CNN are fixed, and the training loss from the candidate region network and the training loss of Fast R-CNN are combined in the shared layer through pre-calculation. (24) The bounding box of the human target is calculated parametrically and is expressed as , , , , , , , ,in Respectively represent the horizontal axis coordinate, vertical axis coordinate, width and height of the predicted bounding box center point, Represents the horizontal axis coordinate, vertical axis coordinate, width and height of the center point of the anchor box bounding box, Represents the horizontal axis coordinate, vertical axis coordinate, width and height of the center point of the real bounding box; (25) The optimization goal of the target detection network model is to reduce the error in predicting the bounding box and category of human targets and minimize the multi-task training objective function. ,in , , , , Is smooth Loss function, represents the subscript index of the anchor box, Indicates The probability that an anchor box contains the target object, is the sample label used for learning, is a vector representing a set of parameterized coordinates of the bounding box of the human target. is a set of parameterized coordinates of the anchor boxes with positive sample labels, is the logarithmic classification loss for predicting whether a human target exists. is the adjusted loss function The output of the target detection network model is the position and size of the human target bounding box in the video instance.

4. The multi-stage human abnormal motion detection method based on residual network according to claim 1, characterized in that: The step (4) is specifically implemented as follows: (41) A multi-layer perceptron is used as the anomaly score learning model, with the category and confidence score of human actions as input, and a multi-instance learning method is used. The anomaly score obtained for each video instance in the training and prediction instance packages, where It is an abnormal video instance package. It is a normal video instance package. are the corresponding abnormal video instances and normal video instances, is the category and confidence score of the human action in the video instance, It is a multi-layer perceptron model, consisting of three fully connected layers, with 512 neurons in the input layer, 32 neurons in the hidden layer, and 1 neuron in the final output layer. It predicts the anomaly score of a given video instance, and the output value ranges from 0 to 1. (42) The optimization goal of the anomaly score learning model is to maximize the distance between samples of the two categories between abnormal video instance bags and normal video instance bags so that the multi-layer perceptron model can classify them, using a hinge-based ranking loss. , and considering that in real situations, the probability and frequency of normal events are much greater than abnormal events, a sparse constraint is added to the loss function , so that some abnormal human actions such as falling and beating, which occur less frequently than other normal events, have higher abnormal scores. is a hyperparameter that adjusts the weight of hinge ranking loss and sparsity constraint; (43) Jointly train the target detection network model, human action recognition model and anomaly score learning model to optimize Accumulate the loss function and balance the weights, are regularization hyperparameters that control each loss function.

Images