A small sample abnormal behavior recognition method based on key regions and scene depth

Abstract

The application discloses a small sample abnormal behavior recognition method based on key areas and scene depth. Firstly, random sparse sampling is performed on a video, and global feature extraction is performed on the sampled video frames; then the global feature map is input into a key area selection module based on weighted offset to obtain a key area containing an abnormal behavior subject and perform local feature extraction; then global features and local features are fused to obtain video-level RGB features; the corresponding scene depth map is extracted from the video frames, and the above feature extraction steps are repeated on the scene depth map to obtain video-level scene depth features; finally, the video-level RGB features and the scene depth features are fused to obtain the final video-level features, and the video-level features are input into a small sample classifier to obtain an abnormal behavior recognition result. The application improves the accuracy, calculation efficiency and robustness for abnormal behavior recognition in a monitoring scene, and is suitable for monitoring videos with multiple moving targets and complex backgrounds.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of video image processing, and particularly relates to a small sample abnormal behavior recognition method based on key regions and scene depth. BACKGROUND

[0002] Abnormal behavior recognition uses deep learning and other field technologies to realize intelligent recognition of abnormal behaviors in monitoring videos. Timely recognizing abnormal behaviors through monitoring cameras and making early warnings are important means to maintain public safety. However, the traditional manual abnormal behavior recognition method is prone to false detection and missed detection due to fatigue caused by long-time work, so it is of important practical significance to use computers to realize automatic and intelligent recognition of abnormal behaviors in monitoring videos.

[0003] A big difficulty of abnormal behavior recognition is that the probability of abnormal behavior is small compared with normal behavior, and the number of abnormal samples is small. In view of this difficulty, the concept of small sample abnormal behavior recognition is introduced, that is, through small sample learning, the abnormal behavior recognition model has the ability to recognize new abnormal behavior categories with very few samples, solving the problem of small number of abnormal samples. Small sample learning is generally based on the principle of meta-learning, that is, learning some commonalities from a large number of other data to identify new categories. The commonly used small sample learning model is generally a metric-based method, that is, the distance between samples is modeled, so that the distance between samples of the same category is close, and the distance between samples of different categories is far, and the similarity between samples is judged by the distance, and the category of unknown samples is further judged. In addition, abnormal behavior recognition also has the difficulty of complex abnormal behavior, that is, the forms of abnormal behavior are different in different monitoring scenes and different behavior subjects. SUMMARY

[0004] The application provides a small sample abnormal behavior recognition method based on key regions and scene depth to overcome the defects of the prior art, aiming at monitoring scenes, and improve the accuracy, computational efficiency and robustness of abnormal behavior recognition.

[0005] To achieve the above purpose, the technical scheme provided by the application is a small sample abnormal behavior recognition method based on key regions and scene depth, comprising the following steps:

[0006] Step 1, randomly sparse sampling of video, dividing the video into N segments according to the number of frames, randomly sampling M frames in each segment, and taking a total of N x M frames as the representative of the video;

[0007] Step 2, using a global feature extraction network to extract features from the video frames generated in step 1 to obtain a two-dimensional global feature map and a one-dimensional global feature vector;

[0008] Step 3, the key region selection based on weighted offset is performed on the two-dimensional global feature map extracted in step 2, to obtain a key region containing the subject of abnormal behavior in the video frame, and a one-dimensional local feature vector of the key region is extracted by using a local feature extraction network;

[0009] Step 3.1, the spatio-temporal feature extraction and motion feature extraction are performed on the two-dimensional global feature map extracted in step 2, to generate a feature map with spatio-temporal information and object motion information;

[0010] Step 3.2, the center point of the key region is selected based on weighted offset;

[0011] Step 3.3, the pixel values of the remaining points of the key region are obtained by using bilinear interpolation;

[0012] Step 3.4, the key region is input into the local feature extraction network to obtain the local feature of the key region;

[0013] Step 4, the global feature vector extracted in step 2 and the local feature vector extracted in step 3 are fused to obtain a video-level RGB feature vector;

[0014] Step 5, the monocular scene depth estimation model is used to process the N×M frames generated in step 1 to obtain the corresponding N×M frame scene depth map;

[0015] Step 6, the scene depth map extracted in step 5 is repeatedly operated in steps 2 to 4 to obtain a video-level scene depth feature vector;

[0016] Step 7, the video-level RGB feature vector extracted in step 4 and the video-level scene depth feature vector extracted in step 6 are fused to obtain a final video-level feature vector;

[0017] Step 8, the video-level feature vector obtained in step 7 is input into a small sample classifier to obtain a final abnormal behavior recognition result.

[0018] Moreover, in step 1, the video is first extracted into continuous video frames by using the ffmpeg software, the number of video frames is counted by using the os library, the video frames are equally divided into N parts, and M frames are randomly extracted in each part; then the N×M frames are further processed by using the PIL library: if the width or height of the video frame is less than a, the shorter side size is adjusted to a, and the longer side is adjusted to b; when the video frame is used for training, the video frame is randomly cropped at a position, the cropping size is a×a, and random vertical flipping is performed with a probability of 50%; when the abnormal behavior in the video frame is predicted, only the center position of the video frame is cropped, and the cropping size is a×a; finally, the N×M frames are vectorized and then normalized to serve as a representative of the video.

[0019] Moreover, in step 2, the N*M video frame data obtained in step 1 is input into the Resnet-50 network for feature extraction, the global feature map is obtained from the output of the last convolutional layer of the Resnet-50 network, and the global feature vector is obtained from the input of the average pooling layer, and the Resnet-50 loads the parameters pre-trained from the Kinetics video dataset.

[0020] Moreover, step 3.1 includes the following steps:

[0021] Step 3.1.1, channel mean normalization is performed on the input video frame feature map, that is, the average value of the input multiple channels is obtained to obtain a single-channel output;

[0022] Step 3.1.2, spatial-temporal feature extraction is performed on the normalized feature map obtained in step 3.1.1;

[0023] First, data reconstruction is performed, the time dimension and the channel dimension of the video frame feature map are interchanged through data reconstruction, then three-dimensional convolution is input, the three-dimensional convolution network can extract the spatial-temporal information of the video frame, then the dimension is restored through data reconstruction, that is, the time dimension and the channel dimension are interchanged again, and finally the spatial-temporal feature map is obtained through Sigmoid function normalization;

[0024] Step 3.1.3, motion feature extraction is performed on the normalized spatial-temporal feature map obtained in step 3.1.1;

[0025] First, time dispersion is performed, that is, the feature map represented by the continuous video frame is separated to obtain the feature map represented by each frame, then the frame-represented feature map is input into the two-dimensional convolution to extract the spatial feature, and the difference between the two-dimensional convolution output of each frame and the adjacent next frame is calculated, that is:

[0026] X out =K*X t+1 -X t (1)

[0027] In the formula, K represents the parameters learned by the two-dimensional convolution through training, X t+1 , and X t represent the feature maps input by the t+1 frame and the t frame respectively;

[0028] Finally, all the differences obtained are connected and normalized through the Sigmoid function to obtain the motion feature map;

[0029] Step 3.1.4, using the residual structure, the feature maps output by steps 3.1.2 and 3.1.3 are added to the feature map generated by step 3.1.1 to obtain a feature map with spatial-temporal information and motion information;

[0030] Step 3.1.5, a two-dimensional softmax operation is performed on the feature map obtained in step 3.1.4, that is, the elements in each row and column of the two-dimensional feature map are input into the softmax function, so that all elements in the two-dimensional feature map add up to 1.

[0031] Moreover, the step 3.2 is to uniformly take L points from the original graph With the original graph center point pointing to each point a i Obtain the offset vector With the element u on the feature map extracted in step 3.1 i As a weight, the weighted sum of all offset vectors is obtained as the sum vector And the point pointed by is the center point of the key region, including the following steps:

[0032] Step 3.2.1, the center point of the key region is selected in the square region with a side length equal to the difference between the side length of the original graph and the side length of the key region and located at the center of the original graph;

[0033] Step 3.2.2, L points are uniformly taken from the boundary of the square region selected in step 3.2.1 The number L of points is the same as the number of elements on the feature map with spatiotemporal information and motion information extracted in step 3.1, and each point corresponds to an element on the feature map in the order of space from left to right and from top to bottom;

[0034] Step 3.2.3, the vector pointing from the center point of the original graph to each point a i is the offset vector

[0035] Step 3.2.4, the corresponding element u i on the feature map is taken as the weight of the offset vector, and the displacement vector is obtained after weighted sum The sum vector pointed by the displacement vector is the center point of the key region.

[0036] Moreover, in the step 3.3, first, the coordinates of other points on the key region are obtained by translating the center point of the key region , that is:

[0037]

[0038] In the formula, represents the coordinates of the offset vector of the other points on the key region relative to the center point;

[0039] Then, the pixel value of each point on the key region is obtained by using the bilinear interpolation method The specific formula is as follows:

[0040]

[0041] In the formula, for The coordinates of the four neighboring points, (m ij ) 00 ,(m ij ) 01 ,(m ij ) 10 ,(m ij ) 11 The pixel value is the value of the corresponding four neighboring points.

[0042] Furthermore, in step 3.4, the local feature extraction network uses a ResNet-50 model pre-trained on the Kinetics video dataset, and the original mean pooling layer with a fixed stride of 7 in the ResNet-50 is replaced with an adaptive mean pooling layer, so as to enable the local feature extraction network to extract features from key regions with relatively small sizes.

[0043] Furthermore, in step 4, the global feature vector and the local feature vector are concatenated end to end, and the dimension of the fused video-level RGB feature vector is the sum of the dimensions of the global feature vector and the local feature vector.

[0044] Furthermore, in step 5, video frames are input one by one into a self-supervised monocular scene depth estimation model. This model uses a combination of a lightweight U2net model encoder and decoder for scene depth estimation. The scene depth estimation model is then combined with a pose estimation model for self-supervised training. The pose estimation model uses a lightweight U2net model encoder. Self-supervised training calculates errors by reconstructing images to train the model. Specifically, this process includes estimating the relative pose transformation matrix T between the target image and the source image using the pose estimation model. t→t′ The scene depth map D of the target image is predicted using a scene depth estimation model. t The encoder weights of the scene depth estimation model are shared with the lightweight U2net encoder in the pose estimation model; assuming that the camera intrinsics remain unchanged in each image, the scene depth map D of the target image is used. t Relative pose transformation matrix T t→t′ And the camera's intrinsic parameter matrix M, from which the reconstructed target image K is calculated. t′→t The calculation formula is:

[0045] K t′→t =K t′ [proj(D t ,T t→t′ ,M)] (4)

[0046] where K t′→t is the reconstructed target image computed from the source images, K t′ is the source image, proj() denotes the projection of a scene depth map D t to the source image K t′ , M is the camera intrinsic matrix, D t is the scene depth map of the target image, T t→t′ is the relative pose transformation matrix between the target image and the source image, and [] denotes the sampling operation.

[0047] The L1 distance between the reconstructed target image K t′→t and the actual target image K t is computed to obtain the reconstruction error L p , which is given by:

[0048]

[0049] where pe() is the photometric reconstruction error, i.e., the L1 distance in the pixel space.

[0050] The gradient descent method is used to minimize the reconstruction error L p to optimize the scene depth model parameters. During the optimization process, data augmentation is performed on the source images and the target images, specifically, a region of 1 / 2 or 1 / 4 of the original image size is randomly selected from the center of the original image as new training data. The N x M video frames obtained in step 1 are input into the optimized model one by one to generate N x M frames of scene depth maps.

[0051] Moreover, the RGB feature and scene depth feature fusion method in step 7 adopts an adaptive fusion method, and the specific process is as follows: the RGB input feature vector is standardized so that its mean is 0 and its standard deviation is 1, then the scene depth information is used to change the mean and standard deviation, and the calculation method is as follows:

[0052]

[0053] where f(T rgb , T d ) is the final video-level feature vector, T rgb and T d represent the RGB feature vector and the scene depth feature vector, respectively, and the parameters fc s and fc b are learned from a fully connected layer network, and μ(T rgb ) and σ(T rgb ) represent the mean and standard deviation of the RGB feature vector in L dimensions, respectively.

[0054] Further, the small sample classifier, i.e., the prototype network, is first trained in step 8, the monitoring video abnormal behavior data set is used as training data, the training data is divided into a support set and a prediction set, N categories are randomly extracted in the data set, K samples in each category, a total of N x K data are input as the support set; a batch of samples are extracted from the remaining data in the N categories as the prediction set of the model; the input data form when predicting unknown abnormal behavior is the same as when training, the model learns to judge the label of the prediction set sample through the support set; the features of the support set and the prediction set sample are first subjected to L2 normalization, and the L2 normalization formula of a vector X is:

[0055]

[0056] Then, the trainable encoding network is used, the encoding network is composed of two fully connected layers and an activation function Relu in the middle, the input vector dimension of the first fully connected layer is 4096, the output vector dimension is 4096, the input vector dimension of the second fully connected layer is 4096, and the output vector dimension is 1024; then the mean of the K samples in each category in the support set is calculated, and the mean is taken as the prototype B i of each category of samples, the cosine similarity of the prediction set sample A and each prototype B i is calculated, and the formula is:

[0057]

[0058] The normalized probability is obtained by using the softmax function on all cosine similarities, the type of abnormal behavior is judged according to the probability size, and the abnormal behavior corresponding to the maximum probability is the final recognition result.

[0059] Compared with the prior art, the present application has the following advantages:

[0060] The global features of abnormal behavior and the local features containing the key region of the subject of abnormal behavior are fused, and the RGB features and the scene depth features containing the motion target and background information are fused, which not only improves the accuracy and calculation efficiency of abnormal behavior recognition, but also has robustness for monitoring videos with multiple motion targets and complex backgrounds. BRIEF DESCRIPTION OF DRAWINGS

[0061] Figure 1 is a flowchart of an embodiment of the present application.

[0062] Figure 2 is a structure diagram of the spatiotemporal feature extraction module and the motion feature extraction module of an embodiment of the present application.

[0063] Figure 3 is a schematic diagram of the key region selection process based on weighted offset of an embodiment of the present application.

[0064] Figure 4 is a scene depth map extraction flowchart of an embodiment of the present application.

[0065] Figure 5 is a small sample classifier flowchart of an embodiment of the present application. DETAILED DESCRIPTION

[0066] The present application provides a small sample abnormal behavior recognition method based on key regions and scene depth. The technical solutions of the present application are further described below in conjunction with the drawings and embodiments.

[0067] As shown in Figure 1 , the flow of an embodiment of the present application includes the following steps:

[0068] Step 1, randomly sparse sampling of video, dividing the video into N segments according to the number of frames, randomly sampling M frames in each segment, and taking a total of N x M frames as the representative of the video.

[0069] The video is extracted into continuous video frames by ffmpeg software, the number of video frames is counted by os library, and the video frames are equally divided into N parts (N is 4 in this embodiment), and M frames are randomly extracted in each part (M is 2 in this embodiment). The N x M frames are further processed by the PIL library: if a side (width or height) of the video frame is less than 224, the shorter side size is adjusted to 224 and the longer side is adjusted to 256. When the video frame is used for training, the video frame is randomly cropped at a position, the cropping size is 224 x 224, and random vertical flipping is performed with a probability of 50%; when predicting abnormal behavior in the video frame, only the center position of the video frame is cropped, the cropping size is 224 x 224. After vectorization, the N x M frames are normalized and used as the representative of the video.

[0070] Step 2, using a global feature extraction network to extract features from the video frames generated in step 1 to obtain a two-dimensional global feature map and a one-dimensional global feature vector.

[0071] The N x M video frame data obtained in step 1 or the N x M scene depth data obtained in step 5 is input into the Resnet-50 network for feature extraction, and the global feature map is obtained from the output of the last convolutional layer of the Resnet-50 network, and the global feature vector is obtained from the input of the average pooling layer. Resnet-50 loads the parameters pre-trained from the Kinetics video dataset.

[0072] Step 3, key region selection based on weighted offset is performed on the two-dimensional global feature map extracted in step 2 to obtain a key region containing the subject of abnormal behavior in the video frame, and a one-dimensional local feature vector of the key region is extracted using a local feature extraction network.

[0073] Step 3.1, spatial-temporal feature extraction and motion feature extraction are performed on the two-dimensional global feature map extracted in step 2 to generate a feature map with spatial-temporal information and object motion information.

[0074] Step 3.1.1, channel mean normalization is performed on the input video frame feature map, that is, the average value of the input multiple channels is obtained to obtain a single-channel output.

[0075] Step 3.1.2, spatial-temporal feature extraction is performed on the normalized feature map obtained in step 3.1.1.

[0076] First, data reconstruction is performed to realize dimension conversion. The input form of the continuous video frame feature map is N x T x C x H x W, where N is the number of batches of samples, T is the time dimension of the video frame, C represents the channel dimension, H is the height of the video frame, and W is the width of the video frame. The time dimension and the channel dimension of the video frame feature map are interchanged to facilitate the subsequent convolution to process the time dimension and two spatial dimensions. Next, a three-dimensional convolution is input, and the three-dimensional convolution network can extract the spatial-temporal information of the video frame. Then, data reconstruction is performed again to restore the dimensions, that is, the time dimension and the channel dimension are interchanged again, and finally the spatial-temporal feature map is obtained by normalization through the Sigmoid function.

[0077] Step 3.1.3, motion feature extraction is performed on the normalized spatial-temporal feature map obtained in step 3.1.1.

[0078] First, time dispersion is performed, that is, the feature map represented by the continuous video frame is separated to obtain a feature map represented by each frame, and then each frame-represented feature map is input into a two-dimensional convolution to extract spatial features, and the difference between the two-dimensional convolution output of each frame and its adjacent next frame is calculated, that is:

[0079] X out =K*X t+1 -X t (1)

[0080] In the formula, K represents the parameters learned by the two-dimensional convolution through training, X t+1 , and X t represent the feature maps input by the t+1 frame and the t frame, respectively;

[0081] Finally, all the calculated differences are connected and normalized through the Sigmoid function to obtain a motion feature map.

[0082] Step 3.1.4, the residual structure is used to add the feature maps output by steps 3.1.2 and 3.1.3 to the feature map generated in step 3.1.1 to obtain a feature map with spatial-temporal information and motion information.

[0083] Step 3.1.5: Perform a two-dimensional softmax operation on the feature map obtained in step 3.1.4, that is, input the elements of each row and column of the two-dimensional feature map into the softmax function so that the sum of all elements in the two-dimensional feature map is 1.

[0084] Step 3.2: Select the center point of the key region based on the weighted offset.

[0085] Take L points evenly from the original image (the video frame obtained in step 1 or the scene depth map obtained in step 5). Pointing from the center point of the original image to each point a i Obtain the offset vector Using the element u on the feature map extracted in step 3.1 i As weights, a weighted summation is performed on all offset vectors to obtain the sum vector. The point it points to is the center point of the critical area.

[0086] Step 3.2.1, as follows Figure 3 As shown in (a), considering that the extracted key area has a certain size, the center point of the key area can only be selected within a certain range in the original image, that is, a square area whose side length is the difference between the side length of the original image and the side length of the key area and is located in the center of the original image.

[0087] Step 3.2.2, as follows Figure 3 As shown in (b), L points are uniformly selected from the boundary of the square region chosen in step 3.2.1. The number of points L is the same as the number of elements on the feature map containing spatiotemporal and motion information extracted in step 3.1. Each point corresponds to an element on the feature map in spatial order from left to right and from top to bottom.

[0088] Step 3.2.3, as follows Figure 3 As shown in (c), points a are taken from the center point of the original image and pointing to each point a. i The vector is the offset vector.

[0089] Step 3.2.4, as follows Figure 3 As shown in (d), the corresponding element u on the feature map i As the weight of the offset vector, the displacement vector The final result is obtained by weighting and summing. and vector The point it points to is the center point of the critical area.

[0090] Step 3.3: Use bilinear interpolation to obtain the pixel values ​​of the remaining points in the key area.

[0091] Since the size of the key region is set to be fixed, once the center point of the key region is determined... Then the whole key region can be determined, and the coordinates of other points on the key region are obtained by translating the center point of the key region

[0092]

[0093] In the formula, the coordinates of the other points on the key region relative to the center point of the key region are represented by the offset vector.

[0094] Since the coordinates of the center point of the key region obtained by the weighted offset in step 3.2 have decimals, the coordinates of the points on the key region obtained are not integers, and there is no corresponding pixel value on the original image. Therefore, a bilinear interpolation method is used, that is, the pixel value of each point on the key region

[0095]

[0096] In the formula, the coordinates of the four neighboring points of (m ij ) 00 (m ij ) 01 (m ij ) 10 (m ij ) 11 are the coordinates of the four neighboring points, and t→t′ are the pixel values of the corresponding four neighboring points. Since the coordinates of the points on the region are continuous rather than discrete, the above method can find the accurate key region for different position abnormal behavior subjects.

[0097] Step 3.4, input the key region into the local feature extraction network to obtain the local feature of the key region.

[0098] The local feature extraction network also uses the Resnet-50 model pre-trained on the Kinetics video dataset. Unlike the global feature extraction network, the mean pooling layer with a fixed step size of 7 in the original Resnet-50 is changed to an adaptive mean pooling layer to realize the feature extraction of the local feature extraction network on the relatively small key region.

[0099] Step 4, fuse the global feature vector extracted in step 2 and the local feature vector extracted in step 3 to obtain the video-level RGB feature vector.

[0100] ​​​​​The local feature fusion and the global feature fusion mode uses a connection mode, specifically, the global feature vector and the local feature vector are connected head to tail, and the dimension of the fused video-level RGB feature vector is the sum of the dimension of the global feature vector and the dimension of the local feature vector.

[0101] Step 5, using a monocular scene depth estimation model to process the N*M frames generated in step 1 to obtain the corresponding N*M frame scene depth map.

[0102] The video frames are input into the monocular scene depth estimation model of the self-supervised learning one by one, and the model uses a combination of a lightweight U2net model encoder and a decoder for scene depth estimation. The scene depth estimation model is combined with another model using a lightweight U2net network for pose estimation for self-supervised training. During training, the source image and the target image are subjected to data enhancement, and the extracted scene depth map is robust to noise and can effectively represent the scene and target information.

[0103] The self-supervised training calculates the error by reconstructing the image to train the model, and the specific process includes estimating the relative pose transformation matrix T t→t′ between the target image and the source image by using the pose estimation model, and the pose estimation model uses a lightweight U2net model encoder. Compared with the general U2net model, the input channel number, the intermediate channel number and the output channel number in each residual U structure block in the lightweight U2net model are less, so the model occupies less space and has faster calculation speed. Then the scene depth map D t of the target image is predicted using the scene depth estimation model, and the scene depth estimation model uses a combination of a lightweight U2net encoder and a decoder, and the encoder weight is shared with the lightweight U2net encoder in the pose estimation model. The reconstructed target image K t is calculated from the source image based on the target image scene depth map D t→t′ , the relative pose transformation matrix T t′→t and the camera intrinsic parameter matrix M (assuming that the camera intrinsic parameter in each image is constant), and the calculation formula is:

[0104] K t′→t =K t′ [proj(D t ,T t→t′ ,M)] (4)

[0105] In the formula, K t′→t is the reconstructed target image calculated from the source image, K t′ is the source image, proj() represents the projection of the scene depth map D t to the coordinates of the source image K t′ , and M is the camera intrinsic parameter matrix, and D tT is the scene depth map of the target image. t→t′ Let be the relative pose transformation matrix between the target image and the source image, and [] denote the sampling operation.

[0106] Calculate the reconstructed target image K t′→t Compared with the actual target image K t The L1 distance between them is used to obtain the reconstruction error L. p The calculation formula is:

[0107]

[0108] In the formula, pe() is the photometric reconstruction error, which is the L1 distance in pixel space.

[0109] Minimize the reconstruction error L using gradient descent. p The scene depth model parameters are optimized. During the optimization process, data augmentation is performed on the source and target images. Specifically, a region of 1 / 2 or 1 / 4 of the original image size is randomly selected from the center of the original image (source or target image) and used as new training data. The N×M video frames obtained in step 1 are then fed into the optimized model one by one to generate N×M scene depth maps. These scene depth maps can effectively represent the information of moving targets and background in the monitored scene.

[0110] Step 6: Repeat steps 2 to 4 on the scene depth map extracted in step 5 to obtain the video-level scene depth feature vector.

[0111] Step 7: Fuse the video-level RGB feature vector extracted in Step 4 and the video-level scene depth feature vector extracted in Step 6 to obtain the final video-level feature vector.

[0112] The fusion of RGB features and scene depth features adopts an adaptive fusion method. Specifically, the RGB input feature vector is standardized so that its mean is 0 and its standard deviation is 1. Then, the mean and standard deviation are changed using scene depth information. The calculation method is as follows:

[0113]

[0114] In the formula, f(T) rgb ,T d T represents the final video-level feature vector. rgb T d These represent the RGB feature vector and the scene depth feature vector, respectively, with parameter fc. s and FC b μ(T) is learned from a fully connected layer network. rgb ), σ(T) rgb ) represent the mean and standard deviation of the RGB feature vector in L dimensions, respectively.

[0115] Step 8, input the video-level feature vector obtained in step 7 into the small sample classifier to obtain the final abnormal behavior recognition result.

[0116] First, the small sample classifier, i.e., the prototype network, is trained, and the monitoring video abnormal behavior dataset is used as the training data. The training data is divided into a support set and a prediction set. In the data set, N categories are randomly extracted, and K samples of each category (a total of N x K data) are input as the support set. A batch of samples are extracted from the remaining data in the N categories as the prediction set of the model. The input data form when predicting unknown abnormal behaviors is the same as when training. The model learns to judge the labels of the prediction set samples through the support set. The features of the support set and the prediction set samples are first L2 normalized. The L2 normalization formula for a vector X is:

[0117]

[0118] Then, the trainable encoding network is used. The encoding network consists of two fully connected layers and an activation function Relu in between. The first fully connected layer has an input vector dimension of 4096 and an output vector dimension of 4096. The second fully connected layer has an input vector dimension of 4096 and an output vector dimension of 1024. The encoding network plays a role in reducing the dimension of the feature vector, further improving the representation ability of the feature vector. At the same time, using two fully connected layers can increase the number of trainable parameters in the encoding network, and the activation function in between the two fully connected layers can enhance the non-linear expression ability. Then, the mean of the K samples in each category in the support set is calculated, and the mean is taken as the prototype B i of each category of samples. The cosine similarity between the prediction set sample A and each prototype B i is calculated, and the formula is:

[0119]

[0120] All cosine similarities are normalized by the softmax function to obtain the normalized probability. According to the probability size, the type of abnormal behavior is judged, and the abnormal behavior corresponding to the maximum probability is the final recognition result.

[0121] The specific embodiments described herein are merely illustrative of the spirit of the present application. Those skilled in the art of the present application can make various modifications or supplements to the described specific embodiments or use similar ways to replace them, without deviating from the spirit of the present application or exceeding the scope defined by the appended claims.

Claims

1. A method for identifying anomalous behavior in small samples based on key regions and scene depth, characterized in that, Includes the following steps: Step 1: Perform random sparse sampling on the video. Divide the video into N segments according to the number of frames, and randomly sample M frames from each segment. Use the total N×M frames as the representative of the video. Step 2: Use a global feature extraction network to extract features from the video frames generated in Step 1 to obtain a two-dimensional global feature map and a one-dimensional global feature vector. Step 3: Select key regions based on weighted offset from the two-dimensional global feature map extracted in Step 2 to obtain key regions containing the abnormal behavior subject in the video frame. Use a local feature extraction network to extract the one-dimensional local feature vector of the key region. Step 3.1: Perform spatiotemporal feature extraction and motion feature extraction on the two-dimensional global feature map extracted in Step 2 to generate a feature map with spatiotemporal information and object motion information; Step 3.2: Select the center point of the key region based on the weighted offset; Step 3.3: Use bilinear interpolation to obtain the pixel values ​​of the remaining points in the key region; Step 3.4: Input the key region into the local feature extraction network to obtain the local features of the key region; Step 4: Fuse the global feature vector extracted in Step 2 and the local feature vector extracted in Step 3 to obtain the video-level RGB feature vector. By concatenating the global and local feature vectors end to end, the dimension of the fused video-level RGB feature vector is the sum of the dimensions of the global and local feature vectors. Step 5: Process the N×M frames generated in Step 1 using a monocular scene depth estimation model to obtain the corresponding N×M frame scene depth map. Step 6: Repeat steps 2 to 4 on the scene depth map extracted in step 5 to obtain the video-level scene depth feature vector. Step 7: Fuse the video-level RGB feature vector extracted in Step 4 and the video-level scene depth feature vector extracted in Step 6 to obtain the final video-level feature vector. Step 8: Input the video-level feature vector obtained in Step 7 into the few-sample classifier to obtain the final abnormal behavior recognition result.

2. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: Step 1 first extracts the video into continuous video frames using ffmpeg software. The number of video frames is counted using the OS library, and the video frames are divided into N equal parts. M frames are randomly selected from each part. These N×M frames are then further processed using the PIL library: if the width or height of a video frame is less than 'a', the shorter side is adjusted to 'a' and the longer side to 'b'. When using the video frames for training, they are randomly cropped at a size of 'a×a' and randomly vertically flipped with a 50% probability. When predicting abnormal behavior in the video frames, only the center position of the video frame is cropped at a size of 'a×a'. Finally, the N×M frames are vectorized and then normalized to represent the video.

3. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: In step 2, the N×M video frame data obtained in step 1 is input into the ResNet-50 network for feature extraction. The global feature map is obtained from the output of the last convolutional layer of the ResNet-50 network, and the global feature vector is obtained from the input of the average pooling layer. The ResNet-50 is loaded with parameters pre-trained from the Kinetics video dataset.

4. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 3, characterized in that: Step 3.1 includes the following steps: Step 3.1.1: Perform channel mean normalization on the input video frame feature map, that is, calculate the average value of multiple input channels to obtain the single-channel output; Step 3.1.2: Extract spatiotemporal features from the normalized feature map obtained in step 3.1.1; First, data reconstruction is performed, in which the time dimension and channel dimension of the video frame feature map are interchanged. Then, it is input into a 3D convolutional network, which can extract the spatiotemporal information of the video frame. Then, data reconstruction is performed again to restore the dimensions, that is, the time dimension and channel dimension are interchanged again. Finally, the spatiotemporal feature map is obtained by normalization through the Sigmoid function. Step 3.1.3: Extract motion features from the normalized spatiotemporal feature map obtained in Step 3.1.1; First, temporal dispersion is performed, which involves separating the feature maps represented by consecutive video frames to obtain the feature map represented by each frame. Then, the feature maps of each frame are input into a two-dimensional convolution to extract spatial features, and the difference between each frame and the output of the two-dimensional convolution of its adjacent next frame is calculated. X out =K*X t+1 -X t (1) In the formula, K represents the parameters learned by the two-dimensional convolution during training, and X... t+1 X t These represent the feature maps input at frame t+1 and frame t, respectively. Finally, all the obtained differences are concatenated and normalized using the Sigmoid function to obtain the motion feature map; Step 3.1.4: Use the residual structure to add the feature maps output from steps 3.1.2 and 3.1.3 to the feature map generated in step 3.1.1 to obtain a feature map with spatiotemporal and motion information; Step 3.1.5: Perform a two-dimensional softmax operation on the feature map obtained in step 3.1.4, that is, input the elements of each row and column of the two-dimensional feature map into the softmax function so that the sum of all elements in the two-dimensional feature map is 1.

5. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: Step 3.2 involves uniformly selecting L points from the original image. Pointing from the center point of the original image to each point a i Obtain the offset vector Using the element u on the feature map extracted in step 3.1 i As weights, a weighted summation is performed on all offset vectors to obtain the sum vector. and with The point being pointed to is the center point of the critical area, and the process includes the following steps: Step 3.2.1: Select the center point of the key region within a square region whose side length is the difference between the side length of the original image and the side length of the key region, and which is located in the center of the original image; Step 3.2.2: Take L points uniformly from the boundary of the square region selected in Step 3.2.

1. The number of points L is the same as the number of elements on the feature map containing spatiotemporal and motion information extracted in step 3.

1. Each point corresponds to an element on the feature map in spatial order from left to right and from top to bottom. Step 3.2.3: Take the points a from the center point of the original image to each point a. i The vector is the offset vector. Step 3.2.4, using the corresponding element u on the feature map i As the weight of the offset vector, the displacement vector The sum vector is obtained by weighted summation. and vector The point it points to is the center point of the critical area.

6. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: Step 3.3 first involves identifying the center point of the key area. Translate to obtain other points on the key area The coordinates, that is: In the formula, Represents the coordinates of other points in the critical area relative to the center point's offset vector; Then, bilinear interpolation is used to obtain the values ​​of each point in the key region. pixel values The specific formula is as follows: In the formula, for The coordinates of the four neighboring points, (m ij ) 00 ,(m ij ) 01 ,(m ij ) 10 ,(m ij ) 11 The pixel value is the value of the corresponding four neighboring points.

7. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: In step 3.4, the local feature extraction network uses a ResNet-50 model pre-trained on the Kinetics video dataset, and the original mean pooling layer with a fixed stride of 7 in the ResNet-50 is replaced with an adaptive mean pooling layer to enable the local feature extraction network to extract features from key regions with relatively small sizes.

8. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: In step 5, video frames are input one by one into a self-supervised monocular scene depth estimation model. This model uses a combination of a lightweight U2net model encoder and decoder for scene depth estimation. The scene depth estimation model is then combined with a pose estimation model for self-supervised training. The pose estimation model uses a lightweight U2net model encoder. Self-supervised training calculates the error by reconstructing the image to train the model. Specifically, this process includes estimating the relative pose transformation matrix T between the target image and the source image using the pose estimation model. t→t′ The scene depth map D of the target image is predicted using a scene depth estimation model. t The encoder weights of the scene depth estimation model are shared with the lightweight U2net encoder in the pose estimation model; assuming that the camera intrinsics remain unchanged in each image, the scene depth map D of the target image is used. t Relative pose transformation matrix T t→t′ And the camera's intrinsic parameter matrix M, from which the reconstructed target image K is calculated. t′→t The calculation formula is: K t′→t =K t′ [proj(D t ,T t→t′ ,M)] (4) In the formula, K t′→t The reconstructed target image is calculated from the source image, K. t′ The source image is proj(), which represents the scene depth map D. t Projected onto source image K t′ The coordinates, M is the camera intrinsic parameter matrix, D t T is the scene depth map of the target image. t→t′ Let be the relative pose transformation matrix between the target image and the source image, and [] denote the sampling operation; Calculate the reconstructed target image K t′→t Compared with the actual target image K t The L1 distance between them is used to obtain the reconstruction error L. p The calculation formula is: In the formula, pe() is the photometric reconstruction error, which is the L1 distance in pixel space; Minimize the reconstruction error L using gradient descent. p The parameters of the scene depth model are optimized. During the optimization process, data augmentation is performed on the source image and the target image. Specifically, a region of 1 / 2 or 1 / 4 of the original image size is randomly selected from the center of the original image as new training data. The N×M video frames obtained in step 1 are input into the optimized model one by one to generate N×M frame scene depth maps.

9. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: In step 7, the fusion of RGB features and scene depth features adopts an adaptive fusion method. Specifically, the RGB input feature vector is standardized so that its mean is 0 and its standard deviation is 1. Then, the mean and standard deviation are changed using scene depth information. The calculation method is as follows: In the formula, f(T) rgb ,T d T represents the final video-level feature vector. rgb T d These represent the RGB feature vector and the scene depth feature vector, respectively, with parameter fc. s and FC b μ(T) is learned from a fully connected layer network. rgb ), σ(T) rgb ) represent the mean and standard deviation of the RGB feature vector in L dimensions, respectively.

10. The method for identifying small-sample abnormal behavior based on key regions and scene depth as described in claim 1, characterized in that: In step 8, the few-shot classifier, i.e. the prototype network, is first trained. The abnormal behavior dataset of surveillance video is used as the training data. The training data is divided into a support set and a prediction set. N categories are randomly selected from the dataset, with K samples in each category, for a total of N×K data, which are used as the input of the support set. Then, a batch of samples is drawn from the remaining data in these N classes as the model's prediction set; when predicting unknown abnormal behavior, the input data format is the same as during training, and the model learns to determine the labels of the prediction set samples through the support set; the features of the support set and prediction set samples are first L2 normalized, and the L2 normalization formula for a vector X is: Next, a trainable encoding network is used, consisting of two fully connected layers and an intermediate ReLU activation function. The first fully connected layer has an input vector dimension of 4096 and an output vector dimension of 4096, while the second fully connected layer has an input vector dimension of 4096 and an output vector dimension of 1024. Then, the mean of the K samples of each class in the support set is calculated, and the mean is used as the prototype B of each class of samples. i Calculate the prediction set sample A and each prototype B i The cosine similarity is calculated using the following formula: All cosine similarities are normalized using the softmax function. The type of abnormal behavior is determined based on the probability, and the abnormal behavior with the highest probability is the final identification result.

Images