A method and system for real-time video multi-person behavior recognition in complex scenes

Abstract

The present invention relates to a method and system for real-time video multi-person behavior recognition in complex scenes, and belongs to the field of machine vision technology. The method includes: collecting video data; establishing a scene face label library for face recognition model training; constructing a lightweight SlowFast model for complex scene multi-person behavior recognition; establishing an abnormal behavior action label library for different personnel categories, and establishing a training behavior action label library; after pre-processing the training data set, input it into the constructed lightweight SlowFast network for training; and use the face recognition model and the lightweight SlowFast network model to detect and identify actual scene behavior actions. The present invention determines the identity of the personnel category through face recognition, establishes the association relationship between the face detection frame and the personnel position positioning frame, detects and identifies the behavior actions of the target personnel in the scene, so as to realize the monitoring and early warning of abnormal behaviors of different categories of personnel in complex scenes.

Description

Technical Field

[0001] The present invention belongs to the field of machine vision technology and relates to a method and system for real-time video multi-person behavior recognition in complex scenarios using a lightweight deep learning network. Background Art

[0002] At present, the supervision of public places usually relies on video surveillance technology. Since traditional video surveillance technology can only monitor and store video recordings, when disputes or improper behavior occur in public places, video surveillance can only be retrieved and reviewed according to the time range. The introduction of complex scene real-time video multi-person behavior recognition methods and devices can monitor people's behavior in complex scenes in real time and issue early warnings so that public place managers can deal with abnormal situations in a timely manner, thereby ensuring the order and safety of public places, which has far-reaching practical significance and application value. At the same time, action recognition of video data that relies on time series and spatial information is also one of the research focuses in the field of computer vision, including two major tasks: video action behavior classification and human positioning and recognition. Existing action recognition methods can be mainly divided into three categories: methods based on two streams, methods based on 3D convolutional neural networks, and methods based on posture skeleton detection.

[0003] The dual-stream based algorithm relies on the use of the optical flow information of the image to characterize the trajectory of the motion. However, the optical flow uses pixel-level motion vectors, which has a large computational overhead, resulting in a relatively slow model based on the optical flow, which cannot meet the real-time requirements of the system. At the same time, since the optical flow information needs to be calculated separately, an end-to-end system cannot be implemented. Therefore, the model is relatively complex in the training and detection process, and its ability to be used in real-time systems is poor.

[0004] Convolution-based methods can use 3D convolution operations to simultaneously obtain temporal and spatial features, and perform end-to-end learning and prediction. However, when the network is deep, 3D convolution contains a large number of parameters, which consumes a lot of resources and is not conducive to large-scale deployment in actual production environments.

[0005] The method based on posture skeleton detection first obtains prior knowledge such as human joints through posture estimation, and then performs subsequent processing to obtain the final prediction result. This also cannot achieve end-to-end training and detection. In addition, the final result of the action recognition module depends on the prior knowledge of posture estimation, which easily leads to additional errors and affects the final accuracy.

[0006] Therefore, there is an urgent need for a detection method that can improve the real-time video multi-person behavior recognition in complex scenarios. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide a lightweight 3D convolutional neural network model for real-time video multi-person behavior recognition detection in complex scenes, solve the problem that the 3D convolutional neural network has large parameters and is not easy to embed into the actual generation environment, effectively reduce the number of parameters and model size of the 3D convolutional network, and improve the real-time performance of behavior detection and recognition. It can be used in the actual generation environment, improve the level of intelligent supervision of people's behavior in complex scenes, and provide effective protection for the order and safety of public places.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] 1. A method for real-time video multi-person behavior recognition in complex scenes, based on the Slowfast framework, improves the 3D-ResNet50 backbone network and replaces it with a lightweight 3D-ShuffleNetV2 backbone network, called ShuffleNetV2_SlowFast network. This solves the problem of high computational overhead and large number of parameters based on 3D convolutional neural networks, and recognizes and locates multi-person behaviors in videos. The method specifically includes the following steps:

[0010] S1: Capture video frames of people in the scene using a capture device;

[0011] S2: Establish a scene face label library to train the existing face recognition model;

[0012] S3: Build a lightweight SlowFast network for multi-person behavior recognition in complex scenarios;

[0013] S4: According to the identities of different personnel categories in different scenarios, establish abnormal behavior action label libraries and training behavior action label libraries for different personnel categories;

[0014] S5: pre-process the training behavior action label library established in step S4, and then input it into the constructed lightweight SlowFast network for training;

[0015] S6: Use the detection model trained by the lightweight SlowFast network to perform behavior recognition detection on actual scenarios: Use the face recognition model to confirm the identity and determine the category of personnel to which they belong; use the trained lightweight SlowFast network to detect behavior actions and locate the position of the personnel; establish the association between the face detection frame and the personnel position positioning frame; according to the personnel category behavior requirements, compare the behavior actions detected by the lightweight SlowFast network with the behavior action label library of this type of personnel, further determine whether there are abnormal behaviors that this type of personnel should not have, and alarm and punish their abnormal behavior.

[0016] Furthermore, step S1 includes the setting of equipment installation and installation conditions:

[0017] S11: Specify the installation location of the acquisition device to capture the video frames of people in the scene for detecting and identifying behavior and confirming identity.

[0018] S12: Specify the hardware conditions of the device to meet the real-time and accuracy requirements of the model.

[0019] Further, step S2 includes the following steps:

[0020] S21: Build a face label library of different categories of people in complex scenarios for face recognition model training.

[0021] S22: Use the scene face label library collected by S21 to train the existing FaceNet face recognition model.

[0022] Further, in step S3, a lightweight SlowFast network is constructed, specifically including: when constructing the lightweight SlowFast network, the Slow branch 3D-ResNet50 backbone network of SlowFast is replaced according to the 3D-ShuffleNetV2 network structure; during the replacement process, according to the characteristics of the low frame rate of the Slow branch, the input video frame is sampled once every 16 frames, and the temporal dimension is not downsampled to construct a new Slow branch network structure; for the characteristics of the high frame rate and low channel capacity of the Fast branch, the input video frame is sampled once every 2 frames, and the temporal dimension is not sampled, and the number of channels is 1 / 8 times that of the Slow branch to construct the Fast branch; finally, the 3D convolution operation is used to scale the Fast branch features, and then cascaded to the Slow branch to achieve the lateral connection of SlowFast.

[0023] Further, in step S3, a lightweight SlowFast network is constructed, which specifically includes the following steps:

[0024] S31: Build the Slow branch of SlowFast. Based on the 3D-ShuffleNetV2 network structure, replace the 3D-ResNet50 backbone network of the Slow branch of SlowFast. During the replacement process, based on the low frame rate characteristics of the Slow branch, sample the input video frames once every 16 frames, and do not perform downsampling calculations on the temporal dimension to build a new Slow branch network structure.

[0025] S32: Build the Fast branch of SlowFast. Based on the 3D-ShuffleNetV2 network structure, replace the Fast branch 3D-ResNet50 backbone network of SlowFast. During the replacement process, based on the characteristics of high frame rate, high temporal resolution and low channel capacity of the Fast branch, sample the input video frames once every 2 frames, and do not downsample the temporal dimension. At the same time, the number of channels is 1 / 8 of that of the slow branch to build a new Fast branch network structure.

[0026] S33: Construct the lateral connection of SlowFast. In the process from the first stage of ShuffleNetV2 to the second stage of ShuffleNetV2 and from the second stage of ShuffleNetV2 to the third stage of ShuffleNetV2, establish the lateral connection from the Fast branch to the Slow branch; use 3D convolution to scale the Fast branch features, and concatenate the converted features to the Slow path;

[0027] S34: After the third stage of ShuffleNetV2, the model classifier module is constructed using the global average pooling layer and the fully connected layer for classification and prediction.

[0028] Further, in step S5, the lightweight SlowFast network is trained, specifically including: first, from the training set of the training behavior action label library established in S4, a set of video data is cyclically input and a clip (64 frames) is randomly sampled; scaling is performed proportionally, and a video frame of 224*224 scale is randomly cropped from each frame, and then preprocessing operations such as horizontal flipping are performed on it, and then the preprocessed video frame fragments are respectively input into the Slow branch and Fast branch network of the ShuffleNetV2_SlowFast network; and the Slow branch network and the Fast branch network are cascaded and aggregated, and the features after the Slow branch network and the Fast branch are aggregated are classified and the loss value is calculated through the ShuffleNetV2_SlowFast Head module; the specified number of training cycles is executed to complete the training of the lightweight ShuffleNetV2_SlowFast network model.

[0029] Further, in step S6, the detection model trained by the lightweight SlowFast network is used to perform behavior recognition detection on the actual scene, which specifically includes the following steps:

[0030] S61: In the detection process, first use the FaceNet face recognition model trained in step S2 to perform face recognition and determine the identity of the person; use the lightweight SlowFast network to detect behavior and locate the person; establish the association between the face detection frame and the person location frame; then, according to the behavior requirements of different categories of people in different scenarios, establish a mapping relationship between the identified person and the behavior requirements of the category to which they belong; measure and calculate the behavior detected by the ShuffleNetV2_SlowFast network and the feature vector of the behavior label library of this type of person; if the cosine distance is less than the threshold K, the identified person has abnormal behavior that should not be seen by this type of person, and it is recorded in the server log, and an alarm is issued for its abnormal behavior to facilitate monitoring and punishment;

[0031] S62: Metric calculation;

[0032] According to the characteristics of the required behavior actions of the category of personnel and the action features extracted based on the ShuffleNetV2_SlowFast model, the cosine distance formula is used for measurement calculation. The cosine distance formula (1) is as follows:

[0033]

[0034] Among them, A and B represent the feature vectors whose similarity needs to be compared.

[0035] 2. A real-time video multi-person behavior recognition system in complex scenarios, comprising:

[0036] Information acquisition module: uses audio and video decoders to convert the collected video information into digital signals and transmit them to the processor;

[0037] Processor: used to process the signals transmitted by the information acquisition module and transmit the processed information to the server through the local area network;

[0038] Storage module: used to store digital information processed by the processor;

[0039] Server: processes the information passed by the processor and is used to run the FaceNet face recognition model and the lightweight ShufffleNetV2_SlowFast action recognition model, and provides analysis and judgment results and alarms for the actions of different people in the scene.

[0040] The specific processing flow of the system is as follows: First, the information acquisition module will collect audio and video files through hardware such as cameras, and then use the audio and video decoder to convert the collected video information into digital signals and transmit them to the processor. After the processor processes the incoming digital signal, it will be stored in the storage module, and the processed command signal will be transmitted to the server through the local area network connection. According to the incoming command information, the server executes the corresponding FaceNet face recognition model and the lightweight ShufffleNetV2_SlowFast action recognition model for detection, recognition and alarm.

[0041] The beneficial effects of the present invention are:

[0042] (1) The lightweight SlowFast network of the present invention, namely the ShuffleNetV2_SlowFast network, is used for the detection and recognition of different types of human behaviors in complex scenes. Compared with the original SlowFast action recognition network based on 3D-ResNet50, it can realize the detection and recognition of different types of human behaviors in complex scenes, and has a significant reduction in the number of parameters and the amount of calculation.

[0043] (2) The present invention adopts a lightweight 3D-ShuffleNetV2 network structure and integrates the SlowFast network concept to obtain the ShuffleNetV2_SlowFast network model, which mainly uses the separable convolution of the ShuffleNetV2 module to replace the standard convolution, and performs separate calculations on the channel dimension at the module input stage, and shuffles and merges the channel dimension when the final cascade is merged. There is no repeated superposition operation, thereby reducing the number of parameters in the backbone network.

[0044] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0046] Figure 1 It is a flow chart of the method for detecting behavior recognition of various types of people based on the ShuffleNetV2_SlowFast network in complex scenarios of the present invention;

[0047] Figure 2The ShuffleNet Block module (left) and the ShuffleNet Block module with spatial channel downsampling (right) of ShuffleNet_V2 in the real-time example of the present invention. DETAILED DESCRIPTION

[0048] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0049] See also Figure 1-2 ,In view of the background technology, in order to improve the ,precision of the recognition of different categories of people ,actions in complex scenes and reduce the number of ,parameters and model size, such as Figure 1 As shown, the present invention provides a method for intelligently identifying the action behaviors of different categories of people in complex scenarios based on a lightweight SlowFast network, comprising the following steps:

[0050] S1: Capturing video frames of people in the scene using a collection device; wherein the installation position and installation conditions of the collection device are set, specifically including:

[0051] S11: Specify the installation location of the acquisition device. The camera is aimed at the entrance of the scene, mainly used to capture the face information and behavior of the person, identify the person through the face information, and determine the person category and category ID identity information. At the same time, the behavior of the detected person is tracked and identified.

[0052] S12: Regarding the conditions of equipment hardware, the present invention requires that the camera deployed on site has a refresh rate of not less than 30fps to ensure the real-time requirement of the slowfast motion recognition model for high frame rate; at the same time, the resolution of the camera is required to be not less than 1080P to meet the requirements of face detection and recognition.

[0053] S2: Establish a scene face label library for face recognition model training; specifically includes:

[0054] S21: Build a face label library of different categories of people in complex scenarios for face recognition model training.

[0055] S22: Use the scene face label library collected by S21 to train the existing FaceNet face recognition model.

[0056] S3: Build a lightweight SlowFast model for multi-person behavior recognition in complex scenarios;

[0057] Table 1 ShuffleNetV2_SlowFast overall network structure

[0058]

[0059] Block:ShuffleNetV2 Block

[0060] Block_S:ShuffleNetV2 Block with Sample

[0061] As shown in Table 1, there are mainly the following steps to build a lightweight SlowFast model backbone network:

[0062] S31: Build the Slow branch of SlowFast;

[0063] Taking the original video frames uniformly processed into 224*224 pixels and input 64 frames each time as an example, according to the Slow Path low frame rate, sampling is performed at the data input layer with a step length of stride=(16,1,1) to obtain an output feature map (T*S*S, T represents the temporal dimension, S represents the spatial dimension) of 4*224*244;

[0064] The 3D convolution kernel size is kernel_size = 3*3*3, stride = (1, 2, 2), and the number of channels is channel. slow_pat = 24, calculate the features of the previous step, and obtain a feature map with an output of 4*112*112; use the max_pooling (maximum pooling layer) with a convolution kernel size of kernel_size=3*3*3 and stride=(1,2,2) to calculate the features of the previous step, and obtain a feature map of 4*56*56;

[0065] Enter the first stage of ShuffleNetV2. In the first stage of ShuffleNetV2, first enter the ShuffleNetV2Block with downsampling. According to the low frame rate and high temporal dimension characteristics of the Slow branch, only the spatial dimension is downsampled, and the temporal dimension is not downsampled; stride = (1,2,2), the number of channels channel slow_path =32, calculate and output a 4*28*28 feature map; then enter 3 consecutive ShuffleNetV2 Blocks, stride = (1,1,1), channel number channelslow_path =32, the calculated output feature map scale remains unchanged as a 4*28*28 feature map;

[0066] Enter the second stage of ShuffleNetV2. In the second stage of ShuffleNetV2, first enter the ShuffleNetV2Block with downsampling, stride = (1, 2, 2), channel number channel slow_path =64, the calculated output is a 4*14*14 feature map; then enter 7 consecutive ShuffleNetV2 Blocks, stride = (1,1,1), channel slow_path =64, the calculated output feature map scale remains unchanged to a 4*14*14 feature map.

[0067] Enter the third stage of ShuffleNetV2. In the third stage of ShuffleNetV2, first enter the ShuffleNetV2Block with downsampling, stride = (1,2,2), channel number channel slow_path =128, the calculated output is a 4*7*7 feature map; then enter 3 consecutive ShuffleNetV2 Blocks, stride = (1,1,1), channel number channel slow_path =128, and finally the Slow branch is calculated, and the output feature tensor of the feature map scale is 4*7*7.

[0068] S32: Build the Fast branch of SlowFast;

[0069] Taking the original video frames uniformly processed into 224*224 pixels and inputting 64 frames each time as an example, according to the high frame rate characteristics of Fast Path, sampling is performed at the data input layer with a step length of stride=(2,1,1) to obtain an output feature map (T*S*S, T represents the temporal dimension, S represents the spatial dimension) of 32*224*244;

[0070] Based on the characteristics of the low channel of the Fast branch, 3D convolution Conv1 is used to calculate the features of the previous step: the 3D convolution kernel scale is kernel_size = 3*3*3, the step length stride = (1,2,2), and the number of channels is 1 / 8 of the Slow branch (therefore, the number of channels channel fast_path =4). The calculated output is a feature map of 32*112*112. The max_pooling layer with kernel size = 3*3*3 and stride = (1,2,2) is used to calculate the features of the previous step and obtain a feature map of 32*56*56.

[0071] Enter the first stage of ShuffleNetV2. In the first stage of ShuffleNetV2, first enter the ShuffleNetV2Block with downsampling. According to the characteristics of the Fast branch, only the spatial dimension is downsampled, and the temporal dimension is not downsampled; stride = (1, 2, 2), the number of channels is channel fast_path =4, calculate and output a 32*28*28 feature map; then enter 3 consecutive ShuffleNetV2 Blocks, stride = (1,1,1), channel number channel fast_path =4, the calculated output feature map scale remains unchanged as a feature map of 32*28*28;

[0072] Enter the second stage of ShuffleNetV2. In the second stage of ShuffleNetV2, first enter the ShuffleNetV2Block with downsampling, stride = (1,2,2), channel fast_path =8, the calculated output is a 32*14*14 feature map; then enter 7 consecutive ShuffleNetV2 Blocks, stride = (1,1,1), channel fast_path =8, the calculated output feature map scale remains unchanged as a 32*14*14 feature map;

[0073] Enter the third stage of ShuffleNetV2. In the third stage of ShuffleNetV2, first enter the ShuffleNetV2Block with downsampling, stride = (1,2,2), channel fast_path =16, the calculated output is a 4*7*7 feature map; then enter 3 consecutive ShuffleNetV2 Blocks, stride = (1,1,1), channel fast_path =16, and finally the Fast branch is calculated, and the output feature tensor with a feature map scale of 32*7*7 is obtained.

[0074] S33: build lateral connections of SlowFast;

[0075] From the first stage of ShuffleNetV2 to the second stage of ShuffleNetV2, a lateral connection is established from the fast path branch to the slow path; the 3D convolution with kernel size = 5*1*1, stride = (8,1,1), and channel number channel = 16 is selected to transform the fast path feature scale to obtain {T,S 2,C}, where C represents the channel dimension, {4,28 2 ,8} features, the obtained features are concatenated to the slow path according to the channel dimension.

[0076] From the second stage of ShuffleNetV2 to the third stage of ShuffleNetV2, a lateral connection is established from the fast path branch to the slow path; similarly, a 3D convolution with kernel size = 5*1*1, stride = (8,1,1), and channel number channel = 16 is selected to transform the fast path feature scale to obtain {T,S 2 ,C},{4,14 2 ,16} features, and concatenate the obtained features to the slow path according to the channel dimension.

[0077] S34: After the third stage of ShuffleNetV2, the Fast branch and Slow branch output feature tensors are concatenated according to the channel dimension; then they are input into the global average pooling layer and the fully connected classifier layer for classification and prediction; the constructed network is named ShuffleNetV2_SlowFast based on the backbone network.

[0078] S4: Establish an abnormal behavior action label library for different personnel categories and establish a training behavior action label library.

[0079] The steps for establishing a video library of abnormal behavior actions of different categories of people and a training behavior action label library are established according to the labeling rules of the Google AVA (aomic visual actions) dataset. Specifically: First, the original collected behavior action video is analyzed for 15 minutes, and the 15-minute video is uniformly divided into 300 non-overlapping 3-second segments. The sampling follows the strategy of maintaining the time order of the action sequence. Then, the LabelImg labeling tool is used to manually annotate the bounding box of the characters in the middle frame of each 3-second segment. For each person in the annotated box, an appropriate label is selected from the pre-made action category table to describe the character's action. Finally, all video clips are fully labeled to establish a video library of abnormal behavior actions of different categories of people and a training behavior action label library.

[0080] S5: After preprocessing, the collected training action label library is input into the constructed lightweight SlowFast network for training;

[0081] In the training phase, first, a set of video data is input cyclically from the training set established by S4 and a clip (64 frames) is randomly sampled. Then 4 frames and 16 frames are input to the Slow branch path and the Fast branch path respectively, and the original video frames are preprocessed (scaled, randomly cropped out 224*224 video frames, and horizontally flipped). Training is performed for 20 rounds, using learning rate hot start, the initial learning rate (learning rate) is set to 0.000125, and the optimization strategy optimizing_method: sgd (stochastic gradient descent). The learning rate hot start round epochs = 5, the decay rate weight_decay = 1e-7, and the learning rate hot start is performed in the first 5 training rounds. After 5 rounds, the learning rate reaches a stable state, and then the model is relatively stable for 15 rounds.

[0082] S6: Use the detection model trained by the lightweight SlowFast network for behavior recognition detection in actual scenarios: Use the face recognition model to confirm the identity and determine the category of personnel to which they belong. Use the lightweight SlowFast network to detect behavioral actions and locate the position of personnel. Establish an association between the face detection frame and the personnel location frame. According to the personnel category behavior requirements, compare the behavioral actions detected by the lightweight SlowFast network with the behavioral action label library of this type of personnel to further determine whether there are abnormal behaviors that this type of personnel should not have, and issue alarms and penalties for their abnormal behaviors. Specifically, the following steps are included:

[0083] S61: In the detection process, first use the face recognition model FaceNet trained in step S22 to perform face recognition and determine the identity of the person. Then, according to the behavioral action requirements of different categories of people in different scenarios, establish a mapping relationship between the identified person and the behavioral action requirements of the category to which they belong. The behavior actions detected by the ShuffleNetV2_SlowFast network are measured and calculated with the feature vector of the behavior action label library of this type of person. If the cosine distance is less than the threshold T, the identified person has abnormal behavior that should not be seen by this type of person, and it is recorded in the server log, and an alarm is issued for its abnormal behavior to facilitate monitoring and punishment.

[0084] S62: During the measurement calculation, the cosine distance formula is used to perform measurement calculation based on the characteristics of the behavioral actions required by the category of personnel and the action characteristics extracted based on the ShuffleNetV2_SlowFast model.

[0085] A method and device for real-time video multi-person behavior recognition in complex scenes, the device comprising:

[0086] The device of the present invention comprises an information collection module, a storage module, a processor module and a server module. The video data collected by the information collection module is input into the processor module for processing.

[0087] During the processing, the compressed video frames will be stored in the unprocessed data area of ​​the storage module according to the audio and video compression algorithm, and the instructions will be sent to the remote server through the LAN connection according to the logical decision obtained by the calculation. After receiving the instructions from the processor, the server will read the video data from the unprocessed data area in the storage module according to the parameter configuration information in the instruction, and enable the FaceNet face recognition model and ShufffleNetV2_SlowFast action recognition model to process the input video data in real time. Then the detected data is stored in the processed data area in the storage module, and then the processor will read the data from the processed data area in the storage module. According to the detected personnel category identity and the characteristics of the identified action behavior and the stored action tag library of the personnel category behavior, the cosine metric calculation is performed, and the relationship between the calculated cosine distance value and the threshold K is used to determine whether there is abnormal behavior.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for real-time video multi-person behavior recognition in complex scenes, characterized in that: The method specifically comprises the following steps: S1: Use the acquisition device to capture the video frames of people in the scene; S2: Establish a scene face label library to train the existing face recognition model; S3: Build a lightweight SlowFast network for multi-person behavior recognition in complex scenarios; Building a lightweight SlowFast network includes the following steps: S31: Build the Slow branch of SlowFast. Based on the 3D-ShuffleNetV2 network structure, replace the 3D-ResNet50 backbone network of the Slow branch of SlowFast. During the replacement process, based on the low frame rate characteristics of the Slow branch, sample the input video frames once every 16 frames, and do not perform downsampling calculations on the temporal dimension to build a new Slow branch network structure. S32: Build the Fast branch of SlowFast. Based on the 3D-ShuffleNetV2 network structure, replace the Fast branch 3D-ResNet50 backbone network of SlowFast. During the replacement process, based on the characteristics of high frame rate, high temporal resolution and low channel capacity of the Fast branch, sample the input video frames once every 2 frames, and do not downsample the temporal dimension. At the same time, the number of channels is 1 / 8 of that of the slow branch to build a new Fast branch network structure. S33: Construct the lateral connection of SlowFast. In the process from the first stage of ShuffleNetV2 to the second stage of ShuffleNetV2 and from the second stage of ShuffleNetV2 to the third stage of ShuffleNetV2, establish the lateral connection from the Fast branch to the Slow branch; use 3D convolution to scale the Fast branch features, and concatenate the converted features to the Slow path; S34: After the third stage of ShuffleNetV2, the model classifier module is constructed using the global average pooling layer and the fully connected layer for classification and prediction, and the constructed lightweight Slowfast model is called ShuffleNetV2_SlowFast network; S4: According to the identities of different personnel categories in different scenarios, establish abnormal behavior action label libraries and training behavior action label libraries for different personnel categories; S5: pre-process the training behavior action label library established in step S4, and then input it into the constructed lightweight SlowFast network for training; S6: Use the detection model trained by the lightweight SlowFast network to perform behavior recognition detection on actual scenarios: Use the face recognition model to confirm the identity and determine the category of personnel to which they belong; use the trained lightweight SlowFast network to detect behavior actions and locate the position of the personnel; establish the association between the face detection frame and the personnel position positioning frame; according to the personnel category behavior requirements, compare the behavior actions detected by the lightweight SlowFast network with the behavior action label library of this type of personnel, further determine whether there are abnormal behaviors that this type of personnel should not have, and alarm and punish their abnormal behavior.

2. The method for real-time video multi-person behavior recognition in complex scenes according to claim 1 is characterized in that: In step S3, a lightweight SlowFast network is constructed, specifically including: when constructing the lightweight SlowFast network, the 3D-ResNet50 backbone network of the Slow branch of SlowFast is replaced according to the 3D-ShuffleNetV2 network structure; during the replacement process, according to the characteristics of the low frame rate of the Slow branch, the input video frame is sampled once every 16 frames, and the temporal dimension is not downsampled to construct a new Slow branch network structure; for the characteristics of the high frame rate and low channel capacity of the Fast branch, the input video frame is sampled once every 2 frames, and the temporal dimension is not sampled, and the number of channels is 1 / 8 of that of the Slow branch to construct the Fast branch; finally, the 3D convolution operation is used to scale the Fast branch features, and then cascaded to the Slow branch to achieve the lateral connection of SlowFast.

3. The method for real-time video multi-person behavior recognition in complex scenes according to claim 1 is characterized in that: In step S5, the lightweight SlowFast network is trained, specifically including: first, from the training set of the training behavior action label library established in S4, a set of video data is cyclically input and a clip is randomly sampled; each frame is scaled and randomly cut out to obtain video frames of the same scale, and then preprocessed, and then the preprocessed video frame fragments are respectively input into the Slow branch and Fast branch network of the ShuffleNetV2_SlowFast network; and the Slow branch network and the Fast branch network are cascaded and aggregated, and the features after the Slow branch network and the Fast branch are aggregated are classified and the loss value is calculated through the ShuffleNetV2_SlowFastHead module; the specified number of training cycles is executed to complete the training of the lightweight ShuffleNetV2_SlowFast network model.

4. The method for real-time video multi-person behavior recognition in complex scenes according to claim 1 is characterized in that: In step S6, the detection model trained by the lightweight SlowFast network is used to perform behavior recognition detection on the actual scene, which specifically includes the following steps: S61: In the detection process, first use the face recognition model trained in step S2 to perform face recognition and determine the identity of the person; use the lightweight SlowFast network to detect behavior and locate the person; establish the association between the face detection frame and the person location frame; then, according to the behavior requirements of different categories of people in different scenarios, establish a mapping relationship between the identified person and the behavior requirements of the category to which they belong; measure and calculate the behavior detected by the ShuffleNetV2_SlowFast network and the feature vector of the behavior label library of this type of person; if the cosine distance is less than the threshold K, the identified person has abnormal behavior that should not be seen by this type of person, and it is recorded in the server log, and an alarm is issued for its abnormal behavior to facilitate monitoring and punishment; S62: Metric calculation; According to the characteristics of the behavioral actions required by this category of personnel and the action features extracted based on the ShuffleNetV2_SlowFast model, the cosine distance formula is used for measurement calculation.

5. A recognition system applicable to the method for real-time video multi-person behavior recognition in complex scenes as claimed in any one of claims 1 to 4, characterized in that: The system includes: Information acquisition module: uses audio and video decoders to convert the collected video information into digital signals and transmit them to the processor; Processor: used to process the signals transmitted by the information acquisition module and transmit the processed information to the server through the local area network; Storage module: used to store digital information processed by the processor; Server: processes the information passed by the processor and is used to run the FaceNet face recognition model and the lightweight ShufffleNetV2_SlowFast action recognition model, and provides analysis and judgment results and alarms for the behavior of different people in the scene.

Images