A method and system for detecting vigorous motion based on adaptively generated training samples

Abstract

The application provides a violent motion detection method and system based on training sample adaptive generation, obtains existing video information, positions a video segment where a pedestrian appears and the approximate position of a motion target in each frame in a video image; the video segment is divided into multiple short video segments, and the behavior category of each short video segment is labeled; the first and last time stamps of the appearance of violent motion in a short video segment suspected of appearing violent motion are detected, and each violent motion behavior picture of a pre-training data set is made; the behavior picture is preprocessed to form a training set; a network model including a feature extraction layer and a behavior classification layer is constructed, the network model is trained by using the training set until a set requirement is met; and the trained network model is compressed by using a knowledge distillation method and is deployed. The application can detect and position violent motion in a monitoring place in real time and timely respond to dangerous emergencies.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence, and relates to a violent motion detection method and system based on training sample adaptive generation. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] In recent years, social public safety and public security stability problems have gradually become the focus of public attention. Video monitoring has played a huge role in maintaining public security due to its recordability and analyzability, and has greatly promoted the construction process of smart city and safe city. Therefore, intelligent video monitoring technology has emerged as the times require and has rapidly become a research hotspot in the academic and industrial circles.

[0004] A large number of monitoring cameras are usually installed in public places to monitor pedestrian behavior, so that a timely response can be made to the occurrence of dangerous incidents or abnormal behavior. When the human body performs violent motion, such as punching, fencing, swinging, wrestling, shooting, high jump, long jump, kicking football, shooting, skateboarding, horseback riding, etc., there is often a certain degree of danger, and it is also an important signal of abnormal behavior. And in different monitoring places, the types of violent motion that may occur are also different, such as in a classroom, it is impossible to have behaviors such as fencing, punching, and swinging, and if such behaviors occur, the supervision vigilance needs to be improved to avoid the occurrence of dangerous or abnormal events in time. In addition, in a classroom, it is also impossible to have violent motions such as high jump, shooting, and horseback riding, and when real-time video monitoring and behavior detection are performed on a classroom, it is not necessary to load the recognition and detection of such motions (high jump, shooting, and horseback riding). Therefore, real-time and correct detection of violent motion of the human body in the monitoring video stream can enable the supervisory personnel to respond to dangerous incidents or abnormal behavior in time and take measures, which is of great significance to public safety.

[0005] Common behavior detection methods include methods based on traditional image processing and methods based on deep learning. In the process of implementing the present application, the inventors found that the current technology has the following technical problems:

[0006] The idea of video monitoring behavior detection based on traditional image processing methods is mainly based on inter-frame difference or background difference detection. This method has fast processing speed, but poor accuracy, and is prone to false detection and false reporting.

[0007] With the development of deep learning technology, the development of intelligent video processing is also growing. The accuracy of abnormal behavior detection in video monitoring is continuously improved.

[0008] However, the existing deep learning algorithm can only identify and classify video segments, and cannot accurately locate the start and end time frames of the violent motion in real time, and has high requirements for training data. The positioning and classification of violent motion in video monitoring currently face great challenges. First, the frequency and time of violent motion in video monitoring are relatively short, so the data is highly imbalanced, and manual positioning and selection of video segments are required. In addition, the amount of video monitoring data is huge, but the labeling is almost non-existent, so large-scale supervised data labeling is required, and the automatic construction of the data set cannot be realized.

[0009] In the behavior detection process of video monitoring, due to the high complexity and large amount of calculation of the network model, the video needs to be detected frame by frame or frame by frame in the actual application process, so the real-time performance of violent motion detection is difficult to guarantee. SUMMARY

[0010] To solve the above problems, the application provides a violent motion detection method and system based on adaptive generation of training samples. In a monitoring place, the application can detect and locate the violent motion in the monitoring place in real time, and respond to dangerous emergencies in a timely manner. The network can gradually improve the recognition accuracy through iterative learning.

[0011] According to some embodiments, the application adopts the following technical scheme:

[0012] A violent motion detection method based on adaptive generation of training samples, which uses a pre-trained network model to process the obtained video and detect violent motion. The training and deployment process of the network model includes the following steps:

[0013] Obtain existing video information, locate the video segment where pedestrians appear, and the approximate position of the moving target in each frame in the video image;

[0014] Divide the video segment into multiple short video segments, and label the behavior category of each short video segment;

[0015] Detect the start and end timestamps of the violent motion in the short video segment where the suspected violent motion appears, and cut the short video segment where the violent motion appears according to the timestamps to produce each violent motion behavior picture of the pre-training data set;

[0016] Preprocess the behavior pictures to form a training set;

[0017] Construct a network model including a feature extraction layer and a behavior classification layer, and train the network model using the training set until the set requirements are met;

[0018] Compress the trained network model using the knowledge distillation method and deploy it.

[0019] As an alternative embodiment, the specific process of acquiring existing video information and locating the specific video segment where the pedestrian appears includes detecting the video segment where the pedestrian appears by using the frame difference method, automatically locating the video frame range where the pedestrian appears, and roughly locating the position of the moving target in the video image in each frame.

[0020] As an alternative embodiment, the specific process of splitting the video segment into multiple short video segments includes uniformly splitting the video segment containing the behavior into short video segments of a set length, separately dividing the part that is less than the set length into a short video segment, and processing it using the multi-behavior video classification model.

[0021] As an alternative embodiment, the specific process of detecting the start and end time stamps of the appearance of the violent motion in the short video segment suspected of appearing violent motion includes: detecting the start and end time frames of the appearance of the violent motion in each short video using the optical flow method, and automatically locating and outputting the start and end time stamps of the appearance of the violent motion in each short video according to the degree of disorder of the human body.

[0022] As an alternative embodiment, the specific process of making each violent motion behavior picture of the pre-training data set includes: extracting the picture of the current video violent motion behavior frame by frame, and uniformly sampling an equal amount of non-behavior pictures in other video frames that do not contain the behavior.

[0023] Based on the mosaic data augmentation, the number of data channels is expanded, and the behavior and non-behavior pictures are respectively labeled as label 1 and label 0, an equal number of binary classification pictures of the behavior are obtained, and the binary classification pictures of the behavior are stored in the folder corresponding to the behavior.

[0024] Respectively make folders for each type of violent motion behavior.

[0025] As an alternative embodiment, the pre-processing of the training set includes: resizing, normalizing, and data augmentation of the pictures, data normalization, and dimensionality reduction of the data set.

[0026] As an alternative embodiment, the network model uses a neural network as a feature extraction layer, extracts relevant features through convolution stacking, inputs the features into another neural network for classification, captures action behavior information, and learns the relationship between sequence elements.

[0027] As an alternative embodiment, the specific process of training the network model using the training set includes: a behavior and non-behavior picture in the training set are simultaneously assigned corresponding labels 1 and 0 for the two types of pictures, and the picture and label form a pre-training set for the behavior.

[0028] Set a random seed, send the training set data into the network model, build a model optimizer and a loss function, calculate the forward propagation loss value through the loss function and the network model parameters, and finally obtain the pre-training weight to determine the network parameter information;

[0029] The network iterative learning is performed by adjusting the probability threshold, that is, repeating the training-test step, and sending the test result into the training set for iterative training of the next round, and retaining the weight meeting the probability threshold standard as the final behavior test weight.

[0030] A violent motion detection system based on adaptive generation of training samples, comprising,

[0031] The network model execution module is configured to utilize the pre-trained network model to process the acquired video, and detect violent motion, and further comprises:

[0032] The data acquisition module is configured to acquire existing video information, locate the video segment in which a pedestrian appears, and locate the approximate position of the motion target in the video image in each frame;

[0033] The division module is configured to divide the video segment into a plurality of short video segments, and label the behavior category of each short video segment;

[0034] The training set manufacturing module is configured to detect the start and end time stamps of the violent motion in the short video segment in which the suspected violent motion appears, cut the short video segment in which the violent motion appears according to the time stamps, manufacture the behavior pictures of each violent motion in the pre-training data set, pre-process the behavior pictures, and form a training set;

[0035] The model training module is configured to build a network model comprising a feature extraction layer and a behavior classification layer, and train the network model by using the training set until a set requirement is met;

[0036] The model deployment module is configured to compress the trained network model by using a knowledge distillation method, and deploy the compressed network model.

[0037] A computer-readable storage medium, wherein a plurality of instructions are stored, the instructions being adapted to be loaded by a processor of a terminal device and to execute the steps in the method.

[0038] A terminal device comprising a processor and a computer-readable storage medium, the processor being configured to implement instructions, and the computer-readable storage medium being configured to store a plurality of instructions, the instructions being adapted to be loaded by the processor and to execute the steps in the method.

[0039] Compared with the prior art, the beneficial effects of the present application are:

[0040] The deep learning algorithm of the application has low requirements for training data, adopts a multi-behavior classification network to finely classify long videos, assigns a short video behavior label, adopts an optical flow method to automatically locate the start and end time stamps of each violent motion, and obtains each behavior data picture, so that the automatic construction of the training set can be realized. Each violent motion behavior is regarded as a binary classification problem, the program automatically obtains and labels the data set, integrates multiple pictures together, expands the number of picture channels, increases the picture information, and is beneficial to the improvement of the subsequent training efficiency. Finally, the detection, classification and positioning of the violent motion behavior can be realized.

[0041] The application adopts a CNN convolutional neural network model constructed by itself to train the data set, simplifies the existing convolutional neural network model, and at the same time, the network model adopts iterative learning, sets multiple threshold criteria, sends the test results into the training set for multiple rounds, trains multiple times, and continuously improves the binary classification accuracy of each behavior. The calculation amount is small, and finally the violent motion can be accurately detected and positioned, and the violent motion time and category can be displayed in the monitoring video in real time, and the effect is intuitive.

[0042] The application can realize real-time detection of violent motion in a monitoring video, and through the method of knowledge distillation, the model is compressed and deployed to an embedded device, and according to the scene characteristics and requirements, the weight can be selectively deployed and loaded, and the function of different scenes can be selectively recognized. Wide application range and strong deployment.

[0043] The advantages of the additional aspects of the application will be partially given in the following description, partially become obvious from the following description, or be known by the practice of the application. BRIEF DESCRIPTION OF DRAWINGS

[0044] The drawings accompanying the specification of the application form part of the application and serve to provide further understanding of the application, the illustrative embodiments of the application and their description serve to explain the application without constituting an inappropriate limitation thereof.

[0045] Figure 1 The flowchart for obtaining the pedestrian frame of the embodiment one of the application;

[0046] Figure 2 The schematic diagram of the classification data set of the embodiment one of the application;

[0047] Figure 3 The schematic diagram of the overall flow of the embodiment one of the application;

[0048] Figure 4 The schematic diagram of the network model system of the embodiment one of the application;

[0049] Figure 5 The flowchart of the network iterative training of the embodiment one of the application;

[0050] Figure 6A knowledge distillation model process schematic diagram of embodiment one of the present application. DETAILED DESCRIPTION

[0051] The present application will be further described with reference to the drawings and embodiments.

[0052] It should be noted that the following detailed description is illustrative only and is intended to provide further description of the present application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0053] It should be noted that the terms used herein are merely for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the term "comprising" and / or "including" is used in the specification, it means that the features, steps, operations, devices, components and / or combinations thereof are present.

[0054] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0055] Term explanation:

[0056] 1. Inter-frame difference method: The inter-frame difference method is a method of obtaining the contour of a moving target by performing a difference operation on two adjacent frames in a video image sequence. It can be well applied to the case where there are multiple moving targets and camera movement.

[0057] 2. Optical flow method: The optical flow method is a concept in the detection of object motion in a field of view. It is used to describe the motion of an observed target, surface or edge caused by motion relative to the observer.

[0058] 3. CNN: Convolutional neural network is a deep neural network with convolutional structure, which can reduce the amount of memory occupied by deep network. The network model adjusts the weight parameters in the network layer by layer through gradient descent method to minimize the loss function, and improves the accuracy of the network through frequent iteration training.

[0059] 4. LSTM: Long Short-Term Memory (LSTM) is a time recurrent neural network, which is specially designed to solve the long-term dependence problem of general RNN (recurrent neural network).

[0060] 5. Knowledge distillation: Knowledge distillation is a method of extracting knowledge from complex models and compressing it into a single model so that it can be deployed in practical applications.

[0061] Embodiment one

[0062] This example discloses an iterative optimization method for a violent motion detection system based on adaptive generation of training samples, which is used to detect and locate violent motion in surveillance videos.

[0063] The classification problem of any action in strenuous exercise can be regarded as a binary classification problem. Therefore, we will take falls as an example to introduce the iterative optimization method of strenuous exercise detection system based on adaptive generation of training samples. For long surveillance videos with various strenuous exercises, C3D model is used to perform detailed classification and annotation of long videos, and optical flow method is used to automatically locate timestamps to obtain pre-training sets of various behaviors, such as Figure 2 As shown in the figure, a self-built CNN network model is iteratively trained on the dataset to detect and locate violent movements in videos in real time. The model is compressed through knowledge distillation, making it suitable for deployment on embedded devices and allowing for the selective weighting of various violent movements.

[0064] like Figure 3 , the specific implementation steps are as follows.

[0065] Step 1: Obtaining the timestamp

[0066] (1) Pedestrian frame positioning - Pedestrians do not appear in all frames of video surveillance, but violent movements appear in video clips where pedestrians are present. Therefore, the frame difference method can be used to detect video clips where pedestrians appear, thereby reducing the monitoring range and computational complexity. The specific principle of the frame difference method is as follows: Figure 1 As shown, the original video is read and preprocessed using bit-plane filtering to remove noise introduced by the camera itself and the image background. Pixel-based temporal differentiation is used between adjacent frames of the video to extract the moving areas in the image through closed-value conversion. First, the adjacent frames are converted into grayscale images. Next, the corresponding pixel values ​​in the images are subtracted to obtain a differential image. The image is then binarized. If the pixel value changes in the image region are large and greater than a threshold, it is caused by a moving object in the image. This allows the automatic location of the video frame range where pedestrians appear and the approximate position of the moving target in each frame. The following research will be based on video clips where pedestrians appear.

[0067] D t (x,y)=|I t (x,y)-I t-1 (x,y)|

[0068] D t (x,y)≤T,M t (x,y)=0

[0069] D t(x, y) > T, M t (x, y) = 1

[0070] Wherein, the frame difference formula is as above, I t (x, y) is the gray value of the current frame (x, y) in the picture, I t-1 (x, y) is the gray value of the previous frame (x, y) in the picture, D t (x, y) is the absolute value of the difference of adjacent frames at (x, y), T is a threshold, M t (x, y) is the gray value of the difference image at (x, y).

[0071] (2) Fine classification label of pedestrian video - When the pre-training data set of the application is made, each violent motion data picture and label needs to be obtained. This step can obtain label information of each behavior. The application uses a C3D multi-behavior video classification model to label the behavior category of the video segment. First, the C3D network is trained, and UCF-101 is used as the training set. The labeling method of UCF-101 is to label a category for a video, that is, the complete multi-behavior classification model and weight can be obtained. Then, the pedestrian video obtained above is uniformly divided into short video segments with consistent time length (15 seconds in this embodiment), and the short video segments less than 15 seconds are separately divided. Then, the short video is sent into the C3D network for testing. Finally, the behavior category to which each short video belongs can be output.

[0072] Step 2: Automatic construction of pre-training data set.

[0073] (1) Time stamp positioning of violent motion - The specific behavior category contained in each short video segment has been obtained above, and the label corresponding to the video is the behavior label in the video, which represents that this video segment contains this violent motion, but cannot specifically locate the specific time stamp of the behavior occurrence, that is, it is impossible to obtain the picture data of each violent motion behavior through the above operation. At this time, the optical flow method can be used to detect the start and end time stamps of each short video that appears violent motion.

[0074] The time domain variation and correlation of pixel intensity data in the image sequence are used to determine the "motion" of each pixel position. That is, first, the obtained short video of each violent motion behavior is subjected to grayscale processing of color frames; the highest 4 bits of each pixel grayscale value are extracted for filtering, the Gaussian mixture model is initialized using the first frame image of the video, and the first frame image is saved for the second frame image to calculate the optical flow; starting from the second frame image, the bit plane filtering is also performed, and then the Gaussian mixture model is used to obtain the motion area, and the areas close to each other are fused into one area; the optical flow features of the motion area obtained in the previous step are calculated; the histogram p(x) of the optical flow in each area and the entropy H(x) of the optical flow histogram are calculated. Then, according to the optical flow vector features, the dynamic analysis of the image is performed, if the flow velocity of adjacent video frames changes greatly, exceeds the threshold, or the speed direction of the moving target is chaotic, the greater the entropy of the optical flow histogram, the greater the degree of disorder of the human body, that is, the first and last time stamps of each short video in which the violent motion occurs can be automatically located and output. According to the time stamp, the video segment in which the violent motion occurs is intercepted, and the pictures of each violent motion behavior of the pre-training data set are made.

[0075] Up to now, the data pictures and behavior labels of each violent motion have been automatically obtained. After the binary classification network training in the following is completed, the optical flow method detection does not need to be repeated, and the first and last time stamps of each violent motion in the monitoring video can be directly located and output according to the test results of the network.

[0076]

[0077]

[0078] wherein I represents the current frame image, I x represents the derivative of the current frame image x direction, I y represents the derivative of the current frame image y direction, I t represents the difference image of the current frame image and the previous frame image, represents the average value of the displacement of each pixel point in the x direction of the four-connected neighborhood at the n th iteration, represents the average value of the displacement of each pixel point in the y direction of the four-connected neighborhood at the n th iteration, n+1 represents the displacement of each pixel point in the x direction after the n+1 th iteration, v n+1 represents the displacement of each pixel point in the y direction after the n+1 th iteration, (x, y) represents the pixel point. n is the number of iterations, and a is the smoothing degree adjustment parameter. The method of iteration obtains the optical flow direction graph (u n+1 ,v n+1 ) of the adjacent two frame images. (u n+1 (x, y), v n+！(x,y)) represents the vector diagram of the pixel point (x, y) in the displacement direction after n+1 iterations, and the optical flow direction diagram is the motion image of the adjacent two frames of images.

[0079]

[0080] wherein u cur (x,y), v cur (x,y) respectively represent the optical flow direction of the pixel point (x, y) in the x direction and the y direction in the current frame motion image, u per (x,y), v per (x,y) respectively represent the optical flow direction of the pixel point (x, y) in the x direction and the y direction in the previous frame motion image.

[0081] (2) Preparation of pre-training data set - The two-class pictures of each violent motion behavior are prepared by the obtained first and last time stamps. Taking the "wrestling" behavior as an example, the short video segment containing the wrestling behavior can be obtained by step 1.2, and the specific time stamp of the wrestling behavior can be further obtained by step 2.1 optical flow method. The video segment in the time stamp is a continuous frame picture of the wrestling behavior, and the wrestling behavior picture is extracted frame by frame. Then, an equal amount of non-wrestling pictures are uniformly sampled in other video frames without wrestling behavior. At the same time, based on Mosai (mosaic) data enhancement, 25 wrestling pictures and 25 non-wrestling pictures are combined into 1 wrestling picture and 1 non-wrestling picture according to a certain proportion, the data channel number is expanded, and the wrestling picture is labeled 1 and the non-wrestling picture is labeled 0. An equal number of two-class pictures of wrestling behavior can be obtained, and the size of the picture data set is 128*128*25. The two-class pictures of wrestling behavior (wrestling, non-wrestling) are stored in the large folder of wrestling behavior. For the acquisition of two-class data sets of other violent motions, it is equivalent to the above-mentioned wrestling behavior, and is respectively stored in the large folder of each type of violent motion.

[0082] (3) Preprocessing of training set - For the picture data set, size adjustment, normalization and data enhancement are performed. For example, simple data expansion is performed by adopting light distortion and geometric distortion, and Cutout is adopted to randomly mask the picture at the first layer of the training network in the following text. Before being sent into the network, the data set needs to be preprocessed, mainly including the following two operations: ① data normalization: each dimension is subtracted by the mean value of the corresponding dimension, so that the data is decentralized; ② PCA dimension reduction: discard the dimensions with less information, and retain the main feature information.

[0083] Step 3: Construction of network model.

[0084] The network model is mainly divided into feature extraction and classification, which is based on CNN and LSTM.

[0085] (1) Feature extraction: CNN is one of the mainstream methods for image classification, which can capture the local properties of pictures, and the classification error is low, therefore, the application adopts the CNN convolutional neural network built by itself as the feature extraction model, the network model is simple, and the feature extraction of wrestling and sub-wrestling can be completed quickly. As shown in the figure Figure 4 The network model system schematic diagram is shown, before building the network, a convolutional layer MyLayers is built for the calling of the network model, and the kernel=(3, 3). Next, the CNN network model is built, first, the processed training set picture (shap: 128*128*25) is sent into the input layer. Then, the first convolutional layer is sent, filter1=128, kernel=(3, 3), that is, the data depth is increased from 25 to 128, and the data shap: 128*128*128 at this time. Next, the MyLayers convolutional layer built by oneself is continuously called. That is, five rounds of calling are accumulated, and the same convolution operation is repeated twice in each round. In the first convolution, the data is input into MyLayers for two times, filter1=256, kernel=(3, 3), shap: 128*128*256. In the second convolution, the similar convolution operation is repeated, and the data depth is increased from 256 to 512, and the data shape: 128*128*512 is the data depth at this time, that is, the number of network layers and the data depth are increased, and the deep features are further extracted. Then in the third, fourth and fifth convolution, the data is repeatedly input into MyLayers, and the data depth is reduced in turn, and the order is changed to 512, 128, 64, 32, and the data shape: 128*128*32 at this time, realizing the feature fusion of different scales and better extracting the low and high level features of the picture. After the above-mentioned 11 convolutional layer operations, the picture data is input into the average pooling layer AveragePooling2D(pool_size=2), the dimension is reduced, the network parameters are reduced, and overfitting is prevented. Finally, the data is sent into the flat layer Flatten, and the data is reshaped into a one-dimensional array (shape: 1*131072). That is, the picture features can be extracted.

[0086] (2) Behavior classification: The entire CNN input model described above is encapsulated in a TimeDistributed layer, which provides image interpretation and image features to the LSTM model. The LSTM learns the relationship between sequence elements, and the data is composed of image sequences, connecting information in frames taken at different times in a video and associating features between previous and subsequent frames. An LSTM layer with N storage units is used, i.e., the LSTM will see N time steps, each representing a one-dimensional vector of 131072 elements as input. Finally, based on the binary classification problem of the invention, the output of the LSTM is fed into five fully connected layers with units of 1024, 512, 128, 32, and 2, respectively, to solve nonlinear problems, integrate features together, and reduce the impact of feature position on classification. In the fifth fully connected layer, the sofmax activation function is used to output the predicted probability of the two classes. After five fully connected layers, the data is reduced to two, corresponding to the binary classification problem of the invention. Dropout is added to the fully connected layer, which temporarily discards neural network units according to a certain probability, reducing overfitting.

[0087] Step 4: Training of the network model.

[0088] The training set of each violent behavior is input into the binary classification network in step 3. The following takes the wrestling behavior binary classification training as an example.

[0089] The training model first needs to make a training set. A number of wrestling and non-wrestling folders have been obtained as described above, 35 wrestling and non-wrestling picture folders are extracted respectively, and 25 pictures are extracted from each folder at intervals to form wrestling and non-wrestling pictures in the training set, with the picture size in the training set being 128*128*25. The wrestling picture is labeled 1 and the non-wrestling picture is labeled 0. The network model in step 3 is called, the random seed is set, the training set data is sent into the network model in rounds, the model optimizer and loss function are constructed, the forward propagation loss value is calculated through the loss function and the network model parameters, and finally the pre-training weight is obtained. The network parameter information is printed. However, the pre-training weight at this time usually has poor accuracy and the network classification ability is weak, and iterative learning of the network is still needed, as shown in the specific process of Figure 5 .

[0090] At this time, the softmax wrestling probability threshold P=0.9 is adjusted, 10 groups of wrestling video segments are tested using the pre-training weights, if the number of wrestling pictures output by the test is greater than the threshold N, the wrestling pictures and an equal number of non-wrestling pictures in the test results are sent to the current wrestling training set, the wrestling probability threshold P=0.85 is adjusted, if the number of wrestling pictures output by the test is less than the threshold N, it means that the threshold is too high and the number of pictures meeting the classification condition is small, in order to prevent overfitting, the wrestling probability threshold P=0.85 is adjusted at this time, if the number of wrestling pictures in the test results is greater than the threshold N at this time, the wrestling pictures and an equal number of non-wrestling pictures in the test results can be sent to the wrestling training set, and the wrestling probability threshold P=0.8 is adjusted. The expanded wrestling training set is used for the second training, and the second wrestling weight is obtained, and the above network iteration steps are repeated until the wrestling probability threshold P<0.5, the network iteration training and test are stopped, and the weight at this time is retained as the final wrestling behavior test weight. The iteration training of the binary classification of other violent motion behaviors is equivalent to the training process of the wrestling behavior. Finally, the test weights of each violent motion behavior can be obtained.

[0091] Step 5: Test of the model

[0092] First, the test accuracy of the current behavior training weight needs to be verified, that is, the behavior training set is randomly sent into the binary classification network, the current test weight of the violent motion behavior is loaded, and the input training picture label value is predicted. The predicted label value is compared with the real label value, and the accuracy rate is output, which can directly reflect the maturity of the trained model. If the model is mature enough, the violent motion in the unknown test video can be detected and located. That is, according to the time sequence, a 25-channel picture is sampled from the video monitoring every 2 seconds and sent into the network, the saved behavior weight of the model is called, and each picture is predicted in turn, if the predicted label value is 1, the specific key frame information and behavior category of the existing violent motion are output, and the prediction result of every 2 seconds is written into the video, the violent motion can be directly and real-timely detected and located through the video, so that the supervisory personnel can timely respond to dangerous emergencies or abnormal behaviors and take measures.

[0093] Step 6: Compression and selective deployment of the model

[0094] Model compression is performed by the method of Teacher-Student knowledge distillation. Firstly, original model training, the above trained CNN model is taken as "Teacher model", namely Net-T, which is relatively complex, and can output Y for input X, where Y is mapped by softmax, and the output value corresponds to the probability value of the corresponding category. Secondly, the training model is simplified, and the parameter amount of the model is reduced. The knowledge of Net-T is distilled to Net-S at high temperature T. The objective function of the high-temperature distillation process is obtained by weighting distillloss (corresponding to soft target) and studentloss (corresponding to hard target), as shown in Figure 6 .

[0095] The traditional training process (hard targets): maximum likelihood is taken for ground truth. The training process of knowledge distillation (soft targets): the class probabilities of the large model are taken as soft targets. Net-T and Net-S are simultaneously input into the training set used by Net-T, and the softmax distribution generated by Net-T is taken as a soft target. The cross entropy between the softmax output of Net-S at the same temperature T and the soft target is the first part L soft of the loss function. The cross entropy between the softmax output of Net-S at temperature T = 1 and the ground truth is the second part L hard of the loss function. The specific composition of the loss function is as follows, and the values of the softmax outputs of Net-T and Net-S at temperature T on the jth category are c j , and the ground truth on the jth category is:

[0096] L = aL soft + bL hard

[0097]

[0098]

[0099] After the Net-S training is completed, the generalization ability of the original model is migrated, and then distilled to extract a simple model "Student model", which is further deployed to an embedded device. In the final application stage, the required weights can be selectively deployed and loaded according to the scene characteristics and requirements, and the selective recognition of different scenes can be realized. The calculation resources are saved, and the deployment and model use are facilitated.

[0100] Embodiment two

[0101] A violent motion detection system based on training sample adaptive generation, comprising,

[0102] A network model execution module configured to utilize a pre-trained network model to process the acquired video and detect violent motion, further comprising:

[0103] A data acquisition module configured to acquire existing video information, locate video segments with pedestrians appearing, and the approximate position of the moving target in the video image in each frame;

[0104] A division module configured to divide the video segments into multiple short video segments and label the behavior category of each short video segment;

[0105] A training set production module configured to detect the start and end time stamps of the appearance of violent motion in the short video segments suspected of appearing violent motion, cut the short video segments appearing violent motion according to the time stamps, produce each violent motion behavior picture of the pre-training data set, pre-process the behavior pictures, and form a training set;

[0106] A model training module configured to construct a network model comprising a feature extraction layer and a behavior classification layer, train the network model using the training set until the set requirements are met;

[0107] A model deployment module configured to compress the trained network model using a knowledge distillation method and deploy it.

[0108] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0109] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0110] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0111] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more flow or blocks. Figure 1 means for functionally implementing the steps listed in the flowchart block or blocks.

[0112] The above only describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0113] The above describes the specific embodiments of the present application in conjunction with the accompanying drawings, but is not intended to limit the protection scope of the present application. It should be understood by those skilled in the art that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A method for detecting violent motion based on adaptive generation of training samples, which uses a pre-trained network model to process the acquired video and detect violent motion, characterized by: The training and deployment process of the network model includes the following steps: Obtain existing video information, locate video clips where pedestrians appear, and the approximate position of moving targets in the video image in each frame; Segment the video clips into multiple short video segments and label the behavior category of each short video segment; Detect the first and last timestamps of the intense movement in the short video segments suspected of intense movement, intercept the short video segments with intense movement according to the timestamps, and create pictures of each intense movement behavior in the pre-training dataset; Preprocessing the behavior pictures to form a training set; Constructing a network model including a feature extraction layer and a behavior classification layer, and training the network model using a training set until the set requirements are met; Use knowledge distillation to compress the trained network model and then deploy it; The specific process of detecting the first and last timestamps of violent movement in a short video segment suspected of violent movement includes: using the optical flow method to detect the first and last time frames of violent movement in each short video, and automatically locating and outputting the first and last timestamps of violent movement in each short video based on the degree of disorder of the human body.

2. The method for detecting intense motion based on adaptive generation of training samples according to claim 1, wherein: The specific process of dividing a video clip into multiple short video segments includes evenly dividing the video clip containing the behavior into short video segments of a set length, dividing the part that is less than the set length into a separate short video segment, and processing it using a multi-behavior video classification model.

3. The method for detecting intense motion based on adaptive generation of training samples according to claim 1, wherein: The specific process of preparing the images of each intense exercise behavior in the pre-training dataset includes: extracting the images of the intense exercise behavior in the current video frame by frame, and uniformly sampling an equal number of non-acting images in other video frames that do not contain the behavior; Based on mosaic data enhancement, the number of data channels is expanded, and the behavior and non-behavior images are labeled 1 and 0 respectively. An equal number of binary classification images of the behavior are obtained and stored in the corresponding behavior folder; Create separate folders for each type of strenuous exercise behavior.

4. The method for detecting intense motion based on adaptive generation of training samples according to claim 1, wherein: The preprocessing of the training set specifically includes: resizing, normalizing and data augmentation of the images, and performing data normalization and dimensionality reduction operations on the dataset.

5. The method for detecting violent motion based on adaptive generation of training samples according to claim 1, wherein: The network model uses a neural network as a feature extraction layer, extracts relevant features through convolution stacking, and inputs the features into another neural network for classification to capture action behavior information and learn the relationship between sequence elements.

6. The method for detecting intense motion based on adaptive generation of training samples according to claim 1, wherein: The specific process of training the network model using the training set includes: assigning corresponding labels 1 and 0 to pictures of a certain behavior and non-behavior in the training set, and then the pictures and labels constitute the pre-training set of the behavior; Set a random seed, feed the training set data into the network model in rounds, build a model optimizer and loss function, calculate the forward propagation loss value through the loss function and network model parameters, and finally obtain the pre-training weights to determine the network parameter information; The network is iteratively learned by adjusting the probability threshold, that is, repeating the training-testing steps, and sending the test results to the training set for the next round of iterative training, and retaining the weight that finally meets the probability threshold standard as the final test weight for the behavior.

7. A system for detecting severe motion based on adaptive generation of training samples, characterized in that: include, The network model execution module is configured to process the acquired video using a pre-trained network model to detect violent motion, and further includes: A data acquisition module is configured to acquire existing video information, locate video clips where pedestrians appear, and the approximate position of the moving target in the video image in each frame; A segmentation module is configured to segment the video clip into multiple short video segments and label the behavior category of each short video segment; a training set preparation module configured to detect the first and last timestamps of the occurrence of intense movement in a short video segment suspected of having intense movement, intercept the short video segment in which the intense movement occurs according to the timestamps, prepare various intense movement behavior images of the pre-training data set, and pre-process the behavior images to form a training set; A model training module is configured to construct a network model including a feature extraction layer and a behavior classification layer, and train the network model using a training set until set requirements are met; The model deployment module is configured to compress the trained network model using the knowledge distillation method and then deploy it; The specific process of detecting the first and last timestamps of violent movement in a short video segment suspected of violent movement includes: using the optical flow method to detect the first and last time frames of violent movement in each short video, and automatically locating and outputting the first and last timestamps of violent movement in each short video based on the degree of disorder of the human body.

8. A computer-readable storage medium, characterized in that: A plurality of instructions are stored therein, and the instructions are suitable for being loaded by a processor of a terminal device and executing the steps of any one of the methods of claims 1-6.

9. A terminal device, characterized in that: The method comprises a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; and the computer-readable storage medium is used to store a plurality of instructions, wherein the instructions are suitable for being loaded by the processor and executing the steps in the method according to any one of claims 1 to 6.

Images