Behavior detection method and system based on spatiotemporal features

Abstract

The application discloses a behavior detection method based on space-time features, which comprises the following steps: preparing a behavior data set: collecting indoor environment non-suicidal self-injury behavior video data, and marking four types of behavior actions in the collected video data; loading a model and initializing model parameters: loading a computer vision and machine learning software library OpenCV, a target detection feature model M d , a behavior detection feature model M a , a face recognition feature library M f , and a non-suicidal self-injury behavior category set L; non-suicidal self-injury behavior detection based on indoor activity space-time features, comprising non-suicidal self-injury behavior NSSI feature extraction and non-suicidal self-injury behavior NSSI classification. The behavior detection system based on space-time features comprises a video collection, a pretreatment module, a behavior detection module and a visualization module. The application has the characteristics that non-suicidal self-injury behavior can be automatically detected through video data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and particularly relates to a behavior detection method and system based on spatiotemporal features. BACKGROUND

[0002] Non-suicide self-injury (NSSI) refers to directly and repeatedly intentionally damaging one's own body without suicidal intent, and typical behaviors include cutting, severe scratching, burning, biting, beating, and climbing. Timely identification and effective intervention of self-injurious behavior can reduce the suicide rate of patients with potential suicide risk, and visual detection has good timeliness and can capture behavioral details to take appropriate intervention methods. At present, Law et al. developed a prison abnormal behavior detection system that can detect abnormal behaviors of prisoners such as gathering, falling, self-injury, and fighting through monitoring cameras to improve the safety of prisoners and handle emergencies. However, the cost of collecting labeled data of abnormal events is high. Therefore, the widely used method is to train the model on normal training data and test the model on abnormal video data. Pawar et al. solved the detection and positioning problem of abnormal events in surveillance videos by using a pipeline deep autoencoder and one-class learning. However, this method can only be used to detect abnormal situations and cannot obtain the type of abnormal situations. In addition, the current data sources for non-suicide self-injury identification mainly include pictures and texts, which cannot detect and identify non-suicide self-injury based on video data, are not convenient for timely discovery and effective prevention of NSSI, and cannot reduce the suicide rate of patients with potential suicide risk. SUMMARY

[0003] The present application aims to overcome the above-mentioned shortcomings and proposes a video data-driven behavior detection method and system based on spatiotemporal features that can timely discover and effectively prevent non-suicide self-injury.

[0004] The present application provides a behavior detection method based on spatiotemporal features, wherein the method comprises the following steps:

[0005] Step 1, making a behavior dataset: from the perspectives of feasibility and typicality, non-suicide self-injury NSSI is divided into four types of actions: biting, cutting, hitting, and climbing with potential danger, i.e., the behavior category set L = {biting, climbing, hitting, cutting}; in an indoor environment, a camera and a mobile phone are used to collect non-suicide self-injury video data, the collected non-suicide self-injury video data is divided into a training dataset, a validation dataset, and a test dataset, and the four types of behavior actions are labeled in the video data;

[0006] Step 2, Loading the model and initializing model parameters: Load the computer vision and machine learning software library OpenCV, and the object detection feature model M. d Behavior detection feature model M a Face recognition feature library M f , a set of non-suicidal self-harm behavior categories L; initial model parameters include sampling rate, test clipping resolution, detection threshold ε, and number of action categories;

[0007] Step 3: Detection of non-suicidal self-harm behaviors based on the spatiotemporal characteristics of indoor activities, including NSSI feature extraction and NSSI classification of non-suicidal self-harm behaviors. Detailed steps are as follows:

[0008] Step 1: Read the RGB camera video stream V real ;

[0009] Step 2: If the video stream V real The number of frames T = 64, that is, V real =f1f1…f k …f 64 If yes, proceed to Step 3; otherwise, proceed to Step 1.

[0010] Step 3: Read the video time t and initialize the human detection bounding box coordinate set.

[0011] Step 4: Due to frame f k Belongs to video stream V real ,Right now According to f k and M d Using object detection methods, predict f sequentially. k The bounding box for human detection is used to obtain the coordinate set G. t ;

[0012] Step 5: Employ behavior detection methods, based on G... t V real and M a Get V real The temporal feature vector F se and spatial eigenvectors F sp ;

[0013] Step 6: Transfer the time series feature vector F se and spatial eigenvectors F sp The data is concatenated and fused, then input into a classifier to calculate the probability distribution p of NSSI behavior categories, and finally obtain the NSSI behavior category c. t; the calculation of the NSSI behavior category probability distribution p: according to the classifier combined with the spatiotemporal features to predict the NSSI category probability, in the training process of the data, the BCEloss function is taken to calculate the distribution difference between the predicted value and the true value, assuming that the probability value output by a single category is p i , wherein i represents the category; in the training process, the binary value of the true label is represented by , and the loss function can require the loss of a single category, which is equivalent to regarding each category as a binary classification problem, and the formula of the loss function is as follows:

[0014]

[0015] Step 7: if the probability distribution p is greater than the detection threshold ε, and the behavior category c t of the video stream belongs to the non-suicidal self-injury behavior category set L, i.e. if p > ε && c t ∈ L, go to Step 8, otherwise go to Step 1;

[0016] Step 8: taking M f and V real as inputs, using a face recognition method to obtain identity information N t ;

[0017] Step 9: reading the position information O t of the camera;

[0018] Step 10: constructing a sequence S describing the event, including video time t, position O t , identity information N t , and NSSI behavior category c t , i.e. sequence S(t, O t , N t , c t );

[0019] Step 11: fusing S and V real to form a new video stream V * ;

[0020] Step 12: outputting the video stream V * and the sequence S for visualization and early warning purposes;

[0021] Step 13: if an end instruction is received, end, otherwise go to Step 1.

[0022] The above behavior detection method based on spatiotemporal features, wherein in Step 1, the collected non-suicidal self-injury behavior video dataset is divided into a training dataset, a validation dataset and a test dataset according to a ratio of 4:1:1;

[0023] The behavior detection method based on spatio-temporal features, wherein: in step 2, the test cutting resolution of the initialized model parameter is 224, the detection threshold ε is 0.8, the number of action categories is 4, and the sampling rate is set to 4, 8, 16 or 32 according to the behavior detection feature model.

[0024] The behavior detection method based on spatio-temporal features, wherein: in step 2, the target detection feature model M d is obtained by training the initialized feature model Faster-RCNN on a non-suicidal self-injury behavior video dataset.

[0025] The behavior detection method based on spatio-temporal features, wherein: in step 2, the face recognition feature library M f is obtained by extracting 68 facial feature points using the machine learning library Dlib to obtain a face recognition feature library.

[0026] The behavior detection method based on spatio-temporal features, wherein: in step 4, the confidence threshold of the human body detection frame bounding box is greater than 0.9.

[0027] A behavior detection system based on spatio-temporal features, comprising a video acquisition and preprocessing module, a behavior detection module, and a visualization module, wherein:

[0028] The video acquisition and preprocessing module is used to read local videos or capture video streams of local network cameras, and to decode the obtained videos;

[0029] The behavior detection module is used to detect the obtained videos and determine whether there is NSSI behavior, uses a face recognition API to obtain identity information of the detected person, and establishes an NSSI behavior event sequence S, including video time t, location O t , identity information N t , and NSSI behavior category c t ;

[0030] The visualization module stores the behavior event sequence S in a Json format and is used to display the detected behavior events.

[0031] The behavior detection system based on spatio-temporal features, wherein: the visualization module also responds to data queries and management.

[0032] Compared with the prior art, the application has obvious beneficial effects, as can be seen from the above scheme, the Faster-RCNN model is initialized by using the feature model of the Voc2007 data set, and then the Faster-RCNN is used for training on the NSSI BehaviorDataset, so as to obtain a target detection model. The long-term feature library LFB of detailed video understanding, the slow network architecture SlowOnly used for video recognition, the slow-fast network architecture SlowFast used for video recognition, the actor-centered relationship network architecture ACRN, and the unified neural network architecture YOWO based on real-time spatiotemporal action positioning are five methods, which are fused with the ResNet50 and ResNet101 backbone networks to form a behavior detection model, and the behavior detection model M a is obtained. The optimal method is determined by experiments. The Dlib library is used to extract 68 feature points of a face to obtain a face recognition feature library. The target detection method used is Faster-RCNN. It should be noted that only the human body detection frame with a confidence greater than or equal to 0.9 is determined as an effective area. The obtained human body detection frame coordinate set is used for subsequent behavior detection and also provides data for a visualization module of the behavior detection system. Since the behavior detection method considered is the behavior detection model formed by fusing the four methods mentioned above with the ResNet50 and ResNet101 backbone networks, the behavior detection system provides a method with optimal performance.

[0033] The detection threshold value epsilon = 0.8 is set according to the actual effect in subsequent experiments, and therefore the detection effect is good.

[0034] The system detects the face information in the video stream, extracts the feature points, and performs Euclidean metric matching with the data in the face recognition feature library M f , and returns the corresponding person information, and the recognition rate is high.

[0035] The position information o t of the camera is the room number where the camera corresponding to the video stream in which the NSSI behavior is detected is located. Usually, the home environment supporting the implementation of the NSSI behavior detection is installed with multiple cameras distributed in different rooms, and therefore can accurately detect the indoor activity spatiotemporal characteristics.

[0036] The sequence S is used for subsequent generation of the video stream V * , and also provides data for a warning function of the behavior detection system.

[0037] In summary, the application can automatically detect the non-suicidal self-injury behavior through video data, so as to timely discover and effectively prevent the NSSI, and reduce the suicide rate of patients with potential suicide risk.

[0038] The advantageous effects of the present application are further illustrated by the specific embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 A flow chart of the method of the present application;

[0040] Figure 2 An interface of the NSSI behavior detection system in the specific embodiments. DETAILED DESCRIPTION

[0041] The specific embodiments, features and effects of a behavior detection method and system based on spatiotemporal features according to the present application are described in detail below in combination with the preferred embodiments and the drawings.

[0042] Referring to Figure 1 , a behavior detection method based on spatiotemporal features according to the present application, wherein the method comprises the following steps:

[0043] Step 1, making a behavior dataset: from the perspectives of implementability and typicality, non-suicidal self-injury behavior NSSI is divided into four types of actions: biting, cutting, hitting, and climbing with potential danger, i.e., the behavior category set L = {biting, climbing, hitting, cutting}; in an indoor environment, a camera and a mobile phone are used to collect non-suicidal self-injury behavior video data, the collected non-suicidal self-injury behavior video dataset is divided into a training dataset, a validation dataset and a test dataset according to a ratio of 4:1:1, and the four types of behavior actions are labeled in the video data;

[0044] Step 2, loading a model and initializing model parameters: loading a computer vision and machine learning software library OpenCV, a target detection feature model M d , a behavior detection feature model M a , a face recognition feature library M f , and a non-suicidal self-injury behavior category set L; initializing model parameters include a sampling rate, a test cutting resolution of 224, a detection threshold e = 0.8, and a number of action categories of 4, wherein the sampling rate can be set to 4, 8, 16 or 32 according to the behavior detection model; the target detection feature model M d is obtained by training the initialized feature model Faster-RCNN on the non-suicidal self-injury behavior video dataset; the face recognition feature library M f is obtained by extracting 68 feature points of a face using a machine learning library Dlib;

[0045] Step 3, non-suicidal self-injury behavior detection based on spatiotemporal features of indoor activities, the detailed steps are as follows:

[0046] Step 1: read the RGB camera video stream V real ;

[0047] Step 2: if the number of frames T = 64 of the video stream V real , i.e. V real = f1f1…f k …f 64 , go to Step 3, otherwise go to Step 1;

[0048] Step 3: read the video time t, and initialize the human body detection frame coordinate set

[0049] Step 4: since frame f k belongs to video stream V real , i.e. According to f k and M d , use the target detection method to predict the human body detection frame bounding box of f k in turn, and obtain the coordinate set G t ; the confidence threshold of the human body detection frame bounding box is greater than 0.9;

[0050] Step 5: using the behavior detection method, according to G t , V real and M a , obtain the time sequence feature vector F real and the spatial feature vector F sp of V se ;

[0051] Step 6: splice and fuse the time sequence feature vector F se and the spatial feature vector F sp , and input them into the classifier to calculate the NSSI behavior category probability distribution p, and obtain the NSSI behavior category c t ; calculate the NSSI behavior category probability distribution p: according to the classifier combined with the space-time features to predict the NSSI category probability, in the training process of data, take the BCEloss function to calculate the distribution difference between the predicted value and the true value, assuming that the probability value output by a single category is p i , where i represents the category; in the training process, the binary value of the true label is represented by , the loss function can require the loss of a single category, which is equivalent to treating each category as a binary classification problem, and the formula of the loss function is as follows:

[0052]

[0053] Step 7: If the probability distribution p is greater than the detection threshold ε, and the behavior class c of the video stream belongs to the set of non-suicidal self-injury behavior classes L, i.e. if p > ε && c t ∈ L, go to Step 8, otherwise go to Step 1; the detection threshold ε = 0.8; t

[0054] Step 8: Take M f and V real as input, and use a face recognition method to obtain the identity information N t ;

[0055] Step 9: Read the location information O t of the camera;

[0056] Step 10: Construct a sequence S describing the event, including the video time t, the location O t , the identity information N t , and the NSSI behavior class c t , i.e. the sequence S(t, O t , N t , c t );

[0057] Step 11: Fuse S with V real to form a new video stream V * ;

[0058] Step 12: Output the video stream V * , the sequence S(t, O t , N t , c t ) for visualization and warning purposes;

[0059] Step 13: If an end instruction is received, end, otherwise go to Step 1.

[0060] A behavior detection system based on spatiotemporal features, comprising a video acquisition, preprocessing module, behavior detection module, and visualization module, wherein:

[0061] The video acquisition and preprocessing module: used to read local videos or capture video streams from local network cameras, and to decode the obtained videos;

[0062] The behavior detection module: used to detect the obtained videos and determine whether there is NSSI behavior, uses a face recognition API to obtain the identity information of the detected person, and establishes a NSSI behavior event sequence S, including the video time t, the location O t , the identity information N t , and the NSSI behavior class c t ​;

[0063] Visualization module: Stores the sequence of behavioral events S in JSON format and is used to display the detected behavioral events.

[0064] The visualization module also responds to data queries and management.

[0065] Specific Implementation Case Analysis

[0066] The NSSI behavior detection prototype system mainly includes a video acquisition and preprocessing module, a behavior detection module, and a visualization module. The video acquisition and preprocessing module is used to read local video or capture video streams from local network cameras and decode the acquired video. The behavior detection module is used to detect the acquired video and determine whether NSSI behavior exists. A face recognition API is used to obtain the identity information of the detected person and to establish an NSSI behavior event sequence S (video time t, location O). t Identity information N t NSSI Behavioral Category c t The visualization module stores the behavioral event sequence S in JSON format and is used to display the detected behavioral events. The prototype detection system also responds to data queries and management. The proposed NssiDetection is implemented using Python with the PyTorch framework and OpenCV 4.6.0. The NSSI behavioral detection prototype system is implemented as a web application using the Layui, WebSocket, and Flask frameworks on Ubuntu 20.04 LTS. A MySQL database is used to store the data. Figure 2 The interface of the NSSI behavior detection system is displayed.

[0067] To test the actual performance of the NSSI behavior detection system, three test schemes were designed:

[0068] Option a. The volunteers and their home environment are the same as the training dataset. The camera is used to shoot from above, 3.2m above the ground, just like in real life.

[0069] Option b. The volunteers and their home environment are the same as the training dataset. The camera is shot from bottom to top and is 0.6m above the ground.

[0070] Option c involves volunteers and home environments that differ from the training dataset. Testing is conducted by five volunteers in a completely new home environment, using a different dataset. Options a and b primarily detect whether the monitoring perspective affects the detection results, while option c primarily tests the robustness of the model to entirely new scenarios.

[0071] To facilitate statistical detection results, the corresponding video is recorded in advance according to the above scheme and input into the system. 40 videos of each scheme are recorded, 10 groups of videos for each action, and the length of each video is 5-10s, a total of 120 test videos. It should be noted that: in order to judge the application effect of the algorithm model, when the behavior detection result output by the model is consistent with the actual action, it is determined that the recognition is successful.

[0072] Table 1 is the recognition accuracy statistical result of schemes a-c.

[0073] Table 1. Accuracy of NSSI behavior recognition system under different test schemes

[0074]

[0075] As can be seen from Table 1, the lowest recognition accuracy of scheme a is more than 80%, and the average accuracy is 85%. The lowest recognition accuracy of scheme b is 90%, and the average accuracy reaches 95%. The lowest recognition accuracy of scheme c is only 60%, and the highest is only 80%, and the average recognition accuracy is 70%.

[0076] These data show that: 1) the camera shooting angle has an impact on the recognition accuracy, and the shooting from bottom to top is better than the shooting from top to bottom. This is because the mode of shooting from top to bottom is easy to cause the shielding phenomenon of the upper body to the lower body, and then affect the recognition accuracy. 2) When the person and the home environment are different from the training data set, the recognition accuracy of the system will decrease. In summary, the self-injury detection system developed can recognize the designed situation, and the recognition accuracy is 84.18%. The system has good robustness, and can predict new self-injury behaviors for videos of the training data set under different backgrounds, different angles and different angles, indicating that this method can be applied to the detection of self-injury behaviors.

[0077] In order to discover and effectively intervene NSSI in time, thereby reducing the suicide rate of patients with potential suicide risk, the present application proposes a non-suicidal self-injury behavior detection method NssiDetection based on indoor activity space-time characteristics on the basis of establishing NSSI behavior data set. Then, NssiDetection is integrated and developed into NSSI detection system. According to the case study experiment, the system shows good robustness and 84.18% recognition accuracy. This study provides an intelligent method for automatically identifying NSSI. The algorithm proposed can be combined with some mathematical tools and recognition methods to develop a robot system.

[0078] The above is only the preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification of the above embodiment without departing from the technical solution content of the present application and according to the technical essence of the present application still belongs to the scope of the technical solution of the present application.

Claims

1. A behavior detection method based on spatiotemporal features, characterized in that: The method includes the following steps: Step 1: Create a behavioral dataset: From the perspective of feasibility and typicality, non-suicidal self-harm behaviors (NSSI) are divided into four categories: biting, cutting, hitting, and potentially dangerous climbing, i.e., the set of behavioral categories L = {biting, climbing, hitting, cutting}. In an indoor environment, non-suicidal self-harm behaviors are captured using cameras and mobile phones. The captured non-suicidal self-harm behavior video data is divided into training dataset, validation dataset, and test dataset, and the four types of behaviors are labeled in the video data. Step 2, Loading the model and initializing model parameters: Load the computer vision and machine learning software library OpenCV, and the object detection feature model M. d Behavior detection feature model M a Face recognition feature library M f , a set of non-suicidal self-harm behavior categories L; initial model parameters include sampling rate, test clipping resolution, detection threshold ε, and number of action categories; Step 3: Detection of non-suicidal self-harm behaviors based on the spatiotemporal characteristics of indoor activities, including NSSI feature extraction and NSSI classification of non-suicidal self-harm behaviors. Detailed steps are as follows: Step 1: Read the RGB camera video stream V real ; Step 2: If video stream V real The number of frames T = 64, that is, V real =f1f1…f k …f 64 Proceed to Step 3 if the above steps are not taken; otherwise, proceed to Step 1. Step 3: Read the video time t and initialize the human detection bounding box coordinate set G. t It is an empty set; Step 4: Due to frame f k Belongs to video stream V real ,Right now According to f k and M d Using object detection methods, predict f sequentially. k The bounding box for human detection is used to obtain the coordinate set G. t ; Step 5: Employ behavior detection methods, based on G... t V real and M a Get V real The temporal feature vector F se and spatial eigenvectors F sp ; Step 6: Transfer the time series feature vector F se and spatial eigenvectors F sp The data is concatenated and fused, then input into a classifier to calculate the probability distribution p of NSSI behavior categories, and finally obtain the NSSI behavior category c. t The calculation of the NSSI behavior category probability distribution p involves predicting the NSSI category probability based on the classifier combined with spatiotemporal features. During data training, the BCE loss function is used to calculate the distribution difference between the predicted and true values. It is assumed that the probability value of a single category output is p. i , where i represents the category; During training, the binary values ​​of the true labels are used... This means that the loss function can be calculated for a single class, which is equivalent to treating each class as a binary classification problem. The formula for the loss function is as follows: Step 7: If the probability distribution p is greater than the detection threshold ε, and the behavior category c of the video stream is... t Belonging to the set L of non-suicidal self-harm behaviors, i.e., if p > ε && c t If ∈L, proceed to Step 8; otherwise, proceed to Step 1. Step 8: Place M f and V real As input, facial recognition methods are used to obtain identity information N. t ; Step 9: Read the camera's location information. t ; Step 10: Construct a sequence S describing the event, including video time t and location O. t Identity information N t NSSI Behavioral Category c t That is, the sequence S(t, O) t N t c t ); Step 11: Connect S and V real The fusion forms a new video stream V * ; Step 12: Output video stream V * Sequence S is used for visualization and early warning purposes; Step 13: If an end command is received, the process ends; otherwise, proceed to Step 1.

2. The behavior detection method based on spatiotemporal features as described in claim 1, characterized in that: In step 1, the collected non-suicidal self-harm video dataset is divided into a training dataset, a validation dataset, and a test dataset in a 4:1:1 ratio.

3. The behavior detection method based on spatiotemporal features as described in claim 1, characterized in that: In step 2, the test clipping resolution of the initialized model parameters is 224*224, the detection threshold ε = 0.8, the number of action categories is 4, and the sampling rate is set to 4, 8, 16 or 32 according to the behavior detection feature model.

4. The behavior detection method based on spatiotemporal features as described in claim 1, characterized in that: In step 2, the target detection feature model M d It is obtained by training the initialized feature model Faster-RCNN on a non-suicidal self-harm video dataset.

5. The behavior detection method based on spatiotemporal features as described in claim 1, characterized in that: In step 2, the face recognition feature database M f It uses the machine learning library Dlib to extract 68 feature points of a face to obtain a face recognition feature library.

6. The behavior detection method based on spatiotemporal features as described in any one of claims 1 to 5, characterized in that: In Step 4, the confidence threshold of the human body detection bounding box is greater than or equal to 0.

9.

7. A behavior detection system based on spatiotemporal features, comprising a video acquisition module, a preprocessing module, a behavior detection module, and a visualization module, characterized in that: Video capture and preprocessing module: used to read local video or capture video streams from local network cameras, and decode the acquired video; The behavior detection module is used to detect acquired videos and determine whether NSSI behavior exists. It uses a face recognition API to obtain the identity information of the person being detected and establishes an NSSI behavior event sequence S, including video time t and location O. t Identity information N t NSSI Behavioral Category c t ; Visualization module: Stores the sequence of behavioral events S in JSON format and uses it to display the detected behavioral events; The system performs non-suicidal self-harm behavior (NSSI) identification and detection according to the method described in claim 1.

8. The behavior detection system based on spatiotemporal features as described in claim 7, characterized in that: The visualization module also responds to data queries and management.

Images