Video classification tumble behavior identification method based on event camera

Abstract

The invention relates to a video classification tumble behavior identification method based on an event camera, and the method comprises the following steps: 1, capturing the edge and motion information of a human body tumble behavior through the event camera, and obtaining a dense optical flow image of the human body tumble behavior; step 2, training an S3DFAST double-flow model, and identifying a tumble behavior in the video sequence; step 3, training the tumble behavior detection model and the tumble behavior classification model to obtain a tumble detection classification comprehensive result; and step 4, performing comprehensive judgment on results of the falling behaviors in the step 2 and the step 3, and fusing a video recognition result obtained by utilizing the space information and the time information in the step 2, a detection result which focuses on the use of the space information in the step 3 and a classification result to obtain a final falling behavior recognition result. According to the method, the human right problem of privacy protection in special scenes can be solved, and the technical problem that the falling behavior is inaccurate due to the fact that a single feature is adopted in an existing method can also be solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video surveillance, and relates to a video classification fall behavior recognition method, in particular to a video classification fall behavior recognition method based on an event camera. Background Art

[0002] At present, image-based fall detection technology has been widely used in many different fields, such as life, transportation, security and other fields.

[0003] The shortcomings of the existing video classification fall behavior recognition method are as follows:

[0004] (1) Existing video classification fall behavior recognition methods generally use standard cameras, which generally use raw images or raw videos to obtain relevant information. The images may contain some privacy information. Privacy protection is extremely important in the field of video surveillance, and standard cameras cannot be used in some scenarios.

[0005] (2) Existing video classification fall behavior recognition methods only use the spatial dimension information of the image, but not the temporal dimension information. The feature dimension is single and the information is incomplete.

[0006] (3) The existing video classification fall behavior recognition method does not fully learn the features of spatial dimension information, and it is necessary to manually find key features that can represent the fall behavior. For example, the fall behavior needs to be combined with the height and speed of the human body to complete the recognition of the fall behavior, which increases the manual cost of selecting features.

[0007] Therefore, in order to solve the above technical problems, the present invention proposes a video classification fall behavior recognition method based on event camera.

[0008] After searching, no public documents of the prior art identical or similar to the present invention were found. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a video classification fall behavior recognition method based on event camera, which can not only solve the human rights issue of privacy protection in special scenes, but also solve the technical problem that the existing method adopts a single feature to cause inaccurate fall behavior.

[0010] The present invention solves the practical problem by adopting the following technical solutions:

[0011] A method for identifying falling behaviors by video classification based on event cameras comprises the following steps:

[0012] Step 1: First, the edge and motion information of the human falling behavior is captured by the event camera, and then a dense optical flow image of the human falling behavior is obtained;

[0013] Step 2: Based on step 1, a dense optical flow image of a human falling behavior is obtained, and the S3DFAST two-stream model is trained to identify the falling behavior in the video sequence;

[0014] Step 3: Based on the dense optical flow image of the human fall behavior obtained in step 1, a fall behavior detection model and a fall behavior classification model are trained to obtain a comprehensive result of fall detection and classification;

[0015] Step 4: Make a comprehensive judgment on the results of the falling behavior in steps 2 and 3, and integrate the video recognition result (VideoResult) obtained in step 2 using spatial information and time information, the detection result (DetResult) obtained in step 3 focusing on spatial information, and the classification result (ClsResult) to obtain the final falling behavior recognition result.

[0016] Moreover, the objective function of the dense optical flow image of the human body falling behavior in step 1 is as follows:

[0017]

[0018] Where: T(x,y) is the reference image (the first image), I(x,y) is the current image (the second image); the offset u(x,y), v(x,y) is the offset of each point on the image, It is an error function. According to different requirements, it can take absolute value function, quadratic function and other non-negative symmetric functions with monotonically increasing positive semi-axis. is the modulus of the gradient, is a similar The error function; min represents the minimum value of the region; E(u,v) is the objective function; u and v are variables;

[0019] Moreover, the specific steps of step 2 include:

[0020] (1) Construct deep learning training sets and test sets from dense optical flow samples of fallen people and normal moving people;

[0021] (2) Based on the training set and test set constructed in step (1), the S3DFAST framework is used to train the S3DFAST dual-stream fall behavior recognition model, and then the SoftMax function is used to obtain the fall behavior classification score, thereby completing the recognition of human fall behavior;

[0022] Moreover, the objective function of the S3DFAST dual-stream fall behavior recognition model in step (2) of step 2 is as follows:

[0023]

[0024] Among them, b ij is the bias of the feature map; m represents the number of feature maps in the i-1th layer connected to the jth feature map in the i-th layer; P represents the value of the convolution kernel associated with the coordinate point (p, q, r) on the mth feature map; i ,Q i ,R i They represent the size of the convolution kernel in the spatial and temporal dimensions respectively.

[0025] Moreover, the specific calculation formula for obtaining the fall behavior classification score using the SoftMax function in step 2 (2) is as follows:

[0026]

[0027] Where x is the output vector of the neural network, usually represented as an N-dimensional matrix, where each row corresponds to a category and each column corresponds to each feature of a sample; i Represents the i-th element in vector x, representing the original output value of a certain category; y i represents the i-th element in the vector after being processed by the Softmax function, which represents the probability distribution of the i-th category; Represents the exponential function, which is used to convert the vector x i Convert each element in to a positive number; The summation function is used to calculate the sum of the exponential function values ​​of all elements in the vector xx to ensure that the sum of the output values ​​of the Softmax function is 1.

[0028] Moreover, the specific steps of step 3 include:

[0029] (1) Label the dense optical flow samples obtained in step 1, which include both dense optical flow samples of normal human activities and dense optical flow samples of human bodies in falling activities. The labeled samples are divided into positive samples and negative samples, and then the training set and test set of the fall behavior detection model are constructed; the yolov8 detection model is used to detect the fall behavior;

[0030] (2) The labeled dense optical flow samples of falling people and the dense optical flow samples of falling people falsely detected by the fall behavior detection model are used to construct the training set and test set of the fall classification model, and resnet18 is used to classify the fall behavior.

[0031] Moreover, the specific method of step 4 is:

[0032] When one of the scores of the video recognition result, detection result and classification result is very high (>=0.9), the method is considered to be an "expert" in the current situation, and the second fusion strategy is adopted, that is, the output result of the "expert" is taken as the recognition result of the fall behavior; when the scores of all three are low (<0.9), the first fusion strategy is adopted, the results of the three are averaged, and the average score is used as the recognition result of the fall detection behavior.

[0033] Advantages and beneficial effects of the present invention:

[0034] 1. The present invention uses event camera images to detect fall behavior, solves the privacy protection problem in special scenarios, and provides information security for users.

[0035] 2. The present invention uses 3D convolution to construct a deep learning network framework, making full use of the information of the space + time dimensions of the event camera, improving the expression ability of features and improving the accuracy of the algorithm.

[0036] 3. The present invention uses detection + classification to focus on learning spatial information, solves the problem of insufficient spatial information learning due to lack of video sequences in video-based deep learning, and improves the recall rate of the algorithm;

[0037] 4. The present invention proposes a video classification fall behavior recognition method based on event camera, which can complete fall behavior detection while protecting people's privacy information; extract corresponding edge information and operation information through the event camera; collect feature information in time dimension and space dimension according to the above information, and fuse them to obtain video recognition results; perform image detection and classification through dense optical flow images, and extract image-related features in a targeted manner; and combine video and image processing to greatly improve the recognition rate of fall behavior and reduce the error rate of fall behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of image output of an event camera and a standard camera of the present invention;

[0039] Figure 2 It is a schematic diagram of event camera imaging and feature extraction of the present invention;

[0040] Figure 3 It is a flow chart of the method for identifying falling behavior based on video classification of event camera of the present invention;

[0041] Figure 4 This is a network structure diagram of the fall detection model of the present invention. DETAILED DESCRIPTION

[0042] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings:

[0043] A video classification fall behavior recognition method based on event camera, such as Figure 3 and Figure 4 As shown, the following steps are included:

[0044] Step 1: First, the edge and motion information of the human falling behavior is captured by the event camera, and then a dense optical flow image of the human falling behavior is obtained;

[0045] The objective function of the dense optical flow image of the human body falling behavior in step 1 is as follows:

[0046]

[0047] Where: T(x,y) is the reference image (the first image), I(x,y) is the current image (the second image); the offset u(x,y), v(x,y) is the offset of each point on the image, It is an error function. According to different requirements, it can take absolute value function, quadratic function and other non-negative symmetric functions with monotonically increasing positive semi-axis. is the modulus of the gradient, is a similar The error function; min represents the minimum value of the region; E(u,v) is the objective function; u and v are variables;

[0048] Step 2: Based on step 1, a dense optical flow image of a human falling behavior is obtained, and the S3DFAST two-stream model is trained to identify the falling behavior in the video sequence;

[0049] The specific steps of step 2 include:

[0050] (1) Construct deep learning training sets and test sets from dense optical flow samples of fallen people and normal moving people;

[0051] (2) Based on the training set and test set constructed in step (1), the S3DFAST framework is used to train the S3DFAST dual-stream fall behavior recognition model, and then the SoftMax function is used to obtain the fall behavior classification score, thereby completing the recognition of human fall behavior;

[0052] The objective function of the S3DFAST dual-stream fall behavior recognition model in step (2) of step 2 is as follows:

[0053]

[0054] Among them, b ij is the bias of the feature map; m represents the number of feature maps in the i-1th layer connected to the jth feature map in the i-th layer; P represents the value of the convolution kernel associated with the coordinate point (p, q, r) on the mth feature map;i ,Q i ,R i They represent the size of the convolution kernel in the spatial and temporal dimensions respectively.

[0055] The specific calculation formula for obtaining the fall behavior classification score using the SoftMax function in step (2) of step 2 is as follows:

[0056]

[0057] Where x is the output vector of the neural network, usually represented as an N-dimensional matrix, where each row corresponds to a category and each column corresponds to each feature of a sample; i Represents the i-th element in vector x, representing the original output value of a certain category; y i represents the i-th element in the vector after being processed by the Softmax function, which represents the probability distribution of the i-th category; Represents the exponential function, which is used to convert the vector x i Convert each element in to a positive number; The summation function is used to calculate the sum of the exponential function values ​​of all elements in the vector xx to ensure that the sum of the output values ​​of the Softmax function is 1.

[0058] After step 2, the falling behavior in the identified video sequence can be obtained;

[0059] Step 3: Based on the dense optical flow image of the human fall behavior obtained in step 1, a fall behavior detection model and a fall behavior classification model are trained to obtain a comprehensive result of fall detection and classification;

[0060] The specific steps of step 3 include:

[0061] (1) Label the dense optical flow samples obtained in step 1 (the dense optical flow samples include both the dense optical flow samples of normal human bodies and the dense optical flow samples of human bodies in falling activities); divide the labeled samples into positive samples and negative samples, and then construct the training set and test set of the fall behavior detection model; use the yolov8 detection model to detect the fall behavior;

[0062] like Figure 4As shown in the figure, the input resolution of the fall behavior detection model is 576*320, and the basic feature extraction part (Backbone) is composed of a deep convolution layer (DarknetConv2D_BN_Mish), 5 groups of residual layers (ResBlock), a convolution layer (Conv), three maximum pooling layers of different sizes (5, 9, 13), and a full connection (Concat) + convolution (Conv); the middle part is composed of a full connection (Concat) + convolution (Conv), convolution (Conv) + upsampling (Upsampling), convolution (Conv) + downsampling (Downsampling), which constitutes the feature combination part (Neck); finally, the final feature output module (Head) is composed of 3 heads (Head);

[0063] (2) The labeled dense optical flow samples of falling people and the dense optical flow samples of falling people falsely detected by the fall behavior detection model are used to construct the training set and test set of the fall classification model, and resnet18 is used to classify the fall behavior.

[0064] In this embodiment, the recognition method in step 3 has the following advantages over the method of learning the falling behavior based on the recognition video sequence in step 2: the detection and classification method based on the picture can make full use of the spatial information and has a fast processing speed; the disadvantage is that the time information dimension in the video sequence is lost, so the effect of the algorithm based on the behavior class is relatively worse;

[0065] Step 4: Make a comprehensive judgment on the results of the falling behavior in steps 2 and 3, and integrate the video recognition result (VideoResult) obtained in step 2 using spatial information and time information, the detection result (DetResult) obtained in step 3 focusing on spatial information, and the classification result (ClsResult) to obtain the final falling behavior recognition result.

[0066] The specific method of step 4 is:

[0067] When one of the video recognition results, detection results, and classification results has a high score (>=0.9), the method is considered to be an "expert" in the current situation, and the second fusion strategy is adopted, that is, the output result of the "expert" is taken as the recognition result of the fall behavior; when the scores of the three are all low (<0.9), the first fusion strategy is adopted, the results of the three are averaged, and the average score is used as the recognition result of the fall detection behavior;

[0068] The working principle of step 4 is:

[0069] The fusion strategies currently used include probability integration and MoE mixed expert model.

[0070] in:

[0071] 1. Probabilistic Ensemble

[0072] Probabilistic ensemble is a technique that draws on traditional machine learning fusion methods. Specifically, it averages the logit results predicted by multiple models to obtain more stable and accurate predictions.

[0073] 2. MoE Mixed Expert Model

[0074] MoE (Mixture of Experts) is a model architecture method that combines multiple sub-models (i.e. "experts"), aiming to improve the overall prediction effect through the collaborative work of multiple experts.

[0075] For the above two fusion strategies, the specific usage scheme of the present invention is: 1) When one of the three scores is very high (>=0.9), it is considered that the method is an "expert" in the current situation, and the second fusion strategy is adopted, that is, the output result of the "expert" is taken as the recognition result of the fall behavior; when the scores of the three are all low (<0.9), the first fusion strategy is adopted, the results of the three are averaged, and the average score is used as the recognition result of the fall detection behavior;

[0076] The fall behavior recognition technology based on video classification of event cameras in the present invention has been applied in practice, but the technology has not been made public. The fall behavior of people under monitoring is recognized, and finally a real-time fall algorithm alarm is realized.

[0077] The working principle of the event camera of the present invention is:

[0078] An event-based camera is a new type of biologically inspired visual sensor, sometimes also called a dynamic vision sensor (DVS) or DAVIS (Dynamic and Active-Pixel Vision Sensor). Compared to standard cameras that capture images at a fixed frame rate, it has the characteristics of low latency, high dynamic range, low power consumption, and high temporal resolution. At the same time, the image output of an event camera is information about the edge and motion of the target, which has a better privacy protection effect.

[0079] Moreover, event cameras are biomimetic sensors with microsecond response times that record asynchronous streams of brightness changes per pixel, called "events". Event cameras generate an event by detecting brightness changes in each pixel. Compared with standard cameras, they are more suitable for use in high-dynamic and high-speed environments. They have the advantages of high dynamic range (HDR), high temporal resolution, and no motion blur. Event camera imaging, such as Figure 1 shown.

[0080] At the same time, the event camera can effectively capture the edge and movement information of the target. For this purpose, the dense optical flow algorithm is used in the effective feature extraction of the image. The movement is expressed by vectors to highlight the movement process of the target falling behavior. The imaging and feature extraction diagram is shown in the figure. Figure 2 shown.

[0081] It should be emphasized that the embodiments of the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation modes. Any other implementation modes derived by those skilled in the art based on the technical solutions of the present invention also fall within the scope of protection of the present invention.

Claims

1. A video classification fall behavior recognition method based on event camera, characterized by: The following steps are involved: Step 1: First, the edge and motion information of the human falling behavior is captured by the event camera, and then a dense optical flow image of the human falling behavior is obtained; Step 2: Based on step 1, a dense optical flow image of a human falling behavior is obtained, and the S3DFAST two-stream model is trained to identify the falling behavior in the video sequence; Step 3: Based on the dense optical flow image of the human fall behavior obtained in step 1, a fall behavior detection model and a fall behavior classification model are trained to obtain a comprehensive result of fall detection and classification; Step 4: Make a comprehensive judgment on the results of the falling behavior in step 2 and step 3, and integrate the video recognition result VideoResult obtained in step 2 using spatial information and time information, the detection result DetResult obtained in step 3 focusing on spatial information, and the classification result ClsResult to obtain the final falling behavior recognition result.

2. The method for identifying fall behaviors by video classification based on event camera according to claim 1, characterized in that: The objective function of the dense optical flow image of the human body falling behavior in step 1 is as follows: Where: T(x,y) is the reference image, I*x,y) is the current image; the offset u(x,y), v(x,y) is the offset of each point on the image, It is an error function. According to different requirements, it can take absolute value function, quadratic function and other non-negative symmetric functions with monotonically increasing positive semi-axis. is the modulus of the gradient, is a similar The error function; min represents the minimum value of the area; E(u,v) is the objective function; u and v are variables.

3. The method for identifying fall behaviors by video classification based on event camera according to claim 1, characterized in that: The specific steps of step 2 include: (1) Construct deep learning training sets and test sets from dense optical flow samples of fallen people and normal moving people; (2) Based on the training set and test set constructed in step (1), the S3DFAST framework is used to train the S3DFAST dual-stream fall behavior recognition model, and then the SoftMax function is used to obtain the fall behavior classification score, thereby completing the recognition of human fall behavior.

4. The method for identifying fall behaviors by video classification based on event camera according to claim 3 is characterized in that: The objective function of the S3DFAST dual-stream fall behavior recognition model in step (2) of step 2 is as follows: Among them, b ij is the bias of the feature map; m represents the number of feature maps in the i-1th layer connected to the jth feature map in the i-th layer; P represents the value of the convolution kernel associated with the coordinate point (p, q, r) on the mth feature map; i ,Q i ,R i They represent the size of the convolution kernel in the spatial and temporal dimensions respectively.

5. The method for identifying fall behaviors by video classification based on event camera according to claim 3 is characterized in that: The specific calculation formula for obtaining the fall behavior classification score using the SoftMax function in step (2) of step 2 is as follows: Where x is the output vector of the neural network, usually represented as an N-dimensional matrix, where each row corresponds to a category and each column corresponds to each feature of a sample; i Represents the i-th element in vector x, representing the original output value of a certain category; y i represents the i-th element in the vector after being processed by the Softmax function, which represents the probability distribution of the i-th category; Represents the exponential function, which is used to convert the vector x i Convert each element in to a positive number; The summation function is used to calculate the sum of the exponential function values ​​of all elements in the vector xx to ensure that the sum of the output values ​​of the Softmax function is 1.

6. The method for identifying fall behaviors by video classification based on event camera according to claim 1, characterized in that: The specific steps of step 3 include: (1) Label the dense optical flow samples obtained in step 1, which include both dense optical flow samples of normal human activities and dense optical flow samples of human bodies in falling activities. The labeled samples are divided into positive samples and negative samples, and then the training set and test set of the fall behavior detection model are constructed; the yolov8 detection model is used to detect the fall behavior; (2) The labeled dense optical flow samples of falling people and the dense optical flow samples of falling people falsely detected by the fall behavior detection model are used to construct the training set and test set of the fall classification model, and resnet18 is used to classify the fall behavior.

7. The method for identifying fall behaviors by video classification based on event camera according to claim 1, characterized in that: The specific method of step 4 is: When one of the scores among the video recognition result, detection result and classification result is very high>=0.9, the method is considered to be an "expert" in the current situation, and the second fusion strategy is adopted, that is, the output result of the "expert" is taken as the recognition result of the fall behavior; when the scores of the three are all low <0.9, the first fusion strategy is adopted, the results of the three are averaged, and the average score is used as the recognition result of the fall detection behavior.

Images