Behavior detection method and system based on space-time dimension characteristics

Abstract

The invention belongs to the field of behavior detection, and relates to a behavior detection method and system based on space-time dimension characteristics, and the method comprises the steps: obtaining an original video frame, and carrying out the preprocessing, and obtaining a global characteristic; adding position information to the global features by adopting a position coding module to obtain the global features with the position information; performing redundancy removal processing on the global features with the position information by adopting a feature selection module to obtain redundancy-removed visual features; a spatial enhancement module is adopted to process the features processed by the feature selection module to obtain spatial feature representation; inputting the features processed by the feature selection module into a time enhancement module to obtain time feature representation; fusing the redundancy-removed visual features, the spatial feature vector and the time feature vector representation to obtain fused features; inputting the fusion feature into a behavior detection module to obtain a detection result; according to the method, importance selection is carried out on the video features, so that the utilization of the features is improved, and the influence of redundant features is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of behavior detection technology, and specifically relates to a behavior detection method and system based on spatiotemporal dimensional features. Background Art

[0002] The behavior detection task is a computer vision task that aims to identify specific behaviors or actions in a video. The task combines behavior recognition and localization, which requires the algorithm to not only recognize the behavior, but also accurately indicate the time and spatial location of the behavior. This task has a wide range of applications in many fields, including security monitoring, intelligent transportation systems, motion analysis, healthcare, etc. Behavior detection is not only an important topic in computer vision, but also a research hotspot in the field of artificial intelligence.

[0003] Most of the existing behavior detection algorithms are divided into three types: detection methods based on auxiliary information, detection methods based on attention mechanisms, and detection methods based on spatiotemporal fusion. Using auxiliary information to enhance behavior changes helps improve the detection effect of behavior. These methods require large consumption of computing resources, and the training and inference processes require more computing resources. At the same time, useful auxiliary information is difficult to obtain; the method based on the attention mechanism enables the model to focus on the key areas or time periods in the video. Through the attention mechanism, the model can more accurately identify the key features of the action. However, this type of method requires a carefully designed attention module to ensure its effectiveness, which is also the main method currently used; the spatiotemporal fusion network processes time and space information at the same time, which can make full use of spatiotemporal information and improve detection accuracy. This type of method generally has a high model complexity, requires the design of a complex network structure, is difficult to train, and is difficult to meet real-time requirements; therefore, how to effectively identify the motion information of the target object in complex scenes and improve the model calculation speed is an urgent problem to be solved. Summary of the invention

[0004] In order to solve the problems existing in the above prior art, the present invention proposes a behavior detection method based on spatiotemporal dimension features, which comprises: obtaining video frame segments, inputting the video frame segments into a trained behavior detection model, and obtaining user behavior detection results;

[0005] The training of the behavior detection model includes: obtaining the original video frame, preprocessing the original video frame to obtain the global feature; using the position encoding module to add the position information to the global feature to obtain the global feature with the position information; using the feature selection module to perform redundancy processing on the global feature with the position information to obtain the de-redundant visual feature; using the spatial enhancement module to process the features processed by the feature selection module to obtain the spatial feature representation; inputting the features processed by the feature selection module into the time enhancement module to obtain the time feature representation; fusing the de-redundant visual feature, the spatial feature vector and the time feature vector representation to obtain the fused feature; calculating the loss function of the model according to the fused feature, adjusting the parameters of the model, and completing the training of the model when the loss function converges.

[0006] A behavior detection system based on spatiotemporal dimension features, the system comprising: a data acquisition module, a feature selection module, a space enhancement module, a time enhancement module, a feature fusion module and a behavior detection module;

[0007] The data acquisition module is used to acquire the video frame data to be detected and pre-process the video frame data;

[0008] The feature selection module is used to perform weight analysis on the preprocessed video frame features to obtain redundant visual features;

[0009] The spatial enhancement module is used to extract local details of the features processed by the feature selection module to obtain spatial features;

[0010] The time enhancement module is used to extract motion change information from the features processed by the feature selection module to obtain time features;

[0011] The feature fusion module is used to fuse spatial features and temporal features to obtain fused features;

[0012] The behavior detection module is used to perform behavior detection on the fusion features and output the results of positioning and identifying the user behavior.

[0013] Beneficial effects of the present invention:

[0014] The present invention can realize frame-level positioning and identification of behaviors in complex scenes, while ensuring high-precision reasoning, increasing the applicability of the product while making it have good economic benefits. The present invention improves the utilization of features and reduces the impact of redundant features by selecting the importance of video features; the fusion of spatial enhancement and temporal enhancement features makes the final video features contain not only local detail information but also frame-level timing information, which improves the quality of video features and can also improve detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of a behavior detection method based on spatiotemporal dimension features of the present invention;

[0016] Figure 2 This is a schematic diagram of the feature selection network structure of the present invention;

[0017] Figure 3 It is a schematic diagram of the spatial feature network structure of the present invention;

[0018] Figure 4 It is a schematic diagram of the time characteristic network structure of the present invention;

[0019] Figure 5 It is a schematic diagram of the spatial and temporal feature fusion network structure of the present invention;

[0020] Figure 6 It is a schematic diagram of the recognition effect of the behavior detection system of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] A behavior detection method based on spatiotemporal dimensional features, the method comprising: obtaining video frame segments, inputting the video frame segments into a trained behavior detection model, and obtaining user behavior detection results; training the behavior detection model comprises: obtaining original video frames, preprocessing the original video frames, and obtaining global features; using a position encoding module to add position information to the global features, and obtaining global features with position information; using a feature selection module to perform redundancy processing on the global features with position information, and obtaining redundancy-free visual features; using a spatial enhancement module to process the features processed by the feature selection module, and obtaining spatial feature representation; inputting the features processed by the feature selection module into a temporal enhancement module, and obtaining temporal feature representation; fusing the redundancy-free visual features, spatial feature vectors, and temporal feature vector representations, and obtaining fused features; calculating the loss function of the model according to the fused features, adjusting the parameters of the model, and completing the model training when the loss function converges.

[0023] In this embodiment, preprocessing the original video frame includes: dividing the original video frame into video frame segments of the same length, adjusting the size of the video frame, randomly cropping, denoising and normalizing.

[0024] The position encoding module is used to add position information to global features, including: dividing the video frame data into fixed-size patch blocks, introducing position encoding to retain the spatial position of the patch blocks, using learnable position encoding, and adding the embedded patch blocks to their corresponding position embeddings to form the final feature sequence.

[0025] The feature selection module is used to perform redundancy processing on the global features with position information, including: dividing the video frame segments into various feature groups; using a neural network to calculate the weight of each feature group; screening the feature groups according to the weights of the feature groups; generating corresponding masks for the screened feature groups; and performing dot multiplication of the feature groups and the corresponding masks to obtain global features.

[0026] The time enhancement module processes the input features by: dividing the features processed by the feature selection module into n feature segments; using feature processing to perform feature compression on the first n-1 feature segments; using a linear layer to process the nth feature segment; using a time feature fuser to fuse the nth feature segment that has undergone linear processing and the first n-1 feature segments that have undergone feature compression to obtain a time feature representation.

[0027] Further, such as Figure 5 As shown, the temporal feature fuser fuses feature segments by mapping the nth feature segment to a new dimension using linear transformation, reducing the dimension of the first n-1 feature segments using feature compression, fusing the linearly processed feature segments with the feature compressed feature segments using the Transformer encoder, and capturing the dependency between segments containing motion change information using the attention mechanism.

[0028] The method of fusing the de-redundant visual features, spatial feature vectors and temporal feature vector representations comprises: encoding the de-redundant visual features, spatial feature vectors and temporal feature vector representations using an encoder; transforming and adding the encoded features, and inputting the added features into a decoder to obtain fused features.

[0029] A specific implementation method of a behavior detection method based on spatiotemporal dimension features, such as Figure 1As shown, the method includes: obtaining the original video frame, arranging the video by frame to obtain video frame segments of the same length, preprocessing the video frame segments to obtain the preliminarily processed video segment features, and using them as the input of the model architecture. The input model first passes through the position encoding module to add the corresponding position information to the global features. The global features with position information are input into the feature selection module to remove redundant features and obtain redundant visual features. The features processed by the feature selection module are input into the spatial enhancement module to capture more accurate local details and obtain a more effective spatial feature representation. The features processed by the feature selection module are input into the temporal enhancement module to capture longer-term motion change information and obtain a more effective temporal feature representation. The redundant visual features, spatial features and temporal features are fused together to obtain the final feature representation. The supervised loss is calculated for the final feature representation.

[0030] In this embodiment, the video training set of behaviors is obtained as follows: the UCF101-24 dataset contains 3,207 videos with an average length of 15 seconds for training, and 800 videos with an average length of 15 seconds for testing, with a total of 24 behavior categories. The ratio of the training set to the test set is 4:1.

[0031] The present invention can use SGD optimizer for training adjustment, using the learning strategy of first increasing and then decreasing. After multiple rounds of training, the neural network tends to be stable and the iterative training ends. The flowchart of the whole method is as follows: Figure 1 As shown in the figure: get the original video frame, arrange the video frame by frame to get the video frame segments of the same length, preprocess the video frame segments to get the preliminarily processed video segment features, and use them as the input of the model architecture; the input model first passes through the position encoding module to add the corresponding position information to the global features; the global features with position information are input into the feature selection module to remove redundant features and obtain the redundant visual features. The network structure diagram of the feature selection module is shown in the figure Figure 2 As shown; the features processed by the feature selection module are input into the spatial enhancement module to capture more accurate local details and obtain a more effective spatial feature representation; the features processed by the feature selection module are input into the temporal enhancement module to capture longer-term motion change information and obtain a more effective temporal feature representation; the redundant visual features, spatial features and temporal features are fused together to obtain the final feature representation; the supervised loss calculation is performed on the final feature representation, the neural network is trained and the loss is solved, and the loss is fed back to the neural network for iterative optimization; after the training is completed, the video frame fragment is input into the trained network model to detect the position box of the object at the frame level and obtain the corresponding category score.

[0032] In this embodiment, if Figure 3As shown in the figure, the spatial enhancement module is used to process the de-redundant visual features, including: dividing the de-redundant visual feature map into patch blocks of size 16×16×4, and linearly projecting the patch blocks; inputting the linearly projected patch blocks into the linear layer to obtain Q, K, and V of the patch blocks; using Multi-Head Attention to calculate the similarity of patch blocks at different spatial positions according to Q, K, and V to obtain importance features; performing Concat operations on all patch blocks according to the importance features to obtain local detail information.

[0033] like Figure 4 As shown, the time enhancement module processes the input features by: dividing the redundant visual features into n feature segments; performing feature compression processing on the first n-1 feature segments by feature processing; processing the nth feature segment by a linear layer; and fusing the nth feature segment after linear processing and the first n-1 feature segments after feature compression by a time feature fuser to obtain a time feature representation.

[0034] The behavior detection method based on spatiotemporal dimension features of the present invention can be applied to medical care. In elderly care or rehabilitation treatment, behavior detection can monitor the daily activities of patients and promptly detect dangerous situations such as falls. Install cameras in public places and use behavior detection technology to monitor abnormal behaviors in real time, such as intrusions and fights, and issue timely warnings. The behavior detection model based on spatiotemporal dimension features can be deployed in scenarios that need to be monitored to improve the retrieval efficiency of monitoring personnel. The recognition effect diagram of the behavior detection system is shown in the figure below. Figure 6 shown.

[0035] Behavior detection can also be applied to the following scenarios: security monitoring, smart transportation, and sports analysis.

[0036] In the context of security monitoring, behavior detection includes: Through cameras installed at borders or important areas, the behavior detection system can identify unauthorized entrants and trigger alarms in a timely manner. In places with large traffic (such as schools, shopping malls, stations, etc.), the system can automatically identify fighting behaviors and immediately notify security personnel to intervene. When monitoring large gatherings or festivals, behavior detection technology can help identify abnormal situations of crowds and prevent mass incidents such as stampedes.

[0037] In the context of intelligent transportation, behavior detection includes: In traffic monitoring, behavior detection can identify vehicle violations (such as running red lights, speeding) or pedestrians crossing the road. Through video surveillance, the behavior detection system can automatically identify traffic violations such as running red lights, speeding, and driving in the wrong direction, and assist traffic management departments in law enforcement. Install cameras at crosswalks to detect when pedestrians cross the road, and combine with traffic light control to ensure pedestrian safety. Use behavior detection technology to analyze traffic flow, dynamically adjust traffic light times, and alleviate traffic congestion. Identify the behavior of vehicles entering parking lots, automatically allocate parking spaces, and record vehicle entry and exit times to facilitate billing management. Intelligent transportation is also an application scenario for behavior detection.

[0038] In the field of sports, behavior detection technology can provide accurate movement analysis to help coaches and referees make more scientific decisions: by recording athletes' training and competitions through video, the behavior detection system can analyze parameters such as standardization, strength and speed of movements to guide training. In football matches, behavior detection technology can assist referees in judging offside situations and improve the accuracy of penalties. It can identify possible fouls in the game, such as blocking in basketball and foul tackles in football, to help referees make fair decisions. After the game, behavior detection technology can be used to replay key moments and analyze tactical arrangements and technical movements in the game.

[0039] A behavior detection system based on spatiotemporal dimension features, the system comprising: a data acquisition module, a feature selection module, a space enhancement module, a time enhancement module, a feature fusion module and a behavior detection module;

[0040] The data acquisition module is used to acquire the video frame data to be detected and pre-process the video frame data;

[0041] The feature selection module is used to perform weight analysis on the preprocessed video frame features to obtain redundant visual features;

[0042] The spatial enhancement module is used to extract local detail features from the features processed by the feature selection module to obtain spatial features;

[0043] The time enhancement module is used to extract motion change information features from the features processed by the feature selection module to obtain time features;

[0044] The feature fusion module is used to fuse spatial features and temporal features to obtain fused features;

[0045] The behavior detection module is used to perform behavior detection on the fusion features and output the results of positioning and identifying the user behavior.

[0046] The specific implementation of the system of the present invention is the same as the specific implementation of the method.

[0047] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A behavior detection method based on spatiotemporal dimension features, characterized in that: include: Obtain video frame segments, and input the video frame segments into the trained behavior detection model to obtain user behavior detection results; The training of the behavior detection model includes: obtaining the original video frame, preprocessing the original video frame to obtain the global feature; using the position encoding module to add the position information to the global feature to obtain the global feature with the position information; using the feature selection module to perform redundancy processing on the global feature with the position information to obtain the de-redundant visual feature; using the spatial enhancement module to process the features processed by the feature selection module to obtain the spatial feature representation; inputting the features processed by the feature selection module into the time enhancement module to obtain the time feature representation; fusing the de-redundant visual feature, the spatial feature vector and the time feature vector representation to obtain the fused feature; calculating the loss function of the model according to the fused feature, adjusting the parameters of the model, and completing the training of the model when the loss function converges.

2. According to the behavior detection method based on spatiotemporal dimension features according to claim 1, it is characterized in that: Preprocessing the original video frames includes: dividing the original video frames into video frame segments of the same length, adjusting the size of the video frames, randomly cropping, denoising and normalizing.

3. The behavior detection method based on spatiotemporal dimension features according to claim 1 is characterized in that: The position encoding module is used to add position information to global features, including: dividing the video frame data into fixed-size patch blocks, introducing position encoding to retain the spatial position of the patch blocks, using learnable position encoding, and adding the embedded patch blocks to their corresponding position embeddings to form the final feature sequence.

4. The behavior detection method based on spatiotemporal dimension features according to claim 1 is characterized in that: The feature selection module is used to perform redundancy processing on the global features with position information, including: using a neural network to calculate the weights of each feature group; screening the feature groups according to the weights of the feature groups, generating corresponding masks for the screened feature groups, and performing dot multiplication of the feature groups and the corresponding masks to obtain the de-redundant features.

5. The behavior detection method based on spatiotemporal dimension features according to claim 1 is characterized in that: The spatial enhancement module is used to process the features processed by the feature selection module, including: dividing the features processed by the feature selection module into patch blocks of size 16×16×4, and linearly projecting the patch blocks; inputting the patch blocks after linear projection into the linear layer to obtain Q, K, and V of the patch blocks; using Multi-Head Attention to calculate the similarity of patch blocks at different spatial positions based on Q, K, and V to obtain importance features; performing Concat operations on all patch blocks based on the importance features to obtain local fine-grained information.

6. The behavior detection method based on spatiotemporal dimension features according to claim 1 is characterized in that: The temporal enhancement module processes the input features by: dividing the de-redundant visual features into n feature segments; performing feature compression on the first n-1 feature segments using feature processing; processing the nth feature segment using a linear layer; and fusing the nth feature segment that has undergone linear processing and the first n-1 feature segments that have undergone feature compression using a temporal feature fuser to obtain a temporal feature representation.

7. The behavior detection method based on spatiotemporal dimension features according to claim 5 is characterized in that: The temporal feature fuser fuses feature segments by mapping the nth feature segment to a new dimension using linear transformation, reducing the dimensions of the first n-1 feature segments using feature compression, fusing the linearly processed feature segments with the feature compressed feature segments using the Transformer encoder, and capturing the dependencies between segments containing motion change information using the attention mechanism.

8. The behavior detection method based on spatiotemporal dimension features according to claim 1 is characterized in that: Obtaining the fused features includes: using an encoder to encode the de-redundant visual features, spatial feature vectors, and temporal feature vector representations; transforming and adding the encoded features, and inputting the added features into a decoder to obtain the fused features.

9. The behavior detection method based on spatiotemporal dimension features according to claim 1 is characterized in that: The loss function of the model consists of three parts: spatial bounding box regression loss for each frame, temporal behavior boundary localization loss, and behavior classification loss for all frames.

10. A behavior detection system based on spatiotemporal dimension features, the system being used to execute the behavior detection method based on spatiotemporal dimension features as described in any of claims 1 to 8, characterized in that: include: Data acquisition module, feature selection module, space enhancement module, time enhancement module, feature fusion module and behavior detection module; The data acquisition module is used to acquire the video frame data to be detected and pre-process the video frame data; The feature selection module is used to perform weight analysis on the preprocessed video frame features to obtain redundant visual features; The spatial enhancement module is used to extract local details of the features processed by the feature selection module to obtain spatial features; The time enhancement module is used to extract motion change information from the features processed by the feature selection module to obtain time features; The feature fusion module is used to fuse spatial features and temporal features to obtain fused features; The behavior detection module is used to perform behavior detection on the fusion features and output the results of positioning and identifying the user behavior.

Images