Real-time detection method for crossing behavior based on spatio-temporal information fusion and related components
By combining AlphaPose and 2s-AGCN models to perform real-time trespassing behavior detection in surveillance videos, the problem of low detection efficiency in traditional methods is solved, achieving efficient trespassing behavior recognition and security assistance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENZHEN ALL THINGS CLOUD TECH CO LTD
- Filing Date
- 2023-01-28
- Publication Date
- 2026-07-03
AI Technical Summary
Existing machine learning-based methods for detecting vaulting behavior cannot achieve real-time and effective detection, resulting in low efficiency for manual monitoring.
The method combines AlphaPose and 2s-AGCN. By extracting frames from the surveillance video, the AlphaPose model is used to extract human skeletal information and store it in a fixed queue. After the queue reaches a predetermined length, the information is input into the 2s-AGCN model for behavior recognition, and the behavior category and its confidence level are output.
It enables real-time and effective detection of climbing behavior, improves detection accuracy and speed, reduces manpower consumption, and enhances security efficiency.
Smart Images
Figure CN115984753B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and in particular to a real-time detection method and related components for climbing behavior based on spatiotemporal information fusion. Background Technology
[0002] With the rapid development of urban industrialization and the increasing density of urban populations, various video surveillance infrastructures have been built. However, in the past, community security and prevention work mainly relied on manual identification of abnormal behavior through real-time video monitoring, which consumed a lot of human resources and was inefficient. As the monitoring area and scale continue to expand, the traditional video surveillance management method that relies on manual decision-making is clearly unsustainable. Instead, intelligent video surveillance technologies with artificial intelligence, computer vision, and deep learning as their core have emerged. However, traditional climbing behavior detection based on machine learning methods requires cumbersome feature engineering and cannot detect climbing behavior in real time and effectively. Summary of the Invention
[0003] This invention provides a real-time detection method and related components for vaulting behavior based on spatiotemporal information fusion, aiming to solve the problem that existing vaulting behavior detection methods based on machine learning cannot detect vaulting behavior in real time and effectively.
[0004] In a first aspect, the present invention provides a real-time detection method for vaulting behavior based on spatiotemporal information fusion, wherein:
[0005] The system acquires surveillance video and continuously extracts frames. Each extracted frame is then input into the AlphaPose model in sequence to extract the human skeleton information of a single frame. The human skeleton information includes the pixel coordinates and confidence scores of key points of the human skeleton.
[0006] The human skeleton information of each frame of the image is stored sequentially into a fixed queue;
[0007] Determine whether the length of the fixed queue is greater than the predetermined length. If so, retrieve the human skeleton information of the earliest stored frame image in the fixed queue according to the time sequence.
[0008] The human skeleton information of a frame of image is constructed into an array representing the connection of the human physical structure according to the predefined skeletal point numerical index, and then input into the 2s-AGCN model to output the behavior category and its confidence.
[0009] Secondly, the present invention also provides a real-time detection device for vaulting behavior based on spatiotemporal information fusion, comprising:
[0010] The extraction unit is used to acquire surveillance video and continuously extract frames. Each extracted frame image is input into the AlphaPose model in sequence to extract the human skeleton information of a single frame image. The human skeleton information includes the pixel coordinates and confidence scores of key points of the human skeleton.
[0011] The storage unit is used to sequentially store the human skeleton information of each frame of image into a fixed queue.
[0012] The judgment unit is used to determine whether the length of the fixed queue is greater than the predetermined length. If so, the human skeleton information of the earliest frame image stored in the fixed queue is retrieved in chronological order.
[0013] The output unit is used to construct an array representing the physical structure connections of the human skeleton from a frame of image according to a predefined skeletal point index, and input it into the 2s-AGCN model to output the behavior category and its confidence level.
[0014] Thirdly, the present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the real-time detection method for vaulting behavior based on spatiotemporal information fusion as described above.
[0015] Fourthly, the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to perform the real-time detection method for vaulting behavior based on spatiotemporal information fusion as described above.
[0016] This invention provides a real-time detection method and related components for vaulting behavior based on spatiotemporal information fusion. The main approach uses a combination of AlphaPose and 2s-AGCN. Specifically, a single-frame image obtained by extracting frames from a video is first input into the AlphaPose model for skeletal point detection. The resulting human skeletal information is stored in a fixed queue. Once the queue reaches a predetermined length, the skeletal information is retrieved and input into the 2s-AGCN model, which outputs the behavior category and its confidence level. This invention can detect whether a human is performing a vaulting behavior in real time and effectively, improving both the accuracy and speed of detection. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 A flowchart illustrating the real-time detection method for vaulting behavior based on spatiotemporal information fusion provided in an embodiment of the present invention;
[0019] Figure 2 This is a schematic diagram of a sub-process of the real-time detection method for vaulting behavior based on spatiotemporal information fusion provided in an embodiment of the present invention;
[0020] Figure 3 This is a network structure diagram of the MobileOne module provided in an embodiment of the present invention;
[0021] Figure 4 The network structure diagram of the three output branches of YOLOv7-Tiny-MobileOne provided in the embodiments of the present invention;
[0022] Figure 5 This is a principle block diagram of the real-time detection method for vaulting behavior based on spatiotemporal information fusion provided in an embodiment of the present invention;
[0023] Figure 6 A schematic block diagram of a real-time detection device for vaulting behavior based on spatiotemporal information fusion provided in an embodiment of the present invention;
[0024] Figure 7 This is a schematic block diagram of the extraction unit provided in an embodiment of the present invention. Detailed Implementation
[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.
[0027] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.
[0028] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.
[0029] Please see Figure 1 , Figure 1 A flowchart illustrating a real-time detection method for vaulting behavior based on spatiotemporal information fusion, provided in an embodiment of the present invention, includes steps S101-S104:
[0030] S101. Acquire surveillance video and continuously extract frames. Input each extracted frame image into the AlphaPose model in sequence to extract human skeleton information of a single frame image. The human skeleton information includes the pixel coordinates and confidence of key points of the human skeleton.
[0031] S102. Store the human skeleton information of each frame image into a fixed queue in sequence.
[0032] S103. Determine whether the length of the fixed queue is greater than the predetermined length. If so, retrieve the human skeleton information of the earliest frame image stored in the fixed queue according to the time sequence.
[0033] S104. Construct an array representing the physical structure connections of the human skeleton from the extracted frame image according to the predefined skeletal point digital index, and input it into the 2s-AGCN model to output the behavior category and its confidence level.
[0034] This invention primarily employs a combined AlphaPose and 2s-AGCN technique for detection. Specifically, single-frame images obtained by extracting frames from a video are first input into the AlphaPose model for skeletal point detection. The resulting human skeletal information is stored in a fixed queue. Once the queue capacity reaches a predetermined length, the human skeletal information is then input into the 2s-AGCN model, which outputs the behavior category and its confidence level. This invention can detect whether a person is performing a climbing behavior in real time and effectively to ensure security within the park. This method improves both the accuracy and speed of detection.
[0035] In step S101, the footage recorded by the surveillance video can be captured, and then the human skeleton information of a single frame image can be extracted using the AlphaPose model.
[0036] First, at the park entrance or guard post, image data from surveillance videos is collected using video frame extraction. This image data includes footage of people illegally climbing over turnstiles, barriers, or gates to enter the park. Then, a time-related image sequence is generated, with the climbing action taking approximately 3 seconds from start to finish; that is, a complete climbing action is usually completed within 3 seconds.
[0037] Secondly, each extracted frame image is sequentially input into the AlphaPose model to extract the human skeleton information of a single frame image.
[0038] Specifically, such as Figure 2 As shown, step S101 includes steps S201-S204:
[0039] S201. Detect each frame of the image using an object detection algorithm to obtain pedestrian detection boxes;
[0040] S202. Input the pedestrian detection box into the STN module and SPPE module to automatically generate a pose box;
[0041] S203. Input the attitude frame into the P-NMS module for refinement;
[0042] S204. Generate pedestrian bounding boxes with the same distribution as the pose boxes using the pose generator for data augmentation.
[0043] The original AlphaPose used YOLOv3 as the detector (i.e., the detection algorithm) to ensure that the entire person area is extracted first. However, in order to detect pedestrian bounding boxes better and faster, this embodiment uses YOLOv7-Tiny as the detector.
[0044] That is, in step S201, YOLOv7-Tiny is used as a detector, which can detect pedestrian detection boxes better and faster. YOLOv7-Tiny has a faster and more efficient network architecture, and on this basis, the MobileOne module is introduced to balance the accuracy and speed of the detector, namely YOLOv7-Tiny-MobileOne.
[0045] In this embodiment, as Figure 3As shown, the left side constitutes a complete component of the MobileOne module. It consists of two parts: the upper part is based on depthwise convolution, and the lower part is based on pointwise convolution. The MobileOne module includes a depthwise convolution module and a pointwise convolution module arranged sequentially. The kernel size of the depthwise convolution module is 3×3, and the kernel size of the pointwise convolution module is 1×1. The depthwise convolution module has 3 branches, and the pointwise convolution module has 2 branches. The MobileOne module employs overparameterized branches, which provide further benefits during model training. During inference, the MobileOne module has no branches, and these branches are removed during the reparameterization process. By introducing simple overparameterized branches, the MobileOne module leaves only a simple feedforward structure during inference, effectively reducing memory access costs and expanding parameters in the model to achieve better expressive power.
[0046] In this embodiment, as Figure 4 As shown, three Mob il eOne modules can be set up, which are connected before the convolutional modules (Conv) of the last three output branches in YOLOv7-T iny, and replace the CBL modules before the original three convolutional modules respectively. Figure 4 In this context, Concat refers to the concatenation module, CBL refers to the convolution module (including Conv+BN+Leakyrelu, where BN refers to the batch normalization module and Leakyrelu refers to an activation function), and C5 refers to 5 convolutions.
[0047] In this embodiment, step S201 uses an activation function to optimize the model training process.
[0048] Specifically, the H-Swish activation function is used in YOLOv7-Tiny-MobileOne. This function has good numerical stability and can be implemented in almost all software and hardware frameworks. The Swish activation function has been proven to be a better activation function than ReLU6, but its computation is more complex than ReLU6. To enable the application of Swish on mobile devices and reduce its computational overhead, the H-Swish activation function was proposed. The H-Swish activation function is implemented as a piecewise function, which reduces the number of memory accesses, thereby significantly reducing latency and thus increasing computation speed.
[0049] The H-Swi sh activation function formula is as follows:
[0050]
[0051] It can also be expressed as:
[0052]
[0053] In the formula: x is the input feature vector.
[0054] In step S202, the pedestrian detection boxes are input into the STN module and the SPPE module to automatically generate pose boxes. Specifically, the pedestrian detection boxes are first input into the STN module, and then into two parallel lightweight single-person pose estimation network (SPPE) modules. One of the lightweight single-person pose estimation network (SPPE) modules is used to generate pose boxes, and the other lightweight single-person pose estimation network (SPPE) module acts as a regularization function to further enhance the STN's extraction of high-quality pose boxes.
[0055] In step S203, the generated pose boxes are refined using the P-NMS module to eliminate redundant pose boxes.
[0056] In step S204, the PGPG (pose generator) performs data augmentation by generating pedestrian bounding boxes with the same distribution as the labeled pose boxes, thereby further improving the frame performance.
[0057] In step S102, a sliding window is set up and implemented through a fixed queue. Human skeletal information is saved to the fixed queue, thus enabling the sliding window concept to be dynamically implemented within the fixed queue. This embodiment only saves human skeletal information (i.e., key points), which reduces model computation and protects personal privacy.
[0058] In step S103, as Figure 5 As shown, it is determined whether the queue capacity meets the preset length. If it does, the human skeleton information of the earliest frame image stored in the fixed queue is retrieved in chronological order. If it does not meet the requirement, real-time detection cannot be performed, normalization cannot be performed, and the data is re-entered into the AlphaPose model to extract the human skeleton information again.
[0059] The preset length can be 64 frames. The sliding window moves across the array, removing an element from the left and adding an element from the right, requiring only the calculation of the element values within the current window. The fixed queue is a first-in, first-out (FIFO) data structure, with insertion operations at the tail and removal operations at the head.
[0060] In step S104, the human skeleton information stored in the fixed queue of the preset length is used to construct an array representing the physical structure connections of the human body from the human skeleton information of a retrieved frame image according to the predefined bone point index. This array is then input into the 2s-AGCN model, and finally, the behavior category and its confidence score are output. The sliding window described above achieves the purpose of connecting two deep learning models (AlphaPose and 2s-AGCN).
[0061] The 2s-AGCN behavior recognition mentioned in this embodiment involves constructing an array representing the connections of the human body's physical structure based on the human skeletal key point features (i.e., human skeletal information) stored in the aforementioned queue, according to a predefined numerical index of skeletal points. This array is then input into the 2s-AGCN model, which outputs the behavior category and its confidence score.
[0062] The 2s-AGCN model consists of a graph neural network (GCN) and a temporal convolutional neural network (TCN). Its main advantages are twofold: firstly, the adaptive graph convolutional network can adaptively learn the topology of different GCN layers and skeleton samples end-to-end; secondly, the two-stream framework extracts temporal and spatial information, utilizing joint coordinates, bone length, and orientation to extract rich behavioral information, thus enhancing model performance. The model's network structure is a two-stream mode (2s-AGCN) composed of an adaptive graph convolutional network (AGCN).
[0063] The formula for adaptive graph convolution is as follows:
[0064]
[0065] Among them, K v Indicates the division of regions, K v Setting it to 3 means dividing the neighborhood into 3 parts, W k f is the parameter of the k-th region. in Given the feature matrix (human pose information) of the input network, the topology of the graph is determined by the adjacency matrix and the mask. A k B k C k These are matrices about the graph, all of size N×N.
[0066] Among them, A k Equivalent to the convolution kernel in ST-GCN, the sum of the adjacency matrix and the identity matrix represents the natural convolution operation on the joints of the human skeleton, i.e., the physical structure of the human body. And B... k This is a relationship with A k Fully parameterized matrices of the same size. During training, B k It will learn the impact of joints on motion recognition, regardless of whether these joints are directly connected on the human skeleton map. Note B kThe values in the table not only indicate whether two joints will interact during a certain action, but also the strength of that interaction. And C... k For each sample, a graph is learned, and the connection between two key points of the human body is learned from the sample, as well as the strength of the connection.
[0067] The adaptive graph convolution module consists of spatial domain GCN, batch normalization, ReLU, Dropout, temporal domain GCN, batch normalization, and ReLU in that order. The spatial GCN is concatenated with the output residual of the second ReLU operation. The convolution of the temporal GCN is a TCN convolution operation. The input data dimensions are [N, C, T, V, M], representing the number of samples, channels, frames, nodes, and people, respectively. Skeletal information is calculated as the second feature. Each bone is connected by two nodes. The node closest to the centroid of the skeleton is used as the source node, while the node farther from the centroid is used as the target node. The source node is represented as v1 = (x1, y1, z1), and the target node as v2 = (x2, y2, z2). The bone is represented as... Since the number of nodes always exceeds the number of edges, a zero-length edge with a self-loop at the center point is added. 2s-AGCN consists of two network output branches, B-Stream and J-Stream, and the softmax outputs of the two branches are added together to obtain the behavior recognition result.
[0068] The Softmax formula is as follows:
[0069]
[0070] Softmax maps the results to the range [0,1] and sums them to 1, outputting the probability of the behavior category. First, Softmax raises the power of each unnormalized prediction, then divides each raised power result by the sum of all predictions.
[0071] In practical applications, surveillance cameras at the park entrance or guard posts continuously extract frames in real time and send these frames to a trained model for inference. The model detects whether there is any climbing behavior in the images and outputs the detection results in real time. If the detection results indicate climbing behavior, an alarm message is sent to the backend.
[0072] In summary, this invention adopts the sliding window concept and uses a queue data structure as a tool to connect deep learning models with different inputs to detect whether there is climbing behavior in the camera monitoring screen in real time. This invention proposes a method that combines the AlphaPose model and the 2s-AGCN model, and improves the original AlphaPose model to effectively detect people climbing behavior, assist property management personnel in monitoring community security, free up human resources, and empower security upgrades.
[0073] like Figure 6 As shown, this embodiment of the invention also provides a real-time detection device 600 for vaulting behavior based on spatiotemporal information fusion, which includes:
[0074] The extraction unit 601 is used to acquire surveillance video and continuously extract frames. Each extracted frame image is input into the AlphaPose model in sequence to extract the human skeleton information of a single frame image. The human skeleton information includes the pixel coordinates and confidence of key points of the human skeleton.
[0075] Storage unit 602 is used to sequentially store the human skeleton information of each frame image into a fixed queue.
[0076] The judgment unit 603 is used to determine whether the length of the fixed queue is greater than the predetermined length. If so, the human skeleton information of the earliest frame image stored in the fixed queue is retrieved in chronological order.
[0077] Output unit 604 is used to construct an array representing the physical structure connection of the human skeleton information of a frame of image taken out according to a predefined skeletal point digital index, and input it into the 2s-AGCN model to output the behavior category and its confidence.
[0078] In one embodiment, such as Figure 7 As shown, the extraction unit 601 includes:
[0079] The detection unit 701 is used to detect each frame of the image using an object detection algorithm to obtain a pedestrian detection box.
[0080] The pose box generation unit 702 is used to input the pedestrian detection box into the STN module and the SPPE module to automatically generate the pose box.
[0081] The refining unit 703 is used to input the attitude frame into the P-NMS module for refining.
[0082] The data augmentation unit 704 is used to generate pedestrian boxes with the same pose box distribution as the pose generator for data augmentation.
[0083] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed, can implement the methods provided in the above embodiments.
[0084] The present invention also provides a computer device, which may include a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the methods provided in the above embodiments.
[0085] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this invention.
[0086] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
Claims
1. A real-time detection method for vaulting behavior based on spatiotemporal information fusion, characterized in that, include: The system acquires surveillance video and continuously extracts frames. Each extracted frame is then input into the AlphaPose model in sequence to extract the human skeleton information of a single frame. The human skeleton information includes the pixel coordinates and confidence scores of key points of the human skeleton. The human skeleton information of each frame of the image is stored sequentially into a fixed queue; Determine whether the length of the fixed queue is greater than the predetermined length. If so, retrieve the human skeleton information of the earliest stored frame image in the fixed queue according to the time sequence. The human skeleton information of a frame of image is constructed into an array representing the connection of the human physical structure according to the predefined skeletal point numerical index, and then input into the 2s-AGCN model to output the behavior category and its confidence. The process involves acquiring surveillance video and continuously extracting frames. Each extracted frame is then sequentially input into the AlphaPose model to extract the human skeleton information from a single frame, including: Each frame of the image is detected using an object detection algorithm to obtain pedestrian detection boxes; The pedestrian detection box is first input to the STN module, and then input to two parallel SPPE modules. One SPPE module is used to generate the pose box, and the other SPPE module is used for regularization. The attitude frame is input into the P-NMS module for refinement; Pedestrian bounding boxes with the same distribution as the pose boxes are generated using a pose generator for data augmentation; The target detection algorithm is YOLOv7-Tiny; The YOLOv7-Tiny includes a MobileOne module to balance detection accuracy and speed. Three MobileOne modules are configured, each connected before the convolutional modules of the last three output branches in the YOLOv7-Tiny, and each replacing the CBL module before the original three convolutional modules.
2. The real-time detection method for vaulting behavior based on spatiotemporal information fusion according to claim 1, characterized in that, The MobileOne module includes a depthwise convolutional module and a pointwise convolutional module arranged sequentially; the kernel size of the depthwise convolutional module is 3×3, and the kernel size of the pointwise convolutional module is 1×1; the depthwise convolutional module has 3 branches, and the pointwise convolutional module has 2 branches.
3. The real-time detection method for vaulting behavior based on spatiotemporal information fusion according to claim 1, characterized in that, The H-Swish activation function is used in YOLOv7-Tiny.
4. The real-time detection method for vaulting behavior based on spatiotemporal information fusion according to claim 3, characterized in that, The calculation formula for the H-Swish activation function is as follows: Where x is the input.
5. A real-time detection device for climbing behavior based on spatiotemporal information fusion, characterized in that, include: The extraction unit is used to acquire surveillance video and continuously extract frames. Each extracted frame image is input into the AlphaPose model in sequence to extract the human skeleton information of a single frame image. The human skeleton information includes the pixel coordinates and confidence scores of key points of the human skeleton. The storage unit is used to sequentially store the human skeleton information of each frame of image into a fixed queue. The judgment unit is used to determine whether the length of the fixed queue is greater than the predetermined length. If so, the human skeleton information of the earliest frame image stored in the fixed queue is retrieved in chronological order. The output unit is used to construct an array representing the physical structure connection of the human skeleton information of a frame of image taken out according to the predefined skeletal point numerical index, and input it into the 2s-AGCN model to output the behavior category and its confidence. The process involves acquiring surveillance video and continuously extracting frames. Each extracted frame is then sequentially input into the AlphaPose model to extract the human skeleton information from a single frame, including: Each frame of the image is detected using an object detection algorithm to obtain pedestrian detection boxes; The pedestrian detection box is first input to the STN module, and then input to two parallel SPPE modules. One SPPE module is used to generate the pose box, and the other SPPE module is used for regularization. The attitude frame is input into the P-NMS module for refinement; Pedestrian bounding boxes with the same distribution as the pose boxes are generated using a pose generator for data augmentation; The target detection algorithm is YOLOv7-Tiny; The YOLOv7-Tiny includes a MobileOne module to balance detection accuracy and speed. Three MobileOne modules are configured, each connected before the convolutional modules of the last three output branches in the YOLOv7-Tiny, and each replacing the CBL module before the original three convolutional modules.
6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the real-time detection method for vaulting behavior based on spatiotemporal information fusion as described in any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the real-time detection method for vaulting behavior based on spatiotemporal information fusion as described in any one of claims 1 to 4.