Fall detection algorithm based on multi-scale space and time sequence double-flow enhancement

Abstract

The application provides a fall detection algorithm based on multi-scale space and time sequence double-flow enhancement, and relates to the technical field of computer vision and intelligent health monitoring.The application takes Video Swin Transformer as a backbone network, and in view of the dual defects of the window self-attention mechanism of the Video Swin Transformer, that is, the insufficient modeling efficiency of the cross-window long-range spatial correlation and the indirect capture of inter-frame time sequence dynamic information, a multi-scale large-core attention module and a double-flow time sequence difference enhancement module are designed.The multi-scale large-core attention module realizes the direct coverage of multi-scale spatial features from local details to global context in a single-layer network through the cooperation of hierarchical receptive field multi-branch extraction units and scale self-adaptive feature aggregation units.The double-flow time sequence difference enhancement module realizes the comprehensive time sequence description of the fall behavior from the inter-frame correlation to the motion mutation through the parallel processing and adaptive fusion of the time sequence offset branch and the multi-scale frame difference branch.The two modules are systematically integrated in a four-stage hierarchical feature extraction architecture to form spatial and temporal double enhancement.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and intelligent health monitoring technology, specifically to a fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement, which is applicable to real-time identification and early warning of accidental falls in scenarios such as elderly care institutions, home environments, and public communities. Background Technology

[0002] Fall detection is a key technology for ensuring the safety of the elderly and people with mobility impairments. Existing methods mainly fall into three categories: wearable device-based, environmental sensor-based, and computer vision-based. Vision-based methods have become a research hotspot due to their advantages such as not requiring wearable devices, reusability of monitoring facilities, and wide monitoring range. Deep learning, especially convolutional neural networks (CNNs) and the Transformer architecture, has been widely applied to this task.

[0003] However, existing CNN-based methods are limited by their local receptive fields, making it difficult to model long-range spatiotemporal dependencies in video sequences. While Vision Transformer (ViT)-based methods possess global modeling capabilities, they lack local inductive biases and are susceptible to background interference. In recent years, the Video Swin Transformer has balanced local and global modeling to some extent through windowing and shifted window self-attention mechanisms, but its direct application to fall detection still faces two inherent bottlenecks: (1) Fixed window self-attention requires multiple layers to establish long-range spatial associations across windows, and a single-scale receptive field cannot fully cover the multi-scale motion characteristics of falling behavior; (2) The inter-frame temporal correlation is not directly captured by simply modeling the instantaneous velocity change and motion boundary change during the fall process, and it is difficult to accurately depict the temporal evolution characteristics of the fall behavior.

[0004] Therefore, there is an urgent need for a detection model that can simultaneously and comprehensively capture multi-scale spatial features and accurately characterize the instantaneous dynamic information of a fall. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing Video Swin Transformers in fall detection tasks, such as insufficient efficiency in long-range spatial correlation modeling across windows, inadequate capture of multi-scale motion features, and indirect capture of inter-frame temporal dynamic information, this invention provides a fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement. The algorithm aims to achieve direct modeling of long-range spatial dependencies across windows through a multi-scale large-kernel attention module and high-precision capture of instantaneous dynamic changes during falls through a dual-stream temporal difference enhancement module, thereby significantly improving the accuracy and scene robustness of fall detection.

[0006] (II) Technical Solution To achieve the above objectives, this invention provides a fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement. The core technical concept of this invention is as follows: using the Video Swin Transformer as the backbone network, and addressing the inefficiency of its window self-attention mechanism in modeling cross-window spatial correlations, a multi-scale large-kernel attention module (MSLKA) is designed. This module directly covers the multi-scale receptive field within a single layer through hierarchical multi-branch large-kernel convolution, and is supplemented by scale-adaptive aggregation to achieve efficient modeling of long-range spatial dependencies across windows. Addressing the weakness in its inter-frame temporal dynamic perception, a dual-stream temporal difference enhancement module (DSTDE) is designed. Through the parallel collaboration and adaptive fusion of temporal offset branches and multi-scale frame difference branches, a comprehensive temporal characterization from inter-frame correlation to motion abrupt changes is achieved. The two modules complementarily enhance the backbone network from both spatial and temporal dimensions, and are systematically integrated into a four-stage hierarchical architecture, forming a comprehensive and in-depth representation of the spatiotemporal characteristics of fall behavior. Specifically, the following steps are included: Step 1: Video Preprocessing and Feature Embedding To address the mismatch between the dimensions of the original video frame sequence and the input space of the backbone network, while preserving sufficient spatiotemporal resolution for subsequent feature extraction, this step performs 3D patch embedding and linear projection on the input video.

[0007] Specifically, a continuous RGB video frame sequence to be detected is acquired, assuming the sequence contains T frames, each with a height of H and a width of W. A 3D patch embedding layer is used to extract spatiotemporal features and transform the dimensions of the input video: the temporal dimension is sampled with a stride of 2, and the spatial dimension with a stride of 4, achieving non-overlapping patch partitioning. This configuration reduces the temporal length to 50% of the original sequence and the spatial resolution to 25% of the original image. Each 3D patch is mapped to a 96-dimensional feature space, serving as the initial input representation for the subsequent ST-Stage four-stage network. Through the above configuration, effective mapping of the input dimensions is achieved while ensuring feature representation capabilities.

[0008] Step 2: Spatiotemporal feature extraction based on the ST-Stage four-stage hierarchical architecture To progressively deepen feature representation and comprehensively characterize the fall behavior from low-level edge textures to high-level semantic actions, while addressing the dual shortcomings of insufficient efficiency in cross-window spatial correlation modeling and weak inter-frame temporal dynamic perception in the backbone network, this step sequentially inputs the initial spatiotemporal feature map into a hierarchical network consisting of four ST-Stage feature extraction stages. Each stage integrates a dual-stream temporal differential enhancement module (DSTDE) and a multi-scale large kernel attention module (MSLKA), forming a serial processing architecture where temporal enhancement is pre-processed, and window encoding and spatial enhancement iterate sequentially with each encoder block. The number of encoder blocks in the four stages are 2, 2, 6, and 2, respectively, with the network depth increasing progressively to match the complexity requirements of feature extraction at different levels. Each ST-Stage stage specifically includes the following sequential processing steps: (2.1) PatchMerge downsampling (except for the first stage) To construct the feature pyramid, expand the receptive field of subsequent operations, and control computational cost, spatial downsampling and channel dimensionality upsampling are performed on the input feature map. Specifically, the spatial resolution of the feature map is reduced (e.g., downsampled by 2x) through the PatchMerge layer, while the channel dimension is increased accordingly (e.g., doubled). This reduces the hierarchical size of the feature map while aggregating local contextual information, resulting in higher-level features having a larger equivalent receptive field and stronger semantic representation capabilities.

[0009] (2.2) Dual-stream temporal difference enhancement preprocessing (DSTDE) To address the shortcomings of the Video Swin Transformer, which relies solely on window shifting to indirectly model inter-frame temporal correlations and fails to directly capture explicit dynamic information such as instantaneous velocity changes and motion boundary variations during a fall, this step embeds a dual-stream temporal difference enhancement module (DSTDE) at the front end of the encoder assembly to comprehensively capture temporal dynamic information from the input features. This module achieves a comprehensive characterization of fall behavior, from inter-frame correlations to sudden motion changes, through two parallel and complementary temporal modeling paths: The first path: Temporal Offset Branch (TSM) – To establish long-range temporal correlations between frames with zero computational overhead without introducing any learnable parameters, this branch divides the input features along the channel dimension into a forward offset group, a backward offset group, and an invariant group in a 1 / 3 ratio. These groups are responsible for perceiving future information, perceiving historical information, and preserving current details, respectively. By shifting features forward, backward, and preserving them along the time dimension, inter-frame information exchange is achieved, outputting features with long-range temporal correlations. The advantage of this design is that it allows subsequent encoders to perceive the temporal context across frames without additional parameters or complex calculations.

[0010] The second path: Multi-scale frame difference branch—To accurately capture the velocity abrupt changes in different stages of a fall (initial instability, rapid descent, and impact with the ground), a single frame difference cannot cover the variable velocity characteristics of the fall behavior. This branch calculates the feature differences between each frame and the preceding 1st, 2nd, and 4th frames, corresponding to three temporal granularities: instantaneous dynamics, short-term evolution, and long-term trend, respectively. This comprehensively captures multi-scale motion information from rapid abrupt changes to slow evolution. For the frame difference feature at each scale, global average pooling is performed along the spatial dimension to generate channel-level statistics. Channel transformation is then used to generate channel attention weights, which are multiplied element-wise with the corresponding frame difference feature to suppress background interference and enhance the motion response related to the fall. Finally, the weighted frame difference features of the three scales are concatenated along the channel dimension and compressed via convolution to output multi-scale motion difference features.

[0011] The adaptive fusion mechanism—the temporal offset branch establishes inter-frame correlation but is insensitive to motion amplitude, while the inter-frame difference captures motion changes but is susceptible to noise interference; a single mechanism cannot simultaneously satisfy the requirements of correlation and discriminability. Therefore, this module concatenates the two output features along the channels and extracts the global context through global average pooling. Dynamic fusion weights are generated through two independent channel transformations. The two weights are normalized (making the sum of the weights in each channel equal to 1) and then weighted and applied to their respective branches, achieving adaptive fusion that dynamically adjusts the contribution of the two paths based on the input video content. The weighted features are then summed element-wise and batch normalized, finally superimposed on the original input features using a residual connection method to ensure smooth gradient flow and output temporally enhanced features. Through this design, this module significantly enhances the network's perception accuracy of instantaneous dynamic changes during falls without increasing the model depth.

[0012] (2.3) Iterative encoding of encoder group To further capture local spatial details within video frames and global spatiotemporal dependencies between frames based on the temporally enhanced features, this step iteratively encodes the temporally enhanced features output by DSTDE through multiple encoder blocks within this stage. Each encoder block consists of a pair of Window Multi-Head Self-Attention (WMSA) and Shift Window Multi-Head Self-Attention (SWMSA), achieving a dynamic balance between local feature focusing and global context modeling by alternating between the two attention mechanisms. The specific process is as follows: the input features are first processed by WMSA to calculate spatial self-attention within the local window, and then pass through the first residual connection to retain the original information; then through the first feedforward network (FFN) for position-wise nonlinear feature transformation, and then through the second residual connection; then through SWMSA to achieve cross-window information interaction through window shifting and establish long-range dependencies, and then through the third residual connection; then through the second FFN to enhance the nonlinear expressive power, and then through the fourth residual connection, completing the feature extraction of one encoder block.

[0013] (2.4) Multiscale Large Kernel Attention Postprocessing (MSLKA) To address the shortcomings of the Swin Transformer's windowed self-attention, which requires multiple stacked layers to establish long-range spatial associations across windows and where a single-scale receptive field cannot cover the multi-scale features of fall behavior, this step sequentially inserts a multi-scale large kernel attention module (MSLKA) after the fourth residual connection output of each encoder block to enhance the spatial features of the encoded output at multiple scales. This module contains two cooperating units: The hierarchical receptive field multi-branch extraction unit—to directly cover multi-scale spatial information from local details to global context within a single-layer network—feeds input features in parallel to three branches. Each branch uses a 7×7 depthwise convolution as its base layer, then concatenates depthwise separable dilated convolutions with corresponding dilation rates. Specifically, the small-scale branch concatenates depthwise separable dilated convolutions with the first dilation rate, the medium-scale branch concatenates with depthwise separable dilated convolutions with the second dilation rate, and the large-scale branch concatenates with depthwise separable dilated convolutions with the third dilation rate. The dilation rates increase sequentially, ultimately forming a three-tiered progressive equivalent receptive field of 7×7, 15×15, and 27×27. This design avoids the efficiency loss of the traditional Swin Transformer, which requires stacking four stages to establish global correlations. A 3D linear attention module is embedded after the convolution operation in each branch. This module merges the batch dimension and temporal dimension, flattens the spatial dimension into a sequence, and normalizes the query matrix along the spatial dimension and the key matrix along the feature dimension. This transforms the quadratic computational complexity of standard self-attention into linear computational overhead, further strengthening the long-range spatial dependencies within each branch and achieving collaborative optimization of large kernel coarse-grained coverage and fine-grained attention association.

[0014] The scale-adaptive feature aggregation unit—to integrate multi-scale context and allow the model to focus on the human body region while reducing background noise—concatenates the features output from the three branches along the channel dimension. It then performs channel compression and activates enhanced nonlinearity via point convolution, followed by spatial convolution to predict the scale weight distribution at each spatial location. Softmax normalization yields adaptive scale selection weights, ensuring that the sum of the weights for the three scales at each spatial location is 1. Finally, these weights are used to perform a weighted summation of the features from the three branches, achieving position-wise adaptive scale fusion and outputting multi-scale enhanced spatial features.

[0015] The enhanced features serve as input to the next encoder block or as the final output of the ST-Stage. After four stages of progressive feature extraction, the network ultimately outputs a deep feature map rich in temporal dynamics and comprehensive spatial correlation information, providing strong feature support for subsequent accurate classification.

[0016] Step 3: Classification Prediction To map the extracted deep spatiotemporal features to the final fall discrimination result, this step performs global adaptive average pooling on the feature map output from the last ST-Stage stage, aggregating the features of each spatial location into a 768-dimensional global feature vector. This vector is then input into a fully connected layer with a dimension of 768×2, and after Softmax activation, it outputs the corresponding binary classification probability prediction result of fall / non-fall.

[0017] (III) Beneficial Effects This invention provides a fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement. It has the following beneficial effects: 1. This invention achieves excellent detection performance on standard public datasets. As shown in the examples, the accuracy reaches 99.51% on the Le2i fall detection dataset and 99.72% on the UR fall detection dataset, both outperforming fall detection algorithms based on CNN, 3D CNN, and the basic ViT architecture of the same period. This verifies the effectiveness and advancement of the multi-scale spatial and temporal dual-stream enhancement architecture proposed in this invention for fall detection tasks.

[0018] 2. This invention addresses the shortcomings of the Video Swin Transformer's native window self-attention mechanism in modeling long-range spatial relationships across windows by designing a multi-scale large-kernel attention module. This module utilizes a hierarchical receptive field multi-branch extraction unit, employing three progressively increasing equivalent receptive fields of 7×7, 15×15, and 27×27. This allows for direct coverage from local details to the global context within a single network layer, avoiding the efficiency loss of the traditional Swin Transformer which requires stacking four stages to establish global relationships. Simultaneously, a scale-adaptive feature aggregation unit enables position-by-position adaptive scale selection, allowing the model to accurately focus on human regions related to fall behavior and effectively mitigate background noise interference. Experiments show that this design significantly improves the model's ability and robustness in capturing multi-scale human pose and motion features in complex scenes.

[0019] 3. This invention addresses the shortcomings of the Video Swin Transformer, which relies solely on window shifting to indirectly model inter-frame temporal correlations and is indirect in capturing explicit dynamic information such as instantaneous velocity changes and motion boundary variations during a fall. A dual-stream temporal difference enhancement module is designed to address these shortcomings. This module achieves efficient information exchange and long-range temporal correlation modeling of inter-frame features with zero parameters through a temporal offset branch; it calculates feature differences between the current frame and the preceding 1st, 2nd, and 4th frames through a multi-scale frame difference branch, capturing instantaneous dynamics, short-term evolution, and long-term trends respectively; and it suppresses background interference and enhances fall-related motion responses through a channel attention weighting mechanism; and it dynamically balances the contributions of the two branches through an adaptive fusion mechanism. The synergistic effect of these three paths enables the model to accurately depict the complete temporal evolution of a fall from initiation and instability to impact with the ground, effectively solving the problem of existing models being indirect and insensitive to capturing instantaneous dynamic changes during falls.

[0020] 4. This invention uses the Video Swin Transformer-Tiny as its backbone network. Based on this, significant performance improvements are achieved simply by sequentially inserting multi-scale large-kernel attention modules and embedding dual-stream temporal difference enhancement modules at the front end of each stage. The multi-scale large-kernel attention module, through hierarchical receptive field multi-branch extraction and 3D linear attention co-design, achieves efficient capture of multi-scale spatial features while maintaining linear computational complexity. The temporal offset branch in the dual-stream temporal difference enhancement module does not introduce any learnable parameters. The overall architecture effectively enhances the model's spatiotemporal feature representation capabilities while maintaining low computational and parameter overhead, demonstrating good deployment feasibility and practical application value.

[0021] 5. This invention's vision-based fall detection method achieves non-invasive, continuous monitoring without requiring the subject to wear any dedicated sensing devices. It can be rapidly deployed using existing smart monitoring hardware in communities, elderly care facilities, and homes, significantly reducing implementation costs. Furthermore, it boasts a wider monitoring range than wearable devices and can completely retain contextual information throughout the entire fall event process. Its four-stage hierarchical feature extraction architecture and modular design ensure stable detection performance under various lighting conditions, occlusion scenarios, and complex background environments, making it suitable for diverse practical applications such as home-based elderly care, medical institutions, and public communities. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall framework of the fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement of the present invention; Figure 2 This is a schematic diagram of the ST-Stage feature extraction stage in this invention; Figure 3This is a schematic diagram of the structure of the Multi-Scale Large Kernel Attention Module (MSLKA) in this invention; Figure 4 This is a schematic diagram of the structure of the dual-stream timing difference enhancement module (DSTDE) in this invention; Figure 5 This is a schematic diagram of the multi-scale frame difference branch within the dual-stream timing difference enhancement module of the present invention; Figure 6 This is a schematic diagram of the adaptive fusion mechanism within the dual-stream timing difference enhancement module of the present invention; Figure 7 The accuracy curves of SPTDE-VSwin and its variants on Le2i in this invention are shown. Figure 8 This is the accuracy curve of SPTDE-VSwin and its variants on UR in this invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Example: like Figures 1 to 6 As shown, to capture a large amount of spatial information and temporal dynamics in videos, this invention proposes a fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement. Using the Video Swin Transformer as the backbone network, two core innovative modules are designed to complement each other: a Multi-Scale Large Kernel Attention (MSLKA) module for capturing large-scale information outside the Swin window; and a Dual-Stream Temporal Difference Enhancement (DSTDE) module for single-frame channel fusion and highlighting inter-frame differences, increasing detection accuracy. By systematically integrating these modules into the backbone network, together with the Transformer Encoder, a novel dual-stream multi-scale stage (ST-Stage) is formed, accurately identifying fall behavior in complex scenes.

[0025] SPTDE-VSwin, for example Figure 1As shown, it mainly consists of three core parts: video preprocessing, hierarchical feature extraction, and classification prediction. The feature extraction model adopts a four-stage architecture, with 2, 2, 6, and 2 encoder blocks in each stage, respectively. The encoder blocks are paired based on WMSA and SWMSA to form encoder groups. In terms of module integration strategy, the DSTDE module is applied at the beginning of each stage for temporal enhancement, while the MSLKA module is inserted serially after each encoder block to enhance the spatial feature representation.

[0026] The model's input is a sequence of video frames, containing T frames, each with a height of H and a width of W, in RGB three-channel format. The input video is first divided into non-overlapping 3D patches, which are then projected into the embedding space through a linear embedding layer. First, a 3D patch embedding layer is used to extract spatiotemporal features and transform the dimensions of the input video. Specifically, the temporal dimension is sampled with a stride of 2, and the spatial dimension with a stride of 4, achieving non-overlapping patch partitioning. This configuration reduces the temporal length to 50% of the original sequence and the spatial resolution to 25% of the original image, mapping each 3D patch to a 96-dimensional feature space as the initial input representation for subsequent Swin Transformer layers. This preprocessing process ensures feature representation while effectively mapping the input dimensions. After preprocessing, the features are fed into the ST-Stage four-stage network designed in this embodiment for hierarchical feature extraction, fully utilizing various feature information in the video.

[0027] For the features output from the fourth stage of ST-Stage, adaptive average pooling is first used to perform global feature aggregation and dimensionality regularization. For the binary classification task of fall detection, this embodiment of the invention uses a fully connected layer with a dimension of 768×2 to achieve classification mapping, where 768 corresponds to the channel dimension of the input features after pooling, and 2 represents the two target classification labels: fall and non-fall. The pooled features are input to this fully connected layer, and the final output is the classification prediction result for the corresponding video sequence.

[0028] Subsequently, embodiments of the present invention will describe in detail the feature extraction process and two innovative modules built on this backbone network, namely ST-Stage, MSLKA and DSTDE.

[0029] like Figure 2 As shown, in order to capture features more precisely, this embodiment of the invention designs ST-Stage as the core feature extraction unit of the architecture, and deeply integrates DSTDE and MSLKA modules into the feature extraction process to form a collaborative optimization mechanism for precise temporal characterization and global spatial capture. Figure 2As an ST-Stage module, the input feature map first passes through the PatchMerge module (there is no PatchMerge module in the first stage), which performs spatial dimension downsampling, channel dimension upsampling, and aggregates local contextual information. Subsequently, to address the problem that traditional Video Swin Transformer encoders rely solely on window shifting to indirectly model temporal correlations, this embodiment first uses the DSTDE module to comprehensively capture the temporal information of the input features. The temporal discriminative power of the feature map is significantly improved, providing a precise dynamic information foundation for subsequent spatial feature extraction.

[0030] Let the feature map after passing through the PatchMerge module be... The timing enhancement process through DSTDE is as follows: The process then proceeds through the encoder block in this stage to ensure that temporal information is fully utilized in feature extraction at each level. Within the encoder block, windowed self-attention first captures local information by calculating attention weights within a local feature window, and then preserves this information through residual connections. FFN performs feature transformation independently for each spatial location, enhancing the model's nonlinear expressive power. Simultaneously, a periodic window shifting strategy is employed to enable cross-window information interaction, ensuring long-term dependency modeling capabilities. Then, FFN was used again to enhance the model's nonlinear expressive power: To address the inefficiencies and limited receptive field scale of windowed attention space correlation modeling using the Swing transformer, this invention inserts an MSLKA module sequentially after the FFN to specifically enhance inter-window features. The enhanced output features are then fed back into the Transformer Encoder and MSLKA modules in the same group to complete multiple rounds of feature iteration optimization for more refined feature extraction.

[0031] The features output by ST-Stage possess both rich temporal dynamic information and comprehensive spatial correlation information, breaking through the structural limitations of Video Swin Transformer in temporal dynamic perception and long-range spatial correlation modeling, and providing strong feature support for the accurate identification of falling behavior in complex scenes.

[0032] like Figure 3 As shown, to comprehensively capture spatial dynamic information in videos, this embodiment of the invention designs a multi-scale large kernel attention module (MSLKA). The MSLKA consists of a hierarchical receptive field multi-branch extraction unit (HRF-MBEU) and a scale-adaptive feature aggregation unit (SA-FAU). Through a three-level progressive receptive field design of 7×7, 15×15, and 27×27, multi-scale feature extraction from local details to global context is achieved.

[0033] HRF-MBEU Unit: To expand the receptive field and capture multi-scale information, this embodiment of the invention designs the HRF-MBEU unit, which utilizes large-kernel convolution to perform long-distance dependency modeling with linear complexity. This unit employs a three-level progressive receptive field of 7×7, 15×15, and 27×27, achieving direct coverage from local details to global context within a single layer, avoiding the efficiency loss of the Swin Transformer, which requires four stages to establish global associations. Specifically, each branch begins with a 7×7 depthwise convolution, followed by the introduction of depthwise separable dilated convolution to expand the receptive field. Although standard convolution can extract features at different scales, its receptive field is limited. Depthwise separable dilated convolution, by introducing a dilation factor, can effectively expand the receptive field without increasing the kernel size, enabling the model to capture a longer range of contextual information. This design offers significant computational complexity advantages.

[0034] While large-kernel convolution expands the receptive field, it lacks fine-grained positional interactions. This invention embeds a 3D linear attention module after each branch to further enhance the spatial feature correlation between branches at different scales. Simultaneously, it avoids the high computational complexity of standard self-attention mechanisms, achieving synergistic optimization of multi-scale convolutional feature extraction and linear attention enhancement. This mechanism employs a spatial-dimensional attention computation strategy: merging batch and temporal dimensions, flattening the spatial dimension into a sequence, and keeping the channel dimension unchanged, thus focusing on long-range spatial dependencies while maintaining temporal independence. By applying Softmax normalization to Q along the spatial dimension and K along the feature dimension, the quadratic complexity of standard self-attention is transformed into K... T Matrix multiplication of V and Q achieves linear computational overhead. This collaborative design of large-kernel coarse-grained coverage and parallel attention fine-grained association enables features at each scale to possess both global context and precise local associations.

[0035] SA-FAU Module: To integrate the multi-scale spatial context output by HRF-MBEU, allowing the model to focus on the human body region, weaken background noise, and enhance the expression of important features, this embodiment of the invention designs the SA-FAU module. This module first concatenates the features from the small-scale, medium-scale, and large-scale branches along the channel dimension, effectively covering the three receptive fields to obtain fused features after multi-scale concatenation. To achieve position-by-position scale selection, a lightweight convolutional network is used to predict the scale weights at each spatial location. First, a 1×1×1 point convolution reduces the dimensionality of the concatenated features from 3C to C, and GELU enhances its non-linear expressive power. Subsequently, a spatial convolution with a 1×7×7 kernel is set to predict the weight distribution of the three scale branches. This convolutional layer contains learnable bias terms to provide initialization flexibility.

[0036] Next, the predicted weights are normalized using the Softmax function to ensure that the sum of the weights at the three scales at each spatial location is 1, thus obtaining the adaptive scale selection weights. .

[0037] Finally, the three scale features are stacked by channel, multiplied element-wise with the expanded weight matrix, and summed along the scale dimension to achieve weighted fusion.

[0038] MSLKA effectively overcomes the receptive field limitation of Video Swin window attention by combining parallel multi-scale extraction of HRF-MBEU with spatial adaptive fusion of SA-FAU. It achieves direct modeling of long-range dependencies across windows and accurate aggregation of multi-scale features while maintaining linear complexity.

[0039] like Figure 4As shown, to comprehensively capture temporal dynamic information in video, this embodiment of the invention designs a DSTDE module. DSTDE comprises three core components: a temporal offset branch (TSM), a multi-scale frame difference branch (Multi-Scale Frame Difference Branch), and an adaptive fusion mechanism. Through the synergy of channel offset and multi-scale frame difference, efficient capture of multi-granular temporal dynamics is achieved, improving the accuracy of fall detection.

[0040] While TSM (Time-Self-Attention Window) can establish inter-frame associations, it lacks sensitivity to motion boundaries and struggles to distinguish between moving foreground and stationary background. To enhance inter-frame information interaction without introducing learnable parameters, TSM is introduced, enabling the model to establish long-range temporal associations with extremely low computational overhead, providing temporal pre-enhancement features for subsequent motion analysis. It first divides the channel dimension into a forward group, a backward group, and an invariant group, each accounting for one-third of the total. These groups respectively handle future information perception, historical information perception, and current detail preservation, completing information transfer through feature shifting along the temporal dimension.

[0041] Multi-Scale Frame Difference Branch: A single frame difference is insufficient to cover the variable speed characteristics of the fall process. Therefore, this embodiment of the invention designs a multi-scale frame difference to enhance feature differences. Figure 5 The specific process of the multi-scale frame difference branch is given, assuming that the input of the multi-scale frame difference branch is the original temporal features. This branch captures motion information at different time scales through refined operations such as multi-scale frame difference calculation, channel attention weighting, and cross-scale feature fusion.

[0042] In this embodiment of the invention, for each frame in the time dimension, the feature differences between it and multiple historical interval frames are calculated to comprehensively capture the information of each frame. Let the set of time interval frames be... Interval 1 is used to capture instantaneous dynamics, interval 2 to capture short-term evolution, and interval 4 to capture long-term trends. For time t and interval frames... The backward frame difference feature is calculated as follows: Among them, for The boundary positions are zero-padding, and then the frame difference features at each scale are obtained. .

[0043] To suppress background interference and enhance fall-related motion response, this embodiment of the invention applies channel attention weighting to the frame difference features at each scale. First, global average pooling is performed along the spatial dimension to obtain channel-level statistics. : Then, dimensionality reduction and convolution are performed. Upgrade to a higher dimension, and then through Generate channel-level attention weights: By multiplying the weights element-wise with the original frame difference features, we can obtain a feature map containing inter-frame difference information. ,in These are the attention weights generated earlier.

[0044] In this embodiment of the invention, the obtained inter-frame difference features at different scales are stitched together. Corresponding to the "Multi-Scale Fusion" feature fusion module in the figure, the weighted features of the three scales are stitched together along the channel dimension. : Finally, it is compressed to the C channel through a 1×1×1 convolution, and then normalized and... After activation, the final output includes timing information and is a branch output. This multi-scale difference design enables the model to adaptively perceive feature changes at different stages of a fall.

[0045] Adaptive Fusion: Channel offset establishes inter-frame correlation but is insensitive to motion amplitude, while inter-frame difference captures motion changes but is susceptible to noise interference. Neither single mechanism can simultaneously satisfy the requirements of correlation and discriminativeness. Therefore, this embodiment of the invention designs Adaptive Fusion to fuse the outputs of TSM and MSFDB. Figure 6 This describes the specific process of Adaptive Fusion.

[0046] like Figure 6 Let the input of this mechanism be the output of the timing offset branch. The output of the multi-scale frame difference branch is First, and By splicing along the channel dimension, we obtain .

[0047] Subsequently, Perform global average pooling to extract global context information, better balance contributions, and output as... .

[0048] To reduce computational overhead, By using a 1×1×1 convolution, the number of channels is compressed from 2C to C / 4, resulting in... : Then, the dimensions are increased back to the C channels through two independent 1×1×1 convolutions, and two different sets of channel attention weights are generated by the sigmoid activation function: The fusion weights for the temporal offset branch are used to adjust the contribution of inter-frame temporal correlation features; The fusion weights for the multi-scale frame difference branch are used to modulate the contribution of the fall motion abrupt feature: To avoid weight imbalance, the channel attention weights are normalized.

[0049] Among them, parameters The purpose is to prevent the number from being reduced to zero.

[0050] Finally, the normalized weights are multiplied element-wise by the corresponding branch features, and batch normalization is completed through a BatchNorm layer. This lightweight gating design allows the network to dynamically adjust the weights of different branches based on the input video. The output features are then obtained. This achieved efficient feature fusion.

[0051] Final output: as shown Figure 4 As shown, this embodiment of the invention uses residual connections to add features from the original input. This is to ensure the module retains the original information. The final output of DSTDE was obtained. : The module achieves cross-frame information exchange at the feature level through TSM, highlights inter-frame differences through multi-scale frame difference branches, and dynamically balances the contributions of a lightweight adaptive fusion network. It can significantly enhance temporal discriminative power without increasing the model depth, and provide robust and interpretable feature support for complex video understanding tasks.

[0052] Experimental Example The experimental examples of this invention were conducted on a server equipped with a 10 vCPU Intel Xeon Processor and an NVIDIA Tesla V100 GPU. The models used in these experimental examples were built using the PyTorch 2.2.0 deep learning framework, CUDA 12.4, and Python 3.10.

[0053] Dataset The experimental examples of this invention were validated on two publicly available fall detection datasets: the Le2i dataset and the UR dataset.

[0054] Le2i contains 191 videos captured by RGB cameras at 25fps with a resolution of 320×240, covering four scenes: home, coffee shop, office, and lecture. Fall behaviors include forward falls, side falls, and slips while sitting. Normal activities include walking, sitting, bending over, and picking up objects. The Le2i dataset is briefly described as follows: there are six folders representing different scenes. These folders are further divided into two subfolders: Videos and Annotations. The former contains several video files, including those depicting falls and daily activities. The latter contains a .txt tag file with the same name as the video file. The tag file primarily contains the starting frame number of the fall, the category, and the bounding box of the person. UR records 70 RGB videos (40 videos of daily activities and 30 videos of falls) at a frame rate of 32fps with a resolution of 640×480. Both datasets consider environmental effects such as lighting changes, shadow areas, and occlusion, and are widely authoritative in fall detection. The UR dataset is described below: The complete sequence of this dataset includes visual data acquired by a Microsoft Kinect camera, raw accelerometer data acquired by the PS Move and x-IMU devices worn by the subjects, and time synchronization information for both types of data. The fall event sequence was simultaneously acquired by two Kinect cameras: camera 0 was mounted parallel to the ground, and camera 1 was suspended from the ceiling with its lens perpendicular to the ground. The daily activity sequence was acquired only by camera 0. The visual data for each sequence includes RGB and depth image sequences from the corresponding camera. All video frames are stored sequentially in PNG image format in a separate folder. Synchronization ensures timestamp alignment between camera video frames and accelerometer data, guaranteeing temporal matching of multimodal data. This experiment uses only the RGB image PNG sequences from the UR dataset for algorithm training and performance verification. The experimental examples of this invention use VideoSwin Transformer-Tiny as the backbone. Given that the dataset is relatively small compared to VideoSwin Transformer-Tiny, the experimental examples of this invention employ data augmentation techniques, including random cropping, horizontal flipping, random rotation, brightness adjustment, and contrast adjustment. After data preprocessing, the experimental example of this invention sets the input frame size to 224×224. To stably evaluate the generalization ability of SPTDE-VSwin, the experimental example divides the data into training and test sets in an 8:2 ratio, and uses a sliding window to divide the complete video into 8-frame video segments. When a fall label is present in a segment, the experimental example classifies that segment as a fall. To ensure the reliability of the experimental results, the experimental example uses a five-fold cross-validation method to obtain the average experimental results.

[0055] Experimental setup In the experiments, the initial learning rate was set to 0.00001, the batch size to 64, and AdamW was selected as the optimizer, CosineAnnealingLR as the learning rate scheduler, using the cross-entropy loss function. Furthermore, pre-trained weights from ImageNet-1K were used to initialize the model to improve its generalization ability. The proposed network has a total of 58.29M parameters and 38.12G GFlops. All subsequent experiments are based on these parameters.

[0056] Evaluation indicators Fall detection is typically considered a binary classification task to determine whether a fall event has occurred in a video sequence. The experimental examples in this invention use precision, sensitivity, accuracy, and F-score as evaluation metrics to assess the model's performance.

[0057] Here, TP, FP, TN, and FN represent the number of true positives, false positives, true negatives, and false negatives, respectively.

[0058] ablation experiment This invention includes an ablation experiment to verify the effectiveness and independent contribution of each core innovative component in the proposed algorithm. The experiment constructs three variant models by sequentially removing modules from the complete algorithm SPTDE-VSwin proposed in this invention: the SP-VSwin algorithm with the DSTDE module removed and only the MSLKA module retained; the TDE-VSwin algorithm with the MSLKA module removed and only the DSTDE module retained; and the baseline algorithm VSwin with both modules removed and only the Video Swin Transformer backbone network retained. The experimental examples of this invention compare the detection performance of the four models on the UR and Le2i datasets. The training hyperparameters, experimental environment, and data preprocessing procedures of all models are kept completely consistent to ensure the uniqueness of experimental variables.

[0059] The learning trends of the four algorithms are similar: in the first 10 epochs, the accuracy increases rapidly, and then the growth rate slows down in the next 10 epochs, and the performance differences between the algorithms gradually become apparent; after 20 epochs, the accuracy increases slowly and tends to stabilize. Figure 7 Figure 8 shows the accuracy curves of the four algorithms as the number of training epochs changes. The final performance of the four algorithms is shown in Tables 1 and 2.

[0060] Specifically, the proposed SPTDE-VSwin complete algorithm integrates both the MSLKA and DSTDE modules, possessing powerful multi-scale global and local spatial feature extraction capabilities. Furthermore, the DSTDE module significantly enhances its temporal dynamic feature capture capabilities, enabling accurate capture of the fall event occurrence process. Therefore, it performs best among the four algorithms. When the DSTDE module is removed from the complete algorithm, the resulting SP-VSwin algorithm retains only its spatial feature enhancement capabilities, lacking temporal optimization, resulting in a significant performance drop. It ranks second among the four algorithms, thus verifying the crucial role of the DSTDE module in modeling temporal dynamic information. When the MSLKA module is removed from the complete algorithm, the resulting TDE-VSwin algorithm retains only its temporal enhancement capabilities, weakening its multi-scale spatial feature extraction capabilities. While its performance is better than the pure backbone baseline VSwin, it is inferior to the complete algorithm that retains both modules and SP-VSwin that only retains MSLKA, ranking third among the four algorithms. This demonstrates the effectiveness of the MSLKA module in spatial feature optimization. The VSwin algorithm lacks targeted multi-scale spatial feature extraction and temporal dynamic enhancement capabilities, resulting in the weakest performance in fall detection tasks, with the lowest performance across all four evaluation metrics among the four groups of algorithms.

[0061] Comparative experiment To verify the effectiveness of SPTDE-VSwin, this invention employs several advanced fall detection methods for comparative experiments on the UR and Le2i datasets. These methods include those based on manual feature extraction and advanced deep learning methods, and the evaluation metrics from the paper are used. The experimental comparison results are shown in Tables 3 and 4. The experimental results show that, compared with existing mainstream methods, the method of this invention performs well in terms of accuracy, sensitivity, precision, and F-score, achieving competitive results. In comparison with methods based on manual features and traditional physical geometric features, such as the method of Beddiar et al. based on human geometry features, and the method of Dentamaro et al. combining human posture and kinematics theory, SPTDE-VSwin, unlike these traditional methods that rely on manually designed rules, automatically extracts features through deep networks. This saves on tedious manual extraction steps and is less susceptible to interference from image noise, lighting changes, and occlusion, greatly improving detection accuracy and robustness. In comparison with other deep learning methods, many existing cutting-edge algorithms heavily rely on pre-processed multimodal features: for example, Núñez et al. and Carlier et al. use optical flow features to capture inter-frame dynamic information, while GAO et al. and Raza et al. rely on skeleton sequences for spatial modeling. Unlike these algorithms that rely on additional features such as skeletons and optical flow, the model of this invention directly uses RGB frames as input without needing any additional data. Furthermore, when compared with network architectures based on visual data (Li et al. based on feature fusion pyramids, and the latest Cai et al. based on visual Transformers), the algorithm of this invention still achieves leading results. Finally, the algorithm of this invention performs stably on the UR and Le2i datasets. Even when faced with complex native scenes such as occlusion, lighting changes, and camera shake inherent in the datasets, it maintains excellent performance; the model also demonstrates fast convergence when faced with data augmentation techniques such as flipping and cropping. Moreover, the model of this invention uses pre-trained weights for initialization, indicating that the model has excellent feature representation capabilities and can adapt to different data distributions, not limited to the UR and Le2i datasets.

[0062] Table 1. SPTDE-VSwin ablation study and application of related algorithms in Le2i

[0063] Table 2. SPTDE-VSwin ablation study and application of related algorithms in UR.

[0064] Table 3 Comparison of SPTDE-VSwin with six advanced methods on Le2i

[0065] Table 4 Comparison of SPTDE-VSwin with five advanced methods on UR

[0066] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement, characterized in that, Includes the following steps: Obtain the video frame sequence to be detected, and perform 3D patch embedding and linear projection on the video frame sequence to obtain an initial spatiotemporal feature map; The initial spatiotemporal feature map is sequentially input into a hierarchical network consisting of multiple ST-Stage feature extraction stages to perform multi-stage spatiotemporal feature extraction. Each ST-Stage feature extraction stage first performs temporal enhancement preprocessing, and then performs iterative feature extraction through multiple cascaded encoder blocks. Window self-attention encoding is performed within each encoder block, and the output of each encoder block is subjected to spatial multi-scale enhancement postprocessing. The temporal enhancement preprocessing is used to enhance the temporal differences of the input features in a dual-stream manner. The window self-attention encoding process is based on the alternating combination of window multi-head self-attention and shifted window multi-head self-attention to model local and global spatiotemporal correlations. The spatial multi-scale enhancement postprocessing is used to expand the receptive field of the encoded output in multiple scales and enhance the long-range dependencies across the window space. The features output from the last ST-Stage feature extraction stage are sequentially processed by global adaptive average pooling and a fully connected classification layer to output a binary classification prediction result of falling or not falling. The fully connected classification layer has an input channel dimension of 768 and an output dimension of 2.

2. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 1, characterized in that, Each ST-Stage feature extraction stage includes a dual-stream temporal difference enhancement module, an encoder group, and a multi-scale large kernel attention module; Furthermore, each ST-Stage feature extraction stage, except for the first ST-Stage feature extraction stage, also includes a PatchMerge module, whose processing flow is as follows: The input features are first spatially downsampled and channel upsampled by the PatchMerge module, and the output features are downsampled. The downsampling features are input into the dual-stream temporal difference enhancement module to fully capture temporal dynamic information and output temporal enhancement features. The temporal enhancement features are iteratively encoded sequentially through multiple encoder blocks in the encoder group. Each encoder block consists of a window multi-head self-attention network, a shifted window multi-head self-attention network, and a feedforward network, and outputs encoded spatial features. The encoded spatial features output by each encoder block are fed into the multi-scale large kernel attention module for multi-scale spatial enhancement, outputting enhanced spatial features, which are then used as the input to the next encoder block or the final output of this stage.

3. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 1, characterized in that, The temporal enhancement preprocessing is implemented through a dual-stream temporal difference enhancement module, which includes a temporal offset branch, a multi-scale frame difference branch, and an adaptive fusion mechanism. Its processing flow is as follows: The input feature maps are fed into a temporal offset branch and a multi-scale frame difference branch for parallel processing. The temporal offset branch divides the input features into a forward offset group, a backward offset group, and an invariant group along the channel dimension. It completes inter-frame information interaction through feature shifting in the time dimension and outputs temporal correlation features. The multi-scale frame difference branch calculates the feature differences between the current frame and historical frames at multiple preset time intervals. After channel attention weighting, these differences are fused into multi-scale motion difference features. The temporal correlation features and the multi-scale motion difference features are input into the adaptive fusion mechanism. The adaptive fusion mechanism generates two dynamic fusion weights by extracting global context information and transforming two sets of independent channels. After normalizing the two fusion weights, they are weighted to the corresponding branch features. The weighted features are added element by element and batch normalized to output the fused temporal features. The input feature map is added to the fused temporal features using a residual connection method, and this is used as the final output of the dual-stream temporal difference enhancement module.

4. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 1, characterized in that, The spatial multi-scale enhancement post-processing is implemented through a multi-scale large-kernel attention module, which includes a hierarchical receptive field multi-branch extraction unit and a scale-adaptive feature aggregation unit. Its processing flow is as follows: The input feature map is fed in parallel into the three branches of the hierarchical receptive field multi-branch extraction unit. The three branches respectively use the equivalent receptive fields of small scale, medium scale and large scale to perform combined feature extraction of depth convolution and dilated convolution. At the end of each branch, a 3D linear attention module is embedded to enhance spatial long-range dependency, and output small-scale spatial features, medium-scale spatial features and large-scale spatial features respectively. The small-scale, medium-scale, and large-scale spatial features are concatenated along the channel dimension and input into the scale-adaptive feature aggregation unit. The scale weight distribution of each spatial location is predicted by channel compression and spatial convolution. The adaptive scale selection weight is obtained by normalization. The three branch features are weighted and summed using the adaptive scale selection weight to output multi-scale enhanced spatial features.

5. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 3, characterized in that, In the multi-scale frame difference branch, the multiple preset time intervals are specifically 1-frame interval, 2-frame interval and 4-frame interval, which are used to capture instantaneous dynamics, short-term evolution and long-term trends, respectively. The multi-scale frame difference branch calculates the feature difference between each frame and the preceding 1st, 2nd, and 4th frames to obtain the initial frame difference features at three scales. Channel-level statistics are extracted along the spatial dimension from the initial frame difference features at the three scales respectively. Channel attention weights are generated for each scale through channel transformation. The channel attention weights at each scale are multiplied element-wise with the initial frame difference features at the corresponding scale to obtain the weighted frame difference features at the three scales. The weighted frame difference features of the three scales are concatenated along the channel dimension and then compressed by convolution to output multi-scale motion difference features.

6. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 3, characterized in that, In the adaptive fusion mechanism, the specific method for normalizing the two dynamic fusion weights is as follows: divide the attention weights of the two channels by their sum and the sum of the preset anti-zero parameters, so that the sum of the two normalized weights in each channel dimension is 1, thereby achieving a dynamic and balanced distribution of the contribution of the two branches.

7. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 4, characterized in that, In the three branches of the hierarchical receptive field multi-branch extraction unit, a 7×7 depthwise convolution is used as the base convolutional layer, and then depthwise separable dilated convolutions with corresponding dilatancy rates are connected in series to achieve a three-level progressive equivalent receptive field of 7×7, 15×15, and 27×27.

8. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 4, characterized in that, The 3D linear attention module adopts a spatial-dimensional attention computation strategy, merging the batch and temporal dimensions, flattening the spatial dimension into a sequence, and keeping the channel dimension unchanged. By normalizing the query matrix along the spatial dimension and the key matrix along the feature dimension, the quadratic computational complexity of self-attention is transformed into linear computational overhead, realizing lightweight modeling of long-range spatial dependencies within each branch.

9. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 1, characterized in that, The hierarchical network adopts a four-stage architecture, with the number of encoder blocks in the four stages being 2, 2, 6, and 2, respectively. During the 3D patch embedding process, the temporal dimension is downsampled with a step size of 2, and the spatial dimension is downsampled with a step size of 4. Each 3D patch is mapped to a 96-dimensional feature space.

10. The fall detection algorithm based on multi-scale spatial and temporal dual-stream enhancement according to claim 2, characterized in that, The processing flow for each encoder block is as follows: the input features are sequentially connected to the first residual via window multi-head self-attention, the first feedforward network via the second residual, the shifted window multi-head self-attention network via the third residual, and the second feedforward network via the fourth residual, thus completing the feature extraction of one encoder block; the output features of the encoder block are fed into the multi-scale large kernel attention module for multi-scale spatial enhancement; multiple encoder blocks within the same ST-Stage are cascaded sequentially, iteratively optimizing the features round by round.

Images