Real-time multi-modal elderly behavior monitoring method and system based on pruning compression UniFormerV2

Abstract

The present application relates to the technical field of intelligent old-age care, in particular to a real-time multi-modal old-age care behavior monitoring method and system based on pruning and compression of UniFormerV2, which first performs spatiotemporal alignment and spatial mapping of dynamic time base sequence correction on visible light, depth and radar multi-source sensing data; then generates a multi-modal comprehensive token sequence through cross fusion of mode-specific embedded branches and sparse mask gating; then constructs a progressive pruning framework based on timing sensitivity guidance, performs grouping structured compression on the UniFormerV2 backbone network, and assists with distillation quantization repair to generate a fixed-point lightweight model; finally, it is deployed on the edge, and uses an adaptive confidence accumulation mechanism based on state residence perception to make timing smoothing decisions and hierarchical early warnings. The present application significantly reduces the model calculation amount while maintaining high-precision timing behavior discrimination ability, realizing real-time, robust high-risk behavior monitoring in the old-age care scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent elderly care technology, and in particular to a real-time multimodal elderly care behavior monitoring method and system based on the pruning and compression UniFormerV2. Background Technology

[0002] As the global population ages, the demand for intelligent monitoring of elderly people living alone and in nursing homes is becoming increasingly urgent. Behavioral monitoring, a core component of smart elderly care, aims to automatically identify and issue warnings for key behaviors in the daily lives of the elderly, such as falls, wandering, prolonged bed rest, and irregular eating habits. Currently, behavior recognition technology based on visual sensors has made significant progress. UniFormerV2, as a unified architecture integrating convolutional local perception capabilities and Transformer global modeling capabilities, has demonstrated excellent performance in action recognition tasks. However, in real-world elderly care scenarios, relying solely on RGB visual modalities has significant limitations: changes in lighting, viewpoint occlusion, and similar clothing textures among the elderly can easily lead to visual misjudgments; simultaneously, a single modality struggles to capture auxiliary vital signs such as subtle changes in respiration, heart rate, or depth.

[0003] In existing technologies, some solutions attempt to introduce multimodal data, such as combining RGB images with optical flow, depth maps, or millimeter-wave radar point clouds, to improve the robustness of monitoring. However, these solutions generally face two major engineering challenges: First, the lack of efficient mechanisms for temporal synchronization and feature alignment of multimodal heterogeneous data leads to extremely high feature redundancy after fusion. Second, Transformer-type models such as UniFormerV2 have a large number of parameters and high computational complexity, making them difficult to deploy directly on edge computing devices with limited computing power (such as embedded AI boxes or smart gateways). The inference latency cannot meet the second-level real-time response requirements for sudden accidents (such as falls) in the elderly. In addition, traditional model pruning methods often adopt a globally uniform sparsity strategy, ignoring the differences in the contribution of different temporal stages and different attention heads to behavioral semantics in UniFormerV2. This results in a break in the temporal continuity of key actions in the pruned model, leading to a significant degradation in recognition accuracy. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time multimodal elderly care behavior monitoring method and system based on pruning and compression UniFormerV2, which realizes real-time and robust monitoring of high-risk behaviors in elderly care scenarios.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a method for real-time multimodal elderly behavior monitoring based on pruning and compression UniFormerV2, comprising the following steps: Multimodal raw data streams, including visible light images, depth and distance information, and radar micro-motion echoes, are asynchronously captured by a multimodal sensor cluster. Spatiotemporal alignment processing is then performed on the multimodal raw data streams based on a dynamic time sequence correction strategy to generate a time-synchronized and spatially overlapping multimodal spatiotemporally aligned tensor sequence. The multimodal spatiotemporal aligned tensor sequences are fed into the lightweight embedding branch and mapped to a unified semantic feature space. Based on the sparse mask-guided gated cross-fusion mechanism, the mapped modal token sequences are selectively fused in the foreground region to generate a multimodal integrated embedded token sequence. A time-sensitive progressive pruning framework is constructed, and the standard UniFormerV2 backbone network is compressed to output a compact model after pruning. Using the original, unpruned UniFormerV2 network as the teacher network and the pruned compact model as the student network, a multi-level joint distillation loss containing an output semantic layer, an intermediate feature layer, and a temporal relation layer is constructed for knowledge transfer. Then, outlier-aware channel-wise fixed-point quantization calibration is performed to generate a fixed-point lightweight model. The fixed-point lightweight model is deployed on an edge computing terminal. The multimodal spatiotemporal aligned tensor sequence is continuously input in a sliding time window manner to perform forward inference and output the probability distribution of behavior categories. An adaptive confidence accumulation mechanism with state dwell awareness is introduced to perform temporal smoothing and steady-state judgment on the continuous inference results. Based on the hierarchical discrimination strategy, a multi-level early warning response is triggered for the high-risk behavior categories in the final judgment.

[0006] Specifically, a time-sensitive, progressive structural pruning framework is constructed, and the standard UniFormerV2 backbone network is compressed to output a compact, pruned model, including: First, the temporal sensitivity scores of each attention head and feedforward channel are calculated by combining the activation response temporal variance and the back gradient accumulation. Then, based on the temporal sensitivity scores, grouped structured pruning is performed and hardware alignment reduction is applied to the pruning quantity. Finally, progressive iterative pruning is performed according to the gradually increasing sparsity step size, and short-cycle weights are embedded for fine-tuning after each pruning step to output the pruned compact model.

[0007] The spatiotemporal alignment process specifically includes: using the visible light image frame rate as the main time reference, performing coarse-grained nearest neighbor time matching on the depth map and radar point cloud to obtain initial candidate frames; based on this, performing sub-frame-level displacement scanning within a narrow-band search window based on the temporal cross-correlation between the radar micro-motion energy envelope and the visible light texture change intensity coefficient, and using the displacement corresponding to the maximum cross-correlation value as the dynamic time offset compensation value to perform timestamp correction on the radar candidate frames; simultaneously, using the cross-correlation between the depth map and the visible light image in the spatial gradient to perform time offset optimization on the depth candidate frames, generating multimodal calibration frames with motion semantic synchronization; finally, projecting all modal calibration frames onto the visible light image pixel coordinate system through a pre-calibrated extrinsic parameter matrix to generate spatially registered multimodal alignment data.

[0008] The lightweight embedding branch includes: setting independent depth-separable convolutional subnetworks for texture feature maps, depth projection maps, and radar projection maps respectively; setting the kernel size and dilation rate of each branch according to the modal physical characteristics; and projecting the number of channels of each branch to the preset global embedding dimension through pointwise convolution; after completing the independent embedding, encoding the learnable spatial location of all modal tokens, and using the texture modal token sequence as the anchor reference sequence.

[0009] Specifically, the sparse mask-guided gated cross-fusion mechanism involves calculating the magnitude of the token vector at each spatial location and its directional consistency with neighboring vectors on the anchor reference sequence, generating a spatial saliency activation map, and then applying the activation map... Figure 2 The values ​​are converted into a sparse gated mask, marking only the token positions corresponding to the foreground motion region; the anchor tokens selected by the mask are used as the query end, and the tokens at the corresponding positions in the depth mode and radar mode are used as the key end to perform multi-head cross-attention operation, and a mode confidence weight factor dynamically adjusted according to the radar echo signal-to-noise ratio and depth map hole rate is introduced when the attention is weighted and summed; the cross-attention output is superimposed with the original anchor tokens using a learnable gated residual to generate the enhanced fusion tokens for the foreground region, and the original texture tokens are directly retained for the background region outside the mask. The multi-modal integrated embedded token sequence is output after being spliced ​​in spatial order.

[0010] The specific method for calculating the temporal sensitivity score is as follows: freeze the network weights, perform forward propagation using calibration samples, and capture the activation response variance of the attention distribution matrix of each multi-head self-attention unit along the temporal dimension, as well as the temporal variance of the activation value of each output channel of each feedforward multilayer perceptron unit; simultaneously perform backpropagation to track the projection matrix of each attention head and the gradient accumulation of each feedforward channel; multiply the normalized activation response variance by the gradient accumulation, and use the product as the temporal sensitivity score of the corresponding network substructure.

[0011] Specifically, the grouped structured pruning and hardware alignment specification involves: globally sorting all attention heads and feedforward channels from low to high according to the time sensitivity score; marking low-scoring objects as pruning candidates according to the preset global sparsity rate; then grouping the pruning candidates according to their respective UniBlock modules; rounding down the number of channels to be pruned within each group to an integer multiple of the alignment base that matches the single instruction multiple data stream width of the target edge processor; and outputting a pruning mask table that simultaneously satisfies the global sparsity requirement and the hardware dimension alignment requirement.

[0012] The process involves progressive iterative pruning with gradually increasing sparsity step sizes, followed by short-cycle weight fine-tuning after each pruning step, including: The target sparsity is divided into multiple progressively increasing sparsity steps. At each step, the low temporal sensitivity substructures corresponding to the current batch are removed according to the pruning mask table. Then, short-cycle fine-tuning is performed using the complete training dataset to restore the feature representation. Throughout the iteration process, the weight update rate of the attention heads and feedforward channels with the highest temporal sensitivity scores is set to extremely low values ​​to freeze their core structure until the final preset sparsity target is reached.

[0013] Among these measures, an adaptive confidence accumulation mechanism with state dwell awareness is introduced to perform temporal smoothing and steady-state decision-making on continuous inference results. Furthermore, based on a hierarchical discrimination strategy, multi-level early warning responses are triggered for high-risk behavior categories in the final decision, including: Maintain a fixed-length historical probability buffer queue, and perform a weighted moving average of the probability distribution of the current frame output and the historical values ​​in the queue to obtain a smooth confidence vector; only when the highest-scoring category in the smooth confidence vector exceeds the steady-state confidence threshold and the cumulative value of its corresponding state dwell timer reaches the preset dwell confirmation frame number threshold, will the final decision category switch be performed. When an alert is triggered, a high-risk event is determined based on the instantaneous spatial displacement change amount corresponding to the behavior category to trigger the highest-level alert, or a cumulative high-risk event is determined based on the cumulative number of consecutive frames maintained by the behavior category to trigger the second-highest-level alert; after the alert is triggered, local audible and visual alarms, peripheral device linkage, and asynchronous uploading of encrypted data packets to the cloud are executed simultaneously.

[0014] Secondly, the present invention provides a real-time multimodal elderly behavior monitoring system based on pruning and compression UniFormerV2, for implementing a real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as provided in the first aspect, comprising: The multimodal data acquisition front end includes a visible light imaging unit, a depth sensing unit, and a radar micro-motion sensing unit, which are used to asynchronously capture multimodal raw data streams including visible light images, depth and distance information, and radar micro-motion echoes through a multimodal sensor cluster. The edge-end main control processing module is connected to the multimodal data acquisition front end and is used to perform spatiotemporal alignment, embedding fusion, pruning compression, and distillation quantization processing to generate a fixed-point lightweight inference model. The system also includes a deployment inference and early warning execution module, which is used to deploy the fixed-point lightweight model on an edge computing terminal. The module continuously inputs the multimodal spatiotemporally aligned tensor sequence in a sliding time window manner to perform forward inference and output the probability distribution of behavior categories. It introduces an adaptive confidence accumulation mechanism with state dwell awareness to perform temporal smoothing and steady-state judgment on the continuous inference results. Based on the hierarchical discrimination strategy, it triggers multi-level early warning responses for high-risk behavior categories in the final judgment. The output of the deployment inference and early warning execution module is connected to a local sound and light alarm and a cloud management platform to form a closed-loop monitoring and early warning system.

[0015] This invention discloses a real-time multimodal elderly care behavior monitoring method and system based on pruning and compression of UniFormerV2. First, it performs spatiotemporal alignment and spatial mapping on multi-source sensor data (visible light, depth, and radar) with dynamic temporal sequence correction. Then, it generates a multimodal comprehensive token sequence through modality-specific embedding branches and sparse mask gating. Next, it constructs a time-sensitivity-oriented progressive pruning framework, performing grouped structured compression on the UniFormerV2 backbone network and supplementing it with distillation quantization repair to generate a fixed-point lightweight model. Finally, it is deployed at the edge, employing a state-resident awareness-based adaptive confidence accumulation mechanism for temporal smoothing decision-making and hierarchical early warning. This invention significantly reduces model computation while maintaining high-precision temporal behavior discrimination capabilities, achieving real-time and robust monitoring of high-risk behaviors in elderly care scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0017] Figure 1 This is a schematic diagram illustrating the steps of a real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 according to the first embodiment of the present invention.

[0018] Figure 2 This is a flowchart illustrating a real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 provided by the present invention.

[0019] Figure 3 This is a structural schematic diagram of a real-time multimodal elderly behavior monitoring system based on the pruning and compression UniFormerV2 according to the second embodiment of the present invention.

[0020] Figure 4 This is a schematic diagram of the electronic device of the present invention. Detailed Implementation

[0021] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0022] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0023] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0024] The first embodiment of this application is as follows: Please see Figures 1-3 This invention provides a real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2, comprising the following steps: S1. Asynchronously capture multimodal raw data streams containing visible light images, depth and distance information, and radar micro-motion echoes through a multimodal sensor cluster, and perform spatiotemporal alignment processing on the multimodal raw data streams based on a dynamic time sequence correction strategy to generate a time-synchronized and spatially overlapping multimodal spatiotemporally aligned tensor sequence.

[0025] Specifically, a multimodal front-end acquisition cluster is first deployed at fixed locations in the elderly care scenario. This cluster includes at least one visible light imaging unit, one time-of-flight depth sensing unit, and one millimeter-wave radar micro-motion sensing unit. When the system starts up, the edge-end main control chip broadcasts a unified initial reference clock to all acquisition units through a hardware synchronization signal line, enabling the internal clock counters of the three sensing units to complete initial zeroing and synchronized oscillation, thereby establishing a globally unique physical time reference system.

[0026] Once in the real-time operation phase, the three acquisition units independently capture data streams according to their inherent internal driving rhythms: the visible light imaging unit outputs a sequence of raw image frames containing color and texture information at a fixed nominal frame rate; the depth sensing unit outputs a sequence of depth map frames containing spatial distance information at a different nominal frame rate; and the radar micro-motion sensing unit outputs raw point cloud data packets containing target radial velocity, distance, and angle information at a higher pulse repetition frequency. At the instant each unit captures a complete frame of raw data, a high-precision timestamp marker at the edge reads the current global reference clock count as the physical capture time of that frame and binds it to the raw data in the form of metadata tags. Then, all the timestamped raw data is temporarily stored in an asynchronous input queue in memory. Because the trigger times and frame intervals of the three units are different, these raw data streams appear on the timeline as a sparse, discrete sequence of sampled points that are misaligned and have uneven density, rather than naturally aligned synchronous frame groups.

[0027] In order to integrate the aforementioned asynchronous and misaligned multimodal raw data streams into a time-synchronized and spatially registered tensor sequence that can be directly input into the backbone network, this invention abandons the traditional fixed interpolation framework and adopts a two-stage spatiotemporal alignment strategy with motion perception capabilities. This strategy consists of three progressive steps: coarse-grained time window matching, fine-grained motion-driven time basis correction, and unified spatial coordinate mapping.

[0028] In the coarse-grained time window matching stage, the system uses the output frame rate of the visible light imaging unit as the master control time reference, defining the acquisition time of each visible light image frame as a target alignment time point. For each target alignment time point, the system retrieves the timestamp tags of the depth data and radar data from the asynchronous input queue, and extracts the original data frame with the smallest absolute time difference from the target alignment time point, as the initial candidate frame for that mode at this alignment time. After this stage, each target alignment time point obtains an original triplet consisting of the visible light image, the depth candidate frame, and the radar candidate frame. However, the triplet still contains microsecond-level residual time offsets caused by the inherent trigger delays and transmission jitter of each unit. Directly using these offsets will lead to motion phase disorder in the fused features.

[0029] In the fine-grained motion-driven temporal sequence correction stage, this invention introduces a dynamic time-off compensation mechanism based on radar micro-motion energy peak detection. Specifically, the system first uses several adjacent frames of the visible light image frame as references to calculate the overall texture change intensity coefficient in the spatial gradient domain, while simultaneously extracting the Doppler velocity spectrum energy distribution of radar candidate frames in the vicinity of the time axis. Then, a narrow-band search window centered on the current target alignment time is constructed, within which sub-frame-level micro-displacement scanning is performed on the original radar timestamp. For each displacement, the system calculates the temporal cross-correlation between the radar micro-motion energy envelope after displacement and the visible light texture change intensity coefficient, and determines the displacement corresponding to the maximum cross-correlation value as the optimal dynamic time-off compensation value of the radar mode relative to the visible light mode at the current alignment time. This compensation value directly reflects the fixed phase lag caused by the difference in the electronic shutter activation delay between the two sensors. After obtaining the optimal compensation value, the system does not perform simple numerical interpolation on the radar data. Instead, it performs overall offset correction on the radar point cloud timestamps based on the time offset compensation value, and uses the corrected timestamps to re-extract or weightedly combine the nearest valid frames in the original radar sequence, thereby generating a radar calibration frame that is truly synchronized with the current visible light time in terms of motion semantics. Similarly, by leveraging the spatial gradient cross-correlation between the depth map and the visible light image on the edge contours, a similar small time offset optimization is performed on the depth candidate frames. However, since the depth map and visible light both belong to the optical imaging category, the search range of its time offset compensation value is much smaller than that of the radar modes. Through this correction process, the data of each mode are not only close in terms of physical clock, but also highly consistent in terms of the phase of motion changes.

[0030] The width of the narrowband search window is set to three original radar sampling intervals before and after the current target alignment time. Subframe-level displacement scanning is performed within this window with a step size of one-tenth of the original sampling interval. The search range for the temporal offset compensation value of the depth candidate frame is set to one depth map frame interval before and after, with a step size of one-fifth of the frame interval. When the displacement corresponding to the maximum temporal cross-correlation value between the radar micro-motion energy envelope and the visible light texture change intensity coefficient exceeds a preset temporal offset tolerance threshold, the system determines that there is a significant sensor asynchronous mismatch at the current moment. In this case, the target alignment time point is marked as a temporal offset anomalous frame, and a lower fusion confidence weight is assigned to the multimodal data unit at that moment during subsequent fusion processes to prevent mismatched data from contaminating feature representation.

[0031] After completing the fine alignment in the temporal dimension, the system proceeds to the unified spatial coordinate mapping stage. This stage pre-establishes a fixed transformation relationship between the extrinsic parameter matrices of the depth sensing unit and radar sensing unit relative to the visible light imaging unit through joint calibration experiments. In real-time processing, the system does not perform real-time iterative optimization calculations. Instead, it directly utilizes the calibrated extrinsic parameter matrices and perspective projection transformation formulas to project the distance values ​​represented by each pixel in the depth map to their corresponding positions in the visible light image pixel coordinate system, generating a spatially registered dense depth projection map. For radar point clouds, they are projected onto the same two-dimensional image plane based on their ranging and angular measurement information. Weighted aggregation is then performed based on the reflection intensity and radial velocity of multiple radar point clouds falling within the same pixel region, generating a sparse radar feature projection map that includes micro-motion velocity attributes. After spatial mapping, the data from all modalities are normalized to an image coordinate system with the same two-dimensional spatial resolution.

[0032] The perspective projection transformation uses bilinear interpolation to fill in the non-integer coordinate pixels generated during the depth map projection process. When multiple scattering points fall into the same pixel grid when the radar point cloud is projected onto the image plane, the system performs weighted convergence based on the product of the reflection intensity and the absolute value of the radial velocity at each point. The weighted average of the principal components of the three point clouds with the largest product is used as the final velocity attribute value for that pixel location, while the remaining point cloud information is discarded. This controls the sparsity of the radar feature projection map to remain stable within a preset range.

[0033] After completing the aforementioned two-stage spatiotemporal alignment and spatial mapping, the system obtains a set of strictly synchronized and spatially overlapping multimodal data units for each target alignment time point. These units include a visible light texture map, a depth projection map, and a radar feature projection map. To ensure the numerical stability of subsequent network training and eliminate the bias effect caused by differences in the dimensions of each modality, the system performs zero-mean and unit-variance normalization operations on these three projection maps, respectively. Subsequently, at the edge, multimodal normalized data units spanning several frames are extracted from the circular buffer at a fixed number of consecutive alignment time points. The three modal projection maps in these units are then simply stacked along the channel dimension to construct a spatiotemporally continuous multimodal fundamental tensor sequence as the final output, which is then used by the subsequent lightweight embedding layer for further high-order semantic extraction.

[0034] S2. The multimodal spatiotemporal aligned tensor sequences are fed into the lightweight embedding branch respectively, mapped to a unified semantic feature space, and the mapped modal token sequences are selectively interacted and fused in the foreground region based on the sparse mask-guided gating cross-fusion mechanism to generate a multimodal integrated embedded token sequence.

[0035] Specifically, given the fundamental differences in data form and physical meaning among the three types of input modalities—texture feature maps reflect apparent color and gradients, depth projection maps carry spatial distance and geometric contours, and radar micro-motion projection maps characterize velocity energy and weak displacement—this method abandons a unified convolutional mapping approach and constructs three independent but structurally similar lightweight embedding branches specific to each modality. Each branch uses depthwise separable convolution as the basic operator to control the number of parameters, but the size of the convolutional kernel, dilation rate, and number of stacked layers within each branch are differentiated according to the modal characteristics.

[0036] For texture feature maps, which are information-dense and rich in high-frequency details, their dedicated embedding branch employs a standard-sized depthwise separable convolution kernel with a small dilation rate to maintain the fine structure of local textures, initially extracting edge response features of color and shape. For depth projection maps, whose numerical distribution reflects a continuous range field with sharp edges and gentle interior regions, their branch, based on depthwise separable convolution, embeds an additional Laplacian gradient-guided attention mask, forcing the branch to prioritize the preservation of geometrically abrupt regions where the human silhouette meets the ground during mapping, suppressing redundant range information in flat backgrounds. For radar micro-motion projection maps, whose effective echo points exhibit high spatial sparseness and concentrated energy, their branch employs a depthwise separable convolution kernel with a large dilation rate to expand the spatial receptive field, thereby aggregating and diffusing the discrete scattering point echo energy to the surrounding continuous region, generating a smooth velocity response heatmap.

[0037] After the initial extraction of the differentiated features, each of the three branches passes through a pointwise convolutional layer, projecting the number of feature channels within each branch onto a preset global embedding dimension. At this point, the three modalities achieve normalized alignment of numerical scale in the channel dimension. However, since the distribution range of the feature values ​​is still affected by their respective physical dimensions, this method introduces a modality-specific learnable scaling factor and bias term at the end of the projection layer. This scaling factor adaptively adjusts the contribution ratio of the activation amplitude of different modal features, ensuring that during subsequent interactions, no modality will dominate attention calculations due to excessively large numerical magnitudes, nor will it be submerged in noise due to excessively small numerical magnitudes. Ultimately, it outputs texture embedding feature maps, depth embedding feature maps, and micro-motion embedding feature maps with consistent spatial resolution but independent physical semantics.

[0038] After completing the independent embedding mapping, the three feature maps need to be converted into a token sequence format that can be processed by the attention mechanism. In this method, instead of simply flattening the feature maps into unordered tokens during serialization, spatial location priors are explicitly injected. Specifically, a set of shared, learnable coordinate codes corresponding to the spatial grid of the feature maps is first generated. This code table is universally applicable to all three modalities, and its physical meaning is the absolute positional identifier on the two-dimensional image plane. When the three embedded feature maps are respectively unfolded into two-dimensional token matrices, the system synchronously superimposes these shared coordinate codes into the token vector of each spatial location in each modality using additive embedding. This ensures that during subsequent cross-modal interactions, tokens located at the same spatial grid point in different modalities can establish a strong correspondence based on the same absolute positional code, thereby avoiding spatial semantic mismatch caused by disordered tokenization order.

[0039] Subsequently, the system arranges the three sets of token sequences carrying positional codes independently along the sequence length dimension. This method uses the token sequence corresponding to the texture embedding feature map as the anchor reference sequence in the subsequent fusion process because the texture modality contains the richest spatial semantic category information, which facilitates guiding other modalities to perform targeted feature supplementation.

[0040] To generate the final integrated embedded token sequence, this method does not employ full multi-head self-attention to allow all modal tokens to mix freely globally, as full interaction would introduce a large number of invalid associations from background or noisy regions, increasing the computational burden at the edge. Instead, this method designs a gated cross-fusion mechanism guided by a priori sparse masks.

[0041] The mechanism first performs a lightweight spatial saliency self-evaluation on the anchor point reference sequence. Specifically, the system calculates the magnitude of the token vector at each spatial location in the texture embedding feature map and its directional consistency with adjacent token vectors, thereby generating a spatially saliency activation map. Regions with higher response values ​​in this map identify the human foreground and regions of interest with significant motion or texture changes. Subsequently, based on a preset retention ratio, this activation map is... Figure 2 The values ​​are converted into a sparse gated mask, which only marks the foreground token positions with the most discriminative value. The consistency of the magnitude and direction of the token vectors at each position in the spatial saliency activation map is determined by the following criteria: the direction cosine similarity between each token vector and the token vectors in its eight neighborhoods is calculated, and the direction consistency is defined as the proportion of neighborhoods with a similarity greater than 0.7; the normalized vector magnitude and the direction consistency are multiplied to obtain the saliency score of each spatial position; according to the preset retention ratio, the saliency scores of the entire image are sorted from high to low, and the score value corresponding to the sorting quantile is used as the binarization threshold. Token positions above the threshold are marked as foreground motion regions, and those below or equal to the threshold are marked as background regions.

[0042] After obtaining the sparse gating mask, the system performs a three-step progressive fusion. First, the embedded token sequences of the depth modality and the micro-motion modality are spatially gating based on the sparse gating mask, sending only the tokens corresponding to the mask-marked positions into subsequent interaction calculations, while a large number of background tokens outside the mask are directly masked, significantly reducing the total number of tokens participating in the interaction. Second, the system uses the tokens selected by the mask in the anchor point reference sequence as the query end, and simultaneously uses the corresponding selected tokens in the depth modality as the first source of the key-value pair and the corresponding selected tokens in the micro-motion modality as the second source of the key-value pair, performing a multi-head cross-attention operation. In this operation, the geometric structural features from depth and the micro-motion velocity features from radar are projected into independent sets of key-value pairs, and attention weights are calculated with the same query vector for each. It is worth noting that this method introduces a modal confidence weighting factor, which is specifically calculated based on the radar echo signal-to-noise ratio (SNR) and depth map hole rate of the current frame. The effective lower limit of the radar echo SNR is set to 1 / 2 decibel. When the measured SNR is lower than this limit, the confidence weighting factor of the radar mode decreases to zero linearly. The depth map hole rate is set to a threshold of 30%. When the hole rate exceeds this threshold, the confidence weighting factor of the depth mode also decreases linearly. Under normal signal conditions, the ratio of the basic weighting factors of the radar mode to the depth mode is set to 1:1.5 to strengthen the dominant role of depth geometric information in fusion. Both weighting factors are calculated independently at each frame inference time and apply to the weighted summation stage of the multi-head cross-attention operation. This factor dynamically adjusts the contribution ratio of the two key pairs in the attention-weighted summation based on the radar echo signal-to-noise ratio and the depth map hole rate of the current frame: when the radar echo is weak, the output weight of the radar key pair is reduced; when there are large areas of invalid holes in the depth map, the influence of the depth mode is actively reduced, and the fusion focus is automatically shifted to the mode with better signal quality.

[0043] The third step involves gating the enhanced fusion vector obtained from the cross-attention calculation and then performing residual superposition with the original texture token vector at the corresponding position in the anchor point reference sequence. This gating is controlled by a lightweight, learnable scalar parameter to balance the retention ratio between the original texture information and the supplementary fusion information. After these three steps, the system generates an enhanced fusion token that aggregates texture appearance, distance geometry, and micro-motion velocity attributes for each foreground spatial location selected by the mask. For background spatial locations outside the mask, the system directly retains the original texture token vector without any modality mixing to maintain a stable representation of the background environment. The learnable scalar parameter is set to zero at the initial stage of training, ensuring that the gating residual superposition completely retains the original texture token information in the early stages of training, avoiding drastic oscillations in the fusion features due to modality distribution differences in the early stages of training. During training, this scalar parameter employs an independent learning rate scheduling strategy, with its update rate set to one-tenth of the network's main learning rate, to suppress excessively rapid changes in the fusion weights and ensure that the cross-attention enhancement information is injected into the anchor point sequence in a slow and gradual manner.

[0044] Finally, the system rearranges and concatenates all foreground enhancement fusion tokens and original background texture tokens according to their original spatial order, forming a complete multimodal integrated embedding token sequence with the same length as the total number of pixels in the input feature map. This sequence deeply couples the complementary advantages of multimodal modes in key human activity regions while maintaining efficient single-modal lightweight representation in non-critical background regions, providing a high-purity and computationally clear input foundation for the subsequent feature encoding and behavior discrimination of the backbone network.

[0045] S3. Construct a progressive structural pruning framework based on time sensitivity, perform compression processing on the standard UniFormerV2 backbone network, and output a compact model after pruning.

[0046] Specifically, while keeping the pre-training weights completely frozen, a small number of representative elderly care behavior calibration sample sets are used to quantitatively evaluate the time-series dynamic expression capabilities of each internal substructure in the complete UniFormerV2 backbone network.

[0047] The system first feeds the multimodal synthesis embedded token sequences from the calibration sample set into an unpruned UniFormerV2 network and performs a complete forward and backward propagation process. During the forward propagation, for each multi-head self-attention unit in the network, the system does not focus on the magnitude of its final output. Instead, it extracts the attention distribution matrix generated when the input sequence is weighted and aggregated at different time steps, and calculates the activation response variance after expanding this matrix along the temporal dimension. A larger variance indicates that the attention head exhibits significant focus switching between different frames, meaning it has strong temporal discriminative power regarding action evolution. Simultaneously, for each feedforward multilayer perceptron unit in the network, the system captures activation values ​​at the output of each channel within the network and similarly calculates its temporal variance.

[0048] In the subsequent backpropagation process, the system tracks the cumulative gradient of the projection matrix corresponding to each feedforward network channel and each attention head. This cumulative gradient not only reflects the contribution of this substructure to the final classification loss, but more importantly, when the gradient exhibits a continuous and consistent propagation trend in the temporal direction, it indicates that this substructure plays an indispensable bridging role in maintaining the temporal coherence of actions.

[0049] After completing the forward and backward computations, the activation variance of each network substructure in its corresponding temporal dimension is normalized and multiplied by the accumulated backward gradient. This product is defined as the temporal sensitivity score of that substructure. This score takes into account both the forward response magnitude and the backward error propagation, accurately distinguishing which attention heads and feedforward channels truly participate in the encoding of motion dynamics and which are merely overfitting responses to a static spatial background. Finally, the system assigns a temporal sensitivity score between zero and one to all multi-head self-attention heads and feedforward multilayer perceptron channels in the network. A higher score indicates that the substructure is more critical to modeling the continuous movements of the elderly.

[0050] The activation response temporal variance and back gradient accumulation are scaled using their respective maximum values ​​across all network substructures during normalization, mapping them to a value range of zero to one. The temporal sensitivity score obtained after the product operation is further logarithmically compressed to suppress the impact of the order-of-magnitude difference between extremely high and low scores on subsequent ranking. For substructures with temporal sensitivity scores below one-thousandth, the system directly marks them as static background response structures, placing them at the very end of the subsequent ranking and excluding them from grouping reduction protection. Let the activation response temporal variance of the i-th network substructure (attention head or feedforward channel) be v. i The back gradient accumulator is g i The system calculates the maximum value of each of the two values ​​across all N substructures: , ; Mini-maximum normalization is performed on each substructure: , ; The normalized activation variance is multiplied by the gradient accumulator, and the product is then logarithmically compressed to obtain the temporal sensitivity score S of the substructure. i : ; Logarithmic compression is used to suppress the order-of-magnitude difference between extremely high and extremely low scores, preventing a few extremely high-scoring structures from dominating subsequent pruning and sorting. For substructures with Si < 0.001, the system directly marks them as static background response structures and places them at the very end of the global ranking.

[0051] Based on the timing sensitivity scores of all substructures obtained in the first stage, this stage is responsible for selecting the set of low-contribution substructures to be pruned according to the preset overall sparsity target, and at the same time completing the adaptation specification of the pruning mode to the edge hardware.

[0052] The system first globally sorts all multi-head self-attention heads and feedforward perceptron channels according to their temporal sensitivity scores from low to high, forming a sequence to be evaluated. Based on a pre-set global sparsity rate, such as retaining 60% of the total parameters, the system sequentially marks objects to be pruned starting from the low-scoring end of the sequence, while marking the high-scoring end as objects to be forcibly retained. However, directly performing isolated pruning channel-by-channel or individual attention head-by-attention head pruning according to this fine-grained sorting result will disrupt the regularity of the network tensor dimensions, causing additional padding overhead in matrix multiplication operations by the edge processors due to dimension misalignment, severely reducing inference efficiency.

[0053] To address this, the present invention introduces a grouping reduction strategy. Specifically, the system groups all candidate objects to be pruned according to their respective UniBlock modules. Within each group, the system counts the total number of channels to be pruned and rounds this total down to an integer multiple of a preset alignment base, which is set to four or eight based on the single instruction multiple data stream width of the target edge processor. Subsequently, within each group, the system, based on the proximity principle of timing sensitivity scores, additionally removes several low-scoring objects corresponding to non-integer multiples of the difference, until the actual number of prunes within the group reaches an integer multiple of the alignment base. Although this operation slightly sacrifices a very small number of boundary substructures that were originally intended to be retained, it ensures that all tensor dimensions of the entire network are perfectly aligned with the hardware memory access bit width during inference, maximizing cache utilization and instruction pipeline efficiency. After this stage, the system outputs a deterministic pruning mask table that satisfies both the global sparsity target and the hardware alignment requirements. This mask table clearly indicates which attention heads and which feedforward channels in each UniBlock module will be permanently removed.

[0054] To avoid drastic shifts in network feature representation and accuracy collapse caused by removing a large number of substructures at once, this invention adopts a multi-step progressive pruning strategy and embeds a short-cycle adaptive fine-tuning process between each pruning operation. This is the decisive factor in ensuring the final accuracy of the compressed model.

[0055] The system divides the pruning objective into multiple progressively increasing sparsity steps. The objective is set to remove 40% of the total parameters, and this objective is divided into eight progressively increasing sparsity steps. The first four steps each increase the pruning ratio by 3%, and the last four steps each increase the pruning ratio by 7%, approaching the final sparsity target at a gradual pace. The number of iterations for each short-cycle fine-tuning step is set to one-tenth of the total training rounds, and the learning rate is set to one percent of the initial pre-training learning rate. When the model's behavior recognition accuracy on the validation set drops by more than one percentage point after a certain step fine-tuning compared to before fine-tuning, the system automatically rolls back the pruning operation for that step and halves the sparsity increment before re-executing. For example, each step increases the pruning ratio by 10% compared to the previous step. At the first step, the system removes the lowest temporal sensitivity substructure corresponding to the current batch according to the pruning mask table, obtaining a preliminary sparsified intermediate model. At this point, the system immediately uses the complete elderly care behavior training dataset to perform a limited number of weight fine-tuning iterations on the intermediate model. During this fine-tuning process, the system adopts a low learning rate to avoid drastic oscillations. At the same time, the weight update rates of the top five attention heads and feedforward channels in the first stage of temporal sensitivity scores are set to extremely low values, which almost freezes these core structures, thereby ensuring that the foundation of the model's overall dynamic modeling ability is not disturbed by the fine-tuning process.

[0056] After fine-tuning the current step size, the system moves on to the next long pruning iteration: based on the currently fine-tuned and stable intermediate model, it continues to remove the next batch of low-sensitivity substructures and performs short-cycle fine-tuning again. This iterative process, with each alternation of pruning and fine-tuning, allows the model to gradually approach the final sparsity target while continuously readjusting the collaborative relationships between the remaining substructures through data-driven methods, effectively bridging the representation gaps caused by structure removal. Throughout the progressive process, the system consistently enforces hard protection constraints: all attention heads with top-ranking temporal sensitivity scores do not participate in any pruning candidate step during the entire iteration process, ensuring that even at the highest sparsity, the network retains a sufficient number of the most discriminative temporal coding channels.

[0057] After iterating through all step sizes, the system finally outputs a highly sparse UniFormerV2 model that maximizes the retention of temporal dynamic expression capabilities. Compared to the original network, this model achieves a significant reduction in both the number of parameters and computational cost. At the same time, the remaining distribution of its attention heads and feedforward channels strictly follows the hardware alignment convention, laying a solid foundation for subsequent accuracy compensation and fixed-point deployment with a compact structure and robust performance.

[0058] S4. Using the original unpruned UniFormerV2 network as the teacher network and the pruned compact model as the student network, a multi-level joint distillation loss including an output semantic layer, an intermediate feature layer, and a temporal relation layer is constructed for knowledge transfer. Then, outlier-aware channel-by-channel fixed-point quantization calibration is performed to generate a fixed-point lightweight model.

[0059] Specifically, the original, unpruned, high-precision UniFormerV2 model is used as the teacher network, and the pruned, compact model output in the third step is used as the student network. The distillation process does not simply transfer the output probabilities of the teacher network to the student network. Instead, it constructs a three-level progressive distillation loss system from the output semantic layer to the intermediate feature layer and then to the temporal relation layer. By introducing a temporal relation distillation term, the student network is forced to still be able to mimic the dynamic perception of the action evolution process of the teacher network after compression.

[0060] At the output semantic layer, the system performs forward inference on the same batch of multimodal input samples using both the teacher and student networks, obtaining their original logical outputs across all elderly care behavior categories. Subsequently, a temperature coefficient softening process is applied to this logical output, making the probability distribution smoother and revealing the teacher network's subtle discriminative tendencies towards easily confused behavior categories. The system calculates the difference between the softened category probability distributions of the student network and the teacher network, defining it as the soft-label distillation loss. Simultaneously, the system preserves the standard cross-entropy between the student network output and the actual behavior labels, defining it as the hard-label discriminative loss to ensure that the student network does not deviate from the labeled information of the actual samples. The soft-label distillation loss is responsible for conveying the teacher network's prior knowledge of inter-class similarity, while the hard-label discriminative loss is responsible for maintaining the responsiveness to basic facts; together, they constitute the basic supervision at the output level.

[0061] In the intermediate feature layer, the system selects the output response maps of several key spatiotemporal transformation modules at corresponding positions in the teacher and student networks as feature alignment anchors. Since the number of channels in the teacher and student networks is no longer consistent due to pruning, at each selected alignment anchor, the system adds a learnable linear projection adapter to the student network's feature map, temporarily mapping its channel dimension to the same size as the teacher network. Subsequently, the system calculates the element-wise cumulative difference between the mapped student and teacher feature maps and defines it as the feature map distillation loss. To enhance attention to foreground motion regions, this cumulative difference is not uniformly weighted but spatially weighted according to the sparse gating mask generated in the first step. This ensures that the fitting error in the human foreground region dominates the loss, while the fitting error in the background region is significantly attenuated, thus concentrating the limited distillation capacity on the core regions relevant to behavior discrimination.

[0062] At the temporal relation layer, this invention introduces a dynamic relation distillation term. This design stems from an insight into the essence of behavior recognition—actions are not only manifested as spatial poses in a single frame, but also as state transition trajectories between frames. The system extracts the attention transfer matrices of the teacher network and the student network at adjacent time steps. This matrix describes the attention mapping relationship between each spatial position in the current frame and each spatial position in the next frame, essentially characterizing the motion evolution path on the spatiotemporal manifold. Subsequently, the system calculates the difference between the teacher and student on this attention transfer matrix and uses this difference value as the temporal relation distillation loss. The existence of this loss enables the student network to not only learn to imitate the static response of the teacher network to each frame, but also to learn to imitate the dynamic reasoning trajectory of the teacher network in the temporal dimension, thereby effectively preventing misjudgments of action abrupt changes caused by temporal expression discontinuities in the pruned model.

[0063] Finally, the system assigns preset balanced weight coefficients to the soft-label distillation loss, hard-label discriminative loss, feature map distillation loss, and temporal relation distillation loss, and then sums them to obtain the comprehensive distillation loss function. During distillation training, the teacher network weights are completely frozen, and only the student network weights and adapter parameters are updated. By minimizing this comprehensive loss, the pruned compact model maintains a lightweight size while maximizing the inheritance of the teacher network's dual composite discriminative capabilities in spatial semantics and temporal dynamics.

[0064] The balancing weight coefficients for the soft-label distillation loss, hard-label discrimination loss, feature map distillation loss, and temporal relation distillation loss are set as follows: soft-label distillation loss coefficient is 1.0, hard-label discrimination loss coefficient is 0.5, feature map distillation loss coefficient is 0.8, and temporal relation distillation loss coefficient is 1.2. The temporal relation distillation loss coefficient is set to the highest value to highlight the core role of temporal dynamics throughout the distillation process. The temperature coefficient is set to four to balance distribution smoothness and semantic distinguishability during the category probability softening process.

[0065] The system denotes the teacher network as T and the student network as S. The distillation loss is composed of a weighted sum of four components, with the total distillation loss being L. total The expression is: ; Among them, L soft For soft label distillation loss, L hard For hard label discrimination loss, L feat For the feature map distillation loss, L temp For the time-series distillation loss, the balance weight coefficients are set as α=1.0 (soft label distillation loss), β=0.5 (hard label discrimination loss), γ=0.8 (feature map distillation loss), and δ=1.2 (time-series distillation loss).

[0066] After knowledge distillation, the pruned compact model still maintains floating-point precision, making it unable to efficiently utilize the integer acceleration instructions provided by the edge neural processing units. To address this, an end-to-end fixed-point quantization deployment process was designed. This process consists of three closely linked sub-steps: static statistical calibration, quantization-aware fine-tuning, and inference graph optimization and fusion. It introduces an outlier-aware, channel-by-channel scaling factor selection strategy to avoid the propagation of quantization truncation errors caused by retaining large activation values ​​in some channels of the pruned network.

[0067] In the static statistical calibration sub-stage, the system selects a small batch of calibration sample sets covering various typical action scenarios, places the distilled student network in evaluation mode, and performs several forward inference iterations. During this process, the system collects the activation value distribution histograms of each output channel in each layer in a layer-by-layer, channel-by-channel manner. Unlike conventional methods that directly use the absolute maximum value as the scaling basis, this invention first performs outlier detection on the distribution histogram of each channel to identify sporadic maxima points far from the main distribution cluster. These maxima points usually originate from the accidental resonance between the large norm weights remaining after pruning and specific input patterns. If these are included in the scaling range, it will cause the activation of most regular amplitudes to be squeezed into an extremely low resolution range, resulting in severe accuracy degradation. Therefore, the system adopts a quantile statistical strategy, removing extreme outlier percentiles at both ends, and then using the absolute maximum value of the effective numerical range as the benchmark scaling amplitude for that channel. At the same time, the system also performs the same channel-by-channel outlier removal and amplitude statistical operation on the weight tensor of each layer to ensure that the scaling factor can cover the main numerical range while ignoring harmful outlier interference. The outlier detection employs a two-sided quantile removal strategy, removing numerical samples below the 1% quantile and above the 99% quantile in the activation value distribution histogram of each channel. The absolute maximum value of the remaining valid numerical range is then used as the baseline scaling magnitude for that channel. For the weight tensor, a more stringent two-sided 0.5% quantile removal is used. The forward inference process on the calibration sample set is performed in ten rounds. In each round, the scaling magnitude for each channel is calculated independently. The final baseline scaling magnitude for that channel is the exponential moving average of the ten rounds' results, thus eliminating statistical bias caused by accidental input fluctuations in a single forward inference iteration.

[0068] After obtaining the baseline scaling magnitude for each layer and channel, the system proceeds to the quantization-aware fine-tuning stage. Pseudo-quantized nodes are inserted at the input and output of each convolutional and fully connected layer in the distilled student network. These nodes simulate fixed-point integer rounding and saturation truncation operations during forward propagation, but employ a pass-through estimator during backpropagation to maintain the continuity of gradient propagation. The system uses the weights at the end of the distillation stage as initial values ​​and performs a small number of iterations of fine-tuning training using the elderly behavior training dataset. During this fine-tuning process, the system employs a gradual annealing strategy: initially, pseudo-quantization noise is relatively high, forcing the model to adapt to the low-bit numerical space; later, the pseudo-quantization noise gradually decreases, allowing the weights to find a local optimum within the fixed-point representation range. Specifically, for the core attention heads marked as having the highest temporal sensitivity in the third step, the system sets the update step size of their quantization scaling factor to an extremely low value and prunes their gradients during backpropagation to prevent the fine-tuning process from destroying the accurate temporal alignment capabilities already learned by these key structures due to quantization noise. The gradual annealing strategy uses the total number of iterations for fine-tuning as a reference. In the first 30% of iterations, pseudo-quantization noise is kept fully enabled. In the middle 40% of iterations, pseudo-quantization noise is gradually reduced linearly. In the final 30% of iterations, pseudo-quantization noise is reduced to a minimum maintenance level. The gradient pruning threshold for the core attention head with the highest time sensitivity is set to three times the root mean square value of the layer's weights. Gradients exceeding this threshold are scaled proportionally to prevent excessive disturbance to the core structure during the quantization fine-tuning stage.

[0069] After quantization-aware fine-tuning, the system enters the inference graph optimization and fusion sub-stage. This stage no longer involves training but focuses on graph transformation operations for actual deployment. The system pre-folds and merges the parameters (including mean, variance, scaling factor, and bias) of each batch-normalized layer in the fine-tuned model into the convolutional kernel weights and biases of the previous layer according to mathematical equivalence rules, thereby eliminating the computational overhead of batch normalization during inference. After batch normalization fusion, the system truly quantizes the floating-point values ​​of all weight tensors and activation tensors in the entire model into eight-bit fixed-point integer values ​​based on the previously determined channel-by-channel scaling factor and zero-point offset. All pseudo-quantized nodes are then removed from the computation graph, ultimately exporting a lightweight inference model file composed entirely of integer operators. When this model is loaded at the edge, it can be directly mapped to the processor's fixed-point matrix multiply-accumulate unit for execution, without any runtime dynamic quantization conversion, thus achieving extremely high forward inference speed. Thus, the pruned, compressed, and fixed-point UniFormerV2 model fully meets the engineering requirements of the elderly care scenario for real-time performance and ease of deployment.

[0070] S5. Deploy the fixed-point lightweight model on an edge computing terminal, continuously input the multimodal spatiotemporal aligned tensor sequence in a sliding time window manner to perform forward inference and output the probability distribution of behavior categories, introduce an adaptive confidence accumulation mechanism with state dwell awareness to perform temporal smoothing and steady-state judgment on the continuous inference results, and trigger multi-level early warning response for high-risk behavior categories in the final judgment according to the hierarchical discrimination strategy.

[0071] Specifically, after the edge terminal starts, it continuously executes a circular cache update strategy. The system extracts the most recent aligned multimodal spatiotemporal tensor sequences from the circular cache described in the first step, according to a preset fixed time window span, as input samples for the current inference time. This tensor sequence is fed into an 8-bit integer pruning compression model for forward inference. The compressed backbone encoder extracts multi-scale spatiotemporal features, which are then aggregated into a global description vector using global average pooling. Finally, the fully connected classification layer at the end outputs a raw logical value vector with a dimension equal to the preset total number of behavior categories.

[0072] Since the model has been fully quantized into integer operations, the forward inference process is efficiently completed in the fixed-point matrix multiplication and addition unit of the edge processor, and the latency of a single inference is strictly controlled within the millisecond range. The system inputs the original logical value vector into a lightweight normalized exponential function, which converts it into a probability distribution vector with clearly defined physical meaning. Each element in this vector represents the initial confidence level of the input sample belonging to each preset elderly care behavior category within the current time window. At this point, only the instantaneous discrimination result of a single inference is obtained, and temporal context constraints have not yet been introduced.

[0073] To eliminate accidental misjudgments caused by instantaneous sensor noise, brief human posture fluctuations, or sudden changes in lighting within a single frame or time window, this invention abandons the traditional simple exponential smoothing method and designs a state-resident perception confidence accumulation mechanism that combines the advantages of short-term response speed and long-term stability. This mechanism maintains two core data structures in the edge memory: a fixed-length historical probability cache queue and an array of state-resident timers that count independently for each category.

[0074] At each new inference time step, the system first pushes the current output probability distribution vector into the historical probability cache queue and pops the oldest probability vector from the head of the queue. Then, the system calculates the weighted moving average of all probability vectors in the queue, assigning higher weights to probability vectors closer to the current time step, thus smoothing out noise while ensuring sensitive tracking of sudden actions. The result of the weighted moving average is defined as the smoothed confidence vector, serving as the initial decision criterion for the current time step.

[0075] At this point, the system does not directly use the category corresponding to the maximum value in the smooth confidence vector as the final decision. Instead, it introduces state dwell awareness logic. Specifically, the system marks the category with the highest score in the current smooth confidence vector as the candidate behavior category and compares its corresponding score with a pre-set steady-state confidence threshold. If the candidate category's score is lower than the threshold, the system considers the current evidence insufficient, does not perform any behavior state switch, and resets the state dwell timer corresponding to the candidate category to zero, maintaining the previously stable behavior decision result. If the candidate category's score is higher than the threshold, the system further checks the state dwell timer of the candidate category: if the timer has not yet reached the preset dwell confirmation frame threshold, it only increments the timer by one frame, but still does not switch the final decision result, continuing to maintain the behavior category output at the previous moment; only when the candidate category's score is consistently higher than the threshold and its corresponding state dwell timer's accumulated value reaches the dwell confirmation frame threshold, does the system officially update the final decision result to the candidate behavior category and reset the dwell timers of all other categories to zero.

[0076] The core advantage of this state-dwelling perception mechanism lies in its imposing duration constraints on the triggering of high-risk behaviors. A brief, instantaneous forward lean or rapid hand wave will not be judged as a fall due to a single high probability event; only abnormal postures that persist for a sufficient duration and maintain a consistently high probability will trigger a state switch. Simultaneously, because the dwell timer for low-risk background categories is quickly reset to zero when scores fluctuate, the model can rapidly recover from erroneous transient trends, avoiding cumulative lag effects. This approach effectively filters out random false alarms without sacrificing the sensitivity to detect genuine, persistent dangerous behaviors.

[0077] After obtaining the final judgment behavior category confirmed by the state-based perception system, the system compares this category with a predefined set of high-risk behaviors. This set of high-risk behaviors covers situations that pose a direct threat to the life safety of the elderly, such as falling postures, prolonged immobility in bed, and abnormal body curling. If the final judgment category does not belong to this high-risk set, the system only records the current behavior status in the log, without triggering any warning actions, and keeps the cache queue and timer status updated normally.

[0078] The preset elderly care behavior categories include, but are not limited to, seven high-risk behaviors: falling, wandering, prolonged bed rest, abnormal stillness, getting out of bed, sitting for a long time, and abnormal nocturnal activities. Among them, falling and abnormal nocturnal activities are set as first-level instantaneous high-risk events, and their early warning response link is to synchronously trigger an emergency call by local audible and visual alarms; prolonged bed rest, abnormal stillness, and sitting for longer than the preset cumulative duration are set as second-level cumulative high-risk events, and their early warning response link is a local nursing station reminder combined with cloud message push; wandering and getting out of bed are set as third-level general risk events, and their early warning response link is local log recording combined with cloud trend statistics, without triggering audible and visual alarms. Each category corresponds to an independent steady-state confidence threshold, a residence confirmation frame number threshold, and a cumulative duration warning line. Among them, the residence confirmation frame number threshold for the falling category is set to the lowest value to ensure the fastest response, and the cumulative duration warning line for the sitting-for-a-long-time category is set to the longest value to avoid frequent false alarms in normal rest scenarios.

[0079] If the final judgment category hits the high-risk set, the system further executes a hierarchical discrimination strategy. The present invention divides the urgency of high-risk behaviors into two levels: the first level is an instantaneous high-risk event, including behaviors with obvious sudden impact characteristics such as sudden falls, etc., and its judgment basis is that after the state residence is confirmed, the inter-frame spatial displacement mutation amount of this category exceeds the preset severe threshold, and the system directly enters the highest-level early warning process; the second level is a cumulative high-risk event, including prolonged stillness or continuous abnormal postures, etc., and its judgment basis is that after the judgment of this behavior category remains stable, the system queries the cumulative continuous frame number that this behavior has continuously maintained. When the continuous frame number exceeds the cumulative duration warning line preset for the corresponding category, it enters the second-highest-level early warning process.

[0080] The hierarchical discrimination mechanism enables the early warning system to distinguish between two different types of dangerous modes: sudden impact and gradual deterioration, achieve zero-delay response to sudden situations, give a short observation window to gradual physical abnormalities to avoid frequent false alarms due to the normal rest or slow movements of the elderly, and at the same time ensure that true continuous abnormalities will not be missed due to being misjudged as benign.

[0081] When the hierarchical discrimination determines that an early warning needs to be triggered, the edge terminal immediately executes a locally prioritized linkage response action. The system first activates the audible and visual alarm of the access terminal, emits a beep and flash warning of a specific frequency to attract the immediate attention of on-site personnel or the elderly themselves. At the same time, the system sends a trigger signal to local peripheral devices such as intelligent nursing beds and emergency call button panels through the general input / output interface to reserve an interface for possible subsequent automatic intervention.

[0082] Simultaneously with initiating local alarms, the system asynchronously initiates cloud communication. The edge terminal extracts keyframe thumbnails before and after the triggering of the alarm from the currently aligned multimodal data cache—preferably texture maps containing significant foreground contours and corresponding depth projection maps. These two images are then structured and packaged with metadata information such as the alarm behavior category label, the triggering time, and the cumulative confidence score confirmed by state residency during the final judgment of that category. Subsequently, the system sends the alarm data packet to the remote cloud management platform via an encrypted network transmission protocol and stores an encrypted log backup locally. After receiving the data, the cloud platform can further perform long-term data mining, trend statistics, and remote medical care scheduling. The entire alarm reporting process runs as a low-priority background task, without affecting the real-time processing of subsequent continuous inference tasks by the edge terminal's main thread. This ensures a loosely coupled parallel and collaborative relationship between the triggering of alarm actions and the inference pipeline, ultimately forming a complete closed-loop elderly care behavior monitoring system from perception, analysis, judgment to linkage response.

[0083] The edge computing terminal adopts an embedded AI platform with an integrated neural network processing unit (NPU), including but not limited to edge smart gateways based on ARM architecture and equipped with deep learning accelerators, NVIDIA Jetson series edge computing modules, or domestic AI chip platforms such as Rockchip and Horizon Robotics. The edge inference engine adopts a low-level operator library optimized for fixed-point matrix multiply-accumulate units, and all convolution and fully connected operations are executed in 8-bit integer fixed-point format. Actual testing shows that on a typical hardware platform, the single forward inference latency of the 8-bit integer pruned and compressed UniFormerV2 model can be controlled within 15 milliseconds, the end-to-end behavior discrimination throughput reaches more than 35 frames per second, and the model storage volume is compressed to less than 20% of the original floating-point model, fully meeting the real-time response requirements for sudden high-risk behaviors such as falls in elderly care scenarios.

[0084] The second embodiment of this application is as follows: Please see Figure 3 This invention provides a real-time multimodal elderly behavior monitoring system based on pruning and compression UniFormerV2, applied to a real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as provided in the first embodiment, comprising: The multimodal data acquisition front end includes a visible light imaging unit, a depth sensing unit, and a radar micro-motion sensing unit, which are used to asynchronously capture multimodal raw data streams including visible light images, depth and distance information, and radar micro-motion echoes through a multimodal sensor cluster. The edge-end main control processing module is connected to the multimodal data acquisition front end and is used to perform spatiotemporal alignment, embedding fusion, pruning compression, and distillation quantization processing to generate a fixed-point lightweight inference model. The system also includes a deployment inference and early warning execution module, which is used to deploy the fixed-point lightweight model on an edge computing terminal. The module continuously inputs the multimodal spatiotemporally aligned tensor sequence in a sliding time window manner to perform forward inference and output the probability distribution of behavior categories. It introduces an adaptive confidence accumulation mechanism with state dwell awareness to perform temporal smoothing and steady-state judgment on the continuous inference results. Based on the hierarchical discrimination strategy, it triggers multi-level early warning responses for high-risk behavior categories in the final judgment. The output of the deployment inference and early warning execution module is connected to a local sound and light alarm and a cloud management platform to form a closed-loop monitoring and early warning system.

[0085] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0086] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0087] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described above. Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is part of a real-time multimodal elderly behavior monitoring system based on pruning and compression UniFormerV2 provided in an embodiment of the present invention. (Except for...) Figure 4 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0088] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the real-time multimodal elderly behavior monitoring method based on the pruned and compressed UniFormerV2 described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0089] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0090] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2, characterized in that, Includes the following steps: Multimodal raw data streams, including visible light images, depth and distance information, and radar micro-motion echoes, are asynchronously captured by a multimodal sensor cluster. Spatiotemporal alignment processing is then performed on the multimodal raw data streams based on a dynamic time sequence correction strategy to generate a time-synchronized and spatially overlapping multimodal spatiotemporally aligned tensor sequence. The multimodal spatiotemporal aligned tensor sequences are fed into the lightweight embedding branch and mapped to a unified semantic feature space. Based on the sparse mask-guided gated cross-fusion mechanism, the mapped modal token sequences are selectively fused in the foreground region to generate a multimodal integrated embedded token sequence. A time-sensitive progressive pruning framework is constructed, and the standard UniFormerV2 backbone network is compressed to output a compact model after pruning. Using the original, unpruned UniFormerV2 network as the teacher network and the pruned compact model as the student network, a multi-level joint distillation loss containing an output semantic layer, an intermediate feature layer, and a temporal relation layer is constructed for knowledge transfer. Then, outlier-aware channel-wise fixed-point quantization calibration is performed to generate a fixed-point lightweight model. The fixed-point lightweight model is deployed on an edge computing terminal. The multimodal spatiotemporal aligned tensor sequence is continuously input in a sliding time window manner to perform forward inference and output the probability distribution of behavior categories. An adaptive confidence accumulation mechanism with state dwell awareness is introduced to perform temporal smoothing and steady-state judgment on the continuous inference results. Based on the hierarchical discrimination strategy, a multi-level early warning response is triggered for the high-risk behavior categories in the final judgment.

2. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 1, characterized in that, A time-sensitive progressive pruning framework is constructed, and the standard UniFormerV2 backbone network is compressed to output a compact pruned model, including: First, the temporal sensitivity scores of each attention head and feedforward channel are calculated by combining the activation response temporal variance and the back gradient accumulation. Then, based on the temporal sensitivity scores, grouped structured pruning is performed and hardware alignment reduction is applied to the pruning quantity. Finally, progressive iterative pruning is performed according to the gradually increasing sparsity step size, and short-cycle weights are embedded for fine-tuning after each pruning step to output the pruned compact model.

3. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 1, characterized in that, The spatiotemporal alignment process specifically includes: using the visible light image frame rate as the main time reference, performing coarse-grained nearest neighbor time matching on the depth map and radar point cloud to obtain initial candidate frames; based on this, performing sub-frame-level displacement scanning within a narrow-band search window based on the temporal cross-correlation between the radar micro-motion energy envelope and the visible light texture change intensity coefficient, and using the displacement corresponding to the maximum cross-correlation value as the dynamic time offset compensation value to perform timestamp correction on the radar candidate frames; simultaneously, using the cross-correlation between the depth map and the visible light image in the spatial gradient to perform time offset optimization on the depth candidate frames, generating multimodal calibration frames with motion semantic synchronization; finally, projecting all modal calibration frames onto the visible light image pixel coordinate system through a pre-calibrated extrinsic parameter matrix to generate spatially registered multimodal alignment data.

4. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 1, characterized in that, The lightweight embedding branch includes: setting up independent depth-separable convolutional subnetworks for texture feature maps, depth projection maps, and radar projection maps respectively. The kernel size and dilation rate of each branch are set differently according to the modal physical characteristics, and the number of channels of each branch is uniformly projected to the preset global embedding dimension through pointwise convolution. After completing the independent embedding, the learnable spatial position encoding of all modal tokens is superimposed and shared, and the texture modal token sequence is used as the anchor reference sequence.

5. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 4, characterized in that, The sparse mask-guided gated cross-fusion mechanism is as follows: calculate the magnitude of the token vector at each spatial location and its directional consistency with the neighboring vector on the anchor point reference sequence, generate a spatial saliency activation map, and binarize the activation map into a sparse gated mask to mark only the token position corresponding to the foreground motion region; Using the anchor token selected by the mask as the query end, and the tokens at corresponding positions in the depth mode and radar mode as the key value end, a multi-head cross-attention operation is performed. When the attention is weighted and summed, a modal confidence weight factor that is dynamically adjusted according to the radar echo signal-to-noise ratio and the depth map hole rate is introduced. The cross-attention output is superimposed with the original anchor token using a learnable gated residual to generate an enhanced fusion token for the foreground region. The original texture token is directly retained in the background region outside the mask. The tokens are then spliced ​​together in spatial order and the multi-modal integrated embedded token sequence is output.

6. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 2, characterized in that, The specific method for calculating the temporal sensitivity score is as follows: freeze the network weights, perform forward propagation using calibration samples, and capture the activation response variance along the temporal dimension of the attention distribution matrix of each multi-head self-attention unit, as well as the temporal variance of the activation values ​​of each output channel of each feedforward multilayer perceptron unit; simultaneously perform backpropagation to track the projection matrix of each attention head and the gradient accumulation of each feedforward channel; multiply the normalized activation response variance by the gradient accumulation, and use the product as the temporal sensitivity score of the corresponding network substructure.

7. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 2, characterized in that, The grouped structured pruning and hardware alignment specification is as follows: All attention heads and feedforward channels are globally sorted from low to high according to the time sensitivity score, and low-scoring objects are marked as candidates to be pruned according to the preset global sparsity rate; then the candidates to be pruned are grouped according to their respective UniBlock modules, and within each group, the number of channels to be pruned is rounded down to an integer multiple of the alignment base that matches the single instruction multiple data stream width of the target edge processor, and a pruning mask table that simultaneously meets the global sparsity requirement and the hardware dimension alignment requirement is output.

8. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 2, characterized in that, A progressive iterative pruning process is performed using progressively increasing sparsity steps, with short-cycle weight fine-tuning embedded after each pruning step, including: The target sparsity is divided into multiple progressively increasing sparsity steps. At each step, the low temporal sensitivity substructures corresponding to the current batch are removed according to the pruning mask table. Then, short-cycle fine-tuning is performed using the complete training dataset to restore the feature representation. Throughout the iteration process, the weight update rate of the attention heads and feedforward channels with the highest temporal sensitivity scores is set to extremely low values ​​to freeze their core structure until the final preset sparsity target is reached.

9. The real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 1, characterized in that, An adaptive confidence accumulation mechanism with state dwell awareness is introduced to perform temporal smoothing and steady-state decision-making on continuous inference results. Furthermore, based on a hierarchical discrimination strategy, multi-level early warning responses are triggered for high-risk behavior categories in the final decision, including: Maintain a fixed-length historical probability buffer queue, and perform a weighted moving average of the probability distribution of the current frame output and the historical values ​​in the queue to obtain a smooth confidence vector; only when the highest-scoring category in the smooth confidence vector exceeds the steady-state confidence threshold and the cumulative value of its corresponding state dwell timer reaches the preset dwell confirmation frame number threshold, will the final decision category switch be performed. When an alert is triggered, a high-risk event is determined based on the instantaneous spatial displacement change amount corresponding to the behavior category to trigger the highest-level alert, or a cumulative high-risk event is determined based on the cumulative number of consecutive frames maintained by the behavior category to trigger the second-highest-level alert; after the alert is triggered, local audible and visual alarms, peripheral device linkage, and asynchronous uploading of encrypted data packets to the cloud are executed simultaneously.

10. A real-time multimodal elderly behavior monitoring system based on pruning and compression UniFormerV2, used to implement the real-time multimodal elderly behavior monitoring method based on pruning and compression UniFormerV2 as described in claim 1, characterized in that, include: The multimodal data acquisition front end includes a visible light imaging unit, a depth sensing unit, and a radar micro-motion sensing unit, which are used to asynchronously capture multimodal raw data streams including visible light images, depth and distance information, and radar micro-motion echoes through a multimodal sensor cluster. The edge-end main control processing module is connected to the multimodal data acquisition front end and is used to perform spatiotemporal alignment, embedding fusion, pruning compression, and distillation quantization processing to generate a fixed-point lightweight inference model. The system also includes a deployment inference and early warning execution module, which is used to deploy the fixed-point lightweight model on an edge computing terminal. The module continuously inputs the multimodal spatiotemporally aligned tensor sequence in a sliding time window manner to perform forward inference and output the probability distribution of behavior categories. It introduces an adaptive confidence accumulation mechanism with state dwell awareness to perform temporal smoothing and steady-state judgment on the continuous inference results. Based on the hierarchical discrimination strategy, it triggers multi-level early warning responses for high-risk behavior categories in the final judgment. The output of the deployment inference and early warning execution module is connected to a local sound and light alarm and a cloud management platform to form a closed-loop monitoring and early warning system.

Images