Neuromorphic visual fall detection method based on spatio-temporal graph modeling and regional distillation
The neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation solves the problem of insufficient reflection of event flow spatiotemporal features in existing technologies, and achieves efficient and accurate fall recognition, which is suitable for embedded devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SICHUAN UNIV
- Filing Date
- 2026-06-09
- Publication Date
- 2026-07-14
AI Technical Summary
Existing neuromorphic visual fall detection methods are unable to fully reflect the spatiotemporal distribution characteristics of event streams, resulting in insufficient recognition accuracy and an inability to simultaneously meet the requirements of low power consumption and efficient processing.
A neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation is adopted. By constructing a lightweight student network for bi-branch spatiotemporal relationship modeling and hierarchical downsampling aggregation, and combining the category response distillation and region attention distillation of the teacher network, the responsiveness to key regions is enhanced.
It improves the accuracy and stability of fall detection, is suitable for deployment in embedded and low-power devices, and can effectively characterize complex spatiotemporal features and enhance behavioral modeling capabilities.
Smart Images
Figure CN122392136A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of neuromorphic visual perception technology, specifically to a neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation. Background Technology
[0002] Fall detection is an important research area in human behavior recognition, with significant application value in scenarios such as home care for the elderly, medical monitoring, elderly care services, and security monitoring. Existing fall detection methods utilize sensors that can be broadly categorized into wearable sensors, fixed vision sensors, environmental sensors, and multi-sensor fusion sensors.
[0003] Visual methods, which do not require the monitored subject to actively wear devices and can directly perceive human movement, have become an important research route for fall detection. Current visual fall detection typically uses RGB or RGB-D cameras to acquire videos of human behavior and combines them with models such as convolutional neural networks, 3D convolutional networks, pose estimation networks, and recurrent neural networks to analyze and classify human movements. While these methods can extract information on human posture and movement changes to a certain extent, they generally suffer from significant data redundancy, susceptibility to motion blur in fast movements, sensitivity to changes in lighting and complex backgrounds, and heavy real-time processing burdens, thus limiting their application in edge device deployments and privacy-sensitive scenarios.
[0004] With the development of neuromorphic visual perception technology, event cameras have begun to be used for human motion recognition and fall detection tasks. Unlike traditional frame-based cameras, event cameras do not capture complete images at a fixed frame rate. Instead, each pixel independently senses brightness changes and asynchronously outputs event signals when changes occur. Therefore, they feature high temporal resolution, low latency, low redundancy, high dynamic range, and low power consumption. Furthermore, because event cameras typically do not directly record complete texture and appearance information, they also offer advantages in privacy protection, making them more suitable for applications such as home care, hospital monitoring, and edge intelligence monitoring. Existing research has already incorporated event cameras into human motion recognition and published neuromorphic visual datasets for fall detection, demonstrating that event vision has a practical application basis in this field.
[0005] Because the data output by event cameras differs from traditional images—it is essentially an asynchronously triggered, spatiotemporally sparse, and irregularly distributed event stream—existing methods often transform this data using event frames, time surfaces, or other regular tensors before processing to adapt it to conventional visual network structures. While this approach reduces the complexity of model design, the spatially sparse yet temporally intensive nature of the event stream means that effective information in the samples is often concentrated in locally active regions. Therefore, existing methods still struggle to fully reflect the spatiotemporal distribution characteristics of the event stream in human behavior tasks like fall detection, which involve significant changes in movement. This is especially true when modeling based on regular grids, where the utilization of local structural patterns, dynamic changes, and contextual information within the event stream remains insufficient.
[0006] Furthermore, while neuromorphic visual recognition tasks place higher demands on the feature representation capabilities of models, they also require lightweight and efficient processing for deployment on embedded and low-power devices. However, most existing fall detection technologies cannot simultaneously meet the requirements of low power consumption and efficient processing while maintaining accuracy. Summary of the Invention
[0007] To address the aforementioned shortcomings in existing technologies, the neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation provided by this invention solves the problem that existing solutions cannot fully reflect the spatiotemporal distribution characteristics of the event flow itself, resulting in the inability to guarantee recognition accuracy.
[0008] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows: A neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation is provided, comprising the following steps: S1. Obtain the asynchronous event stream of human fall and non-fall behaviors, perform voxelization on the asynchronous event stream and construct a spatiotemporal relationship graph; S2. Input the spatiotemporal relationship graph into the lightweight student network and perform a forward process of bi-branch spatiotemporal relationship modeling and hierarchical downsampling aggregation to global semantic convergence and classification output. S3. Input the time surface sequence formed by the event flow into the teacher network, and transfer the class discrimination knowledge and key area attention knowledge of the teacher network to the lightweight student network through category response distillation and region attention distillation. S4. Obtain the event stream to be identified, and after voxelizing it, obtain the node set. Then, input it into the optimized lightweight student network for fall recognition, and output the prediction result of fall class or non-fall class.
[0009] Furthermore, the lightweight student network is a hierarchical spatiotemporal relationship modeling network based on graph neural networks, and step S2 further includes: S21. For lightweight student networks, in its first... In the set of layer nodes, the node to be updated is designated as the center node. Non-central nodes are recorded as candidate nodes. In the branch In the middle, the central node of the calculation With nodes Spatiotemporal distance metric The values are 1 and 2, representing the local neighborhood branch and the context neighborhood branch, respectively. S22. Based on the spatiotemporal distance metric, in the branch The central node is constructed using the K-nearest neighbor method. The neighborhood of the central node is determined, and the relative spatiotemporal relationship between the central node and its neighboring nodes is determined. S23. Perform relation-aware fusion on the neighborhood node features and corresponding relative spatiotemporal relationships of the two branches respectively, and then fuse the outputs of the two relation-aware fusions to obtain the dual-branch fusion representation. S24, By linear projection, the first The two-branch fusion representation of the layer is mapped back to the first layer. The feature space of the layer yields the central node. In the The updated node representation after layer update; S25, in the Select the first node from the set of layer nodes Multiple central nodes of the layer are identified, and their neighborhoods are constructed. The features of the neighborhood nodes and their coordinate offsets relative to the central nodes are concatenated and then a neighborhood aggregation operation is performed. S26. After the neighborhood aggregation operation, the features are input into a shared transformation function for feature mapping. All feature mapping representations of the same central node are then aggregated to obtain the... The node representation of the layer's center node; S27. For all layers of the lightweight student network, the bi-branch spatiotemporal relationship modeling in steps S21 to S24 and the hierarchical downsampling aggregation in steps S25 to S26 are alternately updated to obtain the node representation of each layer. S28. Perform average pooling and max pooling on the representations of all nodes in the last layer of the lightweight student network, and concatenate the two to obtain the sample-level global feature representation, which is then input into the classification head.
[0010] Furthermore, in the branches In the middle, the central node of the calculation With nodes The expression for the spatiotemporal distance metric is: in, For branches Central node With candidate nodes The spatiotemporal distance between them; and They are the central nodes. With nodes In the The spatial coordinates of the layer; and They are the central nodes. With nodes Time coordinates; and Branches The scale parameters in the spatial and temporal dimensions are learnable parameters with an initial value of 1.
[0011] Furthermore, methods for fusing the outputs after perceptual fusion of the two branches include: Based on the gating fusion mechanism, the weights of the two branches are adaptively estimated using the features of the current node: in, and They are the central nodes. Fusion weights on local neighborhood branches and contextual neighborhood branches; As the central node In the Layer feature representation; Represents the gated mapping function; This indicates a normalization operation; According to weight Merge local neighborhood branches and contextual neighborhood branches: in, As the central node In the Layered two-branch fusion representation; and They are the central nodes. The output after relation-aware fusion in local neighborhood branches and contextual neighborhood branches; This is a feature splicing operation.
[0012] Furthermore, the expression for the sample-level global feature representation is: in, It represents global features at the sample level; and These are average pooling and max pooling operations, respectively. For feature splicing operations; For the first The node at the th Layer feature representation; For the first The total number of nodes in the layer.
[0013] Furthermore, the teacher network is a spatiotemporal multi-scale representation learning network, which includes: The multi-scale feature encoding module is used to receive event frames from the asynchronous event stream, progressively model the dynamic information in different time ranges, and obtain time multi-scale features. The spatial multi-scale feature fusion module is used to receive temporal multi-scale features to integrate spatial structure information at different levels and obtain high-level spatiotemporal semantic features that take into account both local details and global semantics. The pooling and classification mapping module is used to obtain the class response information of the teacher network by pooling and classifying high-level spatiotemporal semantic features; The category activation map generation module is used to process high-level spatiotemporal semantic features to obtain category activation maps.
[0014] Furthermore, methods for transferring category-based knowledge and key region-focused knowledge from teacher networks to lightweight student networks include: In the category response distillation stage, the category responses of the teacher network and the lightweight student network are temperature scaled to obtain the softening probability distribution; based on the softening probability distribution, the target class distillation loss is constructed using a decoupling method. In the teacher network and lightweight student network, the category responses of all categories except the target category are renormalized to obtain the softening probability output, and then the non-target class distillation loss is constructed based on the softening probability output. We construct a category distillation loss for lightweight student networks to learn teacher networks at the category level, using both target class distillation loss and non-target class distillation loss. During the region attention distillation phase, the teacher network outputs a category activation graph, mapping the region attention information of the teacher network to the node space of the last layer of the lightweight student network, thus obtaining the importance distribution on the teacher side. ; The lightweight student network outputs the importance responses of each node in the last layer, which are then normalized to obtain the distribution of student-side nodes. ; Importance distribution Student-side node distribution The teacher network is constructed to transfer the knowledge of key discriminative regions to the regional distillation loss in the lightweight student network.
[0015] Furthermore, the overall loss function of lightweight student networks for: , in, To monitor and categorize losses; For category distillation loss; This represents regional distillation losses; and These are the weighting coefficients for category distillation loss and region distillation loss, respectively. For target type distillation loss; This is a non-target type of distillation loss; and This is the balance coefficient; Temperature coefficient; Let be the divergence function.
[0016] Furthermore, the method for voxelizing asynchronous event stream data and constructing a spatiotemporal graph further includes: S11. Normalize the timestamps of individual events in the asynchronous event stream, and discretize the asynchronous event stream along the spatial and temporal dimensions, mapping it to a three-dimensional voxel grid. S12. Record the number of events in each non-empty voxel in the 3D voxel mesh, and sort all non-empty voxels in descending order of the number of events. S13, When the total number of non-empty voxels ≥ Preset number of nodes At that time, adopt the former A set of nodes is composed of non-empty voxels, when < At that time, fill in the non-empty voxels to the required level. Each of these forms a node set; S14. Perform local spatiotemporal encoding on the events inside the voxels in the node set, so that each voxel is represented as a local spatiotemporal response tensor, and then mapped to an initial node embedding of a uniform dimension, i.e., a spatiotemporal relation graph.
[0017] Furthermore, the method for local spatiotemporal encoding of voxel internal events in a node set further includes: S141. Calculate the local spatial coordinates and local temporal coordinates of any event within a voxel in the node set: in, and Voxels Inner One event Local spatial coordinates within a voxel; , and Voxels Position index in spatial width, spatial height, and time dimension; for In voxels Relative position on the internal local timeline; and They are respectively Original spatial coordinates; , , These represent the dimensions of a voxel unit in the spatial width, spatial height, and time dimension, respectively. for Normalized time coordinates voxels The number of local time channels obtained by further dividing the interior along the time dimension; S142. Construct voxel feature tensors along the local time axis using linear interpolation: in, voxels In local time and local spatial location voxel feature tensor at the location; The polarity of the event; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. To obtain the maximum value; It is the absolute value symbol; voxels The set of events contained therein; S143. Map the voxel feature tensor to the initial node embedding through 3D convolution and pooling operations: in, For the first Each node is embedded in the initial node of the input layer, with superscript... Indicates the input layer; For the first The voxel feature tensor corresponding to each node; It is a voxel feature encoding mapping composed of three-dimensional convolution and pooling.
[0018] The beneficial effects of this invention are as follows: This solution combines event flow spatiotemporal relationship graph modeling with region attention distillation, which can better characterize the complex spatiotemporal features of neuromorphic visual fall actions. By constructing a bi-branch spatiotemporal neighborhood relationship modeling mechanism in a lightweight student network, relation-aware aggregation of local neighborhood relationships, contextual neighborhood relationships, and their dynamic interaction processes is performed. This enables the lightweight student network to more effectively characterize the local structural relationships, cross-regional dynamic associations, and action evolution features in human actions, thereby improving the ability to model complex behaviors in fall detection tasks.
[0019] Furthermore, this invention introduces category response distillation and region attention distillation from the teacher network, transferring the teacher network's category discrimination knowledge and key region attention knowledge to the lightweight student network, thereby enhancing the lightweight student network's responsiveness to key discrimination regions. Therefore, this invention effectively addresses the problems of insufficient utilization of local spatiotemporal relationships in event flows and limited representation capabilities of key discrimination regions in existing technologies, improving the accuracy and stability of fall detection while controlling model complexity. Since the lightweight student network uses a lightweight neural network, it is also suitable for deployment in embedded and low-power devices.
[0020] This scheme transforms the irregular and sparsely distributed event stream into a compact, stable, and scalable set of event nodes by performing voxel aggregation, active node filtering (i.e., node set filtering), and node feature encoding on the original asynchronous event stream. This reduces redundant input while preserving the local active regions with significant event responses. Attached Figure Description
[0021] Figure 1 This is a flowchart of a neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation.
[0022] Figure 2 This is a block diagram illustrating the principle of local spatiotemporal encoding of events within voxels in a node set.
[0023] Figure 3 This is a schematic diagram of the sampling aggregation at the farthest point in this scheme.
[0024] Figure 4 This is a general framework diagram of a neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation. Detailed Implementation
[0025] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.
[0026] refer to Figure 1 , Figure 1 A flowchart of a neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation is shown; Figure 1 As shown, the method S includes steps S1 to S4.
[0027] In step S1, an asynchronous event stream of human falls and non-fall behaviors is acquired, the asynchronous event stream is voxelized, and a spatiotemporal relationship graph is constructed. The asynchronous event stream in this scheme can use a publicly available neuromorphic visual fall dataset as the data source. This dataset is acquired by a DAVIS346redColor event camera with a resolution of 346×260 and saved in .aedat format. The asynchronous event stream can be represented as: in, For input event stream; For the first 1 event; I is the total number of event streams; Represents pixel coordinate information; Indicates the time when the event occurred; Indicates the polarity of the event.
[0028] In this embodiment, the neuromorphic visual fall dataset contains behavioral recordings of 15 subjects in an indoor office environment. The actions include four categories: falling, bending, slumping down, and tying-shoes. Falling is labeled as a falling action, and the other actions are uniformly labeled as non-falling actions. Each action lasts for about 5 seconds and is repeated 3 times, resulting in a total of 180 recorded samples.
[0029] In one embodiment of the present invention, the method for voxelizing asynchronous event stream data and constructing a spatiotemporal graph further includes: S11. Normalize the timestamps of individual events in the asynchronous event stream: in, Indicates the first One event The time coordinates after normalization; and These represent the earliest and latest event times in the current sample, respectively. It represents the upper bound of the range after the time dimension is discretized.
[0030] Discretize the asynchronous event stream along the spatial and temporal dimensions: in, Indicates the first One event Corresponding voxel index; This indicates the floor function; Through the above mapping, the original asynchronous event stream is organized into several local spatiotemporal units in a three-dimensional voxel grid, that is, the three-dimensional voxel grid is obtained by mapping.
[0031] The number of non-empty voxels obtained after voxelization varies with the sample, and some voxels correspond only to weak event responses. If all voxels are directly used as graph nodes, it will not only lead to inconsistencies in input scale between different samples, but also introduce more low-response regions, increasing the computational burden of subsequent graph structure modeling. Therefore, this scheme further selects representative active voxels from the voxel set formed by the 3D voxel mesh to form a graph node set. The specific implementation process is as follows: steps S12 to S13.
[0032] S12. Record the number of events in each non-empty voxel in the 3D voxel mesh, and sort all non-empty voxels in descending order of event count: in, This represents the non-empty voxel sequence sorted in descending order of event count. This represents the sorted non-empty voxels. This represents the total number of non-empty voxels in the current sample. These represent the number of events in the corresponding voxel.
[0033] S13, When the total number of non-empty voxels ≥ Preset number of nodes At that time, adopt the former A set of nodes is composed of non-empty voxels, when < At that time, fill in the non-empty voxels to the required level. Each node constitutes a set of nodes; the original set of nodes input to the lightweight student network can be represented as: in, This represents the original set of nodes that will eventually be input into the lightweight student network, with the index... Indicates the first Each graph node This indicates the number of nodes retained after filtering. Indicates the first 1 node Indicates the first The feature representation of each node. Indicates the first The three-dimensional position coordinates of each node.
[0034] By constructing the node set in the above manner, the local areas with significant event responses can be preserved while controlling the input scale.
[0035] S14. Perform local spatiotemporal encoding on the events inside the voxels in the node set, so that each voxel is represented as a local spatiotemporal response tensor, and then mapped to an initial node embedding of a uniform dimension, i.e., a spatiotemporal relation graph.
[0036] like Figure 2 As shown, the method for local spatiotemporal encoding of voxel internal events in a node set further includes: S141. Calculate the local spatial coordinates and local temporal coordinates of any event within a voxel in the node set: ; ; in, and Voxels Inner One event Local spatial coordinates within a voxel; , and Voxels Position index in spatial width, spatial height, and time dimension; for In voxels Relative position on the internal local timeline; and They are respectively Original spatial coordinates; , , These represent the dimensions of a voxel unit in the spatial width, spatial height, and time dimension, respectively. for Normalized time coordinates voxels The number of local time channels obtained by further dividing the interior along the time dimension; S142. In order to simultaneously encode event polarity information and local temporal distribution information in node features, this embodiment uses linear interpolation to construct a voxel feature tensor along the local time axis: ; in, voxels In local time and local spatial location voxel feature tensor at the location; The polarity of the event; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. To obtain the maximum value; It is the absolute value symbol; voxels The set of events contained therein.
[0037] Through step S142, the information of a single event can be smoothly distributed across adjacent time channels, thereby better preserving the continuous variation characteristics of the event within a short time range. The resulting voxel feature tensor... It contains both event polarity information and local temporal distribution information. Let the three-dimensional position coordinates of each voxel node be denoted as... By adopting Describe the semantic attributes of the nodes. Describing the geometric location of nodes, the original event stream is transformed into a local spatiotemporal representation jointly described by node features and node coordinates.
[0038] S143. The voxel feature tensor is mapped to the initial node embedding through three-dimensional convolution and pooling operations to enhance the representation ability of short-term dynamic features and local spatial structure information within voxels. This is expressed as follows: in, For the first Each node is embedded in the initial node of the input layer, with superscript... Indicates the input layer; For the first The voxel feature tensor corresponding to each node; It is a voxel feature encoding mapping composed of three-dimensional convolution and pooling.
[0039] After local spatiotemporal feature encoding, the local spatiotemporal response tensor corresponding to each active voxel (the selected set of nodes) is mapped to an initial node embedding of uniform dimension. The original node set is further represented as the initial input node set of the lightweight student network: in, Represents the initial set of input nodes for a lightweight student network, with superscript... Indicates the initial input layer. Indicates the first Initial feature embedding of each node, Indicates the first The coordinates of each node, This indicates the total number of input nodes.
[0040] After obtaining the node features formed by the local spatiotemporal response tensor encoding and the node coordinates determined by the voxel position index, the spatial and temporal proximity relationships between nodes are used as the basis for initial connections. Spatiotemporally adjacent neighboring nodes are selected for each node, and initial connection edges are established between nodes, thus forming an initial spatiotemporal relationship graph composed of node features, node coordinates, and node connection relationships.
[0041] In step S2, the spatiotemporal relationship graph is input into the lightweight student network to perform a forward process of bi-branch spatiotemporal relationship modeling and hierarchical downsampling aggregation to global semantic convergence and classification output; The Lightweight Student Network is a hierarchical spatiotemporal relationship modeling network based on graph neural networks. It includes a two-branch spatiotemporal relationship aggregation layer, a hierarchical downsampling aggregation layer, and a classification head, used for joint modeling of local structural relationships, cross-regional dynamic associations, and action evolution processes within an event node set. Due to the asynchronous triggering, sparse distribution, and irregular dynamic changes of event flows, the event node set formed after voxelization is more suitable for node relationship-based modeling. Effective associations between nodes are related not only to spatial location but also to temporal differences and local dynamic changes.
[0042] In lightweight student networks, its first The set of layer input nodes is: in, Indicates lightweight student network The set of input nodes of the layer, superscript Indicates the first layer, Indicates the first The node at the th Layer feature representation, Indicates the coordinates of the corresponding node. This indicates the number of nodes in this layer. In each layer, node features and node coordinates are used as inputs to model the spatiotemporal relationships between nodes in the local neighborhood branch and the context neighborhood branch, respectively.
[0043] In one embodiment of the present invention, step S2 further includes: S21. For lightweight student networks, in its first... In the set of layer nodes, the node to be updated is designated as the center node. Non-central nodes are recorded as candidate nodes. In the branch In the middle, the central node of the calculation With nodes Spatiotemporal distance metric The values are 1 and 2, which represent the local neighborhood branch and the context neighborhood branch, respectively. The local neighborhood branch is used to characterize the more significant local dynamic response near the central node, while the context neighborhood branch is used to characterize the spatiotemporal correlation information over a larger range.
[0044] During implementation, this solution is preferably implemented in branches. In the middle, the central node of the calculation With nodes The expression for the spatiotemporal distance metric is: in, For branches Central node With candidate nodes The spatiotemporal distance between them; and They are the central nodes. With nodes In the The spatial coordinates of the layer; and They are the central nodes. With nodes Time coordinates; and Branches The scale parameters in the spatial and temporal dimensions are learnable parameters with an initial value of 1.
[0045] S22. Based on the spatiotemporal distance metric, in the branch The central node is constructed using the K-nearest neighbor method. The neighborhood of the central node is determined, and the relative spatiotemporal relationship between the central node and its neighboring nodes is determined: in, Indicates the first Neighborhood nodes in the layer Relative to the central node The characteristics of the relative spatiotemporal relationship, , and These represent the neighboring nodes relative to the center node. Relative displacement in the horizontal, vertical and time dimensions of space.
[0046] By using relative spatiotemporal relationships, the local dynamic relationships that were originally implicit in the differences in node coordinates are transformed into explicit spatiotemporal relationship features. This enables the lightweight student network to not only determine whether nodes are close to each other, but also to perceive the directional changes, temporal order, and intensity of local dynamic changes of neighboring nodes relative to the central node.
[0047] S23. Perform relation-aware fusion on the neighborhood node features and corresponding relative spatiotemporal relationships of the two branches respectively. Specifically, use the neighborhood node features and corresponding relative spatiotemporal relationships together for neighborhood information update, so that the spatiotemporal relationship information can participate in neighborhood weight calculation and neighborhood feature enhancement at the same time, thereby realizing collaborative modeling of local structural information, cross-regional interaction information and behavioral dynamic evolution information.
[0048] The outputs of the two relation-aware fusions are then fused to obtain a bi-branch fusion representation, the detailed implementation of which includes: Based on the gating fusion mechanism, the weights of the two branches are adaptively estimated using the features of the current node: in, and They are the central nodes. Fusion weights on local neighborhood branches and contextual neighborhood branches; As the central node In the Layer feature representation; Represents the gated mapping function; This indicates a normalization operation; According to weight Merge local neighborhood branches and contextual neighborhood branches: in, As the central node In the Layered two-branch fusion representation; and They are the central nodes. The output after relation-aware fusion in local neighborhood branches and contextual neighborhood branches; This is a feature splicing operation.
[0049] S24, By linear projection, the first The two-branch fusion representation of the layer is mapped back to the first layer. The feature space of the layer yields the central node. In the The updated node representation is achieved by adjusting the contribution of local fine-grained information and broader contextual information to the node representation based on the current node state, thus achieving a better balance between local dynamic modeling and contextual semantic modeling.
[0050] like Figure 3 As shown, in step S25, in the first The farthest point sampling method is used to select the first node in the layer node set. Multiple central nodes of the layer: in, Indicates the first The set of coordinates of the center node of the layer Indicates the first Layer node coordinate set express Number of center nodes in the layer This indicates the sampling operation at the farthest point.
[0051] The farthest-point sampling method ensures that the selected center nodes are distributed as evenly as possible within the current node space, preventing excessive concentration of center nodes in local areas and thus improving the coverage and representativeness of the next layer representation. After determining the center nodes for the next layer, a corresponding local neighborhood is constructed around each center node. Specifically, using each center node as a reference, the K-nearest neighbor method is used to select several neighboring nodes that are closest to the center node in the current layer, forming the local neighborhood set of the center node.
[0052] To ensure that the neighborhood representation simultaneously incorporates semantic and geometric information, for each neighborhood node, its features and its coordinate offset relative to the center node are used as common aggregation inputs. This common aggregation can be expressed as: in, Indicates the first Layer center node Feature representation, Indicates the first The local neighborhood set corresponding to each central node Indicates feature splicing, Representing neighboring nodes In the Layer feature representation, Indicates the neighboring node relative to the center node coordinate offset, This indicates a neighborhood aggregation operation.
[0053] S26. After the neighborhood aggregation operation, the features are input into a shared transformation function for feature mapping. All feature mapping representations of the same central node are then aggregated to obtain the... The node representation of the layer's center node; S27. For all layers of the lightweight student network, the bi-branch spatiotemporal relationship modeling of steps S21-S24 and the hierarchical downsampling aggregation of steps S25-S26 are alternately updated to obtain the node representation of each layer. During the process of obtaining the node representation, the lightweight student network can gradually compress the node size, expand the node's receptive range, and form a more compact and discriminative hierarchical feature representation. The node set of the last layer of the lightweight student network is represented as follows: in, Represents the set of nodes in the last layer of a lightweight student network, with superscript... Indicates the last layer. This indicates the number of nodes in this layer. and They represent the first The last layer feature representation and coordinates corresponding to each node.
[0054] S28. Perform average pooling and max pooling on the representations of all nodes in the last layer of the lightweight student network, and concatenate the two to obtain the sample-level global feature representation, which is then input into the classification head.
[0055] In implementation, the preferred expression for the sample-level global feature representation in this scheme is: in, It represents global features at the sample level; and These are average pooling and max pooling operations, respectively. For feature splicing operations; For the first The node at the th Layer feature representation; For the first The total number of nodes in the layer.
[0056] Through the detailed implementation of step S2, the lightweight student network completes the entire forward process from event voxel input, node feature encoding, bi-branch spatiotemporal relationship modeling, hierarchical aggregation to global semantic convergence and classification output. It can not only better preserve the local dynamic features in the event flow, but also obtain discriminative global semantic representation with low computational complexity, and provide the output foundation for subsequent category distillation and region attention distillation.
[0057] In step S3, the time surface sequence formed by the event flow is input into the teacher network, and the class discrimination knowledge and key area attention knowledge of the teacher network are transferred to the lightweight student network through class response distillation and area attention distillation. In step S4, the event stream to be identified is obtained and voxelized to obtain a set of nodes. Then, it is input into the optimized lightweight student network for fall recognition and the prediction result of fall class or non-fall class is output.
[0058] During the inference phase, this solution eliminates the need for a teacher network. Only the trained, lightweight student network is required to independently perform fall detection, and it can be deployed in fall monitoring scenarios that demand real-time performance, lightweight design, and privacy protection.
[0059] like Figure 4 As shown, the teacher network is a spatiotemporal multi-scale representation learning network, which includes: The multi-scale feature encoding module is used to receive event frames from the asynchronous event stream, progressively model the dynamic information in different time ranges, and obtain time multi-scale features. The spatial multi-scale feature fusion module is used to receive temporal multi-scale features to integrate spatial structure information at different levels and obtain high-level spatiotemporal semantic features that take into account both local details and global semantics. The pooling and classification mapping module is used to obtain the class response information of the teacher network by pooling and classifying high-level spatiotemporal semantic features; The category activation map generation module is used to process high-level spatiotemporal semantic features to obtain category activation maps.
[0060] like Figure 4 As shown, the teacher network uses a time-surface sequence. As input, the lightweight student network consists of a set of event nodes. As input, the output of both during the distillation stage can be expressed as: in, A teacher network representing the output category response and category activation graph; A lightweight student network representing the output category response, the importance response of the final layer nodes, and the node coordinates; Represents the category response vector output by the teacher network, with subscripts... Indicates a teacher network; The index represents the category response vector output by the lightweight student network. Indicates a lightweight student network; Indicating teacher network In category The category activation map generated under the given conditions; This represents the nodes of the final layer of the lightweight student network. Importance response, This represents the set of coordinates of the final layer nodes in a lightweight student network. This represents the time surface sequence of the teacher's network input. Represents the initial set of nodes for the input of a lightweight student network, with superscript... This indicates the initial layer.
[0061] Therefore, the teacher network provides category responses. The class activation graph provided by the teacher network is used to constrain the output distribution of the lightweight student network. Used to construct the importance distribution of teacher-side nodes; the lightweight student network outputs category responses. It is used to receive discriminative knowledge at the category level, and on the other hand, it outputs the final layer node importance response. Used to receive key regional level knowledge.
[0062] In one embodiment of the present invention, a method for transferring category discrimination knowledge and key region attention knowledge from a teacher network to a lightweight student network includes: During the category response distillation stage, temperature scaling is applied to the category responses of the teacher network and the lightweight student network to obtain the softening probability distribution: in, Indicates teacher networks in categories The softening probability normalized output; Indicates lightweight student networks in the category The softening probability normalized output; and These respectively represent teacher networks and lightweight student networks in the category Category response values; Indicates the total number of categories; Indicates temperature coefficient, subscript The category index represents the summation when the denominator is normalized; It is an exponential function.
[0063] Based on the softening probability distribution, a decoupling approach is used to construct the target class distillation loss: in, Indicates the knowledge distillation loss for the target category; subscript This represents the category index corresponding to the true label of the sample. and These represent teacher networks and lightweight student networks in real-world scenarios. The softening probability output on the top, This represents the Kullback-Leibler divergence.
[0064] In the teacher network and lightweight student network, the category responses for categories other than the target category are renormalized to obtain the softening probability output: in, and These represent teacher networks and lightweight student networks in non-target categories, respectively. The softening probability output after renormalization, index Indicates a non-target category index that satisfies .
[0065] Then, a non-target class distillation loss is constructed based on the softening probability output. We construct a category distillation loss for lightweight student networks learning teacher networks at the category level, using both target-class and non-target-class distillation losses: in, Indicates category of distillation loss, This represents the knowledge distillation loss for the target category. This represents the loss from distillation of non-target category knowledge. and For balance coefficient, , where is the temperature coefficient. Through the aforementioned categorical distillation constraints, the lightweight student network is able to learn the discriminative relationships of the teacher network at the category level.
[0066] During the region attention distillation phase, the teacher network outputs a category activation graph, mapping the region attention information of the teacher network to the node space of the last layer of the lightweight student network, thus obtaining the importance distribution on the teacher side. The lightweight student network outputs the importance responses of each node in the final layer, which are then normalized to obtain the distribution of student-side nodes. .
[0067] Importance distribution Student-side node distribution Constructing a teacher network transfers the knowledge of key discriminative regions to the region distillation loss in a lightweight student network. Through regional distillation constraints, the teacher network's knowledge of key discriminative regions is transferred to the lightweight student network, thereby enhancing the lightweight student network's ability to represent key regions.
[0068] In implementation, this scheme preferentially uses the overall loss function of a lightweight student network. for: in, To monitor and categorize losses.
[0069] Through the aforementioned joint distillation optimization, the lightweight student network, while maintaining a lightweight structure, can learn the discriminative knowledge of the teacher network at the category level and inherit its ability to focus on key discriminative regions, thereby enhancing its ability to represent key dynamic features of fall actions and improving the accuracy and stability of fall recognition.
[0070] In summary, this solution can better preserve the irregular spatiotemporal topology of event data. While controlling the complexity of the model, it enhances the modeling ability of key dynamic regions of fall actions and their contextual information. It solves the problems in existing technologies where regular grid representations cannot fully depict the local spatiotemporal relationships of event flows and lightweight models do not pay enough attention to key discrimination regions. This improves the accuracy and stability of fall recognition and is suitable for deployment in embedded and low-power devices.
Claims
1. A neuromorphic visual fall recognition method based on spatiotemporal graph modeling and region distillation, characterized in that, Including the following steps: S1. Obtain the asynchronous event stream of human fall and non-fall behaviors, perform voxelization on the asynchronous event stream and construct a spatiotemporal relationship graph; S2. Input the spatiotemporal relationship graph into the lightweight student network and perform a forward process of bi-branch spatiotemporal relationship modeling and hierarchical downsampling aggregation to global semantic convergence and classification output. S3. Input the time surface sequence formed by the event flow into the teacher network, and transfer the class discrimination knowledge and key area attention knowledge of the teacher network to the lightweight student network through category response distillation and region attention distillation. S4. Obtain the event stream to be identified, and after voxelizing it, obtain the node set. Then, input it into the optimized lightweight student network for fall recognition, and output the prediction result of fall class or non-fall class.
2. The neuromorphic visual fall recognition method according to claim 1, characterized in that, The lightweight student network is a hierarchical spatiotemporal relationship modeling network based on graph neural networks, and step S2 further includes: S21. For lightweight student networks, in its first... In the set of layer nodes, the node to be updated is designated as the center node. Non-central nodes are recorded as candidate nodes. In the branch In the middle, the central node of the calculation With nodes Spatiotemporal distance metric The values are 1 and 2, representing the local neighborhood branch and the context neighborhood branch, respectively. S22. Based on the spatiotemporal distance metric, in the branch The central node is constructed using the K-nearest neighbor method. The neighborhood of the central node is determined, and the relative spatiotemporal relationship between the central node and its neighboring nodes is determined. S23. Perform relation-aware fusion on the neighborhood node features and corresponding relative spatiotemporal relationships of the two branches respectively, and then fuse the outputs of the two relation-aware fusions to obtain the dual-branch fusion representation. S24, By linear projection, the first The two-branch fusion representation of the layer is mapped back to the first layer. The feature space of the layer yields the central node. In the The updated node representation after layer update; S25, in the Select the first node from the set of layer nodes For multiple central nodes of a layer, construct their neighborhoods, and then perform neighborhood aggregation by concatenating the features of the neighborhood nodes with their coordinate offsets relative to the central nodes. S26. After the neighborhood aggregation operation, the features are input into a shared transformation function for feature mapping. All feature mapping representations of the same central node are then aggregated to obtain the... The node representation of the layer's center node; S27. For all layers of the lightweight student network, the bi-branch spatiotemporal relationship modeling in steps S21 to S24 and the hierarchical downsampling aggregation in steps S25 to S26 are alternately updated to obtain the node representation of each layer. S28. Perform average pooling and max pooling on the representations of all nodes in the last layer of the lightweight student network, and concatenate the two to obtain the sample-level global feature representation, which is then input into the classification head.
3. The neuromorphic visual fall recognition method according to claim 2, characterized in that, In the branch In the middle, the central node of the calculation With nodes The expression for the spatiotemporal distance metric is: ； in, For branches Central node With candidate nodes The spatiotemporal distance between them; and They are the central nodes. With nodes In the The spatial coordinates of the layer; and They are the central nodes. With nodes Time coordinates; and Branches The scale parameters in the spatial and temporal dimensions are learnable parameters with an initial value of 1.
4. The neuromorphic visual fall recognition method according to claim 2, characterized in that, Methods for fusing the outputs after perceptual fusion of two branches include: Based on the gating fusion mechanism, the weights of the two branches are adaptively estimated using the features of the current node: ； in, and They are the central nodes. Fusion weights on local neighborhood branches and contextual neighborhood branches; As the central node In the Layer feature representation; Represents the gated mapping function; This indicates a normalization operation; According to weight Merge local neighborhood branches and contextual neighborhood branches: in, As the central node In the Layered bi-branch fusion representation; and They are the central nodes. The output after relation-aware fusion in local neighborhood branches and contextual neighborhood branches; This is a feature splicing operation.
5. The neuromorphic visual fall recognition method according to claim 2, characterized in that, The expression for the sample-level global feature representation is: ； in, It represents global features at the sample level; and These are average pooling and max pooling operations, respectively. For feature splicing operations; For the first The node at the th Layer feature representation; For the first The total number of nodes in the layer.
6. The neuromorphic visual fall recognition method according to claim 1, characterized in that, The teacher network is a spatiotemporal multi-scale representation learning network, which includes: The multi-scale feature encoding module is used to receive event frames from the asynchronous event stream, progressively model the dynamic information in different time ranges, and obtain time multi-scale features. The spatial multi-scale feature fusion module is used to receive temporal multi-scale features to integrate spatial structural information at different levels and obtain high-level spatiotemporal semantic features that take into account both local details and global semantics. The pooling and classification mapping module is used to obtain the class response information of the teacher network by pooling and classifying high-level spatiotemporal semantic features; The category activation graph generation module is used to process high-level spatiotemporal semantic features to obtain category activation graphs.
7. The neuromorphic visual fall recognition method according to claim 6, characterized in that, Methods for transferring categorical knowledge and key region focus knowledge from teacher networks to lightweight student networks include: In the category response distillation stage, the category responses of the teacher network and the lightweight student network are temperature scaled to obtain the softening probability distribution; based on the softening probability distribution, the target class distillation loss is constructed using a decoupling method. In the teacher network and lightweight student network, the category responses of all categories except the target category are renormalized to obtain the softening probability output, and then the non-target class distillation loss is constructed based on the softening probability output. We construct a category distillation loss for class-level discriminative relationships between lightweight student networks and teacher networks by employing both target-class distillation loss and non-target-class distillation loss. During the region attention distillation phase, the teacher network outputs a category activation graph, mapping the region attention information of the teacher network to the node space of the last layer of the lightweight student network, thus obtaining the importance distribution on the teacher side. ; The lightweight student network outputs the importance responses of each node in the last layer, which are then normalized to obtain the distribution of student-side nodes. ; Importance distribution Student-side node distribution The teacher network is constructed to transfer the knowledge of key discriminative regions to the regional distillation loss in the lightweight student network.
8. The neuromorphic visual fall recognition method according to claim 7, characterized in that, Overall loss function of lightweight student networks for: ； ， ； in, To monitor and categorize losses; Distillation loss by category; This represents regional distillation losses; and These are the weighting coefficients for category distillation loss and region distillation loss, respectively. For target type distillation loss; This is a non-target type of distillation loss; and This is the balance coefficient; Temperature coefficient; Let be the divergence function.
9. The neuromorphic visual fall recognition method according to claim 1, characterized in that, The method for voxelizing asynchronous event stream data and constructing a spatiotemporal graph further includes: S11. Normalize the timestamps of individual events in the asynchronous event stream, and discretize the asynchronous event stream along the spatial and temporal dimensions, mapping it to a three-dimensional voxel grid. S12. Record the number of events in each non-empty voxel in the 3D voxel mesh, and sort all non-empty voxels in descending order of the number of events. S13, When the total number of non-empty voxels ≥ Preset number of nodes At that time, adopt the former A set of nodes is composed of non-empty voxels, when < At that time, fill in the non-empty voxels to the required level. Each of these forms a node set; S14. Perform local spatiotemporal encoding on the events inside the voxels in the node set, so that each voxel is represented as a local spatiotemporal response tensor, and then mapped to an initial node embedding of a unified dimension, i.e., a spatiotemporal relation graph.
10. The neuromorphic visual fall recognition method according to claim 9, characterized in that, The method for local spatiotemporal encoding of voxel internal events in a node set further includes: S141. Calculate the local spatial coordinates and local temporal coordinates of any event within a voxel in the node set: ； ； in, and Voxels Inner One event Local spatial coordinates within a voxel; , and Voxels Position index in spatial width, spatial height, and time dimension; for In voxels Relative position on the internal local timeline; and They are respectively Original spatial coordinates; , , These represent the dimensions of a voxel unit in the spatial width, spatial height, and time dimension, respectively. for Normalized time coordinates; voxels The number of local time channels obtained by further dividing the interior along the time dimension; S142. Construct voxel feature tensors along the local time axis using linear interpolation: ； in, voxels In local time and local spatial location voxel feature tensor at the location; The polarity of the event; This is an indicator function that takes the value 1 when the condition is true and 0 otherwise. To obtain the maximum value; It is the absolute value symbol; voxels The set of events contained therein; S143. Map the voxel feature tensor to the initial node embedding through 3D convolution and pooling operations: ； in, For the first Each node is embedded in the initial node of the input layer, with superscript... Indicates the input layer; For the first The voxel feature tensor corresponding to each node; It is a voxel feature encoding mapping composed of three-dimensional convolution and pooling.