A Human Posture and Movement Intent Understanding Method Optimized Based on Brain Spiral Neural Networks and Attention Mechanisms

Abstract

This invention provides a method for understanding human posture and action intent based on brain spiking neural networks and attention mechanisms, comprising the following steps: acquiring an input sequence of human action postures; mapping each posture to a high-dimensional embedding vector to generate a posture embedding sequence; a first feature extraction pathway processing the posture embedding sequence through a multi-head self-attention mechanism to extract global action features representing long-term dependencies; a second feature extraction pathway processing the posture embedding sequence through a brain-like spiking neural network to extract local action features representing short-term details; fusing the global and local action features to generate fused features; and performing a residual connection between the fused features and the posture embedding sequence to obtain an enhanced action feature representation. This invention can reduce the computational resource consumption for modeling short-term motion details and enhance sensitivity to rapid posture changes through unique temporal dynamic characteristics.

Description

Technical Field

[0001] This invention relates to the field of human motion posture recognition technology, and in particular to a method for understanding human posture and action intentions based on brain spiking neural networks and attention mechanisms. Background Technology

[0002] Human motion posture recognition, as a core technology for understanding human posture and action intentions, has wide application value in fields such as visual surveillance and human-computer interaction (HRI). This task aims to enhance the system's deep understanding and intelligent response capabilities regarding human behavior by analyzing observed sequences of human actions, not only identifying their surface behavior categories (such as waving or walking) but also gaining a deeper understanding of their underlying intentions (such as reaching out to grab an object).

[0003] In the field of human posture and intention understanding, existing technologies mainly revolve around the analysis of temporal skeleton sequences. Mainstream methods can be broadly categorized into two types: one is based on recurrent neural networks (RNNs), convolutional neural networks (CNNs), or their variants, focusing on extracting local spatiotemporal features from consecutive frames; the other is based on the Transformer architecture, utilizing its powerful self-attention mechanism to capture long-range global dependencies in the sequence. In recent years, to improve efficiency or incorporate bio-inspiration, spiking neural networks (SNNs) have also begun to be explored for action recognition tasks. For example, some studies have attempted to combine attention mechanisms with SNNs to optimize feature selection, while others have utilized non-autoregressive Transformers for parallelized pose prediction. These methods have made progress on specific tasks, but have not yet systematically resolved the core challenges of real-time intention understanding.

[0004] In practical applications, especially in embedded systems, mobile devices, or real-time human-computer interaction scenarios requiring low latency and high energy efficiency, the industry still faces some challenges. On the one hand, while computationally intensive deep learning models offer high accuracy, their enormous computational and memory overhead makes it difficult to meet the stringent power consumption and response speed requirements of these scenarios. On the other hand, lightweight models or traditional temporal models often struggle to balance capturing long-term intent evolution with rapidly changing details. They either ignore the global semantic coherence of actions or are insensitive to key subtle changes in actions, resulting in insufficient judgment of complex and continuous behavioral intents and limiting the universality and practicality of the technology. Summary of the Invention

[0005] This invention provides a method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms. It can reduce the computational resource consumption for modeling short-term motion details and enhance the sensitivity to rapid posture changes through its unique temporal dynamic characteristics.

[0006] The first aspect of this invention provides a method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms, comprising the following steps: Obtain the input sequence of human action poses, map each pose to a high-dimensional embedding vector, and generate a pose embedding sequence. Set up a first feature extraction path and a second feature extraction path; simultaneously input the pose embedding sequence into the first feature extraction path and the second feature extraction path; The first feature extraction pathway processes the pose embedding sequence through a multi-head self-attention mechanism to extract global action features that represent long-term temporal dependencies; The second feature extraction pathway processes the pose embedding sequence through a brain-like spiking neural network to extract local action features that represent short-term details; Global action features and local action features are fused to generate fused features; The fused features are residually connected to the pose embedding sequence to obtain the enhanced action feature representation; the action feature representation is then used for action recognition and intent understanding tasks.

[0007] Furthermore, the step of acquiring the input human action pose sequence, mapping each pose to a high-dimensional embedding vector, and generating a pose embedding sequence includes the following steps: Preprocess the input human motion and posture sequence; Flatten the attitude coordinate data at each time step into a one-dimensional vector; The pose embedding vectors at each time step are obtained by mapping the one-dimensional vector of the learnable linear transformation layer to a high-dimensional space. Add positional encoding to the pose embedding vector at each time step to generate a pose embedding sequence.

[0008] Furthermore, the first feature extraction pathway processes the pose embedding sequence through a multi-head self-attention mechanism to extract global action features representing long-term temporal dependencies, including the following steps: The pose embedding sequence is linearly projected into a query matrix, a key matrix, and a value matrix, and the query matrix, key matrix, and value matrix are split into multiple attention heads; In the attention head, attention weights are calculated based on the relative position encoding, and the value matrix is ​​weighted and summed according to the attention weights to obtain the output of the attention head; The outputs of multiple attention heads are concatenated and fused using a linear projection layer to obtain preliminary attention features; The initial attention features and pose embedding sequence are processed by residual connection and layer normalization, and then transformed by a feedforward neural network to output global action features.

[0009] Furthermore, the second feature extraction pathway processes the pose embedding sequence through a brain-like spiking neural network to extract local action features representing short-term details, including the following steps: The pose embedding sequence is input into the spiking neuron layer to generate a spiking feature sequence; global average pooling and global max pooling are then performed on the spiking feature sequence along the time dimension. The pooled pulse features are input into the convolution module for convolution processing. The outputs of the two convolutional modules are obtained, and the outputs of the two convolutional modules are added together. Then, a channel attention weight vector is generated by the activation function. The pulse feature sequence is weighted by channel attention weight vector, and the weighted features are batch normalized to output local action features.

[0010] Furthermore, the step of fusing global action features and local action features to generate fused features includes concatenating global action features and local action features along the feature dimension; and mapping the concatenated features to the target dimension through a linear projection layer to generate fused features.

[0011] Furthermore, the step of performing a residual connection between the fused features and the pose embedding sequence to obtain the enhanced action feature representation includes the following steps: The fused features and pose embedding sequences are added element-wise, and the addition results are processed by layer normalization and enhanced by a feedforward neural network. The output of the feedforward neural network is then added to the sum and residual-connected and normalized to output an enhanced action feature representation.

[0012] Furthermore, the spiking neuron layer adopts the Leaky Integrate-and-Fire neuron model.

[0013] The second aspect of the present invention provides a human posture and action intention understanding system based on brain spiking neural network and attention mechanism optimization, including a first processing module for acquiring an input human action posture sequence, mapping each posture to a high-dimensional embedding vector, and generating a posture embedding sequence. The second processing module is used to set up a first feature extraction path and a second feature extraction path; and to simultaneously input the pose embedding sequence into the first feature extraction path and the second feature extraction path. The third processing module is used to process the pose embedding sequence through the first feature extraction path using a multi-head self-attention mechanism to extract global action features that represent long-term dependencies. The fourth processing module is used to process the pose embedding sequence through the second feature extraction pathway using a brain-like spiking neural network to extract local action features that represent short-term details. The fifth processing module is used to fuse global action features and local action features to generate fused features; The sixth processing module is used to perform residual connections between the fused features and the pose embedding sequence to obtain an enhanced action feature representation; the action feature representation is then used for action recognition and intent understanding tasks.

[0014] A third aspect of the present invention provides a computer device, comprising: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory, including executing the human posture and movement intention understanding method optimized based on brain spiking neural network and attention mechanism described above; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0015] A fourth aspect of the present invention provides a readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the above-described method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms.

[0016] As can be seen from the above technical solutions, the present invention has the following advantages: This invention constructs a dual-path parallel processing architecture that integrates multi-head self-attention mechanism and brain-like spiking neural network (SNN). The first feature extraction path uses multi-head self-attention mechanism to simulate high-level cognitive functions to capture global long-term posture dependence. The second feature extraction path draws on the sparse pulse coding mechanism of biological neural systems, extracting local short-term motion features through spiking neurons (LIF) and temporal convolutional gating, thereby enhancing the brain-like processing ability for rapid posture changes, reducing the computational resource consumption for modeling short-term motion details, and enhancing the sensitivity to rapid posture changes through unique temporal dynamic characteristics. Thus, it achieves higher recognition accuracy and deeper intent inference capabilities in complex and continuous action understanding tasks.

[0017] Secondly, this invention employs a brain-like spiking neural network with Leaky Integrate-and-Fire (LIF) spiking neurons at its core as a local feature extraction pathway, achieving efficient capture of fast, sparse motion patterns. This invention reduces the computational resource consumption for modeling short-term motion details through event-driven sparse computation, while enhancing the temporal sensitivity to subtle, transient motion changes, thus providing feasibility for deployment in resource-constrained real-time application scenarios.

[0018] Furthermore, by introducing a feature fusion module and a residual connection mechanism, this invention achieves adaptive and complementary fusion of global and local features, ensuring that key information of the original pose is preserved in the deep layers of the network, enhancing the robustness and discriminative power of the final action feature representation, and effectively alleviating the gradient vanishing problem in deep network training, thereby improving the training stability and convergence speed of the model.

[0019] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from an examination of the following, or may be learned from the practice of the invention. Attached Figure Description

[0020] Figure 1 The method flowchart provided by the present invention.

[0021] Figure 2 The structural block diagram of the neural network provided by this invention. Detailed Implementation

[0022] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Example 1

[0024] The implementation method in this embodiment can be implemented in a system, on a server, or on a terminal; no specific limitation is made. The method in this application will be described below from the perspective of system implementation. Figure 1 As shown, a method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms includes the following steps: Obtain the input sequence of human action poses, map each pose to a high-dimensional embedding vector, and generate a pose embedding sequence. Set up a first feature extraction path and a second feature extraction path; simultaneously input the pose embedding sequence into the first feature extraction path and the second feature extraction path; The first feature extraction pathway processes the pose embedding sequence through a multi-head self-attention mechanism to extract global action features that represent long-term temporal dependencies; The second feature extraction pathway processes the pose embedding sequence through a brain-like spiking neural network to extract local action features that represent short-term details; Global action features and local action features are fused to generate fused features; The fused features are residually connected to the pose embedding sequence to obtain the enhanced action feature representation; the action feature representation is then used for action recognition and intent understanding tasks.

[0025] The system receives raw human pose sequences generated by motion capture devices or video analysis algorithms. After preprocessing, the raw human pose sequences are normalized and interpolated. The pose at each time step is independently mapped to a high-dimensional vector through a learnable embedding layer, converting the geometric relationships in space into feature representations that are easier for neural networks to process. Simultaneously, the system adds positional encoding to these vectors, preserving the temporal order of the actions, forming a complete pose embedding sequence.

[0026] The first feature extraction pathway is used to simulate higher cognitive functions, employing a multi-head self-attention mechanism as the core processor. Its purpose is to calculate the similarity between the pose features of each time step in the sequence and the pose features of all time steps in the sequence, thereby achieving automatic learning and establishing dependencies across long-distance time steps. For example, it can associate the initial pose of a reaching motion with the final grasping pose, thus capturing the complete trajectory of the action's intention evolution and long-range contextual dependencies, forming an understanding of the overall purpose and trend of the action.

[0027] The second feature extraction pathway simulates the response of a biological sensory system to transient stimuli. It employs a brain-like spiking neural network (SNN) as its core, utilizing spiking neuron models such as Leaky Integrate-and-Fire (LIF) to convert continuous pose embeddings into sparse sequences of spiking events. This conversion significantly reduces data redundancy and computational load, and the dynamic nature of spiking makes it exceptionally sensitive to rapid and subtle changes in pose. Subsequently, the second feature extraction pathway extracts detailed features from local spiking patterns through temporal pooling and lightweight convolutional gating.

[0028] After the first and second feature extraction pathways are completed, the system fuses the macroscopic intent features output by the global semantic pathway with the microscopic action features output by the local detail pathway. Typically, a concatenation followed by linear projection is used to allow the two types of information to interact and integrate within the same feature space, ensuring that the final feature representation contains complete information about both global intent and local details. Finally, the system adds the fused high-level features to the initial pose embedding sequence via a residual connection, ensuring that the original information is not lost during forward propagation.

[0029] Finally, in the task output stage, the fused and enhanced final action feature representation is used to output specific action category labels. It can also be a more complex sequence generator or regressor used to predict future pose sequences or directly output a semantic description of the behavioral intent.

[0030] Example 2

[0031] The difference between this embodiment and Embodiment 1 is that the step of obtaining the input human action posture sequence, mapping each posture to a high-dimensional embedding vector, and generating a posture embedding sequence includes the following steps: Preprocess the input human motion and posture sequence; Flatten the attitude coordinate data at each time step into a one-dimensional vector; The pose embedding vectors at each time step are obtained by mapping the one-dimensional vector of the learnable linear transformation layer to a high-dimensional space. Add positional encoding to the pose embedding vector at each time step to generate a pose embedding sequence.

[0032] The received raw human pose sequences often contain individual differences and noise interference. For example, data from depth cameras or video analysis may show differences in height and arm length. The system uses coordinate normalization to transform the coordinates of each joint point into a relative coordinate system centered on the human torso or pelvis, eliminating scale differences caused by different user body shapes. At the same time, for missing frames caused by occlusion or detection failure, the system uses linear or spline interpolation algorithms to automatically complete them, ensuring the continuity of the time series.

[0033] Secondly, spatial structure flattening and vectorization are performed, as the pose at each time step is essentially a graph structure. To facilitate subsequent linear transformations, the 3D coordinates of the N key points at each time step are concatenated into a 3N-dimensional one-dimensional vector. During high-dimensional semantic mapping, a learnable linear transformation layer maps the flattened low-dimensional vector to a predefined high-dimensional feature space. The weight matrix of the linear transformation layer is continuously optimized during model training, learning to map the original coordinate space to a feature space more effective for action recognition. For example, learning that specific coordinate combinations of the shoulder, elbow, and wrist should be mapped to a feature dimension representing arm extension, thus transforming geometric data into semantic features.

[0034] Finally, by adding learnable positional encoding, a unique encoding vector can be assigned to each time step in the sequence, which is added element-wise to the pose embedding vector, enabling the model to perceive the position of each feature on the time axis and understand the sequential logic of the actions.

[0035] Example 3

[0036] The difference between this embodiment and Embodiment 2 is that the first feature extraction path processes the pose embedding sequence through a multi-head self-attention mechanism to extract global action features representing long-term temporal dependencies, including the following steps: The pose embedding sequence is linearly projected into a query matrix, a key matrix, and a value matrix, and the query matrix, key matrix, and value matrix are split into multiple attention heads; In the attention head, attention weights are calculated based on the relative position encoding, and the value matrix is ​​weighted and summed according to the attention weights to obtain the output of the attention head; The outputs of multiple attention heads are concatenated and fused using a linear projection layer to obtain preliminary attention features; The initial attention features and pose embedding sequence are processed by residual connection and layer normalization, and then transformed by a feedforward neural network to output global action features.

[0037] First, the system maps the input pose embedding sequence through three independent linear layers into a query matrix, a key matrix, and a value matrix, respectively. Then, each matrix is ​​evenly split into h heads along the feature dimension, allowing different attention heads to focus on different subspaces of the features in parallel; for example, some heads focus on upper limb movement patterns, while others focus on lower limb coordination relationships, to enhance the model's ability to understand actions from different perspectives.

[0038] The expressions for the query matrix (Q), key matrix (K), and value matrix (V) are as follows: ; Where, sequence (Where T is the length of the action sequence and d is the dimension of the pose feature). .

[0039] Next, attention computation using relative position encoding is introduced. For each attention head, the system calculates the similarity between the query and the key, and introduces an encoded vector representing the relative distance at each time step. That is, when calculating the attention of frame i to frame j, the relative position information from j to i is added to the query vector. This allows the model to directly understand how the pose of the past 3 frames affects the current frame, rather than relying on learned absolute positional deviations, thus more accurately modeling the temporal causal relationships of actions.

[0040] The expressions for the multiple attention heads are: ; in, The projection weight dimension is The output of this multi-head attention module is the final output of the right branch. .

[0041] Concat is a concatenation operation that joins h attention heads (the output of each head). The matrices are concatenated along the feature dimension to obtain a concatenated matrix. This operation aggregates the diverse global pose dependencies learned by all attention heads from different representation subspaces.

[0042] Finally, a learnable weight matrix is ​​used. A linear transformation is performed on the concatenated result to fuse the information from all heads and project it onto a specified output dimension, so that the dimension of the final output is consistent with the dimension of the input pose embedding X.

[0043] Example 4

[0044] The difference between this embodiment and Embodiment 3 is that the second feature extraction pathway processes the pose embedding sequence through a brain-like spiking neural network to extract local action features representing short-term details, including the following steps: The pose embedding sequence is input into the spiking neuron layer to generate a spiking feature sequence; global average pooling and global max pooling are then performed on the spiking feature sequence along the time dimension. The pooled pulse features are input into the convolution module for convolution processing. The outputs of the two convolutional modules are obtained, and the outputs of the two convolutional modules are added together. Then, a channel attention weight vector is generated by the activation function. The pulse feature sequence is weighted by channel attention weight vector, and the weighted features are batch normalized to output local action features.

[0045] The pose embedding sequence is input into the spiking neuron layer, and given the input pose embedding sequence... When the input is the same as the right branch, the following formula is used for calculation: ; Where Z is the pulse sequence. The system then performs global average pooling and global max pooling on the pulse sequence Z in the time dimension. The average pooling path extracts the average intensity of the pulse firing within the entire time window, reflecting the overall activity level of the action during that period. The max pooling path captures the peak intensity of the pulse firing, reflecting the most dramatic instantaneous changes during that period.

[0046] The path for global average pooling is as follows: ; ; ; ; in, The core result of time-dimensional average pooling reflects the overall activity level of feature channels; , Yes The nonlinear feature transformation has the core function of enhancing feature representation; It is the result of feature dimension restoration, and its core purpose is to adapt to the dimensional requirements of attention weights.

[0047] The path for global max pooling is: ; ; ; ; in, The core result of time-dimension max pooling reflects the instantaneous peak activity level of the feature channels; , Yes The nonlinear feature transformation has the core function of enhancing feature representation; It is the result of feature dimension restoration, and its core purpose is to adapt to the dimensional requirements of attention weights.

[0048] The features of the two pooling paths are each processed by a lightweight convolutional module, which can consist of two 1×1 convolutional layers and a ReLU activation function. The convolutional module first performs feature compression and non-linear fusion through the first convolutional layer, and then restores the dimensionality through the second convolutional layer. The processed features of the two paths are added together and then processed by the Sigmoid function to generate a channel attention weight vector in the range [0,1].

[0049] Finally, attention-weighted feature enhancement is applied by multiplying the generated channel attention weights with the original pulse sequence channel by channel. This enhances important feature channels and suppresses secondary channels, making the local features more focused on key details. After batch normalization to stabilize the feature distribution, the final local action features are output.

[0050] Example 5

[0051] The difference between this embodiment and embodiment four is that the feature fusion of global action features and local action features to generate fused features includes concatenating global action features and local action features along the feature dimension; and mapping the concatenated features to the target dimension through a linear projection layer to generate fused features.

[0052] When performing feature concatenation, the global action features output by the first feature extraction path and the local action features output by the second feature extraction path are connected along the channel dimension, so that the original feature information from the two paths is completely preserved and a joint feature vector is formed.

[0053] When performing linear projection fusion, the joint features after concatenation have a high dimensionality. The system uses a fully connected layer with learnable parameters to reduce the dimensionality of these features, outputting fused features with the same dimensionality as the single-path features. The projection layer weights are optimized during model training using a backpropagation algorithm. The corresponding learning objective is to find a weighted combination method that allows features from global and local pathways to be adaptively integrated according to specific action patterns, thereby improving the discriminative power of the final feature representation.

[0054] Example 6

[0055] The difference between this embodiment and embodiment five is that the step of performing a residual connection between the fused features and the pose embedding sequence to obtain the enhanced action feature representation includes the following steps: The fused features and pose embedding sequences are added element-wise, and the addition results are processed by layer normalization and enhanced by a feedforward neural network. The output of the feedforward neural network is then added to the sum and residual-connected and normalized to output an enhanced action feature representation.

[0056] By adding the fused features generated by the preceding modules element-wise to the pose embedding sequence of the network's initial input, the network explicitly learns the residual between the fused features and the original input, rather than directly learning a complex end-to-end mapping. During training, this structure provides a direct path for gradient backpropagation that is unaffected by nonlinear transformation weights, helping to mitigate the vanishing gradient phenomenon that may occur in deep networks, thereby stabilizing the training process.

[0057] Secondly, the sum of the residuals is first normalized to stabilize the distribution of the input at that layer. Then, a feedforward neural network performs nonlinear feature transformation to enhance the model's expressive power. Pre-modules are often stacked sequentially to achieve layer-by-layer feature refinement and deepening. The final output features retain both the precise pose information from the lower layers and the high-level semantic information extracted by the deeper networks.

[0058] Example 7

[0059] The difference between this embodiment and Embodiment Six is ​​that the spiking neuron layer adopts the Leaky Integrate-and-Fire neuron model.

[0060] The spiking neuron layer employs the Leaky Integrate-and-Fire neuron model, replacing traditional activation functions. It combines temporal global pooling with lightweight convolutional gating mechanisms to efficiently capture short-term local motion features and rapidly changing details in human motion sequences, better aligning with biological motion perception characteristics and enhancing response to rapid posture changes. It also improves temporal sensitivity when processing input X to generate the pulse signal Z.

[0061] Example 8

[0062] A human posture and action intention understanding system based on brain spiking neural network and attention mechanism optimization includes a first processing module for acquiring the input human action posture sequence, mapping each posture to a high-dimensional embedding vector, and generating a posture embedding sequence. The second processing module is used to set up a first feature extraction path and a second feature extraction path; and to simultaneously input the pose embedding sequence into the first feature extraction path and the second feature extraction path. The third processing module is used to process the pose embedding sequence through the first feature extraction path using a multi-head self-attention mechanism to extract global action features that represent long-term dependencies. The fourth processing module is used to process the pose embedding sequence through the second feature extraction pathway using a brain-like spiking neural network to extract local action features that represent short-term details. The fifth processing module is used to fuse global action features and local action features to generate fused features; The sixth processing module is used to perform residual connections between the fused features and the pose embedding sequence to obtain an enhanced action feature representation; the action feature representation is then used for action recognition and intent understanding tasks.

[0063] Example 9

[0064] A computer device, comprising: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory, including executing the human posture and movement intention understanding method optimized based on brain spiking neural network and attention mechanism described above; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

[0065] Example 10

[0066] A readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the above-described method for understanding human posture and intention based on brain spiking neural networks and attention mechanisms.

[0067] In summary, this invention constructs a dual-path parallel processing architecture that integrates multi-head self-attention mechanism and brain-like spiking neural network (SNN). The first feature extraction path uses multi-head self-attention mechanism to simulate higher cognitive functions to capture global long-term posture dependence. The second feature extraction path draws on the sparse pulse coding mechanism of biological neural systems, extracting local short-term motion features through spiking neurons (LIF) and temporal convolutional gating, thereby enhancing the brain-like processing ability for rapid posture changes, reducing the computational resource consumption for modeling short-term motion details, and enhancing the sensitivity to rapid posture changes through unique temporal dynamic characteristics. Thus, it achieves higher recognition accuracy and deeper intent inference capabilities in complex and continuous action understanding tasks.

[0068] Secondly, this invention employs a brain-like spiking neural network with Leaky Integrate-and-Fire (LIF) spiking neurons at its core as a local feature extraction pathway, achieving efficient capture of fast, sparse motion patterns. This invention reduces the computational resource consumption for modeling short-term motion details through event-driven sparse computation, while enhancing the temporal sensitivity to subtle, transient motion changes, thus providing feasibility for deployment in resource-constrained real-time application scenarios.

[0069] Furthermore, by introducing a feature fusion module and a residual connection mechanism, this invention achieves adaptive and complementary fusion of global and local features, ensuring that key information of the original pose is preserved in the deep layers of the network, enhancing the robustness and discriminative power of the final action feature representation, and effectively alleviating the gradient vanishing problem in deep network training, thereby improving the training stability and convergence speed of the model.

[0070] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of the invention.

[0071] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical functional division. In actual implementation, there may be other division methods, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored.

[0072] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0073] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms, characterized in that, Includes the following steps: Obtain the input sequence of human action poses, map each pose to a high-dimensional embedding vector, and generate a pose embedding sequence. Set up a first feature extraction path and a second feature extraction path; simultaneously input the pose embedding sequence into the first feature extraction path and the second feature extraction path; The first feature extraction pathway processes the pose embedding sequence through a multi-head self-attention mechanism to extract global action features that represent long-term temporal dependencies; The second feature extraction pathway processes the pose embedding sequence through a brain-like spiking neural network to extract local action features that represent short-term details; Global action features and local action features are fused to generate fused features; The fused features are residually connected to the pose embedding sequence to obtain the enhanced action feature representation; the action feature representation is then used for action recognition and intent understanding tasks.

2. The method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms as described in claim 1, characterized in that, The process of acquiring the input human action pose sequence, mapping each pose to a high-dimensional embedding vector, and generating a pose embedding sequence includes the following steps: Preprocess the input human motion and posture sequence; Flatten the attitude coordinate data at each time step into a one-dimensional vector; The pose embedding vectors at each time step are obtained by mapping the one-dimensional vector of the learnable linear transformation layer to a high-dimensional space. Add positional encoding to the pose embedding vector at each time step to generate a pose embedding sequence.

3. The method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms as described in claim 1, characterized in that, The first feature extraction pathway processes the pose embedding sequence through a multi-head self-attention mechanism to extract global action features representing long-term temporal dependencies, including the following steps: The pose embedding sequence is linearly projected into a query matrix, a key matrix, and a value matrix, and the query matrix, key matrix, and value matrix are split into multiple attention heads; In the attention head, attention weights are calculated based on the relative position encoding, and the value matrix is ​​weighted and summed according to the attention weights to obtain the output of the attention head; The outputs of multiple attention heads are concatenated and fused using a linear projection layer to obtain preliminary attention features; The initial attention features and pose embedding sequence are processed by residual connection and layer normalization, and then transformed by a feedforward neural network to output global action features.

4. The method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms as described in claim 1, characterized in that, The second feature extraction pathway processes the pose embedding sequence through a brain-like spiking neural network to extract local action features representing short-term details, including the following steps: The pose embedding sequence is input into the spiking neuron layer to generate a spiking feature sequence; global average pooling and global max pooling are then performed on the spiking feature sequence along the time dimension. The pooled pulse features are input into the convolution module for convolution processing. The outputs of the two convolutional modules are obtained, and the outputs of the two convolutional modules are added together. Then, a channel attention weight vector is generated by the activation function. The pulse feature sequence is weighted by channel attention weight vector, and the weighted features are batch normalized to output local action features.

5. The method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms as described in claim 1, characterized in that, The process of fusing global and local action features to generate fused features includes concatenating global and local action features along the feature dimension; and mapping the concatenated features to the target dimension through a linear projection layer to generate fused features.

6. The method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms as described in claim 1, characterized in that, The step of performing a residual connection between the fused features and the pose embedding sequence to obtain the enhanced action feature representation includes the following steps: The fused features and pose embedding sequences are added element-wise, and the addition results are processed by layer normalization and enhanced by a feedforward neural network. The output of the feedforward neural network is then added to the sum and residual-connected and normalized to output an enhanced action feature representation.

7. The method for understanding human posture and movement intentions based on brain spiking neural networks and attention mechanisms as described in claim 1, characterized in that, The spiking neuron layer adopts the Leaky Integrate-and-Fire neuron model.

8. A human posture and movement intention understanding system optimized based on brain spiking neural networks and attention mechanisms, characterized in that, It includes a first processing module, which is used to acquire the input human action pose sequence, map each pose to a high-dimensional embedding vector, and generate a pose embedding sequence. The second processing module is used to set up a first feature extraction path and a second feature extraction path; and to simultaneously input the pose embedding sequence into the first feature extraction path and the second feature extraction path. The third processing module is used to process the pose embedding sequence through the first feature extraction path using a multi-head self-attention mechanism to extract global action features that represent long-term dependencies. The fourth processing module is used to process the pose embedding sequence through the second feature extraction pathway using a brain-like spiking neural network to extract local action features that represent short-term details. The fifth processing module is used to fuse global action features and local action features to generate fused features; The sixth processing module is used to perform residual connections between the fused features and the pose embedding sequence to obtain an enhanced action feature representation; the action feature representation is then used for action recognition and intent understanding tasks.

9. A computer device, characterized in that, include: Memory, transceiver, processor, and bus system; The memory is used to store programs; The processor is used to execute the program in the memory, including executing the human posture and movement intention understanding method optimized based on brain spiking neural networks and attention mechanisms as described in any one of claims 1 to 7; The bus system is used to connect the memory and the processor to enable communication between the memory and the processor.

10. A readable storage medium storing computer-readable instructions, characterized in that, When the computer-readable instructions are executed by the processor, they implement the steps of the human posture and movement intention understanding method optimized based on brain spiking neural networks and attention mechanisms as described in any one of claims 1 to 7.

Images