Human motion recognition method and device based on inertial sensor and electronic equipment

Abstract

The application discloses a human action recognition method and device based on an inertial sensor and electronic equipment, and relates to the fields of artificial intelligence and wearable technology.The method comprises the following steps: acquiring time-series motion data collected by a plurality of wearable inertial sensors, and synchronously aligning the time-series motion data; constructing a sensor topology graph according to the human body limb connection relationship corresponding to the plurality of wearable inertial sensors; extracting local motion features of each sensor node and relative motion features between adjacent sensor nodes based on the time-series motion data and the sensor topology graph; and performing feature modeling on the local motion features and the relative motion features by using a space-time graph neural network, and determining a human action recognition result according to a modeling result.The application introduces a human kinematics topology structure and relative motion features with rotation invariance, solves the problem that a model is sensitive to sensor wearing differences and has poor generalization ability, and improves the robustness and accuracy of action recognition.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more particularly to the field of wearable devices, specifically to a method, device, and electronic device for human motion recognition based on inertial sensors. Background Technology

[0002] With social development, the demand for continuous monitoring of human movements is increasing in fields such as medical rehabilitation, home care, and sports and health. Simultaneously, in low-visibility or complex scenarios such as firefighting, security, and field operations, real-time identification of personnel movement is crucial for ensuring safety and supporting decision-making. Traditional vision-based action recognition solutions are easily affected by environmental factors such as lighting, occlusion, and smoke, and the collected video information involves both faces and the environment, posing a risk of privacy breaches and limiting their application in long-term monitoring scenarios. To overcome these shortcomings, solutions based on wearable inertial sensors have become an important technological approach. In existing technologies, one type of solution collects data such as acceleration and angular velocity by wearing inertial measurement units on multiple parts of the body, then simply stitches the multi-channel data and inputs it into a deep network (such as a convolutional neural network or a recurrent neural network) for recognition. Another type of solution, mainly applied in the vision field, constructs a topological map based on the joints of the human skeleton and uses a spatiotemporal graph neural network for action recognition. Summary of the Invention

[0003] This invention provides a method, device, and electronic device for human motion recognition based on inertial sensors.

[0004] According to a first aspect of the present invention, a human motion recognition method based on inertial sensors is provided, comprising: acquiring temporal motion data collected by multiple wearable inertial sensors and synchronizing the temporal motion data; constructing a sensor topology map according to the human limb connection relationships corresponding to the multiple wearable inertial sensors; the sensor topology map including at least sensor nodes corresponding one-to-one with the wearable inertial sensors; extracting local motion features of each sensor node and relative motion features between adjacent sensor nodes based on the temporal motion data and the sensor topology map; performing feature modeling on the local motion features and relative motion features using a spatiotemporal graph neural network, and determining the human motion recognition result based on the modeling result.

[0005] Optionally, wearable inertial sensors are worn on at least some parts of the human torso, pelvis, and limbs; acquiring time-series motion data collected by multiple wearable inertial sensors, including: acquiring at least one of acceleration data and angular velocity data output by each wearable inertial sensor, and generating time-series motion data according to the corresponding timestamp.

[0006] Optionally, a sensor topology graph is constructed based on the human limb connection relationships corresponding to multiple wearable inertial sensors, including: setting each wearable inertial sensor as a corresponding sensor node; establishing connection edges between the corresponding sensor nodes according to the human limb connection relationships; the human limb connection relationships are pre-constructed based on human anatomical structure; generating an adjacency relationship matrix based on the sensor nodes and connection edges, and normalizing the adjacency relationship matrix to obtain the sensor topology graph.

[0007] Optionally, based on the temporal motion data and sensor topology, the local motion features of each sensor node and the relative motion features between adjacent sensor nodes are extracted, including: extracting at least one of the acceleration features and angular velocity features of each sensor node from the temporal motion data as local motion features; and calculating at least one of the relative rotation features and relative angular velocity features between adjacent sensor nodes as relative motion features based on the temporal motion data and sensor topology.

[0008] Optionally, calculating the relative rotation features between adjacent sensor nodes includes: extracting the attitude information corresponding to adjacent sensor nodes from the temporal motion data; calculating the relative rotation quaternion between adjacent sensor nodes based on the attitude information; and using the relative rotation quaternion as the relative rotation feature.

[0009] Optionally, a spatiotemporal graph neural network is used to model local motion features and relative motion features, and the human action recognition result is determined based on the modeling results. This includes: using a spatiotemporal graph neural network to model local motion features and relative motion features, determining spatial correlation features and temporal evolution features; and performing category mapping based on the spatial correlation features and temporal evolution features to determine the human action recognition result.

[0010] Optionally, the spatiotemporal graph neural network includes at least: a spatial graph convolutional unit and a temporal convolutional unit; the spatiotemporal graph neural network is used to model local motion features and relative motion features to determine spatial correlation features and temporal evolution features, including: using the spatial graph convolutional unit to aggregate local motion features and relative motion features based on the sensor topology graph to generate spatial correlation features; and using the temporal convolutional unit to model the changes of spatial correlation features in the temporal dimension to generate temporal evolution features.

[0011] Optionally, the human action recognition result is determined by class mapping based on spatial association features and temporal evolution features, including: pooling the spatial association features and temporal evolution features to obtain action representation vectors; generating output probabilities corresponding to each candidate action category based on the action representation vectors; and determining the candidate action category with the highest output probability as the human action recognition result.

[0012] According to a second aspect of the present invention, a human motion recognition device based on inertial sensors is provided, comprising: a data acquisition module for acquiring temporal motion data collected by multiple wearable inertial sensors and synchronizing and aligning the temporal motion data; a topology construction module for constructing a sensor topology map based on the connection relationships of human limbs corresponding to the multiple wearable inertial sensors; the sensor topology map includes at least sensor nodes corresponding one-to-one with the wearable inertial sensors; a feature extraction module for extracting local motion features of each sensor node and relative motion features between adjacent sensor nodes based on the temporal motion data and the sensor topology map; and a motion recognition module for performing feature modeling on the local motion features and relative motion features using a spatiotemporal graph neural network, and determining the human motion recognition result based on the modeling result.

[0013] According to a third aspect of the present invention, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform any of the methods in the embodiments of the present invention.

[0014] By adopting the solution of the present invention, the problems of model sensitivity to sensor wearing differences and poor generalization ability can be solved by introducing human kinematic topology and relative motion features with rotational invariance, thereby improving the robustness and accuracy of action recognition.

[0015] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0016] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of the invention. Wherein: Figure 1 This is a flowchart illustrating a human motion recognition method based on an inertial sensor according to an embodiment of the present invention. Figure 2 This is a schematic diagram of a sensor layout scheme according to an embodiment of the present invention; Figure 3 This is another schematic flowchart of the human motion recognition method based on inertial sensors according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the extraction process of relative motion features according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a human motion recognition device based on an inertial sensor according to an embodiment of the present invention; Figure 6This is a structural diagram of an electronic device used to implement the human motion recognition method based on an inertial sensor according to an embodiment of the present invention. Detailed Implementation

[0017] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] In this document, the term "and / or" merely describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document indicates any combination of at least two of a plurality of elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this document refer to and distinguish between multiple similar technical terms, not to restrict the order or to limit there to only two. For example, "first feature" and "second feature" refer to two categories / two features; the first feature can be one or more, and the second feature can also be one or more.

[0019] Furthermore, to better illustrate the present invention, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that the present invention can be practiced without certain specific details. In some instances, methods, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the spirit of the invention.

[0020] Before providing a further detailed description of the embodiments of this application, some of the nouns and terms involved in the embodiments of this application will be explained. The nouns and terms involved in the embodiments of this application are subject to the following interpretations.

[0021] Temporal motion data refers to a sequence of data collected at consecutive points in time by one or more wearable inertial sensors, reflecting the motion state of human limbs. In this invention, it may specifically include at least one of triaxial acceleration data, triaxial angular velocity data, and / or attitude data (such as attitude quaternions). These data collectively constitute a quantitative description of human dynamic behavior.

[0022] Sensor topology graph: This refers to a graph-structured data used to describe the spatial relationships between multiple sensors. In this invention, the graph is constructed based on human anatomy, where each node corresponds one-to-one with a wearable inertial sensor, and the edges represent physical connections between the corresponding sensors and the attached human limbs. This topology graph encodes prior knowledge of human kinematics into the model, providing a structured foundation for understanding coordinated limb movements.

[0023] Relative motion features: These are characteristic quantities used to characterize the relative motion patterns between two adjacent human limbs. In this invention, these features are obtained by calculating the relative rotation information (such as relative rotation quaternions) and / or relative angular velocity information between adjacent sensor nodes. Their advantage lies in their rotation invariance, which reduces interference with recognition results caused by changes in sensor wearing orientation, and is key to improving model robustness.

[0024] Spatiotemporal graph neural networks (SBRNNs) refer to a class of deep learning models specifically designed for processing spatiotemporal graph data. They can simultaneously model both the spatial structure (graph topology) and temporal dynamics (sequence evolution) of the data. In this invention, this network is used to fuse spatial coordination features between human limbs and temporal evolution features of movements. By aggregating spatial information on the graph structure and capturing dynamic changes in the temporal dimension, it achieves high-precision movement recognition.

[0025] This invention provides a human motion recognition method based on inertial sensors. Figure 1 This is a flowchart illustrating a human motion recognition method based on an inertial sensor according to an embodiment of the present invention, as shown below. Figure 1 As shown, this human motion recognition method based on inertial sensors includes: S101. Acquire time-series motion data collected by multiple wearable inertial sensors and synchronize and align the time-series motion data.

[0026] S102. Construct a sensor topology diagram based on the connection relationships of human limbs corresponding to multiple wearable inertial sensors.

[0027] S103. Based on time-series motion data and sensor topology, extract the local motion features of each sensor node and the relative motion features between adjacent sensor nodes.

[0028] S104. Use a spatiotemporal graph neural network to model local motion features and relative motion features, and determine the human motion recognition result based on the modeling results.

[0029] In this embodiment of the invention, the step of acquiring temporal motion data from multiple wearable inertial sensors and synchronizing and aligning the temporal motion data is fundamental to all subsequent analyses. Its purpose is to ensure that data collected from different body parts can be compared and fused under a unified time reference. In existing technologies, the drift of independent clock crystals in distributed sensor networks can lead to sampling asynchrony. Direct processing of this data can introduce severe time misalignment, resulting in incorrect causal relationship judgments, such as misinterpreting a leg movement response as an arm movement trigger. This solution, by establishing a global reference time axis and using an interpolation algorithm for alignment, helps to solve the problem of data asynchrony and provides high-quality data assurance for accurately capturing the temporal relationships of coordinated limb movements.

[0030] In this embodiment of the invention, the step of constructing a sensor topology map based on the connection relationships between multiple wearable inertial sensors and human limbs explicitly incorporates prior knowledge of human kinematics into the model. Traditional methods often treat multi-sensor data as independent channels and stitch them together, neglecting the inherent physical connections between limbs. The model needs to implicitly learn this structure from massive amounts of data, resulting in low efficiency and unstable performance. This solution, by constructing a topology map, abstracts the human body into a multi-rigid-body kinematic chain system, forcing subsequent feature learning and information transmission to follow a pre-defined path that conforms to biomechanical mechanisms. This enables the model to understand collaborative concepts more efficiently and accurately, overcoming the deficiency of existing technologies in lacking structured modeling.

[0031] In this embodiment of the invention, the step of extracting local motion features of each sensor node and relative motion features between adjacent sensor nodes based on temporal motion data and sensor topology aims to address the problem of poor model generalization ability caused by differences in sensor wearing conditions. Local motion features, such as the acceleration and angular velocity of a single sensor, describe the motion state of that limb segment itself. Meanwhile, relative motion features describe the relative relationship between two connected limbs, such as the bending angle of the knee joint or the rotational speed of the elbow joint. By calculating features such as relative rotation and relative angular velocity, the influence of changes in the overall orientation of the human body can be eliminated. For example, regardless of whether a person is facing east or west when raising their hand, the relative angle of their upper arm relative to their torso remains essentially the same. By extracting these rotationally invariant relative motion features, the model can learn the inherent geometric patterns of the movement, rather than surface signals related to the wearing direction, thereby improving robustness across different wearing conditions and individuals.

[0032] In this embodiment of the invention, in the step of using a spatiotemporal graph neural network to model local motion features and relative motion features, and determining the human action recognition result based on the modeling result, the spatiotemporal graph neural network is suitable for processing structured data. Its spatial graph convolution part can aggregate features of adjacent nodes along the edges of the sensor topology graph, thereby understanding limb coordination patterns in the spatial dimension. At the same time, its temporal convolution part can capture the dynamic laws of each feature's evolution over time. Through this spatiotemporal joint modeling approach, the model can abstract low-level physical signals such as acceleration and angular velocity layer by layer into high-level action semantics such as walking, running, and falling, ultimately achieving accurate action classification. This solves the problem that existing technologies struggle to uniformly consider both temporal dynamics and spatial coordination.

[0033] In some embodiments, wearable inertial sensors are worn on at least a portion of the torso, pelvis, and limbs. Simultaneously, temporal motion data collected by multiple wearable inertial sensors is acquired, specifically including acquiring at least one of acceleration and angular velocity data output by each wearable inertial sensor, and generating temporal motion data based on the corresponding timestamps. This sensor layout, for example, deployment in key locations such as the torso, pelvis, thigh, and upper arm, maximizes the capture of information from the main kinetic chains of the entire body while ensuring ease of wear. By acquiring acceleration and angular velocity data, the linear and rotational motion states of the limbs can be comprehensively described, providing rich raw information for subsequent feature extraction.

[0034] In some embodiments, the specific process of constructing a sensor topology graph includes: assigning each wearable inertial sensor as a corresponding sensor node; establishing connecting edges between the corresponding sensor nodes according to the human limb connection relationships, wherein the human limb connection relationships can be pre-constructed based on human anatomical structures; and finally, generating an adjacency matrix based on the sensor nodes and connecting edges, and normalizing the adjacency matrix to obtain the sensor topology graph. This series of operations transforms abstract human structural knowledge into a mathematical form that the model can compute (a normalized adjacency matrix). Normalization is particularly important, as it avoids gradient explosion or vanishing problems caused by different node degrees (number of connections) in graph convolution calculations, ensuring the stability and convergence speed of model training.

[0035] In some embodiments, the specific process of extracting local motion features and relative motion features includes: extracting at least one of the acceleration features and angular velocity features of each sensor node from the temporal motion data as local motion features; and calculating at least one of the relative rotation features and relative angular velocity features between adjacent sensor nodes based on the temporal motion data and the sensor topology map as relative motion features. This dual-stream feature construction method enables the model to perceive the intensity of movement of each limb segment through local motion features, and to understand the relative posture and motion relationship between limbs based on relative motion features. This combination provides richer and more robust motion information than a single feature stream, which is key to achieving high-precision recognition.

[0036] In some embodiments, calculating the relative rotation features between adjacent sensor nodes includes: extracting the attitude information corresponding to adjacent sensor nodes from temporal motion data; calculating the relative rotation quaternions between adjacent sensor nodes based on the attitude information; and using the relative rotation quaternions as the relative rotation features. Representing attitude information using attitude quaternions effectively avoids the gimbal lock problem in traditional Euler angle representation and can describe arbitrary three-dimensional rotations without singularities. The relative rotation quaternions calculated through quaternion conjugation and multiplication operations accurately express the relative attitude of the joints and naturally possess rotational invariance, making it a preferred technical path for constructing high-quality relative motion features.

[0037] In some embodiments, the process of feature modeling and recognition using a spatiotemporal graph neural network includes: modeling local motion features and relative motion features using the spatiotemporal graph neural network to determine spatial correlation features and temporal evolution features; and then performing category mapping based on the spatial correlation features and temporal evolution features to determine the human action recognition result. Specifically, the spatiotemporal graph neural network generates spatial correlation features by aggregating spatial neighbor information and generates temporal evolution features by analyzing sequence changes. The final recognition result is based on these two highly abstracted and fused features, which helps ensure the comprehensiveness and accuracy of the decision.

[0038] In some embodiments, the spatiotemporal graph neural network includes at least a spatial graph convolutional unit and a temporal convolutional unit. The specific process of feature modeling using the spatiotemporal graph neural network is as follows: the spatial graph convolutional unit aggregates local motion features and relative motion features based on the sensor topology map to generate spatial correlation features; then, the temporal convolutional unit models the changes of these spatial correlation features in the temporal dimension to generate temporal evolution features. This structure, which concatenates spatial and temporal convolutions, is an efficient way to achieve spatiotemporal joint modeling. First, the coordinated postures of different parts of the body are understood at each time point, and then these continuous postures are concatenated to understand their dynamic evolution process, resulting in a clear logical progression.

[0039] In some embodiments, the specific steps for class mapping to determine the human action recognition result include: pooling spatial correlation features and temporal evolution features to obtain action representation vectors; generating output probabilities corresponding to each candidate action category based on the action representation vectors; and finally determining the candidate action category with the highest output probability as the human action recognition result. Pooling (such as global average pooling) can compress variable-length, multi-dimensional feature maps into a fixed-length vector, which not only reduces computational complexity but also enables the model to handle action segments of different lengths. Subsequently, generating a probability distribution through fully connected layers and a Softmax function is a standard practice in deep learning classification tasks, ensuring the standardization and interpretability of the final output.

[0040] In a specific scenario, the first step is to deploy sensors and collect data. Figure 2 A schematic diagram of the sensor layout scheme in an embodiment of the present invention is shown, such as... Figure 2 As shown, this embodiment employs a six-node layout, with six nine-axis inertial measurement units (IMUs) worn on the waist, chest, left upper arm, right upper arm, left thigh, and right thigh nodes, respectively. Each sensor collects multidimensional motion data in real time at a sampling frequency of 50Hz. At any time t, the raw data vector output by each sensor node v is defined as... It includes triaxial acceleration data. Three-axis angular velocity data and attitude quaternions .

[0041] Figure 3 This illustrates another flowchart of the human motion recognition method based on inertial sensors according to an embodiment of the present invention, as shown below. Figure 3 As shown, after acquiring the raw data from the wearable inertial sensor, multi-sensor data acquisition and preprocessing are required. Due to the sampling asynchrony problem caused by independent clock crystal drift in distributed sensor networks, this embodiment designs a strict time alignment mechanism. The system selects the on-chip time of the waist node as the global reference to establish a unified time series. For any other slave node, if its raw sampling time does not coincide with the reference time, the system uses a linear interpolation algorithm to map the heterogeneous time axis to the unified time axis. The calculation formula is as follows:

[0042] in, Indicates node v at the reference time The interpolated data, and Original sampling time and The data, and This process ensures that the motion data of all nodes are physically synchronized at the same time, laying the data foundation for subsequent spatiotemporal graph modeling.

[0043] Next, data preprocessing and windowing are performed. The acquired raw inertial sequence contains noise and needs to be cleaned. First, a medium-range filter is applied to remove transient pulse interference caused by sensor collisions or packet loss. Then, a fourth-order Butterworth low-pass filter (e.g., with a cutoff frequency of 20Hz) is used to perform bidirectional filtering on the signal to eliminate high-frequency jitter while avoiding phase delay. Subsequently, to address the issue of large differences in the numerical ranges of different physical quantities (such as acceleration and angular velocity), the Z-score normalization method is used to map all feature channels to a uniform distribution. Finally, an overlapping sliding window technique is used to divide the long-time series data stream into fixed-length sample tensors. For example, the time window length is set to 2 seconds, and the sliding step size is 1 second (i.e., 50% overlap), generating a tensor of the form... The input data is given by B, where B is the batch size, L is the time step, V is the number of sensor nodes (6 in this example), and C is the number of feature channels for each node.

[0044] Then, a topological graph based on human biomechanical mechanisms is constructed. In this embodiment, human limb segments are abstracted as vertices in graph theory to construct an undirected spacetime graph. The node set V corresponds to six key anatomical locations where inertial sensors are worn. Based on the human kinetic chain mechanism, the waist node, as an approximate location of the body's center of mass, is defined as the root node of the kinetic chain. The edge set E contains only node pairs with physiological joint connections or strong cooperative relationships; for example, the waist node is connected to the left and right thigh nodes, and the chest node is connected to the waist node, left upper arm node, and right upper arm node. Based on this, an adjacency matrix is ​​constructed. middle Represents a node and A connection exists between the nodes; otherwise, the value is 0. To improve the numerical stability of graph convolutional networks during feature aggregation, self-loops are introduced, and the adjacency matrix is ​​symmetrically normalized, as shown in the following formula:

[0045] in, It is the identity matrix. To introduce the degree matrix after the self-loop is introduced, its diagonal elements By constructing this normalized adjacency matrix This yields a sensor topology map, which forces motion features to propagate and aggregate along the natural skeletal pathways of the human body.

[0046] Based on the constructed topology graph, a two-stream feature vector is constructed, consisting of node attributes and joint relative motion edge features. The first-stream feature vector is the node local attribute stream, focusing on describing the local motion intensity of individual limb segments. For any node v in the graph, the node feature vector at time step t is... Defined as its normalized triaxial acceleration and triaxial angular velocity splicing, that is ,in This indicates a splicing operation. This part of the feature is the local motion feature.

[0047] The second type of feature is the relative motion attribute flow, which aims to explore the relative motion patterns between adjacent limbs, i.e., relative motion features. Figure 4 A schematic diagram of the extraction process of relative motion features in an embodiment of this disclosure is shown, such as... Figure 4 As shown, for any parent node with a connection With child nodes Let their attitude quaternions be respectively and The system first calculates the relative rotation quaternion of the child node with respect to the parent node. :

[0048] in, Represents the conjugate of quaternions (for unit quaternions, the conjugate is the inverse). Physically, this represents the bending angle and rotational state of a joint, a feature that helps reduce interference from overall body displacement and absolute orientation. Simultaneously, it allows for the calculation of the relative angular velocity of adjacent limbs. The final constructed edge feature vector It consists of relative rotation features and / or relative angular velocity features. By inputting these relative motion features into the subsequent graph neural network, the model can directly learn the intrinsic geometry of the movement, thereby greatly improving the robustness of the system under different wearing orientations.

[0049] Subsequently, feature fusion and action reasoning are performed using the spatiotemporal graph neural network. This embodiment employs a spatiotemporal graph convolutional network (ST-GCN) as the core computing engine, which consists of multiple stacked spatiotemporal graph convolutional units. A typical spatiotemporal graph convolutional unit includes one spatial graph convolutional unit and one temporal convolutional unit. For the _th_ unit in the network... The feature update process of the layer is as follows: First, the spatial graph convolutional unit uses the normalized adjacency matrix Aggregating information from neighboring nodes in the spatial dimension can be calculated as follows:

[0050] in, It is the output feature of the previous layer. It is the learnable weight matrix of this layer. It is an activation function. This step enables spatial information exchange and generates spatial correlation features. Then, the temporal convolutional unit employs, for example... convolution kernel pairs One-dimensional convolution operations are performed along the time axis to capture the temporal evolution of actions and generate temporal evolution features. To build deep networks and avoid gradient vanishing, residual connections can be introduced to add and fuse the input features with the features after spatiotemporal convolution, serving as the final output of the unit. The entire process can be represented as follows: .

[0051] After processing by multiple spatiotemporal graph convolutional units, the network outputs a high-dimensional feature tensor containing rich semantics. To obtain the final classification result, this feature tensor is first subjected to global average pooling in both temporal and spatial node dimensions, compressing the entire action sequence into a fixed-length action representation vector. Then, the action representation vector is input into the fully connected layer, mapping it to the action category space. Finally, the Softmax function is used to transform the output logistic value into a normalized probability distribution. Each element This indicates that the current action belongs to the first... The confidence level of the class. The system ultimately outputs the class with the highest probability as the recognition result.

[0052] During model training, a hybrid loss function incorporating cross-entropy classification loss and kinematic smoothness constraints is used for optimization. Cross-entropy loss drives the model's predictions to approximate the true action class, while kinematic smoothness constraints suppress physiologically inaccurate prediction "flickering" by penalizing drastic changes in prediction probabilities between adjacent time steps, thus enhancing the smoothness of the output. End-to-end parameter training is performed using adaptive gradient algorithms such as Adam.

[0053] Taking running motion recognition as an example, when a user starts running, six sensors collect periodic high-frequency motion data. After preprocessing, the data is constructed into a graph sequence containing node and edge features. For example, the system calculates the relative posture changes of the thigh's periodic forward and backward swings relative to the waist, and the relative posture changes of the upper arm's coordinated swings relative to the torso. These structured spatiotemporal features are input into the ST-GCN network, which abstracts layer by layer, fusing low-level limb swing features into high-level running motion semantics, ultimately outputting the running category and high confidence. According to experimental verification, the method in this embodiment achieves an average accuracy of 96.8% across subjects on a dataset containing eight daily movements. In robustness tests with artificially introduced wearing direction rotation noise, its recognition rate fluctuation is less than 2%, significantly better than traditional methods, indicating high accuracy and strong robustness.

[0054] In another implementation, a simplified feature input can be used to verify the contribution of the human kinematic topology graph structure itself to improving recognition performance. This embodiment aims to illustrate that even without explicitly calculating relative motion features, the model can implicitly learn the cooperative relationships between limbs simply by propagating information on a predefined graph structure. In this embodiment, the steps of data acquisition, preprocessing, and graph construction are exactly the same as in the previous embodiment. The core difference lies in the feature generation stage: this embodiment does not calculate edge features; the model only uses node features (i.e., the absolute acceleration and angular velocity of each sensor) as input. The structure and training strategy of the spatiotemporal graph neural network remain consistent, and the edge weights are adaptively learned by the model when the network performs spatial graph convolution. When processing the "running" action, the model receives the absolute motion signals of each limb. Since the graph convolution operation is restricted to a predefined kinematic chain, the model is forced to learn the correlation between waist movement and thigh movement, thereby indirectly capturing the cooperative pattern. According to ablation experiments, the average recognition accuracy of this embodiment is approximately 93.2%, which, although lower than the previous embodiment, is still significantly higher than the baseline model without a graph structure, proving that simply introducing the human kinematic topology graph structure can effectively enhance the representation ability of cooperative movements.

[0055] Furthermore, the method of this invention is not limited to a specific six-node layout and can be flexibly extended to more sophisticated human models. In an optional implementation, an extended graph topology can be used. For example, based on the above six-node layout, two inertial measurement units are added, worn on the left and right calves respectively. Thus, the sensor layout becomes a seven-node layout. Correspondingly, the sensor topology graph structure is also extended: the waist node connects to the thigh node, and the thigh node then connects to the calf node on the same side, forming a longer lower limb kinematic chain. Data preprocessing, feature generation (including the calculation of node features and edge features), model structure, and training strategies are all consistent with the methodology of the aforementioned core embodiment, only applied to this seven-node graph. When recognizing easily confused movements such as going up and down stairs, the newly added calf sensor can provide more accurate information on knee flexion and extension. The model can utilize the relative motion features between the thigh and calf (i.e., knee joint angle changes) to more clearly distinguish the differences between these two movement patterns. This demonstrates that the core idea of ​​this invention has good scalability and is applicable to sensor networks and human models of varying complexity.

[0056] In another alternative implementation, the present invention is not strongly bound to a specific spatiotemporal graph neural network architecture and can employ an alternative model architecture. In this embodiment, the data acquisition, preprocessing, graph construction, and two-stream feature generation steps are exactly the same as in the core embodiment. The core difference lies in the model architecture: a combination of Graph Convolution Network (GCN) and Long Short-Term Memory (LSTM) is used instead of ST-GCN. Specifically, for each time step in the sample tensor, spatial features are first aggregated on the human body topology graph through a GCN layer to obtain the full-body pose feature vector at that moment. Then, the pose feature vectors of all time steps are organized sequentially into a sequence and input into an LSTM network, which learns and models the temporal dependencies of the actions. The final classification layer is connected after the output of the LSTM. In this architecture, the GCN module is responsible for "understanding" the coordinated poses of different parts of the body at each time point, while the LSTM module is responsible for connecting these continuous "pose snapshots" to understand their dynamic evolution. This embodiment is expected to achieve recognition performance similar to that of the core embodiment, demonstrating that relative motion features based on kinematic topology, as a powerful feature engineering method, can be combined with various mainstream spatiotemporal data modeling frameworks and produce significant technical effects.

[0057] In summary, the human motion recognition method based on inertial sensors proposed in this invention, by introducing a human kinematic topology, enables the model to fuse information from multiple body parts according to joint connections, enhancing its ability to express coordinated limb movements. Simultaneously, by constructing relative motion features such as relative posture and / or relative angular velocity, the impact of changes in wearing direction and individual differences on recognition results is reduced, improving cross-scene robustness. Furthermore, a spatiotemporal graph neural network is employed to jointly model spatial coordination and temporal dynamics, improving the accuracy and stability of motion classification. Since this invention does not rely on visible light images, it can operate in low-visibility environments such as darkness, dense smoke, and obstructions, and only outputs motion categories and confidence levels, offering good privacy protection. It is suitable for various application scenarios such as fire rescue, field operations, security patrols, medical rehabilitation, and home monitoring.

[0058] This invention provides a human motion recognition device based on an inertial sensor, such as... Figure 5 As shown, the device may include: a data acquisition module 501, used to acquire temporal motion data collected by multiple wearable inertial sensors and synchronize and align the temporal motion data; a topology construction module 502, used to construct a sensor topology map based on the human limb connection relationships corresponding to the multiple wearable inertial sensors; the sensor topology map includes at least sensor nodes corresponding one-to-one with the wearable inertial sensors; a feature extraction module 503, used to extract local motion features of each sensor node and relative motion features between adjacent sensor nodes based on the temporal motion data and the sensor topology map; and an action recognition module 504, used to perform feature modeling on the local motion features and relative motion features using a spatiotemporal graph neural network, and determine the human action recognition result based on the modeling result.

[0059] In some embodiments, wearable inertial sensors are worn on at least some parts of the human torso, pelvis, and limbs; the data acquisition module 501 is specifically used to: acquire at least one of the acceleration data and angular velocity data output by each wearable inertial sensor, and generate time-series motion data according to the corresponding timestamp.

[0060] In some embodiments, the topology construction module 502 is specifically used to: set each wearable inertial sensor as a corresponding sensor node; establish connection edges between the corresponding sensor nodes according to the human limb connection relationship; the human limb connection relationship is pre-constructed based on the human anatomical structure; generate an adjacency relationship matrix according to the sensor nodes and connection edges, and normalize the adjacency relationship matrix to obtain a sensor topology graph.

[0061] In some embodiments, the feature extraction module 503 is specifically used to: extract at least one of the acceleration features and angular velocity features of each sensor node from the time-series motion data as local motion features; and calculate at least one of the relative rotation features and relative angular velocity features between adjacent sensor nodes as relative motion features based on the time-series motion data and the sensor topology map.

[0062] In some embodiments, the feature extraction module 503 is further configured to: extract the attitude information corresponding to adjacent sensor nodes from the temporal motion data; calculate the relative rotation quaternion between adjacent sensor nodes based on the attitude information; and use the relative rotation quaternion as a relative rotation feature.

[0063] In some embodiments, the action recognition module 504 is specifically used to: use a spatiotemporal graph neural network to model local motion features and relative motion features, determine spatial correlation features and temporal evolution features; perform category mapping based on spatial correlation features and temporal evolution features, and determine the human action recognition result.

[0064] In some embodiments, the spatiotemporal graph neural network includes at least: a spatial graph convolutional unit and a temporal convolutional unit; the action recognition module 504 is further configured to: use the spatial graph convolutional unit to aggregate local motion features and relative motion features based on the sensor topology graph to generate spatial correlation features; and use the temporal convolutional unit to model the changes of spatial correlation features in the time dimension to generate temporal evolution features.

[0065] In some embodiments, the action recognition module 504 is further configured to: perform pooling processing on spatial correlation features and temporal evolution features to obtain action representation vectors; generate output probabilities corresponding to each candidate action category based on the action representation vectors; and determine the candidate action category with the highest output probability as the human action recognition result.

[0066] The specific functions and examples of each module and submodule of the device in this embodiment of the invention can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0067] The human motion recognition device based on inertial sensors in this invention introduces a human kinematic topology, enabling the model to fuse information from multiple body parts according to joint connections, thus enhancing its ability to express coordinated limb movements. Simultaneously, by constructing relative motion features such as relative posture and / or relative angular velocity, it reduces the impact of changes in wearing direction and individual differences on recognition results, improving cross-scene robustness. Furthermore, it employs a spatiotemporal graph neural network to jointly model spatial coordination and temporal dynamics, improving the accuracy and stability of motion classification. Since this invention does not rely on visible light images, it can operate in low-visibility environments such as darkness, dense smoke, and obstructions, and only outputs motion categories and confidence levels, offering good privacy protection. It is suitable for various application scenarios such as fire rescue, field operations, security patrols, medical rehabilitation, and home monitoring.

[0068] According to embodiments of the present invention, an electronic device is also provided.

[0069] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0070] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. The RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. An input / output (I / O) interface 605 is also connected to bus 604.

[0071] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0072] The computing unit 601 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as the inertial sensor-based human motion recognition method. For example, in some embodiments, the inertial sensor-based human motion recognition method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the inertial sensor-based human motion recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured by any other suitable means (e.g., by means of firmware) to perform a human motion recognition method based on inertial sensors.

[0073] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0074] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for human motion recognition based on inertial sensors, characterized in that, The method includes: Acquire temporal motion data collected by multiple wearable inertial sensors and synchronize and align the temporal motion data. Based on the human limb connection relationships corresponding to the multiple wearable inertial sensors, a sensor topology diagram is constructed; the sensor topology diagram includes at least sensor nodes that correspond one-to-one with each of the wearable inertial sensors; Based on the time-series motion data and the sensor topology map, the local motion features of each sensor node and the relative motion features between adjacent sensor nodes are extracted. The local motion features and the relative motion features are modeled using a spatiotemporal graph neural network, and the human motion recognition result is determined based on the modeling results.

2. The method of claim 1, wherein, The wearable inertial sensors are respectively worn on the human torso, pelvis and at least some of the limbs. The acquisition of time-series motion data collected by multiple wearable inertial sensors includes: At least one of the acceleration data and angular velocity data output by each of the wearable inertial sensors is acquired, and the time-series motion data is generated based on the corresponding timestamps.

3. The method of claim 1, wherein, The step of constructing a sensor topology map based on the connection relationships of the human limbs corresponding to the multiple wearable inertial sensors includes: Each of the wearable inertial sensors is configured as a corresponding sensor node; According to the described human limb connection relationships, connection edges are established between the corresponding sensor nodes; the human limb connection relationships are pre-constructed based on human anatomical structures. An adjacency matrix is ​​generated based on the sensor nodes and the connecting edges, and the adjacency matrix is ​​normalized to obtain the sensor topology graph.

4. The method of claim 1, wherein, The step of extracting local motion features of each sensor node and relative motion features between adjacent sensor nodes based on the temporal motion data and the sensor topology map includes: From the time-series motion data, at least one of the acceleration features and angular velocity features of each sensor node is extracted as the local motion feature; Based on the time-series motion data and the sensor topology, at least one of the relative rotation feature and relative angular velocity feature between adjacent sensor nodes is calculated as the relative motion feature.

5. The method of claim 4, wherein, Calculate the relative rotation features between adjacent sensor nodes, including: Extract the attitude information corresponding to adjacent sensor nodes from the temporal motion data; Calculate the relative rotation quaternion between adjacent sensor nodes based on the attitude information; The relative rotation quaternion is used as the relative rotation feature.

6. The method of claim 1, wherein, The step of using a spatiotemporal graph neural network to model the local motion features and the relative motion features, and determining the human action recognition result based on the modeling result, includes: The spatiotemporal graph neural network is used to model the local motion features and the relative motion features to determine the spatial correlation features and temporal evolution features; The human action recognition result is determined by performing category mapping based on the spatial association features and the temporal evolution features.

7. The method of claim 6, wherein, The spatiotemporal graph neural network includes at least: spatial graph convolutional units and temporal convolutional units; The step of using the spatiotemporal graph neural network to model the local motion features and the relative motion features, and determining the spatial correlation features and temporal evolution features, includes: The spatial graph convolutional unit is used to aggregate the local motion features and the relative motion features based on the sensor topology graph to generate the spatial correlation features; The temporal convolutional unit is used to model the changes of the spatial correlation features in the temporal dimension to generate the temporal evolution features.

8. The method of claim 6, wherein, The step of determining the human action recognition result by performing category mapping based on the spatial association features and the temporal evolution features includes: Pooling is performed on the spatial correlation features and the temporal evolution features to obtain the action representation vector; Generate the output probability corresponding to each candidate action category based on the action representation vector; The candidate action category with the highest output probability is determined as the human action recognition result.

9. A human motion recognition apparatus based on inertial sensors, characterized by, The device includes: The data acquisition module is used to acquire time-series motion data collected by multiple wearable inertial sensors and to synchronize and align the time-series motion data. The topology construction module is used to construct a sensor topology diagram based on the human limb connection relationships corresponding to the multiple wearable inertial sensors; the sensor topology diagram includes at least sensor nodes that correspond one-to-one with the wearable inertial sensors. The feature extraction module is used to extract the local motion features of each sensor node and the relative motion features between adjacent sensor nodes based on the time-series motion data and the sensor topology map. The action recognition module is used to perform feature modeling on the local motion features and the relative motion features using a spatiotemporal graph neural network, and to determine the human action recognition result based on the modeling result.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory that is communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

Images