A motion recognition monitoring system and method based on multi-sensor cooperation

Abstract

The application relates to a motion recognition monitoring system and method based on multi-sensor cooperation, which comprises sensor node units, a master control unit and a mobile terminal which are sequentially connected in communication, wherein the mobile terminal is provided with a data processing module and a display screen, the sensor node units comprise at least one sensor sub-node, are used for collecting six-axis data in the human body motion process, and perform preliminary attitude calculation and data packaging processing; the master control unit is used for controlling the working state of the sensor sub-node, and transmitting the data information output by the sensor sub-node to the mobile terminal; and the mobile terminal is used for deep learning inference on the data information output by the sensor sub-node, outputting a motion type recognition result and performing data display. Compared with the prior art, the application can accurately capture different motion states of the human body, and automatically recognize and analyze the motion type in real time.

Description

Technical Field

[0001] This invention relates to the field of intelligent motion monitoring technology, and in particular to a motion recognition and monitoring system and method based on multi-sensor collaboration. Background Technology

[0002] With increasing health awareness and the widespread adoption of smart wearable devices, motion monitoring technology has developed rapidly, especially in applications such as sports health monitoring, rehabilitation training guidance, and sports training assistance. Traditional motion monitoring methods mainly rely on single sensors or simple accelerometers, which suffer from low monitoring accuracy, limited recognition types, and inability to accurately capture complex motion postures. Single-sensor monitoring, using a single accelerometer or gyroscope for motion detection, cannot fully reflect the complexity of human movement, resulting in limited monitoring accuracy. Because it cannot comprehensively capture multi-dimensional information about human movement, the accuracy of motion recognition is low.

[0003] In existing technologies, motion monitoring devices also employ the following methods: 1. GPS-based outdoor sports monitoring: Although it can track movement trajectories, its ability to recognize indoor sports or complex movements is limited.

[0004] 2. Wearable device monitoring: such as smart bracelets and smartwatches, although highly integrated, have a limited number of sensors and are limited by device size, making it difficult to achieve high-precision motion posture recognition.

[0005] 3. Camera-based motion capture: Although it has high accuracy, it is affected by factors such as ambient light and occlusion, and there are privacy issues.

[0006] In addition, the existing communication methods between sensor devices and mobile terminals are limited and susceptible to environmental interference. The data transmission stability is insufficient, resulting in poor communication reliability. The lack of an effective time synchronization mechanism between multiple sensor devices leads to inconsistent data acquisition, affecting the accuracy of motion analysis. Furthermore, traditional motion recognition algorithms have high computational complexity, making it difficult to achieve real-time recognition of motion types. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a motion recognition and monitoring system and method based on multi-sensor collaboration, which can accurately capture different motion states of the human body and perform real-time automatic recognition and analysis of motion types.

[0008] The objective of this invention can be achieved through the following technical solution: a motion recognition and monitoring system based on multi-sensor collaboration, comprising a sensor node unit, a main control unit, and a mobile terminal connected in sequence, wherein the mobile terminal is equipped with a data processing module and a display screen, and the sensor node unit includes at least one sensor sub-node for collecting six-axis data during human motion and performing preliminary posture calculation and data packaging processing. The main control unit is used to control the working status of the sensor sub-nodes and to transmit the data information output by the sensor sub-nodes to the mobile terminal. The mobile terminal is used to perform deep learning inference on the data information output by the sensor sub-nodes, output the motion type recognition result, and display the data.

[0009] Furthermore, the sensor sub-node includes an inertial measurement unit, a microcontroller, and a radio frequency communication module connected in sequence. The inertial measurement unit is used to collect six-axis data during human movement, including three-axis accelerometer data and three-axis gyroscope data. The microcontroller is used to perform preliminary attitude calculation and data packaging processing on the six-axis data. The radio frequency communication module is used to transmit the packaged data to the main control unit.

[0010] Furthermore, the main control unit includes a main controller, an RF receiving module, and a Bluetooth communication module. The main controller and the RF receiving module are respectively connected to the sensor node unit, and the Bluetooth communication module is connected to the mobile terminal. The main controller is used to control the working state of the sensor sub-nodes, the RF receiving module is used to receive data from the sensor node unit and perform time synchronization and aggregation processing, and the Bluetooth communication module is used to transmit the time-synchronized and aggregated data to the mobile terminal.

[0011] Furthermore, the mobile terminal is equipped with a Bluetooth module, which is used to receive time-synchronized and aggregated data from the main control unit, and transmit it to the display screen for display and to the data processing module for deep learning inference.

[0012] Furthermore, the data processing module includes a standardization processing unit, a motion recognition unit, and a storage unit. The standardization processing unit is used to filter and standardize the time-synchronized and aggregated data. The motion recognition unit performs forward inference based on the data output by the standardization processing unit and outputs the motion type recognition result. The storage unit is used to store the time-synchronized and aggregated data and the corresponding motion type recognition result.

[0013] A motion recognition and monitoring method based on multi-sensor collaboration includes the following steps: S1. The sensor node unit collects six-axis data during human movement and performs preliminary attitude calculation and data packaging processing. S2. The main control unit receives data packets from the sensor node unit via radio frequency communication and performs time synchronization and aggregation processing. S3. The mobile terminal receives time-synchronized and aggregated data from the main control unit via Bluetooth communication. Through deep learning inference, it outputs the motion type recognition result and displays the data.

[0014] Furthermore, S1 specifically employs the AHRS (Attitude and Heading Reference System) algorithm to perform preliminary attitude calculations on the six-axis data, obtaining attitude data including yaw angle, pitch angle, and roll angle.

[0015] Furthermore, S1 specifically involves packaging the six-axis data, attitude data, and timestamps into a 32-byte data packet, which includes the data packet sequence number, data status flag, sensor ID, and local timestamp information.

[0016] Furthermore, in S2, after the main control unit receives the data packet from the sensor node unit, it first verifies and manages the timing of the data packet, and then adds the main control timestamp after aggregating multiple data packets.

[0017] Further, S3 includes the following steps: S31. The mobile terminal receives the time-synchronized and aggregated data from the main control unit and performs time alignment processing based on the main control timestamp. S32. Standardize the six-axis data, input the standardized six-axis data into the pre-trained neural network model, output the probability distribution of each motion type through forward inference, and select the motion type with the highest probability as the recognition result. S33. Display the posture data and recognition results.

[0018] Compared with the prior art, the present invention has the following advantages: This invention establishes a sensor node unit, a main control unit, and a mobile terminal connected in sequence. The sensor node unit uses sensor sub-nodes to collect six-axis data during human movement, performing preliminary attitude calculation and data packaging. The main control unit controls the working state of the sensor sub-nodes and transmits the data output from the sensor sub-nodes to the mobile terminal. The mobile terminal performs deep learning inference on the data output from the sensor sub-nodes, outputting and displaying the motion type recognition result. This allows for the accurate capture of different human motion states, and the combination of preliminary attitude calculation and data preprocessing at the sensor node and efficient, edge-computing-based deep learning inference on the mobile terminal enables automatic real-time recognition of motion types.

[0019] This invention incorporates an RF communication module at the sensor sub-nodes to transmit packaged data to the main control unit. The main control unit includes an RF receiving module and a Bluetooth communication module. The RF receiving module receives data from the sensor sub-nodes and performs time synchronization and aggregation processing. The Bluetooth communication module transmits the time-synchronized and aggregated data to the mobile terminal. The mobile terminal also includes a Bluetooth module to receive time-synchronized and aggregated data from the main control unit. This dual-layer architecture, using RF and Bluetooth communication, improves the overall system's communication stability and reliability. RF communication handles data transmission between the sensor sub-nodes and the main control unit, while Bluetooth communication handles data transmission between the main control unit and the mobile terminal.

[0020] This invention designs a time synchronization mechanism in which the sensor node unit includes a local timestamp in the data packet during data packaging. When the master control unit receives the data packet from the sensor node unit, it performs verification and timing management on the data packet and adds the master control timestamp when forwarding it to the mobile terminal. The mobile terminal then aligns the data from different sensor nodes according to the timestamp information to ensure data synchronization. This effectively solves the multi-device synchronization problem, ensuring that the data collected by multiple sensor nodes remains consistent in time, which is beneficial for accurately identifying the corresponding motion type. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the system structure of the present invention; Figure 2 This is a schematic diagram of the method flow of the present invention; The markings in the diagram are as follows: 1. Sensor node unit; 2. Main control unit; 3. Mobile terminal; 101. Inertial measurement unit; 102. Microcontroller; 103. RF communication module; 201. Main controller; 202. RF receiving module; 203. Bluetooth communication module; 301. Data processing module; 302. Display screen; 303. Bluetooth module. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0023] Example like Figure 1 As shown, a motion recognition and monitoring system based on multi-sensor collaboration includes a sensor node unit 1, a main control unit 2, and a mobile terminal 3 connected in sequence. The sensor node unit 1 includes at least one sensor sub-node for collecting six-axis data during human motion and performing preliminary attitude calculation and data packaging processing. The sensor sub-node includes an inertial measurement unit 101, a microcontroller 102, and a radio frequency communication module 103 connected in sequence. The inertial measurement unit 101 is used to collect six-axis data during human motion, including three-axis accelerometer data and three-axis gyroscope data. The microcontroller 102 is used to perform preliminary attitude calculation and data packaging processing on the six-axis data; The radio frequency communication module 103 is used to transmit the packaged data to the main control unit 2; The main control unit 2 is used to control the working status of the sensor sub-nodes and transmit the data information output by the sensor sub-nodes to the mobile terminal 3. The main control unit 2 includes a main controller 201, an RF receiving module 202 and a Bluetooth communication module 203. The main controller 201 and the RF receiving module 202 are respectively connected to the sensor node unit 1, and the Bluetooth communication module 203 is connected to the mobile terminal 3. The main controller 201 is used to control the working status of the sensor sub-nodes, the RF receiving module 202 is used to receive data from the sensor node unit 1 and perform time synchronization and aggregation processing, and the Bluetooth communication module 203 is used to transmit the time-synchronized and aggregated data to the mobile terminal 3. The mobile terminal 3 is used to perform deep learning inference on the data information output by the sensor sub-nodes, output the motion type recognition result and display the data. The mobile terminal 3 is equipped with a data processing module 301 and a display screen 302. The mobile terminal 3 is also equipped with a Bluetooth module 303, which is used to receive time-synchronized and aggregated data from the main control unit 2, and transmit it to the display screen 302 for display and to the data processing module 301 for deep learning inference. The data processing module 301 includes a standardization processing unit, a motion recognition unit, and a storage unit. The standardization processing unit is used to filter and standardize the time-synchronized and aggregated data. The motion recognition unit performs forward inference based on the data output by the standardization processing unit and outputs the motion type recognition result. The storage unit is used to store the time-synchronized and aggregated data and the corresponding motion type recognition result.

[0024] Based on the above system, a motion recognition and monitoring method based on multi-sensor collaboration is implemented, such as... Figure 2 As shown, it includes the following steps: S1. The sensor node unit collects six-axis data during human movement and performs preliminary attitude calculation and data packaging processing. In this embodiment, the AHRS algorithm is used to perform preliminary attitude calculation on the six-axis data to obtain attitude data including yaw angle, pitch angle and roll angle; Specifically, the data packaging process involves packaging six-axis data, attitude data, and timestamps into a 32-byte data packet, which includes the data packet sequence number, data status flag, sensor ID, and local timestamp information. S2. The main control unit receives data packets from the sensor node unit via radio frequency communication and performs time synchronization and aggregation processing. Among them, after receiving data packets from the sensor node unit, the main control unit first verifies and manages the timing of the data packets, and then adds the main control timestamp after aggregating multiple data packets. S3. The mobile terminal receives time-synchronized and processed data from the main control unit via Bluetooth communication. Through deep learning inference, it outputs the motion type recognition result and displays the data. Specifically: S31. The mobile terminal receives the time-synchronized and aggregated data from the main control unit and performs time alignment processing based on the main control timestamp. S32. Standardize the six-axis data, input the standardized six-axis data into the pre-trained neural network model, output the probability distribution of each motion type through forward inference, and select the motion type with the highest probability as the recognition result. S33. Display the posture data and recognition results.

[0025] This embodiment applies the above-described scheme to build a multi-sensor cooperative motion recognition and monitoring system based on the ICM45686 sensor kit and ESP32. The main contents include: I. Hardware Architecture 1.1 The sensor node unit includes at least one sensor sub-node, and each sensor sub-node contains: The ICM45686 six-axis inertial sensor, as the main inertial measurement unit, integrates a three-axis accelerometer and a three-axis gyroscope. It has industry-leading vibration suppression capabilities and temperature stability, and extremely low power consumption (up to 220μA in ultra-low power mode). STM32F103C8T6 microcontroller: As the main control chip for sensor nodes, it is equipped with an ARM Cortex-M3 CPU with a frequency of up to 72MHz, responsible for sensor data acquisition, preprocessing and attitude calculation; Si24R1 RF module: Serves as a communication bridge between sensor nodes and the main control unit. It uses 2.4GHz wireless communication, has extremely low standby power consumption (<1μA), supports a communication distance of 240 meters, and supports three adjustable communication rates of 2Mbps / 1Mbps / 250kbps.

[0026] 1.2 The main control unit uses the ESP32-S3 chip, specifically including: ESP32-S3 microcontroller: integrates 2.4GHz Wi-Fi and Bluetooth 5 (LE), uses an Xtensa® 32-bit LX7 dual-core processor with a clock speed of up to 240MHz, and has 512KB of built-in SRAM; Si24R1 RF receiver module: Pairs with the RF module of the sensor node to receive sensor data; Bluetooth communication module: Establishes a BLE connection with the mobile terminal and transmits processed sensor data.

[0027] II. Software Implementation 2.1 The sensor node software includes: Sensor driver: Implements the initialization, data acquisition, and calibration functions of the ICM45686; Attitude calculation algorithm: An improved AHRS algorithm is adopted, which calculates YPR attitude data (yaw angle, pitch angle, roll angle) in real time based on gyroscope and accelerometer data. Data Packaging Protocol: Packs raw sensor data, calculated attitude data, and timestamps into a 32-byte data packet, including data packet sequence number, data status flag, sensor ID, and timestamp information; Radio frequency communication protocol: Enables reliable data transmission with the main control unit, including data packet retransmission mechanism and connection management.

[0028] 2.2 The main control unit software includes: Radio frequency reception processing: Receive data packets from multiple sensor nodes, and perform data packet verification and timing management; Data aggregation: Aggregating and synchronizing data from multiple sensor nodes over time; Bluetooth transmission: The aggregated data is transmitted to the mobile terminal via the BLE protocol.

[0029] 2.3 Mobile terminal software includes: Bluetooth connection management: Enables a stable connection with the main control unit; Data reception and processing: Receive sensor data and display it in real time; Deep learning inference: Load TensorFlow Lite models to perform real-time motion classification on sensor data; User interface: Provides an intuitive motion monitoring interface that displays sensor data and motion recognition results.

[0030] III. Data Acquisition Process After the sensor node is started, the STM32 is configured with ICM45686 and Si24R1 and enters standby mode. The ESP32 starts up and waits for the phone to connect via Bluetooth; After the Bluetooth connection is established, the mobile phone sends control signals to the ESP32; The ESP32 broadcasts a start signal and performs a TCP-like three-way handshake with multiple STM32s to establish a one-to-many communication channel. After STM32 confirmation, ICM45686 is started to acquire sensor data at a frequency of 400Hz. The STM32 performs attitude calculation on the sensor data, packages the data, and sends it to the ESP32 via the Si24R1. The ESP32 receives data from multiple sensor nodes, performs time synchronization and data aggregation, and transmits it to the mobile terminal via BLE.

[0031] IV. Motion Recognition Algorithm Motion recognition is achieved based on a trained deep learning model, and the specific steps include: Step 1: Data Preprocessing Receive six-axis data (three-axis gyroscope data and three-axis accelerometer data) transmitted from sensor nodes. Standardize the data using pre-trained standardized parameters (mean and standard deviation). The data format is: [gyroX_dps, gyroY_dps, gyroZ_dps, accelX_mg, accelY_mg,accelZ_mg].

[0032] Step 2: Model Inference Load a lightweight neural network model that is optimized for real-time inference on mobile devices; Input standardized six-axis sensor data and perform forward inference; Output the probability distribution for each sport category.

[0033] Step 3: Determine the type of exercise Based on the output probability distribution, the motion category with the highest probability is selected as the recognition result; Common types of exercise include: going down stairs, going up stairs, walking, stepping, sitting, etc.

[0034] V. Communication Protocol A two-layer communication architecture is adopted: Between sensor nodes and the main control unit: communication is achieved using a Si24R1 RF module with a custom protocol that supports data packet sequence number management, retransmission mechanism and connection status monitoring. Communication between the main control unit and the mobile terminal is achieved using BLE low-power Bluetooth and data transmission is performed using the standard GATT protocol.

[0035] VI. Time Synchronization Mechanism To solve the problem of multi-device synchronization, the following time synchronization mechanism is adopted: Each sensor node includes a local timestamp in the data packet; The ESP32 master controller records the time of receiving data packets and adds a master controller timestamp when forwarding them to the mobile terminal; The mobile terminal uses timestamp information to time-align data from different sensor nodes to ensure data synchronization.

[0036] In summary, this solution achieves high-precision motion recognition and monitoring through multi-sensor collaborative operation, accurately capturing the motion states of different parts of the human body. It also automatically identifies and analyzes motion types using intelligent algorithms, offering advantages such as good real-time performance, low power consumption, and high reliability. Compared to existing technologies, this solution has the following significant advantages: 1. Improve monitoring accuracy: By working together with multiple sensor nodes, the movement status of different parts of the human body can be captured comprehensively, significantly improving the accuracy and precision of motion recognition.

[0037] 2. Enhanced communication reliability: The system adopts a two-layer architecture of radio frequency (RF) communication and Bluetooth communication. RF communication is responsible for data transmission between sensor nodes and the main controller, while Bluetooth communication is responsible for data transmission between the main controller and the mobile terminal, which improves the communication stability and reliability of the entire system.

[0038] 3. Real-time processing: Preliminary attitude calculation and data preprocessing are performed at the sensor node, and efficient deep learning inference is performed on the mobile terminal, realizing real-time recognition of motion types.

[0039] 4. Solve the problem of multi-device synchronization: Through timestamp synchronization mechanism and data packet sequence number management, ensure that the data collected by multiple sensor nodes are consistent in time.

[0040] 5. Reduced power consumption: The sensor nodes adopt a low-power design, and the RF module has extremely low standby power consumption, which extends the device's battery life.

[0041] 6. High adaptability: The system can flexibly configure the number of sensor nodes to adapt to different monitoring needs and application scenarios (sports and health monitoring, rehabilitation training guidance, sports training assistance, etc.).

Claims

1. A motion recognition monitoring system based on multi-sensor cooperation, characterized in that, The system includes a sensor node unit (1), a main control unit (2), and a mobile terminal (3) connected in sequence. The mobile terminal (3) is equipped with a data processing module (301) and a display screen (302). The sensor node unit (1) includes at least one sensor sub-node for collecting six-axis data during human movement and performing preliminary attitude calculation and data packaging processing. The main control unit (2) is used to control the working state of the sensor sub-nodes and to transmit the data information output by the sensor sub-nodes to the mobile terminal (3). The mobile terminal (3) is used to perform deep learning inference on the data information output by the sensor sub-nodes, output the motion type recognition result and display the data.

2. The motion recognition monitoring system based on multi-sensor cooperation according to claim 1, characterized in that, The sensor sub-node includes an inertial measurement unit (101), a microcontroller (102), and a radio frequency communication module (103) connected in sequence. The inertial measurement unit (101) is used to collect six-axis data during human movement, including three-axis accelerometer data and three-axis gyroscope data. The microcontroller (102) is used to perform preliminary attitude calculation and data packaging processing on the six-axis data; The radio frequency communication module (103) is used to transmit the packaged data to the main control unit (2).

3. The motion recognition and monitoring system based on multi-sensor collaboration according to claim 2, characterized in that, The main control unit (2) includes a main controller (201), an RF receiving module (202), and a Bluetooth communication module (203). The main controller (201) and the RF receiving module (202) are respectively connected to the sensor node unit (1). The Bluetooth communication module (203) is connected to the mobile terminal (3). The main controller (201) is used to control the working state of the sensor sub-nodes. The RF receiving module (202) is used to receive data from the sensor node unit (1) and perform time synchronization and aggregation processing. The Bluetooth communication module (203) is used to transmit the time-synchronized and aggregated data to the mobile terminal (3).

4. The motion recognition and monitoring system based on multi-sensor collaboration according to claim 3, characterized in that, The mobile terminal (3) is equipped with a Bluetooth module (303) for receiving time-synchronized and aggregated data from the main control unit (2), and transmitting it to the display screen (302) for display and to the data processing module (301) for deep learning inference.

5. A motion recognition and monitoring system based on multi-sensor collaboration according to claim 4, characterized in that, The data processing module (301) includes a standardization processing unit, a motion recognition unit, and a storage unit. The standardization processing unit is used to filter and standardize the time-synchronized and aggregated data. The motion recognition unit performs forward inference based on the data output by the standardization processing unit and outputs the motion type recognition result. The storage unit is used to store the time-synchronized and aggregated data and the corresponding motion type recognition result.

6. A motion recognition and monitoring method based on multi-sensor collaboration, applied to the motion recognition and monitoring system based on multi-sensor collaboration as described in claim 1, characterized in that, Includes the following steps: S1, Sensor node unit (1) collects six-axis data during human movement and performs preliminary attitude calculation and data packaging processing; S2. The main control unit (2) receives data packets from the sensor node unit (1) via radio frequency communication and performs time synchronization and aggregation processing. S3. The mobile terminal (3) receives the time-synchronized and aggregated data from the main control unit (2) via Bluetooth communication, and outputs the motion type recognition result and displays the data through deep learning inference.

7. The motion recognition and monitoring method based on multi-sensor collaboration according to claim 6, characterized in that, Specifically, S1 uses the AHRS algorithm to perform preliminary attitude calculation on the six-axis data to obtain attitude data including yaw angle, pitch angle and roll angle.

8. The motion recognition and monitoring method based on multi-sensor collaboration according to claim 7, characterized in that, Specifically, S1 involves packaging the six-axis data, attitude data, and timestamps into a 32-byte data packet, which includes the data packet sequence number, data status flag, sensor ID, and local timestamp information.

9. The motion recognition and monitoring method based on multi-sensor collaboration according to claim 8, characterized in that, In the S2, after the main control unit (2) receives the data packet from the sensor node unit (1), it first performs verification and timing management on the data packet, and then adds the main control timestamp after aggregating multiple data packets.

10. A motion recognition and monitoring method based on multi-sensor collaboration according to claim 9, characterized in that, S3 includes the following steps: S31. The mobile terminal (3) receives the time-synchronized and aggregated data from the main control unit (2) and performs time alignment processing based on the main control timestamp. S32. Standardize the six-axis data, input the standardized six-axis data into the pre-trained neural network model, output the probability distribution of each motion type through forward inference, and select the motion type with the highest probability as the recognition result. S33. Display the posture data and recognition results.

Images