Human motion reconstruction method based on multi-modal signals

Abstract

This invention relates to human motion reconstruction, specifically to a method for human motion reconstruction based on multimodal signals. The method involves acquiring multimodal signals for human motion reconstruction and preprocessing them; aligning the preprocessed multimodal signals in time to ensure that all signals are analyzed under the same time reference, providing accurate time correspondence for subsequent fusion calculations; using sEMG signals as the primary source, analyzing direct data of human motion through AI to initially determine the type of human motion; fusing sEMG signals with IMU data and combining them with image data to further accurately determine the posture of the human motion, ensuring the accuracy and stability of motion modeling; and reconstructing the specific human movements in the environment based on the human motion model to achieve a complete and accurate description of human motion. The technical solution provided by this invention effectively overcomes the shortcomings of existing technologies in achieving complete, accurate, and stable reconstruction of human motion.

Description

Technical Field

[0001] This invention relates to human motion reconstruction, and more specifically to a method for human motion reconstruction based on multimodal signals. Background Technology

[0002] Human motion reconstruction has wide applications in many fields, such as motion capture and posture analysis, rehabilitation training assessment, human-computer interaction systems, and smart wearables and motion analysis. However, the existing signal acquisition methods currently relied upon for human motion reconstruction have many limitations.

[0003] Traditional human motion reconstruction often relies on a single sensing method, such as a camera. Although it can acquire the appearance information of human motion, it is easily affected by occlusion and is difficult to work stably in complex environments. Inertial sensors can collect acceleration and angular velocity data to obtain the motion state of key parts of the human body, but long-term operation will cause data drift, resulting in a gradual increase in measurement error. Although motion capture gloves can capture hand movements relatively accurately, they are expensive and the system is complex, resulting in high usage and maintenance costs.

[0004] The information obtained by these single sensing methods is often incomplete, making it difficult to comprehensively and accurately describe real human movements. At the same time, noise interference and signal obstruction can easily lead to movement recognition failures during the movement recognition process, especially for complex movements, where a single sensing method is even more difficult to achieve accurate reconstruction.

[0005] Furthermore, most existing technologies focus on one or a few traditional sensors (such as image sensors, IMUs, etc.) or concentrate on a specific algorithm model (such as vision-based algorithms), lacking the synergistic use of multiple signals and failing to fully leverage the advantages of different modal signals, thus failing to improve the overall accuracy and stability of human motion reconstruction. Therefore, developing a multimodal signal-based human motion reconstruction method that can completely, accurately, and stably describe real movements is of great significance. Summary of the Invention

[0006] (a) Technical problems to be solved In view of the above-mentioned shortcomings of the existing technology, the present invention provides a human motion reconstruction method based on multimodal signals, which can effectively overcome the defects of the existing technology in that it is difficult to reconstruct human motion completely, accurately and stably.

[0007] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: The human motion reconstruction method based on multimodal signals includes the following steps: S1. Acquire multimodal signals for human motion reconstruction and preprocess the multimodal signals; S2. Time-align the preprocessed multimodal signals to ensure that each signal is analyzed under the same time reference, providing an accurate time correspondence for subsequent fusion calculations; S3. Using sEMG signals as the primary source, AI is used to analyze direct data of human movements to preliminarily determine the type of human movement. S4. The sEMG signal and IMU data are fused and calculated, and combined with image data, the posture of human movement is further accurately determined to ensure the accuracy and stability of motion modeling. S5. Reconstruct the specific movements of the human body in the environment based on the human motion model to achieve a complete and accurate description of human movements.

[0008] Preferably, the multimodal signals acquired in S1 for human motion reconstruction include: sEMG signal acquisition: Surface electromyography (sEMG) sensors are attached to the surface of relevant muscles in the human body to collect electrical signals generated by muscle activity. Surface EMG sensors can record the contraction and relaxation of muscles in real time, reflecting the direct physiological information of human movement. IMU data acquisition: Inertial measurement units are worn on key parts of the human body to collect acceleration and angular velocity data, and to obtain motion state information of key parts of the human body in real time; Image data acquisition: Using a camera to acquire image data of the environment in which the human body is located, including images of the overall posture of the human body and the hand area, providing rich environmental information and appearance features of human movements.

[0009] Preferably, S1 involves preprocessing the multimodal signal, including: For sEMG signals: Filtering: The sEMG signal is filtered using a bandpass filter to remove high-frequency noise and low-frequency interference; Rectification: The filtered sEMG signal is rectified to convert the AC signal into a DC signal for easier subsequent analysis; Smoothing: The rectified sEMG signal is smoothed using a moving average method to further remove noise.

[0010] Preferably, S1 involves preprocessing the multimodal signal, including: For IMU data: Zero bias correction: Since the inertial measurement unit has a zero bias error, data is collected in a stationary state for a period of time, and the average values ​​of acceleration and angular velocity are calculated as the zero bias value. Subsequent data are then corrected by subtracting the corresponding zero bias value. Coordinate system transformation: The calibrated IMU data is transformed from the local coordinate system to the human coordinate system to correspond with human movements. Coordinate system transformation is achieved by establishing a rotation matrix and a translation vector between the local coordinate system and the human coordinate system.

[0011] Preferably, S1 involves preprocessing the multimodal signal, including: For image data: Image enhancement: Histogram equalization is used to enhance image data, improve image contrast, and make human body contours clearer; Human body detection and segmentation: The target detection algorithm is used to detect human bodies in the enhanced image data, and the image segmentation algorithm is used to segment the human body from the background and extract the human body region image; Key point detection: Key point detection algorithms are used to detect key points in human body region images to obtain key information about human posture.

[0012] Preferably, in S2, the preprocessed multimodal signals are time-aligned to ensure that all signals are analyzed under the same time reference, providing accurate time correspondence for subsequent fusion calculations, including: S21. Timestamp marking: When acquiring multimodal signals, a timestamp is added to each data point to record the precise time of signal acquisition; S22. Time Synchronization: A timestamp-based time synchronization algorithm is adopted to align the preprocessed multimodal signals according to the timestamps. Time alignment is performed by finding the data points with the closest timestamps in each timestamp sequence.

[0013] Preferably, in S3, sEMG signals are the primary signal, and direct data of human movements are analyzed using AI to preliminarily determine the type of human movement, including: S31. Feature Extraction: Extract time-domain and frequency-domain features from the preprocessed sEMG signal; S32. Action Classification: Construct an action classification model based on machine learning or deep learning, input the extracted features into the action classification model, and obtain preliminary classification results of human actions.

[0014] Preferably, in S4, the sEMG signal and IMU data are fused and calculated, and the human body posture is further accurately determined by combining image data, ensuring the accuracy and stability of motion modeling, including: S41. Fusion Calculation: The human motion state vector obtained from the preliminary analysis of the sEMG signal is fused with the human motion state vector obtained from the IMU data using the Kalman filter algorithm to obtain the fused motion state vector. S42. Combining image data: The fused motion state vector is combined with the human body key points in the image data. By constructing a human motion model, the fused motion information and human body key point information are matched and optimized to further accurately determine the posture of the human body motion.

[0015] Preferably, the Kalman filter algorithm in the S41 fusion calculation includes: S411. Based on the system state transition matrix F and the process noise covariance matrix Q, predict the current state and covariance matrix: ; ; in, This is a priori estimate of the current state based on the state at the previous time step. This is the posterior estimate of the state at the previous time step. Prior estimate of the current state The corresponding covariance matrix, Posterior estimate of the state at the previous time step The corresponding covariance matrix; S412. Update the current state and covariance matrix based on the observation model matrix H and the observation noise covariance matrix R: ; ; ; in, The Kalman gain at the current moment. This is the posterior estimate of the state at the current moment. The observed value at the current moment, The posterior estimate of the current state The corresponding covariance matrix.

[0016] Preferably, in step S5, the specific movements of the human body in the environment are reconstructed based on the human motion model to achieve a complete and accurate description of human movements, including: S51. Combine the human motion model with the human kinematics model to reconstruct the motion trajectory and posture of various parts of the human body; S52. Place the three-dimensional human geometric model in the environment provided by the image data, and drive the three-dimensional human geometric model to move according to the reconstructed motion information to realize the reconstruction of the specific actions of the human body in the environment.

[0017] (III) Beneficial Effects Compared with existing technologies, the human motion reconstruction method based on multimodal signals provided by this invention has the following beneficial effects: 1) Multimodal fusion and complementarity enhance the integrity of motion reconstruction. Using sEMG signals as the primary signal, while also integrating IMU data and image data, multiple signals mutually verify and complement each other. sEMG signals can reflect the direct physiological information of human movements, without being limited by the environment or usage scenario; IMU data can acquire the motion status of key parts of the human body in real time; and image data provides environmental information and the appearance features of human movements. This multimodal fusion method overcomes the shortcomings of incomplete information from a single sensing method, and can comprehensively acquire various types of information about human movements, thereby completely reconstructing human movements and providing richer data support for subsequent analysis and applications. 2) Innovate main signal analysis to improve the accuracy of action judgment. Using sEMG signals as the primary analysis object, this study analyzes direct data of human movements through AI and constructs a movement classification model based on machine learning or deep learning to initially determine the type of human movement. Since sEMG signals are directly related to human muscle activity, they can more accurately reflect the intention of human movement. On this basis, IMU data is combined for fusion calculation to further accurately determine the posture of human movement. Then, it is combined with human key points in image data for optimization to construct a human movement model. This layered and progressive analysis method with multiple signals working together significantly improves the accuracy of human movement judgment and can more accurately identify various complex movements. 3) Signal processing and synchronization to ensure the stability of motion reconstruction. After signal acquisition, targeted preprocessing is performed on sEMG signals, IMU data, and image data, such as filtering, rectification, and smoothing of sEMG signals, zero-bias correction and coordinate system transformation of IMU data, and enhancement, detection, and segmentation of image data. This effectively removes noise and interference and improves signal quality. At the same time, high-precision time alignment is performed on the preprocessed multimodal signals through timestamp marking and time synchronization algorithms to ensure that each signal is analyzed under the same time reference, providing accurate time correspondence for subsequent fusion calculations. These measures ensure the stability of the motion reconstruction process, reduce errors and fluctuations caused by signal problems, and make the reconstructed human motion more reliable. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0019] Figure 1 This is a schematic diagram of the process of the present invention; Figure 2 This is a detailed flowchart of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0021] The following describes the specific process of the human motion reconstruction method based on multimodal signals provided by this invention, using specific examples (e.g.) Figure 1 and Figure 2 (as shown) and technical effects.

[0022] S1. Acquire multimodal signals for human motion reconstruction and preprocess the multimodal signals.

[0023] Specifically, S1 acquires multimodal signals for human motion reconstruction, including: sEMG signal acquisition: Surface electromyography (sEMG) sensors are attached to the surface of relevant muscles in the human body to collect electrical signals generated by muscle activity. Surface EMG sensors can record the contraction and relaxation of muscles in real time, reflecting the direct physiological information of human movement. IMU data acquisition: Inertial measurement units are worn on key parts of the human body to collect acceleration and angular velocity data, and to obtain motion state information of key parts of the human body in real time; Image data acquisition: Using a camera to acquire image data of the environment in which the human body is located, including images of the overall posture of the human body and the hand area, providing rich environmental information and appearance features of human movements.

[0024] Specifically, S1 performs preprocessing on the multimodal signal, including: 1) For sEMG signals: Filtering: The sEMG signal is filtered using a bandpass filter to remove high-frequency noise and low-frequency interference; Rectification: The filtered sEMG signal is rectified to convert the AC signal into a DC signal for easier subsequent analysis; Smoothing: The rectified sEMG signal is smoothed using a moving average method to further remove noise; 2) For IMU data: Zero bias correction: Since the inertial measurement unit has a zero bias error, data is collected in a stationary state for a period of time, and the average values ​​of acceleration and angular velocity are calculated as the zero bias value. Subsequent data are then corrected by subtracting the corresponding zero bias value. Coordinate system transformation: The calibrated IMU data is transformed from the local coordinate system to the human coordinate system to correspond with human movements. Coordinate system transformation is achieved by establishing a rotation matrix and translation vector between the local coordinate system and the human coordinate system. 3) For image data: Image enhancement: Histogram equalization is used to enhance image data, improve image contrast, and make human body contours clearer; Human body detection and segmentation: The target detection algorithm is used to detect human bodies in the enhanced image data, and the image segmentation algorithm is used to segment the human body from the background and extract the human body region image; Key point detection: Key point detection algorithms are used to detect key points in human body region images to obtain key information about human posture.

[0025] In this technical solution, sEMG signals are the primary signal, while IMU data and image data are also integrated. The multiple signals verify and complement each other. sEMG signals can reflect the direct physiological information of human movements and are not limited by the environment or usage scenario. IMU data can acquire the movement status of key parts of the human body in real time. Image data provides environmental information and the appearance features of human movements. This multimodal fusion method overcomes the shortcomings of incomplete information from a single sensing method, and can comprehensively acquire various types of information about human movements, thereby completely reconstructing human movements and providing richer data support for subsequent analysis and applications.

[0026] S2. Time-align the preprocessed multimodal signals to ensure that all signals are analyzed under the same time reference, providing accurate time correspondence for subsequent fusion calculations, including: S21. Timestamp marking: When acquiring multimodal signals, a timestamp is added to each data point to record the precise time of signal acquisition; S22. Time Synchronization: A timestamp-based time synchronization algorithm is adopted to align the preprocessed multimodal signals according to the timestamps. Time alignment is performed by finding the data points with the closest timestamps in each timestamp sequence.

[0027] The above technical solution performs targeted preprocessing on sEMG signals, IMU data, and image data after signal acquisition. This includes filtering, rectification, and smoothing of sEMG signals; zero-bias correction and coordinate system transformation of IMU data; and enhancement, detection, and segmentation of image data. This effectively removes noise and interference, improving signal quality. Simultaneously, high-precision time alignment is achieved through timestamp marking and time synchronization algorithms, ensuring that all signals are analyzed under the same time reference. This provides accurate time correspondence for subsequent fusion calculations. These measures guarantee the stability of the motion reconstruction process, reduce errors and fluctuations caused by signal problems, and make the reconstructed human motion more reliable.

[0028] S3. Primarily using sEMG signals, AI is employed to analyze direct data on human movements and preliminarily determine the types of human movements, including: S31. Feature Extraction: Extract time-domain and frequency-domain features from the preprocessed sEMG signal; S32. Action Classification: Construct an action classification model based on machine learning or deep learning, input the extracted features into the action classification model, and obtain preliminary classification results of human actions.

[0029] S4. The sEMG signal and IMU data are fused and calculated, and combined with image data, the posture of the human body is further accurately determined to ensure the accuracy and stability of motion modeling, including: S41. Fusion Calculation: The human motion state vector obtained from the preliminary analysis of the sEMG signal is fused with the human motion state vector obtained from the IMU data using the Kalman filter algorithm to obtain the fused motion state vector. S42. Combining image data: The fused motion state vector is combined with the human body key points in the image data. By constructing a human motion model, the fused motion information and human body key point information are matched and optimized to further accurately determine the posture of the human body motion.

[0030] Specifically, the Kalman filter algorithm in the S41 fusion computation includes: S411. Based on the system state transition matrix F and the process noise covariance matrix Q, predict the current state and covariance matrix: ; ; in, This is a priori estimate of the current state based on the state at the previous time step. This is the posterior estimate of the state at the previous time step. Prior estimate of the current state The corresponding covariance matrix, Posterior estimate of the state at the previous time step The corresponding covariance matrix; S412. Update the current state and covariance matrix based on the observation model matrix H and the observation noise covariance matrix R: ; ; ; in, The Kalman gain at the current moment. This is the posterior estimate of the state at the current moment. The observed value at the current moment, The posterior estimate of the current state The corresponding covariance matrix.

[0031] The above technical solution uses sEMG signals as the primary analysis object. It analyzes direct data of human movements through AI and constructs a movement classification model based on machine learning or deep learning to initially determine the type of human movement. Since sEMG signals are directly related to human muscle activity, they can more accurately reflect the intention of human movement. On this basis, it combines IMU data for fusion calculation to further accurately determine the posture of human movement. Then, it combines and optimizes the human body key points in the image data to construct a human movement model. This layered and progressive analysis method with multiple signals significantly improves the accuracy of human movement judgment and can more accurately identify various complex movements.

[0032] S5. Reconstruct the specific movements of the human body in the environment based on the human motion model to achieve a complete and accurate description of human movements, including: S51. Combine the human motion model with the human kinematics model to reconstruct the motion trajectory and posture of various parts of the human body; S52. Place the three-dimensional human geometric model in the environment provided by the image data, and drive the three-dimensional human geometric model to move according to the reconstructed motion information to realize the reconstruction of the specific actions of the human body in the environment.

[0033] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for human motion reconstruction based on multimodal signals, characterized in that: Includes the following steps: S1. Acquire multimodal signals for human motion reconstruction and preprocess the multimodal signals; S2. Time-align the preprocessed multimodal signals to ensure that each signal is analyzed under the same time reference, providing an accurate time correspondence for subsequent fusion calculations; S3. Using sEMG signals as the primary source, AI is used to analyze direct data of human movements to preliminarily determine the type of human movement. S4. The sEMG signal and IMU data are fused and calculated, and combined with image data, the posture of human movement is further accurately determined to ensure the accuracy and stability of motion modeling. S5. Reconstruct the specific movements of the human body in the environment based on the human motion model to achieve a complete and accurate description of human movements.

2. The human motion reconstruction method based on multimodal signals according to claim 1, characterized in that: S1 acquires multimodal signals for human motion reconstruction, including: sEMG signal acquisition: Surface electromyography (sEMG) sensors are attached to the surface of relevant muscles in the human body to collect electrical signals generated by muscle activity. Surface EMG sensors can record the contraction and relaxation of muscles in real time, reflecting the direct physiological information of human movement. IMU data acquisition: Inertial measurement units are worn on key parts of the human body to collect acceleration and angular velocity data, and to obtain motion state information of key parts of the human body in real time; Image data acquisition: Using a camera to acquire image data of the environment in which the human body is located, including images of the overall posture of the human body and the hand area, providing rich environmental information and appearance features of human movements.

3. The human motion reconstruction method based on multimodal signals according to claim 2, characterized in that: S1 performs preprocessing on the multimodal signal, including: For sEMG signals: Filtering: The sEMG signal is filtered using a bandpass filter to remove high-frequency noise and low-frequency interference; Rectification: The filtered sEMG signal is rectified to convert the AC signal into a DC signal for easier subsequent analysis; Smoothing: The rectified sEMG signal is smoothed using a moving average method to further remove noise.

4. The human motion reconstruction method based on multimodal signals according to claim 2, characterized in that: S1 performs preprocessing on the multimodal signal, including: For IMU data: Zero bias correction: Since the inertial measurement unit has a zero bias error, data is collected in a stationary state for a period of time, and the average values ​​of acceleration and angular velocity are calculated as the zero bias value. Subsequent data are then corrected by subtracting the corresponding zero bias value. Coordinate system transformation: The calibrated IMU data is transformed from the local coordinate system to the human coordinate system to correspond with human movements. Coordinate system transformation is achieved by establishing a rotation matrix and a translation vector between the local coordinate system and the human coordinate system.

5. The human motion reconstruction method based on multimodal signals according to claim 2, characterized in that: S1 performs preprocessing on the multimodal signal, including: For image data: Image enhancement: Histogram equalization is used to enhance image data, improve image contrast, and make human body contours clearer; Human body detection and segmentation: The target detection algorithm is used to detect human bodies in the enhanced image data, and the image segmentation algorithm is used to segment the human body from the background and extract the human body region image; Key point detection: Key point detection algorithms are used to detect key points in human body region images to obtain key information about human posture.

6. The human motion reconstruction method based on multimodal signals according to claim 2, characterized in that: In S2, the preprocessed multimodal signals are time-aligned to ensure that all signals are analyzed under the same time reference, providing accurate time correspondence for subsequent fusion calculations, including: S21. Timestamp marking: When acquiring multimodal signals, a timestamp is added to each data point to record the precise time of signal acquisition; S22. Time Synchronization: A timestamp-based time synchronization algorithm is adopted to align the preprocessed multimodal signals according to the timestamps. Time alignment is performed by finding the data points with the closest timestamps in each timestamp sequence.

7. The human motion reconstruction method based on multimodal signals according to claim 6, characterized in that: S3 primarily uses sEMG signals and employs AI to analyze direct data on human movements, initially determining the type of human movement, including: S31. Feature Extraction: Extract time-domain and frequency-domain features from the preprocessed sEMG signal; S32. Action Classification: Construct an action classification model based on machine learning or deep learning, input the extracted features into the action classification model, and obtain preliminary classification results of human actions.

8. The human motion reconstruction method based on multimodal signals according to claim 7, characterized in that: In S4, sEMG signals and IMU data are fused and calculated, and combined with image data, the posture of human movements is further accurately determined to ensure the accuracy and stability of motion modeling, including: S41. Fusion Calculation: The human motion state vector obtained from the preliminary analysis of the sEMG signal is fused with the human motion state vector obtained from the IMU data using the Kalman filter algorithm to obtain the fused motion state vector. S42. Combining image data: The fused motion state vector is combined with the human body key points in the image data. By constructing a human motion model, the fused motion information and human body key point information are matched and optimized to further accurately determine the posture of the human body motion.

9. The human motion reconstruction method based on multimodal signals according to claim 8, characterized in that: The Kalman filter algorithm in S41 fusion computing includes: S411. Based on the system state transition matrix F and the process noise covariance matrix Q, predict the current state and covariance matrix: ； ； in, This is a priori estimate of the current state based on the state at the previous time step. This is the posterior estimate of the state at the previous time step. Prior estimate of the current state The corresponding covariance matrix, Posterior estimate of the state at the previous time step The corresponding covariance matrix; S412. Update the current state and covariance matrix based on the observation model matrix H and the observation noise covariance matrix R: ； ； ； in, The Kalman gain at the current moment. This is the posterior estimate of the state at the current moment. The observed value at the current moment, The posterior estimate of the current state The corresponding covariance matrix.

10. The human motion reconstruction method based on multimodal signals according to claim 8, characterized in that: S5 reconstructs specific human movements in the environment based on human motion models, achieving a complete and accurate description of human movements, including: S51. Combine the human motion model with the human kinematics model to reconstruct the motion trajectory and posture of various parts of the human body; S52. Place the three-dimensional human geometric model in the environment provided by the image data, and drive the three-dimensional human geometric model to move according to the reconstructed motion information to realize the reconstruction of the specific actions of the human body in the environment.

Images