Real-time monitoring method and system based on multi-sensor data fusion and electronic equipment

Abstract

The invention relates to the field of real-time monitoring, and particularly discloses a real-time monitoring method and system based on multi-sensor data fusion and electronic equipment. The method comprises the following steps: firstly, acquiring acceleration signals of the gravity center of a person to be detected at a plurality of preset time points acquired by an inertial measurement unit and angular velocity values of a left ankle node, a right ankle node, a left knee node and a right knee node at a plurality of preset time points acquired by a plurality of sensors; and feature extraction and correlation analysis are carried out on the two features, and finally a classification result is obtained through a classifier to judge whether the motion posture of the to-be-detected person needs to be adjusted, so that accurate motion posture monitoring and adjusting suggestions are provided, the motion skill of the to-be-detected person is improved, and the motion injury of the to-be-detected person is prevented.

Description

Technical Field

[0001] The present application relates to the field of real-time monitoring, and more specifically, to a real-time monitoring method, system and electronic equipment based on multi-sensor data fusion. Background Art

[0002] Posture detection is the process of identifying and tracking the posture and movement of a person or object by using sensors or computer vision technology. It has a wide range of applications in sports training, fitness monitoring, virtual reality, human-computer interaction, etc. Posture detection systems can extract key posture information such as body angles, joint positions, and motion trajectories by analyzing sensor or image data, thereby enabling people to better understand and improve body posture, improve sports skills, and reduce the risk of sports injuries.

[0003] With the development of sports and the increasing intensity of competition, the technical and performance requirements of athletes are getting higher and higher. In order to improve the training effect and reduce sports injuries, it is necessary to monitor the sports posture information of athletes. However, traditional sports posture information monitoring methods usually rely on the experience and judgment of coaches and cannot meet the requirements of accuracy and real-time performance.

[0004] Therefore, a real-time monitoring method, system and electronic device based on multi-sensor data fusion are desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a real-time monitoring method, system and electronic device based on multi-sensor data fusion, which first obtains the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors, and then uses deep learning technology to perform feature extraction and association analysis on the two, and finally obtains the classification result through the classifier to determine whether the movement posture of the person to be detected needs to be adjusted, and then provides accurate movement posture monitoring and adjustment suggestions, thereby improving the sports skills of the person to be detected and preventing sports injuries of the person to be detected.

[0006] According to one aspect of the present application, a real-time monitoring method based on multi-sensor data fusion is provided, which includes:

[0007] Acquire acceleration signals of the center of gravity of the person to be detected at multiple predetermined time points collected by an inertial measurement unit and angular velocity values ​​of a left ankle node, a right ankle node, a left knee node, and a right knee node at multiple predetermined time points collected by multiple sensors;

[0008] Extracting acceleration-related feature vectors and lower limb node-related feature vectors from the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node, and the right knee node at multiple predetermined time points collected by the multiple sensors;

[0009] Based on the acceleration-related feature vector and the lower limb node-related feature vector, determining whether the movement posture of the person to be detected needs to be adjusted;

[0010] Wherein, judging whether the motion posture of the person to be detected needs to be adjusted based on the acceleration-related feature vector and the lower limb node-related feature vector includes:

[0011] Performing scale modulation optimization based on principal component transformation on the acceleration-related feature vector and the lower limb node-related feature vector to obtain a motion posture classification feature vector;

[0012] The motion posture classification feature vector is passed through a classifier to obtain a classification result, and the classification result is used to determine whether the motion posture of the person to be detected needs to be adjusted.

[0013] According to another aspect of the present application, a real-time monitoring system based on multi-sensor data fusion is provided, comprising:

[0014] A motion posture data acquisition module, used to acquire acceleration signals of the center of gravity of the person to be detected at multiple predetermined time points acquired by an inertial measurement unit and angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points acquired by multiple sensors;

[0015] A motion posture feature extraction module, used to extract acceleration-related feature vectors and lower limb node-related feature vectors from the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by the multiple sensors;

[0016] A motion posture adjustment judgment generation module, used to judge whether the motion posture of the person to be detected needs to be adjusted based on the acceleration-related feature vector and the lower limb node-related feature vector;

[0017] Wherein, the movement posture adjustment judgment generation module is used to:

[0018] Performing scale modulation optimization based on principal component transformation on the acceleration-related feature vector and the lower limb node-related feature vector to obtain a motion posture classification feature vector;

[0019] The motion posture classification feature vector is passed through a classifier to obtain a classification result, and the classification result is used to determine whether the motion posture of the person to be detected needs to be adjusted.

[0020] Compared with the prior art, the present application provides a real-time monitoring method, system and electronic device based on multi-sensor data fusion, which first obtains the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally obtains the classification result through the classifier to determine whether the movement posture of the person to be detected needs to be adjusted, and then provides accurate movement posture monitoring and adjustment suggestions, thereby improving the movement skills of the person to be detected and preventing movement injuries of the person to be detected. . BRIEF DESCRIPTION OF THE DRAWINGS

[0021] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0022] Figure 1 It is a flow chart of a real-time monitoring method based on multi-sensor data fusion according to an embodiment of the present application.

[0023] Figure 2 The present invention is a flowchart of preprocessing the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points in the real-time monitoring method based on multi-sensor data fusion according to an embodiment of the present application.

[0024] Figure 3 The present invention is a flowchart of performing feature extraction on the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data in the real-time monitoring method based on multi-sensor data fusion according to an embodiment of the present application.

[0025] Figure 4 4 is a block diagram of a real-time monitoring system based on multi-sensor data fusion according to an embodiment of the present application.

[0026] Figure 5 is a block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0028] Figure 1 FIG. 1 is a flow chart of a real-time monitoring method based on multi-sensor data fusion according to an embodiment of the present application. Figure 1 As shown, according to the real-time monitoring method based on multi-sensor data fusion according to the embodiment of the present application, it includes: S110, acquiring the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors; S120, extracting the acceleration-related feature vector and the lower limb node-related feature vector from the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors; S130, judging whether the movement posture of the person to be detected needs to be adjusted based on the acceleration-related feature vector and the lower limb node-related feature vector.

[0029] In the above-mentioned real-time monitoring method based on multi-sensor data fusion, the step S110 obtains the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors. It should be understood that the center of gravity acceleration signal can provide information on the overall motion state of the person to be detected, and by analyzing the changes in the acceleration signal, the acceleration characteristics of the person to be detected, such as the amplitude, direction and rate of change of the acceleration, etc. can be obtained. This information can be used to determine whether the motion posture of the person to be detected is reasonable or needs to be adjusted. The angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node can provide the motion state information of the lower limb part. By analyzing the changes in the angular velocity values, the angular velocity characteristics of the lower limb part, such as the direction of rotation, the rate of rotation, etc. can be obtained. This information can be used to determine whether the motion state of the lower limb of the person to be detected is coordinated or needs to be adjusted. By the data fusion collected by multiple sensors, the overall motion state of the person to be detected and the motion state of the lower limb part can be comprehensively considered, so as to more accurately determine whether the motion posture needs to be adjusted. Therefore, obtaining information from multiple data sources is the key to determining whether the motion posture of the person to be detected needs to be adjusted in a specific application scenario.

[0030] In the above-mentioned real-time monitoring method based on multi-sensor data fusion, the step S120 extracts the acceleration-related feature vector and the lower limb node-related feature vector from the acceleration signal of the center of gravity of the person to be detected at multiple predetermined time points collected by the inertial measurement unit and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors. It should be understood that the acceleration-related feature vector is obtained by analyzing the feature extraction of the center of gravity acceleration signal. The center of gravity acceleration signal reflects the acceleration change of the overall movement of the person to be detected. By extracting the features of the acceleration signal, such as frequency domain features or time domain features, more specific motion feature information, such as amplitude, frequency, period, etc., can be obtained. These features can be used to determine whether the motion posture of the person to be detected is reasonable or needs to be adjusted. The lower limb node-related feature vector is obtained by analyzing the feature extraction of the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node. The angular velocity value reflects the rotational movement of the lower limb part. By extracting the features of the angular velocity value, the rotation state information of the lower limb part, such as the rotation rate, rotation direction, etc., can be obtained. These features can obtain more specific and detailed motion features so as to more accurately determine whether the motion posture of the person to be detected needs to be adjusted.

[0031] In a specific embodiment of the present application, the acceleration-related characteristic vector and the lower limb node-related characteristic vector are extracted from the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by multiple sensors, including: performing data preprocessing on the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points to obtain multiple acceleration frequency domain sample data and multiple acceleration time domain sample data; performing feature extraction on the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data to obtain the acceleration-related characteristic vector; performing feature extraction on the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by multiple sensors to obtain the lower limb node-related characteristic vector. It should be understood that frequency domain analysis can provide information about the frequency components of the signal. By converting the acceleration signal to the frequency domain, the energy distribution of different frequency components can be obtained. In this way, motion features within different frequency ranges can be detected, such as low-frequency smooth motion, medium-frequency gait features, and high-frequency subtle movements. Therefore, by performing frequency domain analysis on the acceleration signal, acceleration frequency domain sample data can be extracted to better understand the motion features of the person to be detected. Time domain analysis can provide information about the change of the signal over time. By performing time domain analysis on the acceleration signal, information such as the amplitude, change rate, and periodicity of the signal can be obtained. This information can be used to detect the acceleration change pattern, periodic motion features, etc. of the motion. Therefore, by performing time domain analysis on the acceleration signal, acceleration time domain sample data can be extracted to more comprehensively understand the motion state of the person to be detected. By using frequency domain and time domain analysis in combination, the motion data of the person to be detected can be analyzed from different angles and levels, and more feature information can be extracted, so as to more accurately evaluate the motion state and posture. Therefore, in the technical solution of the present application, the center of gravity acceleration signal collected by the inertial measurement unit is preprocessed to obtain acceleration frequency domain sample data and acceleration time domain sample data, which can more comprehensively analyze and understand the motion state of the person to be detected.

[0032] Furthermore, by performing feature extraction on multiple acceleration frequency domain sample data and multiple acceleration time domain sample data, the most representative and distinguishing features can be extracted to describe the motion state and posture of the person to be detected. Through feature extraction, a set of numerical features can be extracted from the original frequency domain and time domain data, and these features can reflect information on the statistical characteristics, periodicity, amplitude change, etc. of the acceleration signal. Common feature extraction methods include statistical features (such as mean, variance, maximum value, minimum value, etc.), frequency domain features (such as spectrum energy, spectrum peak, etc.), time domain features (such as time domain waveform shape, time domain statistics, etc.), etc. By extracting these features, complex acceleration signals can be converted into abstract representations with a higher level, so as to better describe the motion characteristics of the person to be detected, wherein the acceleration-related feature vector can contain multiple features, each feature corresponding to different motion features, and by combining these features, the motion state and posture of the person to be detected can be more comprehensively understood.

[0033] Furthermore, feature information reflecting the motion state and coordination of the lower limbs is extracted from the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by the multiple sensors. The angular velocity values ​​of the lower limb nodes can provide information about the motion of the lower limb joints, including the rotation speeds of the ankle joints and the knee joints. Through feature extraction, a set of representative features can be extracted from these angular velocity values ​​to describe the association and coordination between the lower limb nodes. These features may include, for example, mean value, standard deviation, peak value, peak occurrence time, spectrum features, time domain statistical features, correlation features and other aspects. By extracting these features, a lower limb node associated feature vector can be obtained for further analysis and identification of lower limb motion patterns, postures and abnormal states.

[0034] Figure 2 The present invention is a flowchart of preprocessing the acceleration signals of the center of gravity of the person to be detected at multiple predetermined time points collected by the inertial measurement unit in the real-time monitoring method based on multi-sensor data fusion according to an embodiment of the present application. Figure 2As shown, in a specific embodiment of the present application, the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points is subjected to data preprocessing to obtain multiple acceleration frequency domain sample data and multiple acceleration time domain sample data, including: S210, the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points is subjected to short-time Fourier transformation to obtain the multiple acceleration frequency domain sample data; S220, the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points is subjected to wavelet packet decomposition to obtain the multiple acceleration time domain sample data. It should be understood that in order to convert the acceleration signal in the time domain into a frequency domain representation to obtain the energy distribution and spectrum characteristics of the acceleration signal at different frequencies, it is necessary to perform short-time Fourier transformation on the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points. Short-time Fourier transform (STFT) is a time-frequency analysis method that can perform local analysis on the signal in time and frequency. By performing a short-time Fourier transform on the acceleration signal, the acceleration spectrum information at multiple time points, that is, multiple acceleration frequency domain sample data, can be obtained. Considering that frequency domain analysis can provide information about the energy distribution and spectrum characteristics of the acceleration signal at different frequencies, that is, the energy distribution at different frequencies can reflect the motion or vibration components of different frequencies, and the frequency domain features are usually robust to noise and interference, which can reduce the impact of noise on feature extraction, in the technical solution of the present application, a more comprehensive and rich description of the acceleration signal is obtained by performing a short-time Fourier transform on the acceleration signal and extracting the frequency domain features.

[0035] Furthermore, wavelet packet decomposition can decompose the acceleration signal in the time domain into multiple scales to obtain acceleration time domain sample data at different scales. In this way, by performing wavelet packet decomposition on the acceleration signal, acceleration time domain sample data at multiple scales, i.e., multiple acceleration time domain sub-signals, can be obtained. It should be understood that wavelet packet decomposition is a multi-scale analysis method that can locally analyze the signal in time and frequency, thereby capturing the characteristics of the acceleration signal at different time scales, wherein the time domain sample data at different scales can reflect the changes in different frequency components. Moreover, wavelet packet decomposition can also decompose the original acceleration signal into sub-signals at different scales, thereby extracting features within different frequency ranges.

[0036] Figure 3 The flowchart is a method for extracting features from the plurality of acceleration frequency domain sample data and the plurality of acceleration time domain sample data in a real-time monitoring method based on multi-sensor data fusion according to an embodiment of the present application. Figure 3As shown, in a specific embodiment of the present application, the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data are feature extracted to obtain the acceleration associated feature vector, including: S310, the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data are passed through an acceleration sample feature extractor based on a time series encoder to obtain an acceleration frequency domain feature vector and an acceleration time domain feature vector; S320, associating the acceleration frequency domain feature vector and the acceleration time domain feature vector to obtain the acceleration associated feature vector. It should be understood that the acceleration sample feature extractor based on the time series encoder can be a deep learning model, such as a recurrent neural network or a convolutional neural network. Such a model can model and learn time series data and extract useful feature information from it. Specifically, by inputting the acceleration frequency domain sample data into the time series encoder, the learning ability of the model can be used to extract the representation of the frequency domain features, which can capture the energy distribution and spectral characteristics of the acceleration signal at different frequencies. Similarly, by inputting the acceleration time domain sample data into the time series encoder, the representation of the time domain features can be extracted, and these features can reflect the time domain changes and dynamic characteristics of the acceleration signal. Converting the acceleration frequency domain sample data and the acceleration time domain sample data into the form of feature vectors respectively can provide a more comprehensive and rich description of the acceleration signal. More specifically, the fully connected layer of the acceleration sample feature extractor based on the time series encoder is used to fully connect the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data to extract the high-dimensional implicit features of the feature values ​​of each position in the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data; and the one-dimensional convolution layer of the acceleration sample feature extractor based on the time series encoder is used to one-dimensionally encode the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data to extract the high-dimensional implicit correlation features of the correlation between the feature values ​​of each position in the multiple acceleration frequency domain sample data and the multiple acceleration time domain sample data.

[0037] Furthermore, the frequency domain features and time domain features of the acceleration signal capture different signal information respectively. By associating the frequency domain features and the time domain features, the information of the frequency domain and the time domain can be comprehensively utilized to obtain a more comprehensive and rich feature representation. This helps to understand the changing patterns of the signal in different time scales and frequency ranges, so as to extract more representative features, and then improve the expressiveness and discriminability of the features, thereby establishing a time-frequency relationship model between them. Specifically, the acceleration frequency domain feature vector and the acceleration time domain feature vector are associated with the following cascade formula to obtain the acceleration associated feature vector; wherein the cascade formula is:

[0038] V=Concat[V a ,V b]

[0039] Among them, V a represents the acceleration frequency domain eigenvector, V b represents the acceleration time domain eigenvector, V represents the acceleration association eigenvector, and Concat[·,·] represents a cascade function.

[0040] In a specific embodiment of the present application, the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by multiple sensors are feature extracted to obtain the lower limb node associated feature vector, including: arranging the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by multiple sensors according to the data sample dimension and the time dimension to obtain a lower limb node input matrix; passing the lower limb node input matrix through a lower limb node feature encoder based on a convolutional neural network to obtain the lower limb node associated feature vector. It should be understood that by arranging the angular velocity values ​​of different nodes according to the data sample dimension and the time dimension, the data from multiple sensors can be integrated into one matrix. In this way, the data of the lower limb nodes can be easily processed and analyzed uniformly without the need to process the data of each node separately, while maintaining the structure and consistency of the data. Specifically, the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by multiple sensors are arranged into row vectors according to the sample dimension; and the row vectors of each time point are arranged into the lower limb node input matrix according to the time dimension.

[0041] Furthermore, the lower limb node input matrix is ​​passed through a lower limb node feature encoder based on a convolutional neural network to obtain a lower limb node associated feature vector. It should be understood that a convolutional neural network is a powerful feature extractor that can learn discriminative feature representations from input data. By inputting the lower limb node input matrix into a convolutional neural network, the network can automatically learn the associated features between the lower limb nodes and capture the spatial and temporal correlations between the nodes. In addition, a convolutional neural network is usually composed of multiple convolutional layers and pooling layers. This hierarchical structure can gradually extract the abstract features of the data. For the lower limb node input matrix, the convolutional neural network can extract different levels of feature representations through multiple convolutional layers and pooling layers, from low-level local features to high-level global features, which helps to better express the association between the lower limb nodes. In this way, the associated features between the lower limb nodes can be extracted from the lower limb node input matrix, thereby providing a more representative and discriminative feature representation. Among them, the lower limb node feature encoder based on convolutional neural network includes a first substructure and a second substructure in parallel, wherein the first substructure is configured to perform convolution processing, pooling processing and activation processing based on a first convolution kernel on the input data to output a first feature vector by the last layer of the first substructure; the second substructure is configured to perform convolution processing, pooling processing and activation processing based on a second convolution kernel on the input data to output a second feature vector by the last layer of the second substructure, wherein the first convolution kernel of the first substructure and the second convolution kernel of the second substructure have different sizes; and the first feature vector and the second feature vector are cascaded to obtain a lower limb node associated feature vector.

[0042] In the above-mentioned real-time monitoring method based on multi-sensor data fusion, the step S130 determines whether the motion posture of the person to be detected needs to be adjusted based on the acceleration-related feature vector and the lower limb node-related feature vector. It should be understood that by analyzing the acceleration-related feature vector and the lower limb node-related feature vector, the posture of the person to be detected can be evaluated. For example, it can be detected whether there is an abnormal acceleration pattern or an uncoordinated relationship between the lower limb nodes, so as to determine whether the posture needs to be adjusted.

[0043] In a specific embodiment of the present application, based on the acceleration-related feature vector and the lower limb node-related feature vector, it is judged whether the motion posture of the person to be detected needs to be adjusted, including: performing scale modulation optimization based on principal component transformation on the acceleration-related feature vector and the lower limb node-related feature vector to obtain a motion posture classification feature vector; passing the motion posture classification feature vector through a classifier to obtain a classification result, and the classification result is used to judge whether the motion posture of the person to be detected needs to be adjusted. It should be understood that the acceleration-related feature vector and the lower limb node-related feature vector provide comprehensive feature information about the motion posture of the person to be detected. Among them, the acceleration-related feature vector can reflect the overall motion acceleration characteristics of the human body, while the lower limb node-related feature vector provides posture information about the lower limb part. By comprehensively considering these two characteristics, it can be more comprehensively judged whether the motion posture needs to be adjusted.

[0044] In particular, considering that the acceleration-related feature vectors and the lower limb node-related feature vectors may be distributed in different feature spaces, the reason for the performance degradation when the source domain model is directly applied to the target domain is that different types of sensor data (such as acceleration signals and angular velocity values) and the features extracted after they are processed by different methods (such as short-time Fourier transform, wavelet packet decomposition and convolutional neural network encoding) are essentially different. These differences make the two feature vectors not only different in value, but more importantly, they represent different types of information and structures. The acceleration signal is decomposed by short-time Fourier transform and wavelet packet decomposition to obtain frequency domain and time domain feature vectors, respectively. These two features capture different frequency components and time variation characteristics during the movement process. The angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node collected by multiple sensors are processed by a feature encoder based on a convolutional neural network, focusing mainly on local time series patterns and spatial distribution information. Therefore, even though both are data describing human body movement, due to their different original information sources and different processing methods, the final feature vectors will naturally exist in different feature spaces. When trying to directly transfer the knowledge of one domain (for example, a model trained only for acceleration signals) to another domain (i.e., a new environment that integrates acceleration and angular velocity), it is difficult for the model to directly and effectively transfer its learned knowledge due to the feature space differences between the different modalities mentioned above. Based on this, in the technical solution of the present application, the acceleration-related feature vector and the lower limb node-related feature vector are subjected to scale modulation optimization based on principal component transformation to obtain a motion posture classification feature vector.

[0045] Furthermore, the acceleration-related feature vector and the lower limb node-related feature vector are subjected to scale modulation optimization based on principal component transformation to obtain a motion posture classification feature vector, including: mapping the acceleration-related feature vector and the lower limb node-related feature vector to an intrinsic decomposition space to obtain an acceleration-related intrinsic feature vector and a lower limb node-related intrinsic feature vector; performing inter-domain feature calibration on the acceleration-related intrinsic feature vector and the lower limb node-related intrinsic feature vector to obtain an acceleration-related modulated intrinsic feature vector and a lower limb node-related modulated intrinsic feature vector; constructing an acceleration-lower limb node fine-grained semantic information field between the acceleration-related modulated intrinsic feature vector and the lower limb node-related modulated intrinsic feature vector; and fusing the acceleration-related modulated intrinsic feature vector and the lower limb node-related modulated intrinsic feature vector based on the acceleration-lower limb node fine-grained semantic information field to obtain the motion posture classification feature vector.

[0046] Furthermore, an acceleration-lower limb node fine-grained semantic information field is constructed between the acceleration-associated modulated intrinsic feature vector and the lower limb node-associated modulated intrinsic feature vector, including: multiplying the acceleration-associated modulated intrinsic feature vector and the transposed vector of the lower limb node-associated modulated intrinsic feature vector, and then dividing by the scale of the lower limb node-associated modulated intrinsic feature vector to obtain an acceleration-lower limb node fine-grained interaction matrix; and convolutionally encoding the acceleration-lower limb node fine-grained semantic information field.

[0047] Furthermore, based on the acceleration-lower limb node fine-grained semantic information field, the acceleration-associated modulated intrinsic feature vector and the lower limb node associated modulated intrinsic feature vector are fused to obtain the motion posture classification feature vector, including: multiplying the acceleration-lower limb node fine-grained semantic information field by the acceleration-associated modulated intrinsic feature vector to obtain the acceleration-associated feature vector after information field mapping; multiplying the acceleration-lower limb node fine-grained semantic information field by the lower limb node associated modulated intrinsic feature vector to obtain the lower limb node associated feature vector after information field mapping; weighted fusion of the acceleration-associated feature vector after information field mapping and the lower limb node associated feature vector after information field mapping to obtain the motion posture classification feature vector.

[0048] In the embodiment of the present application, specifically, the acceleration-related feature vector and the lower limb node-related feature vector are subjected to scale modulation optimization based on principal component transformation to obtain a motion posture classification feature vector, including: processing the acceleration-related feature vector and the lower limb node-related feature vector according to the following formula to obtain a motion posture classification feature vector; wherein the formula is:

[0049]

[0050] V1=αv kt +βv st

[0051] Where V1 represents the acceleration-related eigenvector, V2 represents the lower limb node-related eigenvector, PCA(·) represents principal component analysis, U1 represents the sequence of acceleration-related eigendecomposition vectors, Λ1 represents the acceleration-related diagonal matrix, λ 11 and λ 1m denote the eigenvalues ​​of the first and mth positions of the acceleration correlation diagonal matrix, respectively. T denotes the transpose of the vector. concat{·;·} denotes vector concatenation. v 11 、v 12 、v 1m They represent the first, second and mth eigenvectors of the sequence of acceleration-related eigendecomposition vectors, Vx represents the acceleration-related eigenvectors, U2 represents the sequence of lower limb node-related eigendecomposition vectors, Λ2 represents the acceleration-related diagonal matrix, λ 21 and λ 2m They represent the eigenvalues ​​of the first and mth positions of the diagonal matrix of the lower limb node association, respectively. 21 、v 22 、v 2m Respectively represent the first, second and mth eigenvectors of the sequence of the associated eigendecomposition vectors of the lower limb nodes, V s represents the associated intrinsic eigenvector of the lower limb node, v i represents the i-th eigenvalue of the acceleration-related intrinsic eigenvector, v j represents the jth eigenvalue of the associated intrinsic eigenvector of the lower limb node, represents the square of the vector norm, exp represents the natural exponential function, V' k represents the intrinsic eigenvector after acceleration correlation modulation, V' s represents the intrinsic eigenvector after the lower limb node association modulation, represents matrix multiplication, S represents the scale of the vector, M x Represents the fine-grained interaction matrix between acceleration and lower limb nodes, Conv 3×3 (M x ) indicates that M x A convolutional encoding operation based on a convolution kernel of 3×3 is performed, Ω represents the fine-grained semantic information field of the acceleration-lower limb node, and v kt represents the acceleration associated eigenvector after information field mapping, v st represents the associated feature vector of the lower limb nodes after information field mapping, α and β represent weighted hyperparameters, and V1 represents the motion posture classification feature vector.

[0052] That is, in the technical solution of the present application, the acceleration-associated eigenvector and the lower limb node-associated eigenvector are subjected to scale modulation optimization based on principal component transformation. The process first maps the acceleration-associated eigenvector and the lower limb node-associated eigenvector to the intrinsic decomposition space to obtain the acceleration-associated eigenvector and the lower limb node-associated eigenvector. Those of ordinary skill in the art should know that the process of mapping to the intrinsic decomposition space is a combination of dimensionality reduction and feature extraction. In a specific example, the eigenvalue decomposition technique of principal component analysis (PCA) is used to find the eigendecomposition space, and the eigendecomposition space mapping is performed. The acceleration-associated eigenvector and the lower limb node-associated eigenvector after information field mapping not only represent the projection of the original eigenvector in the new coordinate system, but also reflect the main direction of change of the data, which helps to avoid the "dimensionality disaster", that is, the problem that the sample density drops sharply as the number of features increases.

[0053] Next, after mapping the eigenvector to the intrinsic decomposition space, it is necessary to perform inter-domain feature calibration on the acceleration-associated intrinsic eigenvector and the lower limb node-associated eigenvector to adjust the source domain eigenvector to better match the data distribution of the target domain. This operation aims to capture and compensate for the systematic deviation between the two domains to reduce the hidden disturbance of the class expression of the local feature distribution of the acceleration-associated intrinsic eigenvector and the lower limb node-associated eigenvector to the overall class expression. After modulation, the acceleration-associated modulated eigenvector and the lower limb node-associated modulated eigenvector can retain the source domain characteristics while being more in line with the actual needs of the target domain, enhancing the model's understanding and application of the target domain data.

[0054] After completing the scale modulation of the feature vector, the acceleration-lower limb node fine-grained semantic information field is constructed between the intrinsic feature vector after the acceleration association modulation and the intrinsic feature vector after the lower limb node association modulation. The acceleration-lower limb node fine-grained semantic information field is used to describe a more detailed relationship between feature vectors. The introduction of the acceleration-lower limb node fine-grained semantic information field enables the model to go beyond simple point-to-point comparison and consider more complex patterns and contextual information, thereby enhancing the model's sensitivity to subtle differences and improving the quality of feature representation. Finally, the acceleration-associated modulated intrinsic feature vector and the lower limb node associated modulated intrinsic feature vector are fused based on the acceleration-lower limb node fine-grained semantic information field to obtain a motion posture classification feature vector.

[0055] Furthermore, by inputting the motion posture classification feature vector into the classifier, the classifier can learn the characteristic patterns of different motion posture categories based on the previous training data, and assign the posture of the person to be detected to the most matching category. The classification result can provide information about whether the motion posture of the person to be detected needs to be adjusted. According to the predefined categories and corresponding adjustment rules, it can be determined whether the posture of the person to be detected needs to be adjusted. For example, if the classification result shows that the posture of the person to be detected belongs to a category that needs to be adjusted, the system can issue an alarm or provide corresponding feedback to guide the person to be detected to adjust the posture.

[0056] In summary, the embodiment of the present application first obtains the acceleration signal of the center of gravity of the person to be detected at multiple predetermined time points collected by the inertial measurement unit and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors, and then uses deep learning technology to perform feature extraction and correlation analysis on the two, and finally obtains the classification result through the classifier to determine whether the movement posture of the person to be detected needs to be adjusted, and then provides accurate movement posture monitoring and adjustment suggestions, thereby improving the sports skills of the person to be detected and preventing sports injuries of the person to be detected.

[0057] Figure 4 FIG. 1 is a block diagram of a real-time monitoring system based on multi-sensor data fusion according to an embodiment of the present application. Figure 4 As shown, according to the real-time monitoring system 100 based on multi-sensor data fusion according to the embodiment of the present application, it includes: a motion posture data acquisition module 110, which is used to obtain the acceleration signal of the center of gravity of the person to be detected at multiple predetermined time points collected by the inertial measurement unit and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors; a motion posture feature extraction module 120, which is used to extract acceleration-related feature vectors and lower limb node-related feature vectors from the acceleration signal of the center of gravity of the person to be detected at multiple predetermined time points collected by the inertial measurement unit and the angular velocity values ​​of the left ankle node, right ankle node, left knee node and right knee node at multiple predetermined time points collected by multiple sensors; a motion posture adjustment judgment generation module 130, which is used to judge whether the motion posture of the person to be detected needs to be adjusted based on the acceleration-related feature vector and the lower limb node-related feature vector.

[0058] As described above, the real-time monitoring system 100 based on multi-sensor data fusion according to the embodiment of the present application can be implemented in various terminal devices, such as a server deployed with a real-time monitoring control algorithm based on multi-sensor data fusion. In one example, the real-time monitoring system 100 based on multi-sensor data fusion can be integrated into the terminal device as a software module and / or a hardware module. For example, the real-time monitoring system 100 based on multi-sensor data fusion can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the real-time monitoring system 100 based on multi-sensor data fusion can also be one of the many hardware modules of the terminal device.

[0059] Alternatively, in another example, the real-time monitoring system based on multi-sensor data fusion 100 and the terminal device may also be separate devices, and the real-time monitoring system based on multi-sensor data fusion 100 may be connected to the terminal device via a wired and / or wireless network, and transmit interactive information in accordance with an agreed data format.

[0060] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned real-time monitoring system based on multi-sensor data fusion have been referred to above. Figures 1 to 3 The description of the real-time monitoring method based on multi-sensor data fusion has been introduced in detail, and therefore, its repeated description will be omitted.

[0061] An embodiment of the present application provides an electronic device. Figure 5 is a block diagram of an electronic device according to an embodiment of the present application.

[0062] like Figure 5 As shown, the electronic device 10 includes an input device 11, an input interface 12, a central processing unit 13, a memory 14, an output interface 15, an output device 16, and a bus 17. The input interface 12, the central processing unit 13, the memory 14, and the output interface 15 are connected to each other through the bus 17, and the input device 11 and the output device 16 are connected to the bus 17 through the input interface 12 and the output interface 15, respectively, and then connected to other components of the electronic device 10.

[0063] Specifically, the input device 11 receives input information from the outside and transmits the input information to the central processing unit 13 through the input interface 12; the central processing unit 13 processes the input information based on the computer executable instructions stored in the memory 14 to generate output information, temporarily or permanently stores the output information in the memory 14, and then transmits the output information to the output device 16 through the output interface 15; the output device 16 outputs the output information to the outside of the electronic device 10 for user use.

[0064] In one embodiment, Figure 5 The electronic device 10 shown can be implemented as a network device, which may include: a memory configured to store programs; a processor configured to run the programs stored in the memory to execute any one of the real-time monitoring methods based on multi-sensor data fusion described in the above embodiments.

[0065] According to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable storage medium.

[0066] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0067] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present application, but the present application is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of the present application, and these modifications and improvements are also considered to be within the scope of protection of the present application.

Claims

1. A real-time monitoring method based on multi-sensor data fusion, characterized in that: include: Acquire acceleration signals of the center of gravity of the person to be detected at multiple predetermined time points collected by an inertial measurement unit and angular velocity values ​​of a left ankle node, a right ankle node, a left knee node, and a right knee node at multiple predetermined time points collected by multiple sensors; Extracting acceleration-related feature vectors and lower limb node-related feature vectors from the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node, and the right knee node at multiple predetermined time points collected by the multiple sensors; Based on the acceleration-related feature vector and the lower limb node-related feature vector, determining whether the movement posture of the person to be detected needs to be adjusted; Wherein, judging whether the motion posture of the person to be detected needs to be adjusted based on the acceleration-related feature vector and the lower limb node-related feature vector includes: Performing scale modulation optimization based on principal component transformation on the acceleration-related feature vector and the lower limb node-related feature vector to obtain a motion posture classification feature vector; The motion posture classification feature vector is passed through a classifier to obtain a classification result, and the classification result is used to determine whether the motion posture of the person to be detected needs to be adjusted.

2. The real-time monitoring method based on multi-sensor data fusion according to claim 1 is characterized in that: Extracting acceleration-related feature vectors and lower limb node-related feature vectors from the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node, and the right knee node at multiple predetermined time points collected by the multiple sensors, including: Preprocessing the acceleration signals of the center of gravity of the person to be detected at a plurality of predetermined time points collected by the inertial measurement unit to obtain a plurality of acceleration frequency domain sample data and a plurality of acceleration time domain sample data; Performing feature extraction on the plurality of acceleration frequency domain sample data and the plurality of acceleration time domain sample data to obtain the acceleration associated feature vector; Feature extraction is performed on the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by multiple sensors to obtain the lower limb node associated feature vector.

3. The real-time monitoring method based on multi-sensor data fusion according to claim 2 is characterized in that: The acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points are preprocessed to obtain multiple acceleration frequency domain sample data and multiple acceleration time domain sample data, including: Performing a short-time Fourier transform on the acceleration signal of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points to obtain the multiple acceleration frequency domain sample data; The acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points are decomposed by wavelet packets to obtain the multiple acceleration time domain sample data.

4. The real-time monitoring method based on multi-sensor data fusion according to claim 3 is characterized in that: Extracting features from the plurality of acceleration frequency domain sample data and the plurality of acceleration time domain sample data to obtain the acceleration associated feature vector includes: Passing the plurality of acceleration frequency domain sample data and the plurality of acceleration time domain sample data through an acceleration sample feature extractor based on a timing encoder to obtain an acceleration frequency domain feature vector and an acceleration time domain feature vector; The acceleration frequency domain feature vector and the acceleration time domain feature vector are associated to obtain the acceleration associated feature vector.

5. The real-time monitoring method based on multi-sensor data fusion according to claim 4 is characterized in that: The angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at the plurality of predetermined time points collected by the plurality of sensors are subjected to feature extraction to obtain the lower limb node associated feature vector, including: Arrange the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by the multiple sensors according to the data sample dimension and the time dimension to obtain a lower limb node input matrix; The lower limb node input matrix is ​​passed through a lower limb node feature encoder based on a convolutional neural network to obtain the lower limb node associated feature vector.

6. The real-time monitoring method based on multi-sensor data fusion according to claim 5 is characterized in that: The acceleration-related feature vector and the lower limb node-related feature vector are subjected to scale modulation optimization based on principal component transformation to obtain a motion posture classification feature vector, including: Mapping the acceleration-related eigenvector and the lower limb node-related eigenvector to an eigendecomposition space to obtain an acceleration-related eigenvector and a lower limb node-related eigenvector; Performing inter-domain feature calibration on the acceleration-related intrinsic feature vector and the lower limb node-related intrinsic feature vector to obtain an acceleration-related modulated intrinsic feature vector and a lower limb node-related modulated intrinsic feature vector; Constructing an acceleration-lower limb node fine-grained semantic information field between the acceleration associated modulated intrinsic feature vector and the lower limb node associated modulated intrinsic feature vector; The acceleration-related modulated intrinsic feature vector and the lower limb node-related modulated intrinsic feature vector are fused based on the acceleration-lower limb node fine-grained semantic information field to obtain the motion posture classification feature vector.

7. The real-time monitoring method based on multi-sensor data fusion according to claim 6 is characterized in that: Constructing an acceleration-lower limb node fine-grained semantic information field between the acceleration associated modulated intrinsic feature vector and the lower limb node associated modulated intrinsic feature vector, including: After multiplying the acceleration-associated modulated intrinsic eigenvector and the transposed vector of the lower limb node-associated modulated intrinsic eigenvector, the resultant is divided by the scale of the lower limb node-associated modulated intrinsic eigenvector to obtain an acceleration-lower limb node fine-grained interaction matrix; The acceleration-lower limb node fine-grained matrix is ​​convolutionally encoded to obtain the acceleration-lower limb node fine-grained semantic information field.

8. The real-time monitoring method based on multi-sensor data fusion according to claim 7 is characterized in that: Based on the acceleration-lower limb node fine-grained semantic information field, the acceleration-associated modulated intrinsic feature vector and the lower limb node-associated modulated intrinsic feature vector are fused to obtain a motion posture classification feature vector, including: Multiplying the acceleration-lower limb node fine-grained semantic information field by the acceleration-associated modulated intrinsic feature vector to obtain the acceleration-associated feature vector after information field mapping; Multiplying the acceleration-lower limb node fine-grained semantic information field by the lower limb node associated modulated intrinsic feature vector to obtain the lower limb node associated feature vector after information field mapping; The acceleration-related feature vector after the information field mapping and the lower limb node-related feature vector after the information field mapping are weightedly fused to obtain the motion posture classification feature vector.

9. A real-time monitoring system based on multi-sensor data fusion, characterized in that: include: A motion posture data acquisition module, used to acquire acceleration signals of the center of gravity of the person to be detected at multiple predetermined time points acquired by an inertial measurement unit and angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points acquired by multiple sensors; A motion posture feature extraction module, used to extract acceleration-related feature vectors and lower limb node-related feature vectors from the acceleration signals of the center of gravity of the person to be detected collected by the inertial measurement unit at multiple predetermined time points and the angular velocity values ​​of the left ankle node, the right ankle node, the left knee node and the right knee node at multiple predetermined time points collected by the multiple sensors; A motion posture adjustment judgment generation module, used to judge whether the motion posture of the person to be detected needs to be adjusted based on the acceleration-related feature vector and the lower limb node-related feature vector; Wherein, the movement posture adjustment judgment generation module is used to: Performing scale modulation optimization based on principal component transformation on the acceleration-related feature vector and the lower limb node-related feature vector to obtain a motion posture classification feature vector; The motion posture classification feature vector is passed through a classifier to obtain a classification result, and the classification result is used to determine whether the motion posture of the person to be detected needs to be adjusted.

10. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the real-time monitoring method based on multi-sensor data fusion as described in any one of claims 1 to 8 when executing the computer program.

Images