A method, device and equipment for judging human activity status

Abstract

The present invention includes a method, device and equipment for distinguishing the activity state of a human body, the method including: preprocessing the first sensor data collected by different mobile terminals; performing feature extraction and feature selection on the preprocessed data to obtain the first feature data; training multiple initial classifiers based on the first feature data and stacked generalization algorithm to obtain a first activity state distinguishing model; obtaining the physiological characteristic data of the first user; determining the degree of influence of different physiological characteristic data on the first sensor data; determining the set of important characteristic variables in the first sensor data; determining the weight of each characteristic variable; training the first activity state distinguishing model with the sensor data, weight, physiological characteristic data and degree of influence to obtain a second activity state distinguishing model; distinguishing the activity state of a human body based on the second activity state distinguishing model. The method of the present invention can accurately identify the activity state of a human body using a mobile terminal and generate a user portrait at the same time.

Description

Technical Field

[0001] The present invention belongs to the technical field of posture recognition, and in particular relates to a method, device and equipment for distinguishing the activity state of a human body. Background Art

[0002] The promotion and popularization of smartphones has greatly changed the way people record and analyze their daily activities. By installing health applications, smartphones can record a variety of human motion status data including steps, exercise time, distance and energy consumption. Based on this data, smartphones can play a variety of health detection functions, set reminders according to users' activity habits, such as reminding users to stand, drink water or take a rest, integrate data from different applications and devices, provide users with comprehensive activity reports, help users better understand their living habits, and even provide personalized health and exercise suggestions to help users develop more effective health plans. Users can also share activity data on social media to compare progress with friends and increase the fun and motivation of exercise.

[0003] Smartphones can record the motion status of the human body mainly because built-in motion monitoring sensors such as accelerometers and gyroscopes record activity status data such as speed, acceleration and angular velocity in real time. Among them, the accelerometer is a sensor used to measure the linear acceleration changes of the mobile phone on three axes (X, Y, Z). When the acceleration of the mobile phone changes, the accelerometer will sense this change and transmit the acceleration data to the mobile phone processor for storage or analysis. The gyroscope is a sensor used to measure the angular velocity of the mobile phone rotating around three axes (X, Y, Z). When the person carrying the mobile phone turns, the gyroscope will sense the speed and direction of the turn and transmit this data to the mobile phone processor for storage or processing. Through the data of the accelerometer and gyroscope, the smartphone processor can sense the changes in the posture, angle and direction of the mobile phone, and design corresponding algorithms to judge or track the activity mode of the mobile phone user. Compared with other positioning methods, inertial positioning modules such as accelerometers and gyroscopes are not easily disturbed by the external environment, so the positioning accuracy is higher, and the working principle is usually based on micro-electromechanical system (MEMS) technology.

[0004] Specifically, a MEMS accelerometer typically contains a suspended, small mass (called a mass) that is suspended from a fixed frame by springs. When the device is accelerated, the mass moves relative to the frame due to inertia. The displacement of the mass is usually detected through the piezoelectric effect or a change in capacitance. The piezoelectric effect refers to the fact that certain materials generate an electric charge when subjected to pressure, while the change in capacitance refers to the displacement of the mass changing the distance from the fixed electrode, thereby changing the capacitance of the capacitor. The accelerometer converts the detected displacement into electrical signals, which can be further processed to determine the acceleration of the device.

[0005] A gyroscope is a sensor that measures angular velocity, which can detect the rate of rotation of a device around various axes. MEMS gyroscopes typically use a vibrating object, such as a vibrating wheel or a vibrating beam, to detect rotation. These vibrating objects remain stable when not rotating, but when the device is rotated, they deflect relative to the device due to the Coriolis force. Similar to accelerometers, displacement in gyroscopes is detected by changes in capacitance. When the vibrating object deflects, the distance between it and the fixed electrode changes, causing the capacitance to change. The gyroscope converts the detected angular velocity into electrical signals, which can be used to determine the rotation rate and direction of the device. Accelerometers and gyroscopes are often used together to provide more comprehensive motion detection. In smartphones, accelerometers can be used to detect the linear acceleration of the device, while gyroscopes are used to detect the rotational motion of the device. The combination of the two can provide more accurate human motion recognition, that is, human gesture recognition, tracking and analysis capabilities.

[0006] However, since the acceleration measured by the inertial sensor is different from the actual situation, the displacement measured after the second integration will also have a certain deviation. As time accumulates, these errors will continue to accumulate, which will significantly reduce the accuracy of judging the human body's motion state. Since the inertial sensor lacks an adaptive error correction mechanism during the test process, its error effect will continue to accumulate over time. The composition of the inertial sensor error effect is relatively complex, and can be briefly divided into three parts based on the source: random drift, linear acceleration interference, and magnetometer disturbance. In the short term, the inertial sensor error mainly comes from the instrument quantization error, but as time goes on, the random drift error of the instrument itself will have a greater impact on the sensor accuracy. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method, device and equipment for determining the human activity status of a user, which can use sensor data collected by a mobile terminal to quickly and accurately determine the human activity status of a user and simultaneously generate a user portrait.

[0008] The present invention includes a method for distinguishing human activity status, comprising:

[0009] Preprocessing first sensor data collected by different mobile terminals, wherein the sensor data includes acceleration data and angular velocity data on x, y, and z axes;

[0010] Performing feature extraction and feature selection on the preprocessed first sensor data to obtain first feature data corresponding to different activity states of each first user;

[0011] Training a plurality of initial classifiers based on the first feature data, and processing the trained plurality of initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating an activity state of a first user;

[0012] Obtaining physiological characteristic data of the first user;

[0013] Performing multiple linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data;

[0014] Using an integrated feature screening model constructed by a plurality of different feature importance analysis models, importance analysis is performed on different activity data in the first sensor data, so as to respectively determine a set of feature variables that have a significant impact on each of the activity states;

[0015] Performing independence analysis on the characteristic variables in the characteristic variable set to determine the weight of each characteristic variable;

[0016] Using the first sensor data, the weight of the feature variable, the physiological feature data of the first user and the influence degree as training data, the first activity state discrimination model is trained to obtain a second activity state discrimination model capable of generating a user portrait through case-based reasoning;

[0017] Based on the second activity state discrimination model, the second sensor data corresponding to the second user is processed to obtain a corresponding activity state discrimination result and information of a target user matching the second user, wherein the target user is the first user, and the similarity between the user portrait of the target user and the user portrait of the second user meets the requirements.

[0018] In one embodiment, the preprocessing of the first sensor data collected by different mobile terminals includes:

[0019] Performing outlier repair processing on the first sensor data;

[0020] The repaired first sensor data is subjected to Kalman filtering noise reduction processing.

[0021] In one embodiment, the preprocessing of the first sensor data collected by different mobile terminals further includes:

[0022] Using quaternions to perform fusion posture calculation on the first sensor data after noise reduction to obtain a posture angle corresponding to the activity state of the first user;

[0023] The gravity removal process is performed on the noise-reduced first sensor data based on the attitude angle.

[0024] In one embodiment, the feature extraction of the preprocessed sensor data includes:

[0025] Extracting time domain features from the preprocessed first sensor data, wherein the time domain features include mean, standard deviation, mode, maximum value, minimum value, range, and over-mean rate;

[0026] The frequency domain features are extracted from the preprocessed first sensor data, where the frequency domain features include mean, standard deviation, maximum value, minimum value, skewness, and kurtosis.

[0027] In one embodiment, the feature extraction and feature selection of the preprocessed sensor data includes:

[0028] Feature selection is performed on the extracted time domain features and frequency domain features based on the mutual information algorithm to obtain the first feature data.

[0029] In one embodiment, the training of multiple initial classifiers based on the first feature data and processing the trained multiple initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating the activity state of the first user includes:

[0030] Selecting multiple single models, wherein the multiple single models include a random forest model, a SVM model, a KNN model, and a gradient boosting classification model;

[0031] Using a five-fold cross validation algorithm to process the first feature data to generate training data;

[0032] Training a plurality of the single models based on the training data;

[0033] The trained random forest model, SVM model, and KNN model are used as primary classifiers, and the trained gradient boosting classification model is used as a secondary classifier to construct a stacked generalization combination model as the first activity state discrimination model.

[0034] In one embodiment, the physiological characteristic data includes age, height, and weight;

[0035] The performing of multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data includes:

[0036] Forming a dependent variable matrix with the first sensor data, forming an independent variable matrix with the physiological characteristic data, and setting a regression coefficient matrix and an error term;

[0037] Constructing a multivariate linear regression model based on the dependent variable matrix, independent variable matrix, regression coefficient matrix and error term;

[0038] Performing a multiple linear regression analysis on the first sensor data and the physiological characteristic data using the multiple linear regression model, and constructing a visual heat map based on the analysis results;

[0039] The degree of influence of different physiological characteristic data on the first sensor data is determined based on the analysis result and the visualized heat map.

[0040] In one embodiment, the integrated feature screening model constructed by multiple different feature importance analysis models is used to perform importance analysis on different activity data in the first sensor data to respectively determine a set of feature variables that have a significant impact on each of the activity states, including:

[0041] Construct an integrated feature screening model based on decision tree model, random forest model, mutual information algorithm and gradient boosting decision tree model;

[0042] Performing importance analysis on different activity data in the first sensor data based on each model in the integrated feature screening model, respectively, to obtain multiple groups of analysis results corresponding to different models, each group of analysis results including different importance analysis data corresponding to different activity states;

[0043] Normalizing the multiple groups of analysis results to obtain feature weights corresponding to the analysis results of different models;

[0044] A plurality of the feature weights and the corresponding analysis results are weightedly fused to obtain a set of feature variables that have a significant impact on each of the activity states.

[0045] In one embodiment, performing independence analysis on the feature variables in the feature variable set to determine the weight of each feature variable includes:

[0046] The characteristic variables in the characteristic variable set are subjected to independence analysis by using the Spearman rank correlation coefficient algorithm to determine the weight of each characteristic variable.

[0047] Another embodiment of the present invention also provides a human activity state determination device, comprising:

[0048] A preprocessing module, used to preprocess first sensor data collected by different mobile terminals, wherein the sensor data includes acceleration data and angular velocity data on the x, y, and z axes;

[0049] A feature extraction and selection module, used for performing feature extraction and feature selection on the preprocessed first sensor data to obtain first feature data corresponding to different activity states of each first user;

[0050] A first training module, used for training a plurality of initial classifiers according to the first feature data, and processing the trained plurality of initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating an activity state of a first user;

[0051] An acquisition module, used to obtain physiological characteristic data of the first user;

[0052] A first processing module, used for performing multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the influence degree of different physiological characteristic data on the first sensor data;

[0053] A first analysis module, configured to perform importance analysis on different activity data in the first sensor data using an integrated feature screening model constructed by a plurality of different feature importance analysis models, so as to respectively determine a set of feature variables having a significant impact on each of the activity states;

[0054] A second analysis module is used to perform independence analysis on the characteristic variables in the characteristic variable set to determine the weight of each characteristic variable;

[0055] a second training module, configured to train the first activity state discrimination model using the first sensor data, the weight of the feature variable, the physiological feature data of the first user, and the influence degree as training data, so as to obtain a second activity state discrimination model capable of generating a user portrait through case-based reasoning;

[0056] The second processing module is used to process the second sensor data corresponding to the second user according to the second activity state discrimination model to obtain the corresponding activity state discrimination result and the information of the target user matching the second user, wherein the target user is the first user, and the similarity between the user portrait of the target user and the user portrait of the second user meets the requirements.

[0057] Another embodiment of the present invention further provides an electronic device, including:

[0058] at least one processor; and

[0059] a memory communicatively connected to the at least one processor; wherein,

[0060] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the method for determining the human activity state as described in the above embodiment.

[0061] The beneficial effects of the present invention include using quaternions for posture solution, adding posture angle data, removing gravity according to the posture angle, extracting data time domain and frequency domain features, using mutual information for feature selection, and establishing a stacked generalization combination model to realize activity state discrimination, which significantly improves the discrimination accuracy of the model. In addition, it is proposed to realize user portrait based on case reasoning method, so as to successfully match the target personnel case with the same or similar user portrait as the current test user in the existing personnel case, so that it can be more conducive to the user or machine system to visualize the current human activity state, as well as the overall analysis and interpretation of the human activity state and the current user's activity state. The scheme described in this embodiment has a wide range of applications, including health monitoring and motion analysis, such as gait recognition, posture classification, support for individual health management and sports training; or in the field of safety and security, for personnel behavior recognition and anomaly detection; in smart devices and smart homes, to realize intelligent control according to the user's activity state; at the same time, it is also suitable for industrial production and operation monitoring, optimizing the working state and operation behavior of staff, and improving production efficiency and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 Schematic diagram of the flow of a method for determining human activity status according to an embodiment of the present invention.

[0063] Figure 2 This is a schematic diagram of a process for preprocessing first sensor data collected by different mobile terminals in an embodiment of the present invention.

[0064] Figure 3 Schematic diagram of a flow chart of determining the degree of influence of different physiological characteristic data on the first sensor data in an embodiment of the present invention.

[0065] Figure 4 4 is a structural block diagram of a device for distinguishing human activity status according to an embodiment of the present invention. DETAILED DESCRIPTION

[0066] like Figure 1 As shown, an embodiment of the present invention provides a method for determining human activity status, comprising:

[0067] S1: preprocessing first sensor data collected by different mobile terminals, wherein the sensor data includes acceleration data and angular velocity data on the x, y, and z axes;

[0068] S2: performing feature extraction and feature selection on the preprocessed first sensor data to obtain first feature data corresponding to different activity states of each first user;

[0069] S3: training a plurality of initial classifiers based on the first feature data, and processing the trained plurality of initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating an activity state of a first user;

[0070] S4: Obtaining physiological characteristic data of the first user;

[0071] S5: performing multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data;

[0072] S6: using an integrated feature screening model constructed by a plurality of different feature importance analysis models to perform importance analysis on different activity data in the first sensor data, so as to respectively determine a set of feature variables that have a significant impact on each of the activity states;

[0073] S7: performing independence analysis on the characteristic variables in the characteristic variable set to determine the weight of each characteristic variable;

[0074] S8: using the first sensor data, the weight of the feature variable, the physiological feature data of the first user and the influence degree as training data, training the first activity state discrimination model to obtain a second activity state discrimination model capable of generating a user portrait through case-based reasoning;

[0075] S9: Processing the second sensor data corresponding to the second user based on the second activity state discrimination model to obtain a corresponding activity state discrimination result and information of a target user matching the second user, wherein the target user is the first user, and the similarity between the user portrait of the target user and the user portrait of the second user meets the requirements.

[0076] Based on the above content, it can be seen that this embodiment scheme uses the sensor in the mobile terminal to collect data, pre-processes it, then constructs an activity state discrimination model, and uses the model to achieve accurate activity state discrimination, thereby realizing in-depth use of smartphone sensor data to discriminate and analyze human activity states, so as to determine a matching user portrait, and then enable the system to determine the same or similar user cases from the saved user cases based on the user portrait, thereby assisting the user or machine system to better understand its own motion state, so that the system can monitor the user's health based on the information obtained, formulate personalized exercise recommendations for matching users, and so on.

[0077] The solution described in this embodiment can solve the problem of effective clustering of activity states without pre-labeling activity state data collected by sensors in mobile terminals, such as mobile phones, including but not limited to acceleration data, gyroscope sensing data, etc. Secondly, it can solve the problem of how to generate a model that can accurately distinguish and classify activity states based on sensor data (i.e., activity state discrimination model, the activity state can be, for example, but not limited to running, jumping, walking, etc.) when the activity state is known. Thirdly, it can solve the problem of individual differences between users with different physiological characteristics and sensor data and activity states, and improve the discrimination ability of the activity state discrimination model, so that it can more accurately distinguish the user's human activity state based on sensor data, avoid being affected by the body shape of different users and causing incorrect recognition, and can also determine the user portrait. In addition, if the mobile terminal system stores user information of different users, including identity information, user portraits, historical activity state data, etc., the model can also identify the user corresponding to the current sensor data, that is, it can identify which user's action the currently measured sensor data is collected for.

[0078] Therefore, as described above, the solution based on this embodiment can be more conducive to the user or machine system to visualize the current human activity status, as well as the overall analysis and interpretation of the human activity status and the current user's activity status, so that the overall method has a wide range of applications, for example, it can be applied to health monitoring (not disease diagnosis, the health monitoring is achieved through the user's motion status) and motion analysis, such as recognizing gait, posture classification, supporting individual health management and sports training; or applied in the field of safety and security, for personnel behavior recognition and anomaly detection; in smart devices and smart homes, to achieve intelligent control of devices or homes according to the user's activity status; at the same time, it is also suitable for industrial production and operation monitoring, optimizing the working status and operation behavior of staff, reminding staff whether the current operation behavior does not comply with regulations, etc., thereby improving production efficiency and safety, etc.

[0079] like Figure 2 As shown, the first sensor data in this embodiment is data collected in the user's completely real life state, so there are certain inconsistencies, imbalances, noise values, and abnormal values ​​of the data, so the data needs to be preprocessed first to ensure the accuracy of subsequent calculations. In some embodiments, the preprocessing of the first sensor data collected by different mobile terminals includes:

[0080] S10: performing outlier repair processing on the first sensor data;

[0081] S11: Perform Kalman filtering noise reduction processing on the repaired first sensor data.

[0082] When performing outlier repair processing, this embodiment adopts the mean square error outlier identification algorithm, which includes three steps: data localization processing, matrix linear centering, and outlier discrimination and repair. The data localization processing includes dividing each n sensing data into a data frame, obtaining m continuous local data frames in sequence, and then performing outlier judgment and repair on each local frame to obtain a local signal matrix. Matrix linear centering includes linearly transforming the mean and variance of the local signal matrix, and then obtaining the centering matrix by the vector centering formula.

[0083] Furthermore, in some embodiments, the preprocessing of the first sensor data collected by different mobile terminals further includes:

[0084] S12: using quaternions to perform fusion posture calculation on the first sensor data after noise reduction to obtain a posture angle corresponding to the activity state of the first user;

[0085] S13: performing gravity removal processing on the first sensor data after noise reduction based on the attitude angle.

[0086] Since quaternions satisfy both the characteristics of complete description and convenient calculation, this embodiment performs attitude calculation based on quaternions:

[0087] Assume the initial attitude angle vector is , about the rotation axis defined by the unit vector Rotate by θ degrees, and then convert the format to get the new attitude angle calculation as follows: .in Represents the new attitude angle quaternion, Represents the initial attitude angle quaternion, and q represents the rotation axis quaternion. Then, the quaternion matrix is ​​converted to obtain the attitude matrix of the human body coordinate system compared to the default coordinate system. For six-axis data, there are two main methods for calculating the attitude angle, one is to integrate the angular velocity, and the other is to orthogonally decompose the acceleration angle. However, both methods have shortcomings, so in this embodiment, the two data are fused to obtain accurate attitude, specifically by compensating the angular velocity through an accelerometer. For example, the error of the gyroscope data is compensated according to the deviation between the theoretical vector and the actual vector, and then a more accurate attitude is calculated, that is, the angular velocity error implicit in the quaternion is manifested. Specifically, the outer product of the theoretical gravity acceleration vector and the actual gravity acceleration vector is calculated to obtain the vector direction difference, and then the vector direction difference is normalized. Since the deviation angle between the theoretical vector and the actual vector does not exceed 45°, the rotation angle involved in the quaternion is directly determined as the vector direction difference. After obtaining the tea, the angular velocity compensation value can be calculated by constructing a PI compensator. Finally, the compensation value is added to the angular velocity measurement value, and then substituted into the quaternion difference equation to obtain the accurate attitude angle.

[0088] Since the three-axis (X, Y, Z) acceleration sensors integrated in mobile terminals are usually divided into two types: one is the acceleration due to the constant gravity, and the other is the acceleration caused by the user holding the device. In order to obtain the true motion state, this embodiment will use the attitude angles corresponding to the three axes to remove the gravity of the acceleration values ​​on the three axes of X, Y, and Z, that is, to remove the gravity of the acceleration data.

[0089] Furthermore, feature selection is not consistent with feature extraction. Feature extraction is a transformation-based method that maps the original high-dimensional data to a low-dimensional space through coordinate transformation, while feature selection can be seen as the process of searching for an optimal feature subset from the initial feature space. It does not change the original feature space, but only selects some features with good resolution to form a new low-dimensional space. It can remove irrelevant and redundant features while retaining most of the properties of the original feature space, thereby minimizing the impact of noise data on classifier performance to a certain extent.

[0090] Specifically, in some embodiments, extracting features from the preprocessed sensor data includes:

[0091] S14: extracting time domain features from the preprocessed first sensor data, where the time domain features include mean, standard deviation, mode, maximum, minimum, range, and over-mean rate;

[0092] S15: extracting frequency domain features from the preprocessed first sensor data, where the frequency domain features include mean, standard deviation, maximum value, minimum value, skewness, and kurtosis.

[0093] Time domain features refer to features extracted by observing the changes of signals on the time axis in time series data. These features usually reflect the time series characteristics of the signal, such as periodicity, trend, stability, etc. For discrete activity state data frames, time domain features are calculated based on the acceleration, angular velocity and attitude angle data on the X, Y, and Z axes. Specifically, for the sensor data of a certain axis of a sensor in a set of activity state data, N represents the number of samples within the sampling time, Ci represents the absolute value of the i-th sample data, max represents the maximum value, min represents the minimum value, and num represents the number of elements that meet the conditions, and the seven types of time domain features corresponding to the data fragment of a certain axis are obtained, namely, mean, standard deviation, mode, maximum value, minimum value, range and over-mean rate.

[0094] Frequency domain features are extracted from the frequency components of the signal, and they describe the distribution characteristics of the signal in the frequency domain. These features can be obtained through Fourier Transform (FT) or other frequency domain analysis methods. For the segmented activity state data fragments, in addition to extracting time domain features, frequency domain features will also be extracted. Since the obtained activity state data is discrete time domain data, it needs to be subjected to discrete Fourier transform. For the perception data of a certain axis of a sensor, Represents the i-th frequency domain data obtained after Fourier transform of time domain data. After the transformation is completed, the six types of features corresponding to the data segment of a certain axis can be calculated, namely, mean, standard deviation, maximum value, minimum value, skewness and kurtosis. Among them, mean, standard deviation, maximum value and minimum value are amplitude statistical features, which are consistent with the meaning of time domain features; skewness and kurtosis are shape statistical features.

[0095] Furthermore, the feature extraction and feature selection of the preprocessed sensor data includes:

[0096] S16: Perform feature selection on the extracted time domain features and frequency domain features based on the mutual information algorithm to obtain the first feature data.

[0097] Mutual Information (MI) is a statistic used to measure the mutual dependence between two variables. In feature engineering, mutual information can be used to evaluate the correlation between features and target variables, thereby helping the system select the most informative features. The larger the value of the mutual information, the higher the degree of mutual dependence between the two. In this embodiment, the mutual information maximization (MIM) algorithm is specifically used to calculate the mutual information function of each activity data / activity feature and the corresponding preset class label, and sort them from large to small according to their values, and then select the required first feature data.

[0098] After completing the data preprocessing, the system can also perform Kmeans feature clustering on the processed data.

[0099] In some embodiments, the training of multiple initial classifiers based on the first feature data and processing the trained multiple initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating the activity state of the first user includes:

[0100] S17: selecting a plurality of single models, wherein the plurality of single models include a random forest model, an SVM model (Support Vector Machine (SVM)), a KNN model (K Nearest Neighbors (KNN)) and a gradient lifting classification model (GradientLift, GL);

[0101] S18: Processing the first feature data using a five-fold cross validation algorithm to generate training data;

[0102] S19: Training a plurality of the single models based on the training data;

[0103] S20: Using the trained random forest model, SVM model, and KNN model as primary classifiers, and using the trained gradient boosting classification model as a secondary classifier to construct a stacked generalization combination model as the first activity state discrimination model.

[0104] The stacking generalization algorithm (Stacking method) in this embodiment is generalized by a meta-learner (secondary classifier), and the deviation is comprehensively reduced to improve the classification ability of the model. The basic principle of the Stacking method is to select multiple models from a series of models as primary classifiers, and train these primary classifiers based on the original data set to output a new set of classified data. The output of the primary classifier is the input feature of the secondary classifier, and the label of the new data set generated by the primary classifier still uses the label of the initial data. In this embodiment, the random forest model, SVM model, KNN model and gradient boosting classification model with the best classification effect are preferably used to construct a Stacking combination model, and the combination model with random forest model, SVM model, KNN model as primary classifier and gradient boosting classification model as secondary classifier has the best classification effect. After completing the selection of a single model, it needs to be trained. In order to avoid serious overfitting risks, in the training process of the model, in this embodiment, a five-fold cross-validation algorithm is preferably used to process the first feature data to generate training data for the primary classifier. The purpose of the cross-validation mentioned above is different from that of the cross-validation in model evaluation. The Stacking method requires that the original data be divided into a set of training sets Dtrain and test sets Dtest in a certain way, and then the cross-validation algorithm is used on the training set Dtrain to generate the training set of the secondary classifier, and finally the evaluation index is calculated on the test set Dtest to evaluate the performance of the model. This method can effectively avoid overfitting of the model and enhance the stability of the model.

[0105] In some embodiments, the physiological characteristic data includes age, height, weight, and the user's identity information can also be obtained.

[0106] like Figure 3 As shown, the performing of multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data includes:

[0107] S21: forming a dependent variable matrix with the first sensor data, forming an independent variable matrix with the physiological characteristic data, and setting a regression coefficient matrix and an error term;

[0108] S22: constructing a multivariate linear regression model based on the dependent variable matrix, the independent variable matrix, the regression coefficient matrix and the error term;

[0109] S23: performing a multiple linear regression analysis on the first sensor data and the physiological characteristic data using the multiple linear regression model, and constructing a visual heat map based on the analysis results;

[0110] S24: Determine the degree of influence of different physiological characteristic data on the first sensor data based on the analysis result and the visualized heat map.

[0111] In this embodiment, a multivariate linear regression model is constructed to perform data analysis to find the linear relationship between each sensor data variable and the user attribute (age, weight, height), and a heat map is constructed accordingly, where each cell of the heat map represents the regression coefficient between a specific sensor data variable and a specific person attribute. The following results are obtained through multivariate linear regression analysis and heat map visualization, including that height has the greatest impact on the x-axis of the accelerometer, age has the greatest impact on the y-axis of the accelerometer, weight has the greatest impact on the z-axis of the accelerometer, weight has the greatest impact on the x-axis of the gyroscope, age has the greatest impact on the y-axis of the gyroscope, and age has the greatest impact on the z-axis of the gyroscope.

[0112] In some embodiments, the integrated feature screening model constructed by multiple different feature importance analysis models is used to perform importance analysis on different activity data in the first sensor data to respectively determine a set of feature variables that have a significant impact on each of the activity states, including:

[0113] S25: Construct an integrated feature screening model based on decision tree model, random forest model, mutual information algorithm and gradient boosting decision tree model;

[0114] S26: performing importance analysis on different activity data in the first sensor data based on each model in the integrated feature screening model, respectively, to obtain multiple groups of analysis results corresponding to different models, each group of analysis results including different importance analysis data corresponding to different activity states;

[0115] S27: performing normalization processing on the multiple groups of analysis results to obtain feature weights corresponding to the analysis results of different models;

[0116] S28: performing weighted fusion on the plurality of feature weights and the corresponding analysis results to obtain a set of feature variables that have a significant impact on each of the activity states.

[0117] Since the first feature data in this embodiment belongs to multi-sample data, it has the characteristics of a large number of molecular descriptor variables, high nonlinearity, and high coupling, so the embedded feature selection method random forest is preferentially selected for mathematical modeling, in order to obtain the top 10 or other number of molecular descriptor variables that have the most significant impact on the motion state. In addition, since there may be a nonlinear relationship between the feature variables in this embodiment, the traditional correlation analysis is not applicable. For this reason, this embodiment preferentially selects the distance correlation coefficient (DistanceCorrelation) as an indicator for measuring the nonlinear correlation between variables, including if the distance correlation coefficient is 0, the two variables can be judged to be independent of each other, and the larger the distance correlation coefficient, the stronger the correlation between the two. Through the above method, this embodiment uses each single model used to form an integrated feature screening model to perform a correlation analysis on the importance analysis data of different activity data in the first sensor data. Through analysis, it is determined that except for the two methods using the correlation screening method, most of them have a weak correlation, which is conducive to the fusion and integration of the model. Therefore, in order to integrate the modeling capabilities of these methods for linear, nonlinear, and highly coupled features, an integrated feature screening model is established based on the multi-sample high-dimensional data of this embodiment, in order to obtain a more flexible screening model with a lower risk of overfitting. In order to integrate the feature importance analysis data (multiple groups of analysis results, each group of analysis results includes activity variables of different importance corresponding to different activity states) of the above five single models (decision tree model, random forest model, mutual information algorithm and gradient boosting decision tree model), the multiple groups of analysis results are first normalized to obtain the feature weights of the analysis results corresponding to different models, which reflects the importance of the activity variables (activity data) under the corresponding model. Each feature weight is expressed as the ratio of the importance of the activity variable to the sum of the importance of all activity variables. Then, the feature weights corresponding to each model are weighted and fused to obtain the importance vector of the integrated independent variable: . Where Y is the feature importance vector of the analysis results of integrating multiple models, For the The weight of each method in the fusion, and the sum of the weights is 1, For the By solving the established integrated feature screening model, the first 10 or other number of features can be selected as feature variables that have an important impact on each of the activity states, and a feature variable set can be formed.

[0118] Furthermore, in order to determine the feature weights and realize the optimal weight distribution scheme for different features, in this embodiment, independence analysis is performed on the feature variable set obtained above. Specifically, independence analysis is performed on the feature variables in the feature variable set to determine the weights of each feature variable, including:

[0119] S29: Performing independence analysis on the characteristic variables in the characteristic variable set by using the Spearman rank correlation coefficient algorithm to determine the weight of each characteristic variable.

[0120] Case-based reasoning (CBR) is a type of analogical reasoning and an experience-based intelligent learning and problem-solving method. Case-based reasoning solves problems by comparing and analyzing the target case with previous historical cases and adjusting the solution. The general steps of using case-based reasoning to solve problems include four stages: case retrieval, case reuse, case revision, and case retention, also known as the "4R" life cycle model. When encountering a new problem, the case-based reasoning system first converts the problem into a case representation format to generate a target case, and then retrieves the source case similar to the target case in the case library. After retrieving the source case, if the source case solution can solve the target case problem, the problem is solved; if the source case solution cannot solve the target case problem, then the source case solution must be adjusted to obtain the solution to the target case. After the problem is solved, the target case is stored in the case library, and the case library is dynamically updated for the next problem solving.

[0121] Based on the reasoning ideas of the above cases, the Stacking ensemble learning algorithm, that is, the above activity state discrimination model, can be used as the basis for similar case retrieval. The similar cases are the first sensor data, the activity state label and the corresponding user data. When a new motion feature data is input, the Stacking ensemble learning algorithm randomly discriminates the input data, and finally determines the final classification result through the integration effect, that is, discriminating the activity state, such as running, walking, push-ups, jumping on the spot, etc. However, in order to realize that the activity state discrimination model can generate user portraits and even identify the user corresponding to the current input data, such as selecting a case with the same or similar sensor data from the above similar cases. In this embodiment, the first sensor data, the weight of the feature variable, the physiological feature data of the first user, the degree of influence of different physiological feature data on the first sensor data, and the identity information of the first user are used as training data to retrain the current activity state discrimination model, including using Stacking ensemble learning to perform similarity matching of historical data (the above-mentioned similar cases) to achieve user characterization based on case reasoning ideas, that is, to obtain a new activity state discrimination model with case reasoning capabilities that can generate user portraits through case reasoning ideas, namely, the second activity state discrimination model.

[0122] In actual application, sensor data of new users, i.e., second sensor data, can be collected and then preprocessed to obtain corresponding feature data. Next, the feature data can be input into the second activity state discrimination model to obtain the activity state discrimination result. At the same time, the user portrait corresponding to the currently measured user can be determined, i.e., the user portrait of the user who made the current activity. Based on the user portrait, target cases with similar or identical user portraits can be determined from existing cases, so that the information determined by the system, including the activity state and user portrait, can be visualized.

[0123] like Figure 4 As shown, another embodiment of the present invention also provides a human activity state determination device 100, comprising:

[0124] A preprocessing module, used to preprocess first sensor data collected by different mobile terminals, wherein the sensor data includes acceleration data and angular velocity data on the x, y, and z axes;

[0125] A feature extraction and selection module, used for performing feature extraction and feature selection on the preprocessed first sensor data to obtain first feature data corresponding to different activity states of each first user;

[0126] A first training module, used for training a plurality of initial classifiers according to the first feature data, and processing the trained plurality of initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating an activity state of a first user;

[0127] An acquisition module, used to obtain physiological characteristic data of the first user;

[0128] A first processing module, used for performing multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the influence degree of different physiological characteristic data on the first sensor data;

[0129] A first analysis module, configured to perform importance analysis on different activity data in the first sensor data using an integrated feature screening model constructed by a plurality of different feature importance analysis models, so as to respectively determine a set of feature variables having a significant impact on each of the activity states;

[0130] A second analysis module is used to perform independence analysis on the characteristic variables in the characteristic variable set to determine the weight of each characteristic variable;

[0131] a second training module, configured to train the first activity state discrimination model using the first sensor data, the weight of the feature variable, the physiological feature data of the first user, and the influence degree as training data, so as to obtain a second activity state discrimination model capable of generating a user portrait through case-based reasoning;

[0132] The second processing module is used to process the second sensor data corresponding to the second user according to the second activity state discrimination model to obtain the corresponding activity state discrimination result and the information of the target user matching the second user, wherein the target user is the first user, and the similarity between the user portrait of the target user and the user portrait of the second user meets the requirements.

[0133] In one embodiment, the preprocessing of the first sensor data collected by different mobile terminals includes:

[0134] Performing outlier repair processing on the first sensor data;

[0135] The repaired first sensor data is subjected to Kalman filtering noise reduction processing.

[0136] In one embodiment, the preprocessing of the first sensor data collected by different mobile terminals further includes:

[0137] Using quaternions to perform fusion posture calculation on the first sensor data after noise reduction to obtain a posture angle corresponding to the activity state of the first user;

[0138] The gravity removal process is performed on the noise-reduced first sensor data based on the attitude angle.

[0139] In one embodiment, the feature extraction of the preprocessed sensor data includes:

[0140] Extracting time domain features from the preprocessed first sensor data, wherein the time domain features include mean, standard deviation, mode, maximum value, minimum value, range, and over-mean rate;

[0141] The frequency domain features are extracted from the preprocessed first sensor data, where the frequency domain features include mean, standard deviation, maximum value, minimum value, skewness, and kurtosis.

[0142] In one embodiment, the feature extraction and feature selection of the preprocessed sensor data includes:

[0143] Feature selection is performed on the extracted time domain features and frequency domain features based on the mutual information algorithm to obtain the first feature data.

[0144] In one embodiment, the training of multiple initial classifiers based on the first feature data and processing the trained multiple initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating the activity state of the first user includes:

[0145] Selecting multiple single models, wherein the multiple single models include a random forest model, a SVM model, a KNN model, and a gradient boosting classification model;

[0146] Using a five-fold cross validation algorithm to process the first feature data to generate training data;

[0147] Training a plurality of the single models based on the training data;

[0148] The trained random forest model, SVM model, and KNN model are used as primary classifiers, and the trained gradient boosting classification model is used as a secondary classifier to construct a stacked generalization combination model as the first activity state discrimination model.

[0149] In one embodiment, the physiological characteristic data includes age, height, and weight;

[0150] The performing of multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data includes:

[0151] Forming a dependent variable matrix with the first sensor data, forming an independent variable matrix with the physiological characteristic data, and setting a regression coefficient matrix and an error term;

[0152] Constructing a multivariate linear regression model based on the dependent variable matrix, independent variable matrix, regression coefficient matrix and error term;

[0153] Performing a multiple linear regression analysis on the first sensor data and the physiological characteristic data using the multiple linear regression model, and constructing a visual heat map based on the analysis results;

[0154] The degree of influence of different physiological characteristic data on the first sensor data is determined based on the analysis result and the visualized heat map.

[0155] In one embodiment, the integrated feature screening model constructed by multiple different feature importance analysis models is used to perform importance analysis on different activity data in the first sensor data to respectively determine a set of feature variables that have a significant impact on each of the activity states, including:

[0156] Construct an integrated feature screening model based on decision tree model, random forest model, mutual information algorithm and gradient boosting decision tree model;

[0157] Performing importance analysis on different activity data in the first sensor data based on each model in the integrated feature screening model, respectively, to obtain multiple groups of analysis results corresponding to different models, each group of analysis results including different importance analysis data corresponding to different activity states;

[0158] Normalizing the multiple groups of analysis results to obtain feature weights corresponding to the analysis results of different models;

[0159] A plurality of the feature weights and the corresponding analysis results are weightedly fused to obtain a set of feature variables that have a significant impact on each of the activity states.

[0160] In one embodiment, performing independence analysis on the feature variables in the feature variable set to determine the weight of each feature variable includes:

[0161] The characteristic variables in the characteristic variable set are subjected to independence analysis by using the Spearman rank correlation coefficient algorithm to determine the weight of each characteristic variable.

[0162] Another embodiment of the present invention further provides an electronic device, including:

[0163] at least one processor; and

[0164] a memory communicatively coupled to the at least one processor;

[0165] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the method for determining the human activity state described in the above embodiment.

[0166] Another embodiment of the present invention further provides a storage medium, which includes a stored program, wherein when the program is running, a device including the storage medium is controlled to execute the method for determining the human activity state as described in the above embodiment.

[0167] The embodiment of the present invention also provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions, and when the computer-executable instructions are executed, at least one processor executes a method for distinguishing human activity states such as in the above-described embodiment. It should be understood that each solution in this embodiment has the corresponding technical effects in the above-described method embodiment, which will not be described in detail here.

[0168] A person skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of protection of the present application is limited to these examples. In line with the concept of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of different aspects of one or more embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0169] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A method for determining human activity status, characterized in that: include: Preprocessing first sensor data collected by different mobile terminals, wherein the sensor data includes acceleration data and angular velocity data on x, y, and z axes; Performing feature extraction and feature selection on the preprocessed first sensor data to obtain first feature data corresponding to different activity states of each first user; Training a plurality of initial classifiers based on the first feature data, and processing the trained plurality of initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating an activity state of a first user; Obtaining physiological characteristic data of the first user; Performing multiple linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data; Using an integrated feature screening model constructed by a plurality of different feature importance analysis models, importance analysis is performed on different activity data in the first sensor data, so as to respectively determine a set of feature variables that have a significant impact on each of the activity states; Performing independence analysis on the characteristic variables in the characteristic variable set to determine the weight of each characteristic variable; The first activity state discrimination model is trained using the first sensor data, the weight of the feature variable, the physiological feature data of the first user and the influence degree as training data to obtain a second activity state discrimination model capable of determining the user portrait through case-based reasoning; Processing the second sensor data corresponding to the second user based on the second activity state discrimination model to obtain a corresponding activity state discrimination result and information of a target user matching the second user, wherein the target user is the first user, and a user profile of the target user and a user profile of the second user meet similarity requirements; The physiological characteristic data include age, height, and weight; The performing multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data includes: Forming a dependent variable matrix with the first sensor data, forming an independent variable matrix with the physiological characteristic data, and setting a regression coefficient matrix and an error term; Constructing a multivariate linear regression model based on the dependent variable matrix, independent variable matrix, regression coefficient matrix and error term; Performing a multiple linear regression analysis on the first sensor data and the physiological characteristic data using the multiple linear regression model, and constructing a visual heat map based on the analysis results; The degree of influence of different physiological characteristic data on the first sensor data is determined based on the analysis result and the visualized heat map.

2. The method for determining human activity status according to claim 1, characterized in that: The preprocessing of the first sensor data collected by different mobile terminals includes: Performing outlier repair processing on the first sensor data; The repaired first sensor data is subjected to Kalman filtering noise reduction processing.

3. The method for determining human activity status according to claim 2, characterized in that: The preprocessing of the first sensor data collected by different mobile terminals also includes: Using quaternions to perform fusion posture calculation on the first sensor data after noise reduction to obtain a posture angle corresponding to the activity state of the first user; The gravity removal process is performed on the noise-reduced first sensor data based on the attitude angle.

4. The method for determining human activity status according to claim 1, characterized in that: The feature extraction of the preprocessed sensor data includes: Extracting time domain features from the preprocessed first sensor data, wherein the time domain features include mean, standard deviation, mode, maximum value, minimum value, range, and over-mean rate; The frequency domain features are extracted from the preprocessed first sensor data, where the frequency domain features include mean, standard deviation, maximum value, minimum value, skewness, and kurtosis.

5. The method for determining human activity status according to claim 4, characterized in that: The feature extraction and feature selection of the preprocessed sensor data includes: Feature selection is performed on the extracted time domain features and frequency domain features based on the mutual information algorithm to obtain the first feature data.

6. The method for determining human activity status according to claim 1, characterized in that: The step of training a plurality of initial classifiers based on the first feature data and processing the trained plurality of initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating the activity state of the first user includes: Selecting multiple single models, wherein the multiple single models include a random forest model, a SVM model, a KNN model, and a gradient boosting classification model; Using a five-fold cross validation algorithm to process the first feature data to generate training data; Training a plurality of the single models based on the training data; The trained random forest model, SVM model, and KNN model are used as primary classifiers, and the trained gradient boosting classification model is used as a secondary classifier to construct a stacked generalization combination model as the first activity state discrimination model.

7. The method for determining human activity status according to claim 1, characterized in that: The integrated feature screening model constructed by a plurality of different feature importance analysis models is used to perform importance analysis on different activity data in the first sensor data to respectively determine a set of feature variables that have a significant impact on each of the activity states, including: Construct an integrated feature screening model based on decision tree model, random forest model, mutual information algorithm and gradient boosting decision tree model; Performing importance analysis on different activity data in the first sensor data based on each model in the integrated feature screening model, respectively, to obtain multiple groups of analysis results corresponding to different models, each group of analysis results including different importance analysis data corresponding to different activity states; Normalizing the multiple groups of analysis results to obtain feature weights corresponding to the analysis results of different models; A plurality of the feature weights and the corresponding analysis results are weightedly fused to obtain a set of feature variables that have a significant impact on each of the activity states.

8. A human activity state determination device, characterized in that: include: A preprocessing module, used to preprocess first sensor data collected by different mobile terminals, wherein the sensor data includes acceleration data and angular velocity data on the x, y, and z axes; A feature extraction and selection module, used for performing feature extraction and feature selection on the preprocessed first sensor data to obtain first feature data corresponding to different activity states of each first user; A first training module, used for training a plurality of initial classifiers according to the first feature data, and processing the trained plurality of initial classifiers based on a stacked generalization algorithm to obtain a first activity state discrimination model for discriminating an activity state of a first user; An acquisition module, used to obtain physiological characteristic data of the first user; A first processing module, used for performing multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the influence degree of different physiological characteristic data on the first sensor data; A first analysis module, configured to perform importance analysis on different activity data in the first sensor data using an integrated feature screening model constructed by a plurality of different feature importance analysis models, so as to respectively determine a set of feature variables having a significant impact on each of the activity states; A second analysis module is used to perform independence analysis on the characteristic variables in the characteristic variable set to determine the weight of each characteristic variable; A second training module is used to train the first activity state discrimination model using the first sensor data, the weight of the feature variable, the physiological feature data of the first user and the influence degree as training data, so as to obtain a second activity state discrimination model capable of determining the user portrait through case-based reasoning; A second processing module is used to process the second sensor data corresponding to the second user according to the second activity state discrimination model to obtain a corresponding activity state discrimination result and information of a target user matching the second user, wherein the target user is the first user, and a user profile of the target user and a user profile of the second user meet the similarity requirement; The physiological characteristic data include age, height, and weight; The performing multivariate linear regression processing on the first sensor data and the physiological characteristic data to determine the degree of influence of different physiological characteristic data on the first sensor data includes: Forming a dependent variable matrix with the first sensor data, forming an independent variable matrix with the physiological characteristic data, and setting a regression coefficient matrix and an error term; Constructing a multivariate linear regression model based on the dependent variable matrix, independent variable matrix, regression coefficient matrix and error term; Performing a multiple linear regression analysis on the first sensor data and the physiological characteristic data using the multiple linear regression model, and constructing a visual heat map based on the analysis results; The degree of influence of different physiological characteristic data on the first sensor data is determined based on the analysis result and the visualized heat map.

9. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the method for determining the human activity state as described in any one of claims 1-7.

Images