Human activity energy consumption measurement method and device, electronic equipment and storage medium

Abstract

The application provides a human activity energy consumption measurement method and device, electronic equipment and storage medium, comprising: obtaining acceleration information of a user during activity, and determining data feature information of the acceleration information; inputting the acceleration information, the data feature information and user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model; the energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is obtained based on acceleration information samples of user activity and corresponding user activity type labels and sensor wearing position labels; and the second neural network model is obtained based on user activity information samples and corresponding activity energy consumption labels. The application can effectively improve the precision of human activity energy consumption measurement and improve the accuracy of human activity energy consumption measurement results.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, and in particular to a human activity energy consumption measurement method and device, electronic equipment and storage medium. BACKGROUND

[0002] With the continuous improvement of people's living standards, the importance of human activity to health is becoming greater and greater. If the activity amount is too small when people are active, it cannot achieve the purpose of fitness, and too much activity will make people feel tired and damage health. Therefore, in order to be more scientific and reasonable, it is necessary to accurately measure the energy consumption of the human body during the activity process.

[0003] In recent years, with the continuous development of sensor technology and the continuous enrichment of sensor functions, the energy consumption of human activity can be measured through electronic devices such as mobile phones or sports watches provided with acceleration sensors. However, in the current calculation method for measuring human activity energy consumption by sensors, too few factors are considered, the energy consumption calculation method is too single, and the data measured by the acceleration sensor is not used enough, resulting in low measurement accuracy and large error in human energy consumption measurement results.

[0004] Therefore, how to more effectively measure human activity energy consumption during the user's activity process has become a technical problem to be solved in the industry. SUMMARY

[0005] The present application provides a human activity energy consumption measurement method, device, electronic equipment and storage medium, which can more effectively measure human activity energy consumption during the user's activity process.

[0006] The present application provides a human activity energy consumption measurement method, comprising:

[0007] Obtaining acceleration information of a user during an activity process and determining data feature information of the acceleration information;

[0008] Inputting the acceleration information, the data feature information and user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model;

[0009] The energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples and data feature information samples of acceleration information.

[0010] According to the human activity energy consumption measurement method provided by the application, the acceleration information, the data feature information and the user identity information of the user are input into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model, including:

[0011] The acceleration information is input into the first neural network model to obtain user activity type information and sensor wearing position information output by the first neural network model.

[0012] The data feature information, the user identity information of the user, the user activity type information and the sensor wearing position information are input into the second neural network model to obtain activity energy consumption information of the user output by the second neural network model.

[0013] According to the human activity energy consumption measurement method provided by the application, the acceleration information is input into the first neural network model to obtain user activity type information and sensor wearing position information output by the first neural network model, including:

[0014] The acceleration information is input into the convolution layer and the pooling layer of the first neural network model, and the convolution layer and the pooling layer are used to extract features of the acceleration information to output a two-dimensional feature vector corresponding to the acceleration information.

[0015] The two-dimensional feature vector is input into a bidirectional long short-term memory neural network layer in the first neural network model to output a hidden state corresponding to the two-dimensional feature vector.

[0016] The hidden state corresponding to the two-dimensional feature vector is input into a full connection layer of the first neural network model to convert the two-dimensional feature vector into a one-dimensional feature vector, and the one-dimensional feature vector is output.

[0017] The one-dimensional feature vector is input into an output layer of the first neural network model to output a prediction probability corresponding to the acceleration information, so that the user activity type information and the sensor wearing position information corresponding to the acceleration information are obtained according to the prediction probability; the first neural network model includes the convolution layer, the pooling layer, the bidirectional long short-term memory neural network layer, the full connection layer and the output layer.

[0018] According to the human activity energy consumption measurement method provided by the application, before the acceleration information, the data feature information and the user identity information of the user are input into an energy consumption measurement model, the method further includes:

[0019] The acceleration information sample, the corresponding user activity type label and the sensor wearing position label are taken as a set of training samples, and a plurality of sets of training samples are obtained;

[0020] For any set of training samples, the training sample is input into the first neural network model after parameter initialization, and the prediction probability corresponding to the training sample output by the first neural network model is obtained;

[0021] A first loss value is calculated by using a first loss function according to the prediction probability corresponding to the training sample and the user activity type label and the sensor wearing position label corresponding to the training sample, so as to update the network parameters of the first neural network model according to the first loss value;

[0022] If the first loss value is less than a first preset threshold, the first neural network model trained is obtained.

[0023] According to the human activity energy consumption measurement method provided by the application, the second neural network model is constructed based on an adaptive neural network model; before the acceleration information, the data feature information and the user identity information of the user are input into the energy consumption measurement model, the method further comprises:

[0024] The user activity information sample and the corresponding activity energy consumption label are taken as a set of training samples, and a plurality of sets of training samples are obtained;

[0025] For any set of training samples, the training sample is input into the adaptive neural network model after parameter initialization, and the prediction probability corresponding to the training sample output by the adaptive neural network model is obtained;

[0026] A second loss value is calculated by using a second loss function according to the prediction probability corresponding to the training sample and the activity energy consumption label corresponding to the training sample, so as to update the network parameters of the adaptive neural network model according to the second loss value;

[0027] If the second loss value is less than a second preset threshold, the second neural network model trained is obtained.

[0028] According to the human activity energy consumption measurement method provided by the application, the acceleration information comprises a plurality of acceleration data; the data feature information of the acceleration information is determined, comprising:

[0029] Based on each acceleration data, the data corresponding to a preset data feature type is determined; the preset data feature type comprises a mean value and other data feature types, and the other data feature types comprise at least one of upper quartile, median, lower quartile, maximum value, minimum value, kurtosis, skewness, variance and standard deviation;

[0030] obtain data feature information of the acceleration information based on the data corresponding to the preset data feature type.

[0031] The application further provides a human activity energy consumption measuring device, comprising:

[0032] an obtaining module, configured to obtain acceleration information of a user during an activity and determine data feature information of the acceleration information;

[0033] a measuring module, configured to input the acceleration information, the data feature information and user identity information of the user into an energy consumption measuring model to obtain activity energy consumption information of the user output by the energy consumption measuring model;

[0034] The energy consumption measuring model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples and data feature information samples of acceleration information.

[0035] The application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the human activity energy consumption measuring method as described above when executing the program.

[0036] The application further provides a non-transient computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the human activity energy consumption measuring method as described above.

[0037] The application further provides a computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement the human activity energy consumption measuring method as described above.

[0038] The application provides a human activity energy consumption measurement method, device, electronic equipment and storage medium, which considers the influence of human activity types and sensor wearing positions on multiple influence factors of human energy consumption, trains a first neural network model by using acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels, so that the trained first neural network model can accurately identify user activity types and sensor wearing positions; meanwhile, by considering user identities and extracting data features of acceleration information collected by the sensor, rich features of acceleration data are fully mined, a second neural network model is trained by using user activity information samples and corresponding activity energy consumption labels, wherein the user activity information samples include user activity type information samples, corresponding sensor wearing position information samples, user identity information samples and data feature information samples of acceleration information, a multi-input energy consumption measurement model is constructed, the shortcomings of too few influence factors considered and insufficient use of sensor measurement data in the prior art calculation method are solved, and the energy consumption measurement model is constructed based on the first neural network model and the second neural network model, the acceleration information collected by the acceleration sensor, the data feature information of the acceleration information and the user identity information are input into the energy consumption measurement model, and the activity energy consumption information of the user is obtained, so that the accuracy of human activity energy consumption measurement can be effectively improved, and the accuracy of human activity energy consumption measurement results can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0040] Figure 1 is a flowchart of the human activity energy consumption measurement method provided by the application;

[0041] Figure 2 is a model structure schematic diagram in the human activity energy consumption measurement method provided by the embodiment of the application;

[0042] Figure 3 is a structure schematic diagram of the human activity energy consumption measurement device provided by the application;

[0043] Figure 4 is a physical structure schematic diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0045] In the description of the application, it should be noted that, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood in a broad sense, for example, can be fixed connection, can also be detachable connection, or integral connection; can be mechanical connection, can also be electrical connection; can be directly connected, can also be indirectly connected through an intermediate medium, can be the communication inside two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0046] The present application will be described below in combination with Figures 1-4 The human activity energy consumption measurement method, device, electronic equipment and storage medium are described.

[0047] Figure 1 The present application provides a flowchart of the human activity energy consumption measurement method, as shown in Figure 1 The method comprises steps 110 and 120.

[0048] In step 110, the acceleration information of the user in the activity process is obtained, and the data feature information of the acceleration information is determined.

[0049] It should be noted that, in the user activity process, the sensor can be fixedly worn at a specified position, the sensor information in the activity process is detected, and the information of the human activity collected by the sensor is extracted for feature extraction and tracking analysis of acceleration and other information. A large amount of experimental data shows that the integral of body movement acceleration with respect to time is linearly related to energy consumption or oxygen consumption, so that the integral of the absolute value of human acceleration can be measured to better evaluate the energy consumption and calculate the human activity energy consumption. Such sensors have the characteristics of small size and light weight, which are convenient for users to carry and have little interference to the normal activities of the users, and the obtained information is close to the actual value in real life.

[0050] Specifically, the acceleration information described in the embodiments of the present application refers to the acceleration information collected by the sensor such as a gyroscope sensor and an acceleration sensor in the energy consumption measurement device worn by the user in the activity process. Specifically, it can include movement acceleration information, movement angular velocity information and gravity acceleration information in three directions of a spatial coordinate system.

[0051] The data feature information described in the embodiments of the present application refers to feature information obtained by fully mining rich features of acceleration data through data feature extraction on acceleration information collected by a motion sensor.

[0052] Based on the content of the above embodiments, as an optional embodiment, the acceleration information includes a plurality of acceleration data; determining data feature information of the acceleration information includes:

[0053] Based on each acceleration data, data corresponding to a preset data feature type is determined; the preset data feature type includes a mean value and other data feature types, and the other data feature types include at least one of a quartile, a median, a quartile, a maximum value, a minimum value, kurtosis, skewness, variance and standard deviation;

[0054] Based on the data corresponding to the preset data feature type, the data feature information of the acceleration information is obtained.

[0055] Specifically, the preset data feature type described in the embodiments of the present application includes a mean value and other data feature types, and the other data feature types include at least one of a quartile, a median, a quartile, a maximum value, a minimum value, kurtosis, skewness, variance and standard deviation, that is, in the present embodiment, in addition to the mean value, one of the quartile, the median, the quartile, the maximum value, the minimum value, the kurtosis, the skewness, the variance and the standard deviation can be selected as the data feature type, a plurality of items or all items can be selected as the preset data feature type.

[0056] In the embodiments of the present application, by fully mining the data features of the acceleration information data set collected by the sensor, more rich features of the acceleration generated in the user activity process are extracted, which is beneficial to improve the human activity energy consumption measurement accuracy.

[0057] Optionally, in the embodiment, the mean value, upper quartile, median, lower quartile, maximum value, minimum value, kurtosis, skewness, variance and standard deviation are selected as the preset data feature types. By selecting 10 types of preset data feature types, the data features of the acceleration information dataset can be more fully extracted. Meanwhile, each 120-second time period can be divided into a series of non-overlapping windows with durations of 10 seconds, 20 seconds, 30 seconds and 60 seconds. By collecting the acceleration information through the sensor, 10 seconds of acceleration information is collected in the first time window. Assuming that the sensor can collect 100 data in 1 second, the acceleration information collected in the first time window can include 1000 data. Thus, according to the 1000 data, the values corresponding to each preset data feature type are calculated, i.e., the mean value, upper quartile, median, lower quartile, maximum value, minimum value, kurtosis, skewness, variance and standard deviation in the first time window can be calculated.

[0058] Similarly, it can be understood that 20 seconds of acceleration information can be collected in the second time window, which can include 2000 data. The operation in the first time window is also performed, i.e., according to the 1000 data, the values corresponding to each preset data feature type are calculated, i.e., the mean value, upper quartile, median, lower quartile, maximum value, minimum value, kurtosis, skewness, variance and standard deviation in the second time window can be calculated. Similarly, the data corresponding to each preset data feature type in each time window can be obtained.

[0059] Further, in the embodiment of the application, the data corresponding to each preset data feature type is taken as the data feature information of the acceleration information. Thus, the data feature information of the acceleration information in each time window can be obtained, which provides basic data for subsequent measurement of the activity energy consumption of the user in each time window.

[0060] The method of the embodiment of the application fully mines the data features of the acceleration information collected by the sensor, and is used for subsequent measurement and calculation of the activity energy consumption in the human activity process, solves the problem of insufficient use of acceleration data, and is beneficial to improving the measurement accuracy of the human activity energy consumption measurement.

[0061] Further, in the embodiment of the application, the acceleration information of the user in the activity process can be obtained through the sensor in the energy consumption measurement device worn by the user, and the data feature extraction is performed on the dataset composed of the collected acceleration information to obtain the data feature information of the acceleration information.

[0062] In step 120, the acceleration information, the data feature information of the acceleration information and the user identity information of the user are input into the energy consumption measurement model to obtain the activity energy consumption information of the user output by the energy consumption measurement model.

[0063] The energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples, and data feature information samples of acceleration information.

[0064] Specifically, the user identity information described in the embodiments of the present application comprises information such as age, height, and weight of the user.

[0065] The user activity types described in the embodiments of the present application can comprise activity types such as walking, running, going up and down stairs, standing, sitting, swimming, and cycling. There are differences in action amplitude and force between different user activity types, which also have an impact on the measurement results of human energy consumption.

[0066] The sensor wearing position described in the embodiments of the present application refers to the position of the part of the human body close to the sensor in the energy consumption measurement device worn by the user during the activity. For example, due to the wearing setting of the energy consumption measurement device and the wearing habit of the user, the sensor wearing position can be different for different users, such as being worn on different parts of the human body such as the ankle, waist, wrist, and arm.

[0067] The first neural network model described in the embodiments of the present application is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels, and is used to identify the user activity type and the sensor wearing position during the user activity. Specifically, a hybrid model of a convolutional neural network (CNN) and a long short-term memory (LSTM) network can be used, or other neural network models that can be used to identify the user activity type and the sensor wearing position during the user activity can be used.

[0068] The activity energy consumption label described in the embodiments of the present application can be obtained by a gas metabolism instrument. The original data of the user energy consumption data is the data of one breath, and the activity energy consumption label of each 10 seconds, 30 seconds, 1 minute, etc. can be obtained according to the experimental requirements.

[0069] The activity energy consumption information described in the embodiments of the present application refers to activity energy consumption information predicted and calculated based on acceleration information in a user activity process, data feature information of the acceleration information and user identity information of the user, which can represent activity energy consumption of the user in a period of time, such as activity energy consumption information of the user in 10 seconds of activity, activity energy consumption information of the user in 20 seconds of activity, activity energy consumption information of the user in 30 seconds of activity and activity energy consumption information of the user in one minute of activity. It can be understood that the acceleration information in the user activity process can be data collected in a corresponding time period, such as using acceleration information collected in one minute of user activity to correspondingly predict activity energy consumption information of the user in one minute of activity.

[0070] The second neural network model described in the embodiments of the present application is trained based on user activity information samples and corresponding activity energy consumption labels; it is used to identify input user activity type information, sensor wearing position information, user identity information and data feature information of acceleration information, and calculate activity energy consumption information generated by the user in the activity process. Specifically, an adaptive neural network (Adaptive Neural Network, ANN) model can be used, or other neural network models that can identify user activity type information, sensor wearing position information, user identity information and data feature information of acceleration information, and adaptively calculate human activity energy consumption can be used.

[0071] The user activity information sample includes a user activity type information sample, a sensor wearing position information sample, a user identity information sample and a data feature information sample of acceleration information.

[0072] The energy consumption measurement model described in the embodiments of the present application is used to measure the energy consumption of the user in the activity process, which includes a first neural network model and a second neural network model.

[0073] Based on the content of the above embodiments, as an optional embodiment, acceleration information, data feature information and user identity information of the user are input into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model, which includes:

[0074] The acceleration information is input into the first neural network model to obtain user activity type information and sensor wearing position information output by the first neural network model.

[0075] The data feature information, the user identity information of the user, the user activity type information and the sensor wearing position information are input into the second neural network model to obtain activity energy consumption information of the user output by the second neural network model.

[0076] Specifically, in the embodiment of the present application, in step 110, after obtaining the acceleration information of the user during the activity and determining the data feature information of the acceleration information, the acceleration information of the user during the activity is input into the first neural network model, and the user activity type information and the sensor wearing position information can be obtained through the first neural network model.

[0077] Further, in the embodiment, the user activity type information and the sensor wearing position information are input into the second neural network model, and the data feature information of the acceleration information and the user identity information of the user are also input into the second neural network model, and the activity energy consumption information of the user output by the second neural network model can be obtained.

[0078] The method of the embodiment of the present application can real-time monitor the user activity type and the sensor wearing position by considering the influence of the human activity type, the sensor wearing position and the user identity information on the energy consumption during the human activity, dynamically adjust the human energy consumption calculation method according to different human activity state scenes, and effectively improve the accuracy of human energy consumption detection.

[0079] Therefore, the acceleration information of the user during the activity, the data feature information of the acceleration information and the user identity information of the user can be input into the energy consumption measurement model to obtain the activity energy consumption information of the user output by the energy consumption measurement model.

[0080] The human activity energy consumption measurement method provided by the embodiment of the present application considers the influence of the human activity type and the sensor wearing position on multiple influencing factors of human energy consumption, trains the first neural network model by using the acceleration information sample of the user activity and the corresponding user activity type label and sensor wearing position label, so that the trained first neural network model can accurately identify the user activity type and the sensor wearing position; meanwhile, the user identity is considered, the data feature of the acceleration information collected by the sensor is extracted, the rich features of the acceleration data are fully mined, the second neural network model is trained by using the user activity information sample and the corresponding activity energy consumption label, the user activity information sample includes the user activity type information sample and the corresponding sensor wearing position information sample, the user identity information sample and the data feature information sample of the acceleration information, a multi-input energy consumption measurement model is constructed, the shortcomings that too few influencing factors are considered and the sensor measurement data is not used enough in the existing calculation method are solved, and the energy consumption measurement model is constructed based on the first neural network model and the second neural network model, the acceleration information collected by the acceleration sensor, the data feature information of the acceleration information and the user identity information are input into the energy consumption measurement model, and the activity energy consumption information of the user is obtained, which can effectively improve the precision of human activity energy consumption measurement and improve the accuracy of human activity energy consumption measurement result.

[0081] Based on the content of the above embodiment, as an optional embodiment, the acceleration information is input to the first neural network model, and user activity type information and sensor wearing position information output by the first neural network model are acquired, including:

[0082] The acceleration information is input to the convolution layer and the pooling layer of the first neural network model, and the convolution layer and the pooling layer are used for feature extraction of the acceleration information, and a two-dimensional feature vector corresponding to the acceleration information is output;

[0083] The two-dimensional feature vector is input to the bidirectional long short-term memory neural network layer in the first neural network model, and a hidden state corresponding to the two-dimensional feature vector is output;

[0084] The hidden state corresponding to the two-dimensional feature vector is input to the full connection layer of the first neural network model, the two-dimensional feature vector is converted into a one-dimensional feature vector, and the one-dimensional feature vector is output;

[0085] The one-dimensional feature vector is input to the output layer of the first neural network model, a prediction probability corresponding to the acceleration information is output, and user activity type information and sensor wearing position information corresponding to the acceleration information are obtained according to the prediction probability; the first neural network model includes a convolution layer, a pooling layer, a bidirectional long short-term memory neural network layer, a full connection layer and an output layer.

[0086] Specifically, the first neural network model described in the embodiment of the application can include a convolution layer, a pooling layer, a bidirectional long short-term memory neural network layer, a full connection layer and an output layer. It can be understood that the first neural network model in the embodiment is a hybrid model of CNN and LSTM network, which can be described as a CNN-LSTM model.

[0087] In the embodiment, 30 testers aged 19-45 are selected as research objects, acceleration information corresponding to different user activity types and different sensor wearing positions is collected, including motion acceleration, angular velocity and gravity acceleration data in three directions of a space coordinate system. After the acceleration information of the user during the activity is acquired, a large amount of noise information exists in the original data, a complementary filter is used for preprocessing the original acceleration information data, and then the preprocessed data can be segmented into a fixed data window of 2.56 seconds, including 128 time steps. At the same time, the preprocessed data can also be subjected to data normalization processing, such as mean-var normalization, so that the processed data conforms to the standard normal distribution, so that the normalized data is conducive to finding a global optimal solution during model training.

[0088] In the embodiment, the architecture of the CNN-LSTM model involves feature extraction of input data of the LSTM using a CNN layer to support sequence prediction, to realize recognition of the user activity type and determination of the sensor wearing position. The input of the CNN can be expressed as [sample, time, row, column, channel] for a given data window, and 128 time steps are divided into four sub-sequences with a time step of 32. The sample represents all data samples in a fixed data window; the time represents 128 time steps; the row represents one-dimensional input data of each sub-sequence; the column represents 32 time steps in the input sub-sequence; and the channel represents nine input variables (three-axis acceleration data, three-axis angular velocity data, and three-axis gravity acceleration data on the spatial coordinates). Each channel can output current data features as the input of the subsequent LSTM, to avoid the influence of mixed data on feature extraction.

[0089] The CNN can automatically extract spatial features in the acceleration information data, and can map a given signal data window to an activity, to support local dependence and scale invariance. The LSTM can better process the time sequence features between data, and learns to map each window of the acceleration information data collected by the sensor to an activity, in which an observation value in the input sequence is read once, and each time step can include one or more variables. The CNN can capture the local connection of the multi-modal perception data, and the translation invariance introduced by the locality can realize automatic feature extraction, and the LSTM extracts time dependence and learns information through time interval increment.

[0090] In the embodiment, the two-way feedback bidirectional LSTM neural network layer is constructed, which is composed of two front and rear connected LSTM units, and the output is determined by the front and rear connected LSTM units, to ensure the time sequence integrity of the data features.

[0091] Figure 2 is a model structure schematic diagram in the human activity energy consumption measurement method provided by the embodiment, as shown in Figure 2 It can be understood that it is a structure schematic diagram of the first neural network model, and the first neural network model can include two convolution layers, one pooling layer, one bidirectional LSTM neural network layer, two fully connected layers, and an output layer.

[0092] Further, in the embodiment, the acceleration information collected in the user activity process is input to the two-layer convolution layer (Conv1D) of the first neural network model through the input layer (Input) of the first neural network model for convolution calculation. Each convolution layer can adopt a convolution kernel size of 3 and a channel number of 64. The output result of the convolution layer is operated through the activation function ReLU to increase the nonlinear fitting ability of the model. Then, the operation result is input to the pooling layer (Maxpool), and the pooling function adopted by the pooling layer is the maximum pooling function to reduce the parameter quantity of the model. Thus, the convolution layer and the pooling layer are used to extract features from the acceleration information, and finally the two-dimensional feature vector corresponding to the acceleration information is output by the pooling layer.

[0093] Further, the two-dimensional feature vector is input to the bidirectional LSTM neural network layer in the first neural network model, and the number of hidden layer nodes can be set to 100 to learn the time sequence information in the learning data. The hidden state corresponding to the two-dimensional feature vector is output by the bidirectional LSTM neural network layer. The hidden state corresponding to the two-dimensional feature vector is input to the two-layer fully connected layer (Dense) of the first neural network model, the two-dimensional feature vector is converted into a one-dimensional feature vector, and the one-dimensional feature vector is output. Finally, the one-dimensional feature vector is input to the output layer (Output) of the first neural network model for data classification, and the prediction probability corresponding to the acceleration information is output to obtain the user activity type information and the sensor wearing position information corresponding to the acceleration information according to the prediction probability.

[0094] The output layer can specifically adopt a softmax classifier, and the output elements of the softmax classifier are multiple, each output element corresponds to a type of user activity type information and sensor wearing position information, each output element outputs the corresponding prediction probability, the maximum prediction probability is selected from the prediction probabilities output by each output element, and the user activity type information and sensor wearing position information corresponding to the output element with the maximum prediction probability are taken as the output result of the CNN-LSTM model to obtain the user activity type information and sensor wearing position information corresponding to the acceleration information.

[0095] The method of the embodiment can automatically extract the data features of human activity by constructing a hybrid model of CNN and LSTM network, using the acceleration data set of human activity for multi-dimensional input of CNN, processing complex time sequence information in human activity by using the bidirectional propagation LSTM network, combining CNN and LSTM network to learn data space features and predict human activity from original sensor data, accurately identifying different user activity types and sensor wearing positions, and facilitating more accurate calculation of human activity energy consumption based on user activity types and sensor wearing positions.

[0096] Based on the content of the above embodiment, as an optional embodiment, before the acceleration information, the data feature information and the user identity information of the user are input into the energy consumption measurement model, the method further comprises:

[0097] The acceleration information sample, the corresponding user activity type label and the sensor wearing position label are taken as a group of training samples, and a plurality of groups of training samples are obtained.

[0098] For any group of training samples, the training sample is input into the first neural network model after parameter initialization, and a prediction probability corresponding to the training sample output by the first neural network model is obtained.

[0099] The first loss value is calculated according to the prediction probability corresponding to the training sample, the user activity type label and the sensor wearing position label corresponding to the training sample by using the first loss function, so as to update the network parameters of the first neural network model according to the first loss value.

[0100] If the first loss value is less than a first preset threshold, a trained first neural network model is obtained.

[0101] Specifically, the first loss function described in the embodiment of the application refers to a pre-set loss function, which is used for model evaluation in the training process of the first neural network model.

[0102] The first preset threshold described in the embodiment of the application refers to a pre-set loss value threshold, which is used to obtain a minimum loss value and complete the training of the first neural network model.

[0103] In the embodiment of the application, before the acceleration information, the data feature information and the user identity information of the user are input into the energy consumption measurement model, the first neural network model in the energy consumption measurement model needs to be trained, and the specific training process is as follows:

[0104] The acceleration information sample, the corresponding user activity type label and the sensor wearing position label are taken as a group of training samples, and a plurality of groups of training samples can be obtained for a plurality of acceleration information samples.

[0105] It can be understood that in the embodiment, the acceleration information sample, the user activity type label and the sensor wearing position label are one-to-one corresponding, that is, one acceleration information sample corresponds to one user activity type label and one sensor wearing position label.

[0106] Then, for any group of training samples, the acceleration information sample in the training sample and the corresponding user activity type label and sensor wearing position label are simultaneously input into the first neural network model after parameter initialization, and a prediction probability value corresponding to the training sample is output.

[0107] On this basis, the first loss value is calculated according to the prediction probability value corresponding to the training sample, and the user activity type label and the sensor wearing position label carried in the training sample by using the first loss function. The user activity type label and the sensor wearing position label can be represented as one-hot vectors.

[0108] In the embodiments of the present application, the representation of the user activity type label and the sensor wearing position label and the first loss function can be set according to actual needs, and are not specifically limited here.

[0109] After the first loss value is calculated, the current training process is completed, and the model parameters in the first neural network model are updated before the next training is performed. During the training, if the first loss value calculated for a certain training sample is less than the first preset threshold or reaches the preset maximum number of iterations, the first neural network model training is completed, and the trained first neural network model can be obtained.

[0110] The method of the embodiments of the present application controls the loss value of the first neural network model in the preset range by training the first neural network model, thereby facilitating the improvement of the prediction accuracy of the first neural network model and the accurate identification of the user activity type and the sensor wearing position during the user activity.

[0111] Based on the content of the above embodiments, as an optional embodiment, the second neural network model is constructed based on an adaptive neural network model; before the acceleration information, the data feature information and the user identity information of the user are input into the energy consumption measurement model, the method further comprises:

[0112] The user activity information sample and the corresponding activity energy consumption label are taken as a group of training samples, and a plurality of groups of training samples are obtained;

[0113] For any one group of training samples, the training sample is input into the adaptive neural network model after parameter initialization, and the prediction probability corresponding to the training sample output by the adaptive neural network model is obtained;

[0114] The second loss value is calculated according to the prediction probability corresponding to the training sample and the activity energy consumption label corresponding to the training sample by using the second loss function, so as to update the network parameters of the adaptive neural network model according to the second loss value;

[0115] If the second loss value is less than the second preset threshold, a trained second neural network model is obtained.

[0116] Specifically, the second loss function described in the embodiments of the present application refers to a pre-set loss function, which is used for model evaluation in the training process of the second neural network model. It can be defined as the mean square error of the output vector.

[0117] The second preset threshold described in the embodiments of the present application is a preset loss value threshold, which is used to obtain a minimum loss value and complete the training of the second neural network model.

[0118] The second neural network model described in the embodiments of the present application is constructed based on an ANN model, which can adaptively calculate the activity energy consumption of a user based on dynamically changing user activity types and sensor wearing positions.

[0119] In the embodiments of the present application, the ANN model can adopt a three-layer network structure with 10 nodes in the input layer, 20 nodes in the hidden layer and 1 node in the output layer. A hyperbolic tangent curve function is selected as the transfer function of the input layer and the hidden layer.

[0120] In the embodiments of the present application, before starting to train the ANN model, each layer of network weight is assigned a small random value. In order to improve the estimation error, the learning process of the ANN model adopts a feedforward / backpropagation algorithm (BP neural network algorithm). The feedforward process includes calculating the conversion between the input and the output, calculating the data of each time window according to the current weight and the transfer function. The estimated value of the feedforward is compared with the observed value, and the error value, also known as the loss value, is calculated. The gradient (change rate) of this error is used for iteration to adjust the network weight. The process of updating the weight value constitutes the BP backpropagation, which is an optimization process that follows most standard optimization methods. Because the same data is input into the network model multiple times, the weight value will adapt, the error signal will decrease, and a set of optimal weights will be achieved. The model is trained in this way until an acceptable error is reached.

[0121] In the embodiments of the present application, the specific training process of the second neural network model in the energy consumption measurement model is as follows:

[0122] The user activity information sample and its corresponding activity energy consumption label are taken as a set of training samples, wherein the user activity information sample includes a user activity type information sample, a sensor wearing position information sample, a user identity information sample and a data feature information sample of acceleration information. For multiple user activity information samples, multiple sets of training samples can be obtained.

[0123] It can be understood that in the present embodiment, the user activity information sample and the activity energy consumption label are one-to-one corresponding, that is, one user activity information sample corresponds to one activity energy consumption label.

[0124] Then, for any set of training samples, the user activity information sample in the training sample and its corresponding activity energy consumption label are simultaneously input into the second neural network model after parameter initialization, and the prediction probability value corresponding to the training sample is output.

[0125] On this basis, a second loss value is calculated according to the prediction probability value corresponding to the training sample and the activity energy consumption label carried in the training sample by using a second loss function. The activity energy consumption label can also be expressed as a one-hot vector.

[0126] In the embodiments of the present application, the representation of the activity energy consumption label and the second loss function can be set according to actual needs, which are not specifically limited here.

[0127] After the second loss value is calculated, the current training process is completed, and the model parameters in the second neural network model are updated by using the BP back propagation algorithm, and then the next training is performed. During the training, if the second loss value calculated for a certain training sample is less than a second preset threshold or reaches a preset maximum number of iterations, the training of the second neural network model is completed, that is, the trained second neural network model can be obtained.

[0128] The method of the embodiments of the present application trains the second neural network model, uses the feedforward / back propagation algorithm, compares the estimated value of the feedforward with the observation value, calculates the error value, uses the error for iterative adjustment of the network weight, and controls the error of the first neural network model within a preset range, thereby improving the accuracy of the human activity energy consumption measurement and obtaining more accurate activity energy consumption calculation results.

[0129] The human activity energy consumption measurement device provided by the present application is described below, and the human activity energy consumption measurement device described below can be mutually corresponding and referred to the human activity energy consumption measurement method described above.

[0130] Figure 3 is a structural schematic diagram of the human activity energy consumption measurement device provided by the present application, as shown in Figure 3 , comprising:

[0131] The acquisition module 310 is configured to acquire acceleration information of a user during an activity process and determine data feature information of the acceleration information.

[0132] The measurement module 320 is configured to input the acceleration information, the data feature information and user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model.

[0133] The energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples and data feature information samples of acceleration information.

[0134] The human activity energy consumption measurement device provided in the embodiment can be used to execute the human activity energy consumption measurement method, and has similar principles and technical effects, which will not be described herein.

[0135] The human activity energy consumption measurement device provided in the embodiment considers the influence of human activity types and sensor wearing positions on multiple influence factors of human energy consumption, trains the first neural network model by using acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels, so that the trained first neural network model can accurately identify user activity types and sensor wearing positions; meanwhile, the user identity is considered, data features of acceleration information collected by the sensor are extracted, rich features of acceleration data are fully mined, the second neural network model is trained by using user activity information samples and corresponding activity energy consumption labels, the user activity information samples comprise user activity type information samples, corresponding sensor wearing position information samples, user identity information samples and data feature information samples of acceleration information, a multi-input energy consumption measurement model is constructed, the shortcomings of too few influence factors considered and insufficient utilization of sensor measurement data in the prior art are solved, the energy consumption measurement model is constructed based on the first neural network model and the second neural network model, acceleration information collected by an acceleration sensor, data feature information of the acceleration information and user identity information are input into the energy consumption measurement model, and activity energy consumption information of the user is obtained, so that the precision of human activity energy consumption measurement can be effectively improved, and the accuracy of human activity energy consumption measurement results can be improved.

[0136] Figure 4 is a schematic diagram of an entity structure of an electronic device provided in the embodiment, Figure 4As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 complete communication with each other through the communications bus 440. The processor 410 can invoke the logic instructions in the memory 430 to execute the human activity energy consumption measurement method provided by each method described above, which includes: obtaining acceleration information of a user during an activity, and determining data feature information of the acceleration information; inputting the acceleration information, the data feature information, and user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model; the energy consumption measurement model includes a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples include user activity type information samples, sensor wearing position information samples, user identity information samples, and data feature information samples of acceleration information.

[0137] In addition, the logic instructions in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0138] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored in a non-transitory computer readable storage medium, and the computer program being capable of executing the human activity energy consumption measurement method provided by the above-mentioned methods when executed by a processor, the method comprising: obtaining acceleration information of a user during an activity, and determining data feature information of the acceleration information; inputting the acceleration information, the data feature information, and user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model; the energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples, and data feature information samples of acceleration information.

[0139] In another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, the computer program being capable of implementing the human activity energy consumption measurement method provided by the above-mentioned methods when executed by a processor, the method comprising: obtaining acceleration information of a user during an activity, and determining data feature information of the acceleration information; inputting the acceleration information, the data feature information, and user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model; the energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples, and data feature information samples of acceleration information.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0141] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0142] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of measuring energy expenditure of human activity, characterized by, The method comprises: obtaining acceleration information of a user during an activity and determining data feature information of the acceleration information; inputting the acceleration information, the data feature information and user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model; the energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples and data feature information samples of acceleration information; the acceleration information comprises a plurality of acceleration data; determining the data feature information of the acceleration information comprises: based on each of the acceleration data, determining data corresponding to a preset data feature type; the preset data feature type comprises a mean value and other data feature types, and the other data feature types comprise at least one of a maximum value, a minimum value, a kurtosis, a skewness, a variance and a standard deviation; based on the data corresponding to the preset data feature type, obtaining the data feature information of the acceleration information; wherein, based on each of the acceleration data, determining data corresponding to a preset data feature type, based on the data corresponding to the preset data feature type, obtaining the data feature information of the acceleration information comprises: dividing a specific time period into a series of non-overlapping time windows; obtaining a plurality of acceleration data in each time window; based on the plurality of acceleration data in each time window, calculating data corresponding to a preset data feature type in each time window; based on the data corresponding to the preset data feature type in each time window, obtaining the data feature information of the acceleration information; inputting the acceleration information, the data feature information and the user identity information of the user into an energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model comprises: inputting the acceleration information into the first neural network model to obtain user activity type information and sensor wearing position information output by the first neural network model; inputting the data feature information, the user identity information of the user, the user activity type information and the sensor wearing position information into the second neural network model to obtain activity energy consumption information of the user output by the second neural network model.

2. The method of measuring energy expenditure of human physical activity according to claim 1, wherein, inputting the acceleration information into the first neural network model to obtain user activity type information and sensor wearing position information output by the first neural network model comprises: inputting the acceleration information into a convolution layer and a pooling layer of the first neural network model, extracting features of the acceleration information by using the convolution layer and the pooling layer, and outputting a two-dimensional feature vector corresponding to the acceleration information; inputting the two-dimensional feature vector into a bidirectional long short-term memory neural network layer in the first neural network model, and outputting a hidden state corresponding to the two-dimensional feature vector; inputting the hidden state corresponding to the two-dimensional feature vector into a full connection layer of the first neural network model, converting the two-dimensional feature vector into a one-dimensional feature vector, and outputting the one-dimensional feature vector; inputting the one-dimensional feature vector into an output layer of the first neural network model, outputting a prediction probability corresponding to the acceleration information, and obtaining the user activity type information and the sensor wearing position information corresponding to the acceleration information according to the prediction probability; the first neural network model comprises the convolution layer, the pooling layer, the bidirectional long short-term memory neural network layer, the full connection layer, and the output layer.

3. The method of measuring energy expenditure of human physical activity according to any one of claims 1-2, characterized in that, Before inputting the acceleration information, the data feature information, and the user identity information of the user into the energy consumption measurement model, the method further comprises: obtaining a plurality of groups of training samples by taking the acceleration information sample, the corresponding user activity type label, and the sensor wearing position label thereof as a group of training samples; for any one group of training samples, inputting the training sample into the first neural network model after parameter initialization to obtain a prediction probability corresponding to the training sample output by the first neural network model; calculating a first loss value according to the prediction probability corresponding to the training sample and the user activity type label and the sensor wearing position label corresponding to the training sample by using a first loss function, and updating the network parameters of the first neural network model according to the first loss value; if the first loss value is less than a first preset threshold, a trained first neural network model is obtained.

4. The method of measuring energy expenditure of human physical activity according to any one of claims 1 to 2, characterized in that, The second neural network model is constructed based on an adaptive neural network model; before inputting the acceleration information, the data feature information, and the user identity information of the user into the energy consumption measurement model, the method further comprises: obtaining a plurality of groups of training samples by taking the user activity information sample and the corresponding activity energy consumption label thereof as a group of training samples; for any one group of training samples, inputting the training sample into the adaptive neural network model after parameter initialization to obtain a prediction probability corresponding to the training sample output by the adaptive neural network model; calculating a second loss value according to the prediction probability corresponding to the training sample and the activity energy consumption label corresponding to the training sample by using a second loss function, and updating the network parameters of the adaptive neural network model according to the second loss value; if the second loss value is less than a second preset threshold, a trained second neural network model is obtained.

5. A human activity energy expenditure measuring device, characterized by, comprises: an acquisition module configured to acquire acceleration information of a user during an activity process, and determine data feature information of the acceleration information; a measurement module configured to input the acceleration information, the data feature information, and user identity information of the user into an energy consumption measurement model, and obtain activity energy consumption information of the user output by the energy consumption measurement model. The energy consumption measurement model comprises a first neural network model and a second neural network model; the first neural network model is trained based on acceleration information samples of user activities and corresponding user activity type labels and sensor wearing position labels; the second neural network model is trained based on user activity information samples and corresponding activity energy consumption labels; the user activity information samples comprise user activity type information samples, sensor wearing position information samples, user identity information samples, and data feature information samples of acceleration information; The acceleration information comprises a plurality of acceleration data; The data feature information of the acceleration information is determined, comprising: Based on each of the acceleration data, data corresponding to a preset data feature type is determined; the preset data feature type comprises a mean value and other data feature types, and the other data feature types comprise at least one of a quartile, a median, a quartile, a maximum value, a minimum value, a kurtosis, a skewness, a variance, and a standard deviation; Based on the data corresponding to the preset data feature type, the data feature information of the acceleration information is obtained; The method comprises: dividing a specific time period into a series of non-overlapping time windows; obtaining a plurality of acceleration data in each time window; based on the plurality of acceleration data in each time window, data corresponding to a preset data feature type in each time window is calculated; based on the data corresponding to the preset data feature type in each time window, the data feature information of the acceleration information is obtained; The acceleration information, the data feature information, and the user identity information of the user are input into the energy consumption measurement model to obtain activity energy consumption information of the user output by the energy consumption measurement model, comprising: The acceleration information is input into the first neural network model to obtain user activity type information and sensor wearing position information output by the first neural network model; The data feature information, the user identity information of the user, the user activity type information, and the sensor wearing position information are input into the second neural network model to obtain activity energy consumption information of the user output by the second neural network model.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the human activity energy consumption measurement method according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the human activity energy consumption measurement method according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the human activity energy consumption measurement method according to any one of claims 1 to 4.

Images