Environmental Sensor Data-Driven Human Activity Recognition Method

Abstract

The present invention discloses a method for human activity recognition driven by environmental sensor data, comprising the following steps: environmental sensor data acquisition: collecting environmental sensor data, performing sequence synchronization alignment on the environmental sensor data, and constructing a dataset; feature selection: using the sensor data contribution significance analysis (CSA) method based on sensor state frequency to calculate the contribution of sensor categories. Based on this, the sensor category most conducive to identifying behavior is selected, and feature data is obtained and embedded into the state sequence of the object to be identified; data noise constraint: to reduce noise in the activity recognition process, a sensor constraint distance matrix D is constructed based on the sensor layout diagram to form a new feature dataset and encode the sensor data state sequence; activity recognition: constructing a wide time domain convolutional neural network for daily activity recognition. The present invention measures the contribution of different sensor types to activity recognition based on user behavior habits, and has the characteristics of higher recognition rate and shorter recognition time for human activity classification.

Description

Technical Field

[0001] The present invention relates to the field of human activity recognition, and in particular to a human activity recognition method driven by environmental sensor data. Background Art

[0002] With the development of artificial intelligence and the widespread use of sensors, personal human activity recognition (HAR) technology based on noninvasive environmental sensors has attracted widespread attention and demonstrated significant application value. In the prior art, Bianchi et al. proposed an innovative HAR system connected to the internet via a home router to facilitate remote management and data collection. To minimize resource usage, a low-cost activity recognition method was implemented on embedded devices using a CNN for daily activity monitoring. Recurrent Neural Networks (RNNs) can remember early information in the center of a sequence data, making them suitable for processing time series data. When the time series span is large, the vanishing gradient problem can be avoided by using LSTM or GRU layers. Fazle et al. converted an existing univariate time series classification model into a multivariate time series classification model (LSLTM-FCN and ALSLTM-FCN), which is beneficial for various multivariate time series classification tasks. Hassan et al. proposed an accurate and scalable time series classifier in which the Inception module simultaneously slides multiple filters of different lengths over the same input time series and uses filters that are much wider than ResNet (almost ten times wider).

[0003] In summary, methods for identifying human activities using environmental sensors in smart homes still have several challenges. During the data preparation phase, existing methods select the same sensor type for different users, ignoring differences in user habits and sensor layout. During data preprocessing, data overlap is easily generated in multi-person environments. In activity recognition classification, existing classification algorithms are not conducive to ubiquitous computing. Summary of the Invention

[0004] The purpose of the present invention is to overcome the above shortcomings and propose an efficient and accurate human activity recognition method driven by environmental sensor data.

[0005] The present invention provides a method for human activity recognition driven by environmental sensor data, wherein the method comprises the following steps:

[0006] Step 1: Collect environmental sensor data: Collect environmental sensor data, perform sequence synchronization and alignment on the environmental sensor data, and build a data set;

[0007] Step 2: Feature selection: We use the sensor data contribution significance analysis (CSA) method based on sensor state frequency to calculate the contribution of sensor categories. Based on this, we select the sensor category that is most conducive to identifying the behavior, obtain feature data, and embed it into the state sequence of the object to be identified.

[0008] The sensor data contribution significance analysis method CSA based on sensor state frequency is:

[0009] Assume that the activity type of the behavior to be identified is A={A1,A2,…,A i …,A t}, where A i ={a i1 ,a i2 ,…a ij ,…,a in} represents the i-th activity, a ij ={r i j1 ,r i j2 ,…,r i jk} represents the jth time series record of the i-th activity, r i jk Represents the kth state monitoring value of the sensor data window triggered during the behavior, including trigger time, sensor number, and state value;

[0010] The arranged sensor group is represented as S={s1,s2,…,s i ,…,s m}, where m represents the number of sensor types, s i represents the i-th sensor;

[0011] Based on the entropy uncertainty theory, the information gain of different sensor types when measuring an activity occurs is calculated using the following formula: i The trigger frequency of the jth sensor when it occurs

[0012]

[0013] in Indicates activity A i The total triggering time of the jth sensor when it occurs;

[0014] The number of sensor types related to an event determines the chaotic process of the information system. In order to determine the contribution of various sensors in behavior recognition, the following formula is used to calculate the contribution of the jth sensor to activity A i The amount of self-information brought by the occurrence of:

[0015]

[0016] Then, the universality of various sensors in the behavior process is calculated: in the entire activity recognition sensor state dataset, if a certain type of activity usually triggers a specific type of sensor, it means that the sensor is associated with the activity; otherwise, the sensor category that is triggered intermittently can be regarded as noise sensor data; the information of the sensors is balanced and the activity A is calculated using the following formula i The generality of the jth sensor when it occurs:

[0017]

[0018] Where, |A i |Indicates activity A i The total number of occurrences of represents the total number of triggers in the activity occurrence covering the jth sensor;

[0019] Calculate the information contribution of various sensors to behavior recognition: By integrating the self-information and universality of the sensor, the following formula is used to calculate the contribution D of the jth sensor to the i-th activity ij :

[0020]

[0021] Finally, based on the measurement of sensor information contribution, the sensor category that is most conducive to identifying the behavior is selected;

[0022] Step 3: Data noise constraint: To reduce noise in the activity recognition process, a sensor constraint distance matrix D is constructed based on the sensor layout diagram. This excludes irrelevant data in the state sequence of the object to be identified, forms a new feature dataset, and encodes the sensor data state sequence.

[0023] Step 4: Activity Recognition: Based on the environmental sensor data, a wide-temporal convolutional neural network recognition model is constructed. With the goal of minimizing the cross-entropy loss function, the recognition model is trained using the AdaGrad optimizer to perform human activity recognition.

[0024] The above-mentioned human activity recognition driven by environmental sensor data, wherein: the environmental sensor data collected in step 1 includes motion sensors, door sensors, temperature sensors, electricity consumption sensors, burner sensors, object sensors, hot water sensors, and cold water sensors.

[0025] The above-mentioned human activity recognition driven by environmental sensor data, wherein: in step 3, a sensor constraint distance matrix D is constructed based on the sensor layout diagram, and the distance element D (s i , s j) is calculated using the following formula:

[0026]

[0027] |s i , s j | spatial Represents the spatial distance between two instantaneous sensors, and is calculated by the minimum number of intermediate sensors when connected, that is, the Manhattan distance; otherwise, considering the lag of the delayed sensor state change, the corresponding distance is recorded as 1, that is, all data with high significant contribution monitored by the delayed sensor are regarded as non-noise data.

[0028] The above-mentioned environmental sensor data-driven human activity recognition, wherein: the wide time domain convolutional neural network in step 4 includes an embedding layer, four one-dimensional convolutional layers, three fully connected layers, and a softmax classifier.

[0029] In the above-mentioned human activity recognition driven by environmental sensor data, the embedding layer is used to encode the input data and map the data from a low-dimensional space to a high-dimensional space using an embedding encoding method. The formula is as follows:

[0030] x=W E *{s1, s2, ..., s i ,...,s n}

[0031] Among them, s i is the i-th state code value of the sensor data sequence; W E is a randomly initialized learnable embedding matrix W E ∈R V×U , where R is the real domain space, U is the embedding size, V is the number of deployed environmental sensors; x is the sensor feature vector value.

[0032] The above-mentioned human activity recognition driven by environmental sensor data, wherein: the four one-dimensional convolutional layers and the two fully connected layers constitute the feature extraction layer of the wide temporal domain convolutional network: the sensor feature vector value x is input, a small-size convolution kernel is used to extract short-term and detail features, and a large-size convolution kernel is used to extract long-term and context information fusion features, and its convolution feature f(x h )The extraction formula is:

[0033] f(x h )=f(x*w h +b h )

[0034] where w h is a one-dimensional convolution kernel, the size of the convolution kernel is h; b his a random bias value in the real number domain, and the step size is set to 1 to extract accurate information;

[0035] Then, the features extracted by the convolutional layer are spliced and fused to obtain a feature map f(x c ), where the features are downsampled by the maximum pooling method to obtain low-dimensional vectors with significant features. The formula is as follows:

[0036] f(x c )=BN{max[f(x1),...,f(x h )]}

[0037] Among them, BN is standardized normal distribution processing;

[0038] Then, a fully connected layer is added to manage the weight of each feature, and an active layer ReLU is added to perform nonlinear transformation of the features. The operation of the fully connected layer and the active layer is expressed as FA(x) = f(ReLU{w·x+b}). In the process of nonlinear feature transformation, in order to avoid overfitting, the neuron probabilistic learning strategy Drop is adopted, which is formulated as follows:

[0039] f(x o )=FA(Drop{f(x c ),p})

[0040] where p is the activation rate of neurons in the fully connected layer during data fitting, f(x o ) are the output features.

[0041] The above-mentioned environmental sensor data-driven human activity recognition, wherein: the one fully connected layer and the softmax classifier constitute a feature classification layer: human activities are classified according to the number of activity categories.

[0042] Compared with the prior art, the present invention has obvious beneficial effects. As can be seen from the above scheme, a sensor data contribution significance analysis (CSA) method based on the inverse sensor frequency of the sensor state frequency is proposed to measure the contribution of a specific type of sensor to the recognition of a specific type of behavior. Secondly, we construct a spatial distance matrix based on the layout of environmental sensors for context perception and data noise reduction. Then, we propose a human activity recognition method (HAR-WCNN) based on a wide temporal domain convolutional neural network and multi-environmental sensor data for daily behavior recognition. In summary, the present invention measures the contribution of different types of sensors to activity recognition based on the user's behavioral habits. In the data preprocessing stage, a distance matrix is designed according to the user's actual spatial layout to avoid data overlap in a multi-person living environment. It has a higher recognition rate and shorter recognition time for human activity classification.

[0043] The beneficial effects of the present invention are further illustrated below through specific implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0045] The following describes in detail the specific implementation, features and efficacy of a method for human activity recognition driven by environmental sensor data proposed in accordance with the present invention in conjunction with the accompanying drawings and preferred embodiments.

[0046] See also Figure 1 The present invention provides a method for human activity recognition driven by environmental sensor data, wherein: the method comprises the following steps:

[0047] Step 1: Collect environmental sensor data: Collect environmental sensor data including motion sensors (monitoring living body movement within a certain range), door sensors (monitoring door opening and closing), temperature sensors (monitoring temperature), power consumption sensors (monitoring power), burner sensors (monitoring burner working status), object sensors (counting the number of items), hot water sensors (monitoring hot water usage), and cold water sensors (monitoring cold water usage). Perform sequence synchronization and alignment on each environmental sensor data to construct a dataset.

[0048] Step 2: Feature selection: We use the sensor data contribution significance analysis (CSA) method based on sensor state frequency to calculate the contribution of sensor categories. Based on this, we select the sensor category that is most conducive to identifying the behavior, obtain feature data, and embed it into the state sequence of the object to be identified.

[0049] The sensor data contribution significance analysis method CSA based on sensor state frequency is:

[0050] Assume that the activity type of the behavior to be identified is A={A1,A2,…,A i …,A t}, where A i ={a i1 ,a i2 ,…a ij ,…,a in} represents the i-th activity set, a ij ={r i j1 ,r i j2 ,…,r i jk} represents the jth time series record of the i-th activity, r i jkRepresents the kth state monitoring value of the sensor data window triggered during the behavior, including trigger time, sensor number, state value and other information;

[0051] The arranged sensor group is represented as S={s1,s2,…,s i ,…,s m}, where m represents the number of sensor types, s i represents the i-th sensor;

[0052] Based on the entropy uncertainty theory, the information gain of different sensor types when measuring an activity occurs is calculated using the following formula: i The trigger frequency of the jth sensor when it occurs

[0053]

[0054] in Indicates activity A i The total triggering time of the jth sensor when it occurs;

[0055] The number of sensor types related to an event determines the chaotic process of the information system. In order to determine the contribution of various sensors in behavior recognition, the following formula is used to calculate the contribution of the jth sensor to activity A i The amount of self-information brought about by the occurrence of:

[0056]

[0057] Then, the universality of various sensors in the behavior process is calculated: in the entire activity recognition sensor state dataset, if a certain type of activity usually triggers a specific type of sensor, it means that the sensor is associated with the activity; otherwise, the sensor category that is triggered intermittently can be regarded as noise sensor data; the information of the sensors is balanced and the activity A is calculated using the following formula i The commonality of the jth sensor when it occurs.

[0058]

[0059] Where, |A i |Indicates activity A i The total number of occurrences of Represents the total number of triggers in the activity occurrence covering the jth sensor type.

[0060] Calculate the information contribution of various sensors to behavior recognition: By integrating the self-information and universality of the sensor, the following formula is used to calculate the contribution Dij of the j-th sensor to the i-th activity:

[0061]

[0062] Finally, based on the measurement of sensor information contribution, the sensor category that is most conducive to identifying the behavior is selected;

[0063] Step 3: Data noise constraint: To reduce noise in the activity recognition process, a sensor constraint distance matrix D is constructed based on the sensor layout diagram. This excludes irrelevant data in the state sequence of the object to be identified, forms a new feature dataset, and encodes the sensor data state sequence.

[0064] The sensor constraint distance matrix D is constructed based on the sensor layout diagram, D∈R n*n , n represents the number of sensors in the home, and the distance element D(s i , s j ) is calculated using the following formula:

[0065]

[0066] |s i , s j | spatial Represents the spatial distance between two instantaneous sensors, and is calculated by the minimum number of intermediate sensors when connected, that is, the Manhattan distance; otherwise, taking into account the lag of the delayed sensor state change, the corresponding distance is recorded as 1, that is, all data with high significant contribution monitored by the delayed sensor are regarded as non-noise data;

[0067] Step 4: Activity Recognition: Based on the environmental sensor data, a wide-temporal convolutional neural network recognition model is constructed. With the goal of minimizing the cross-entropy loss function, the recognition model is trained using the AdaGrad optimizer to perform human activity recognition.

[0068] The wide time domain convolutional neural network includes an embedding layer, four one-dimensional convolutional layers, three fully connected layers, and a softmax classifier, wherein:

[0069] 1) Embedding layer: Encoding of input data, using embedding encoding to map data from low-dimensional space to high-dimensional space. The formula is as follows:

[0070] x=W E *{s1, s2, ..., s i ,...,s n}

[0071] Among them, s i is the i-th state code value of the sensor data sequence; W E is a randomly initialized learnable embedding matrix W E ∈R V×U, where R is the real domain space, U is the embedding size, V is the number of deployed environmental sensors; x is the sensor feature vector value;

[0072] 2) Feature extraction layer of wide temporal convolutional network: It includes convolutional layer and two fully connected layers. Its input sensor feature vector value x is used to extract short-term and detail features using small-size convolution kernel, and long-term and context information fusion features using large-size convolution kernel. Its convolution feature f(x h )The extraction formula is:

[0073] f(x h )=f(x*w h +b h )

[0074] where w h is a one-dimensional convolution kernel, the size of the convolution kernel is h; b h is a random bias value in the real number domain, and the step size is set to 1 to extract accurate information;

[0075] The features extracted by the convolutional layer are spliced and fused to obtain a feature map f(x c ), where the features are downsampled by the maximum pooling method to obtain low-dimensional vectors with significant features. The formula is as follows:

[0076] F(x C )=BN{max[f(x1),...,f(x h )]}

[0077] Among them, BN is standardized normal distribution processing;

[0078] Then, a fully connected layer is added to manage the weight of each feature, and an active layer ReLU is added to perform nonlinear transformation of the features. The operation of the fully connected layer and the active layer is expressed as FA(x) = f(ReLU{w·x+b}). In the process of nonlinear feature transformation, in order to avoid overfitting, the neuron probabilistic learning strategy Drop is adopted, which is formulated as follows:

[0079] f(x o )=FA(Drop{f(x c ),p})

[0080] where p is the activation rate of neurons in the fully connected layer during data fitting, f(x o ) is the output feature;

[0081] 3) Classification layer: The feature classification layer consists of a fully connected layer and a softmax function. According to the number of activity categories, human activities are classified through probability evaluation of the softmax activation function in the fully connected layer.

[0082] The present invention mainly includes three stages: sensor category selection, data preprocessing, and behavior classification. The input data is collected from various types of environmental sensors, and the monitored states include natural changes in the home environment and changes triggered by biological factors. The data preprocessing stage is mainly to overcome the interference of irrelevant data in the activity recognition process. A key issue is to effectively adjust the new data set when there is a fusion of various data to ensure its compatibility with the classification algorithm. The selection of sensors in the present invention uses the CSA method to calculate the contribution of various sensor data to help select sensor type data suitable for behavior recognition. In addition, based on the consistency characteristics of home layout and human activities, a spatiotemporal constraint distance matrix is constructed to exclude irrelevant data in the state sequence of the object to be identified and form a new classification feature set. The sensor data sequence is then repaired and encoded. The present invention uniformly adopts a binary encoding method to reduce the amount of data encoding and constructs a wide temporal domain convolutional neural network (WCNN) model for activity recognition.

[0083] 1. Sensor categories based on CSA

[0084] Selecting appropriate sensor data ensures sufficient data features and avoids redundancy. There are many types of home environment sensors, including motion sensors, door sensors, temperature sensors, electricity usage sensors, burner sensors, object sensors, hot water sensors, and cold water sensors. Typically, each behavior is associated with a specific type of sensor. There are three main reasons for the difficulty in extracting behavioral features: 1) sensor type, 2) sensor orientation within the home environment, and 3) sensor state changes among individuals participating in the same type of activity, which are influenced by individual lifestyle differences. Therefore, for feature selection, a sensor category saliency analysis method is proposed that integrates the sensor's inherent information content and versatility.

[0085] TF-IDF is a commonly used weighting technique in information retrieval and data mining. TF-IDF can be used to identify keywords within articles. In industry, it is also frequently used for initial text data cleansing. The founders of modern information theory defined information, quantified information quantity, and introduced the concept of entropy within information theory. Inspired by these two approaches, the present invention proposes the CSA method to measure the contribution of a certain sensor type to the recognition of a certain type of activity, ultimately identifying sensor categories that have a significant impact on activity recognition.

[0086] Assume that the activity type of the behavior to be identified is A={A1,A2,…,A t}, where A i ={a i1 ,a i2 ,…a ij ,…,a in} represents the i-th activity set. a ij ={r i j1 ,r i j2 ,…,r i jk} represents the jth time series record of the i-th activity, r i jk Represents the kth state monitoring value of the sensor data window triggered during the behavior (a sensor of the same type may be triggered multiple times), including the trigger time, sensor number, state value, and other information.

[0087] The sensor group arranged in the home is represented as S = {s1, s2, ..., s m}, where m represents the number of sensor types, s i represents the i-th sensor.

[0088] The information gain of different sensor types when measuring an activity is based on entropy uncertainty theory. The activity A is calculated using formula (1). i The trigger frequency of the jth sensor when it occurs

[0089]

[0090] in Indicates activity A i The total triggering time of the jth sensor when it occurs.

[0091] The number of sensor types associated with an event determines the chaotic process of the information system. In order to determine the contribution of various sensors in behavior recognition, formula (2) is proposed to calculate the contribution of the jth sensor to activity A. i The amount of self-information brought about by the occurrence of.

[0092]

[0093] Then, the universality of various sensors in the behavior process is calculated. In the entire activity recognition sensor state dataset, if a certain type of activity usually triggers a specific type of sensor, it means that the sensor is associated with the activity. Otherwise, the sensor category that is triggered intermittently can be regarded as noisy sensor data. The information of the sensors is balanced and the activity A is calculated using formula (3). i The commonality of the jth sensor when it occurs.

[0094]

[0095] Where, |A i |Indicates activity A i The total number of occurrences of Represents the total number of triggers in the activity occurrence covering the jth sensor type.

[0096] Then, the information contribution of various sensors to behavior recognition is calculated. By integrating the self-information and universality of the sensor, that is, the combination of formula (2) and (3), formula (4) is used to calculate the contribution D of the j-th sensor to the i-th activity ij :

[0097]

[0098] Finally, based on the contribution of measured sensor information, the sensor category that is most conducive to identifying the behavior is selected.

[0099] CSA is based on the contribution of frequency measurement of sensor feature information in the behavior recognition process. It reduces the amount of data processing and filters out more unique data because it does not involve changes in the original data values. 2. Contextual information perception based on spatial distance matrix

[0100] Based on the layout of context-aware environmental sensors, a context-aware spatial distance matrix (SDM) method is constructed to reduce data noise caused by multi-person cross-activity.

[0101] Generally speaking, there are three situations that can cause confusion in sensor data in home activity recognition.

[0102] 1) Systematic error is a non-random error caused by the measurement method.

[0103] 2) Accidental errors refer to errors caused by data omissions, recording errors, etc. due to researchers' negligence in data calibration or statistical processes.

[0104] 3) Confused and wrong behavior.

[0105] In our experiments, we selected a public dataset that cannot eliminate the errors in 1) and 2). For the third case, environmental sensors typically monitor a wide range of conditions, making it impossible to directly determine the triggering cause from the sensor sequence. For example, a sensor triggered by a person cooking in the kitchen is typically located near the kitchen. However, if the state values of a bedroom sensor change almost simultaneously, it is likely triggered by a pet or another person. These sensor state changes cannot be triggered by the same person because they do not conform to the gradual temporal and spatial changes of human activity. Therefore, we decided to limit the data to the subject's clutter.

[0106] A distance matrix is a two-dimensional square matrix with extensive applications in mathematics, computer science, and graph theory. The calculation of distance elements in the distance matrix depends on the sensor monitoring range. Based on the layout of environmental sensors in a home environment, a distance matrix (diagonal array) is used to measure the distances between sensors. Given this, a sensor data context-aware distance matrix is constructed.

[0107] Based on their trigger response time, sensors are categorized as real-time sensors and delayed sensors. Real-time sensors directly indicate the location of the trigger. Therefore, distance can be directly calculated based on the physical separation of real-time sensors. In contrast, delayed sensors have a larger detection range and cannot directly determine the location of the triggering object, but they can indirectly reflect changes in the user's daily life. For example, cooking causes the indoor temperature to slowly change. Based on this rule, we established a distance constraint matrix.

[0108] Given a distance matrix D(D∈R n*n ), n represents the number of sensors in the home, where the distance element D(s i , s j ) is calculated by formula (5):

[0109]

[0110] |s i , s j | spatial represents the spatial distance between two instantaneous sensors and is calculated as the minimum number of intermediate sensors when connected, i.e., the Manhattan distance. Otherwise, to take into account the lag in the state change of the delayed sensor, the corresponding distance is recorded as 1. In other words, all data with high significant contribution detected by the delayed sensor are considered non-noise data.

[0111] 3. Wide-temporal convolutional neural network

[0112] Typical deep learning model architectures include CNNs and RNNs. CNNs are powerful for feature extraction. They support parallel computing and parameter sharing with a small number of parameters. RNN-based architectures are particularly adept at processing time series data, especially LSTMs, whose variants have achieved excellent results in activity recognition using home environment sensor data. Thanks to the RNN's memory function, it can remember events that occurred long ago. Furthermore, its unique internal three-gate structure effectively avoids overfitting. However, LSTMs have drawbacks that make them unsuitable for ubiquitous computing. The reasons are: 1) they cannot be computed in parallel because all subsequent states depend on previous results; and 2) their complex structure and high computational complexity.

[0113] The present invention uses a one-dimensional CNN-based network structure for activity recognition. The process of monitoring the human body state based on sensors will produce a strong one-dimensional, highly time-correlated sequence. Complex human activities are composed of a series of partial basic actions or movements. The convolution kernel in CNN can effectively capture local features and has local invariance. Using multiple different convolution kernels can capture different basic actions or movements. Generally speaking, the basic actions or movements in an activity have a certain duration. Reducing the size of the convolution kernel makes it difficult to capture the correlation of long-term features. Increasing the depth of the network can easily lead to gradient disappearance. Therefore, the present invention selects a larger convolution kernel size and a shallower network structure. Then, the captured basic features are stitched together. The present invention uses two fully connected layers for feature fusion. Finally, the activity is classified by Softmax. Therefore, the present invention proposes a wide temporal convolutional neural network (WCNN).

[0114] The input of the convolutional neural network is a series of a={s1, s2, ..., s n During model training, the label value a of the activity sequence is visible. During the recognition process, based on the trained model, a series of sensor state change values are input to predict the label of sequence a, achieving the effect of activity classification.

[0115] 1) Embedding layer

[0116] Encoding input data involves mapping data from one space to another. Two common methods are 1) hot encoding and 2) embedding. Hot encoding is simple and easy to implement. It directly selects appropriate dimensions based on the size of the sample's feature space, ensuring that the dimensions are orthogonal to each other. This is a sparse encoding method. In addition to correctly representing data, embedding can also 1) reduce dimensionality and 2) detect semantic similarities. It is a dense encoding method that has become increasingly popular in recent years. Whether using deep learning classification methods such as RNNs or CNNs, embedding facilitates feature semantic extraction.

[0117] x=W E *{s1, s2, ..., s n} (6) As shown in formula (6), the input of the embedding layer is the representation of the sensor itself. Then, a learnable embedding matrix W is initialized E ∈R V×U , where U is the embedding size and V is the number of sensors deployed in the home environment. After training, the present invention obtains a word embedding model. Similar to a dictionary query, the product of the sensor encoding and the embedding matrix produces the sensor vector x.

[0118] 2) Feature extraction layer of wide temporal convolutional network

[0119] Convolution kernels of different sizes extract features of different time domain lengths. The size of the convolution kernel determines the receptive field of the feature extractor. The larger the convolution kernel, the richer the context information involved, but the more difficult it is to extract detailed information. Therefore, in the network structure design of the present invention, convolution kernels of different sizes are used to capture basic actions or movements of different time lengths. Among them, small-sized convolution kernels are used to extract short-term and detailed features, and large-sized convolution kernels are used to extract long-term and context information fusion features. Convolution feature extraction is shown in formula (7).

[0120] f(x h )=f(x*w h +b h ) (7) Where w is a one-dimensional convolution kernel, the size of the convolution kernel is h, and b∈R. Stride is set to 1 to extract accurate information. Then, the features extracted by the convolution layer are spliced and fused to obtain a feature map containing multiple features. The connected feature layer contains a large number of parameters, and the features are downsampled by the maximum pooling method to obtain a low-dimensional vector with significant features, as shown in formula (8).

[0121] f(x c )=BN{max[f(x1),...,f(x h )]} (8) Different activities have different attention mechanisms for different types of feature vectors. Therefore, a fully connected layer is added to manage the weight of each feature, which facilitates the injection of contextual information. The concatenated features are uniformly linearized and combined. Linear features have limitations in expression, so a ReLU layer is added to perform highly nonlinear transformations of the data. The operations of the fully connected layer and the activity layer are expressed as FA(x) = f(ReLU{w·x+b}), where w and b are parameters. The fully connected layer will contain a large number of parameters, which can easily lead to overfitting of the data. In the process of nonlinear feature representation, in order to avoid overfitting, a neuron probabilistic learning strategy is adopted. As shown in formula (9).

[0122] f(x o )=FA(Drop{FA(x c ), p}) (9)

[0123] where p is the activation rate of neurons in the fully connected layer during data fitting.

[0124] 3) Classification layer

[0125] The feature classification layer consists of a fully connected layer and a softmax function. Depending on the number of activity categories, the fitted data of the fully connected layer fits each category of activity recognition for the subsequent probability evaluation of the softmax function, as shown in Equation (10).

[0126]

[0127] Where n is the number of activity categories.

[0128] The network consists of an embedding layer, four one-dimensional convolutional layers, three fully connected layers, and a softmax classifier. It can be simply described as E(64)-BN-[C(1-32)-C(3-32)-C(5-16)-C(7-8)]–MAX–BN-D(32)-ReLU-Drop(0.4)-D(32)-ReLU-D(c)–S m , where E(n) represents an embedding operation that encodes the data into n-length. BN stands for batch normalization. C(nm) represents a one-dimensional convolution kernel with kernel size n and number of kernels m. MAX represents a one-dimensional max pooling. D(n) represents a fully connected layer consisting of n neurons. c represents the number of activity categories to be confirmed. S m This represents a softmax classifier. The convolution kernels are initialized with he_uniform with random seeds of 1, 2, 3, and 4, and the activation function is the ReLU function. In the second fully connected layer, the dropout value is 0.4.

[0129] Finally, the neural network was trained using the AdaGrad optimizer to minimize the cross-entropy loss function. The AdaGrad optimizer dynamically adjusts the learning rate to smoothly update parameters. Other optimizers were also tested during the experiment, which also had a greater impact on the results. For example, the SGD optimizer converged quickly, but the final classification results were inferior to those of AdaGrad. The Adam family of optimizers also produced unstable classification results, often producing surprising results. Therefore, the AdaGrad optimizer was ultimately chosen.

[0130] 4. Experiment

[0131] 4.1 Dataset

[0132] The CASAS dataset selected for this experiment was collected by the University of Washington. Environmental sensors were placed in the homes of real residents with diverse lifestyles. Therefore, it reflects real-life daily activity data. The number of family members and household layouts vary. Family members include:

[0133] 1) Only one person in the family keeps a pet at the same time;

[0134] 2) There are multiple permanent residents in the family;

[0135] 3) Some families occasionally have relatives and friends visit.

[0136] Furthermore, the amount of data collected varies from household to household. The Milan household dataset contains 11 days of missing data. The Kyoto7 household dataset excludes weekends. The Kyoto8 household dataset contains two days of missing data. The Kyoto11 household dataset contains three weeks of missing data. Different households engage in different activities. The families from whom data was collected have diverse backgrounds, so their living habits, hobbies, and schedules are not entirely consistent. Therefore, a classification model was trained separately for each household's data. Table 1 shows the dataset name, number of household members, number of days monitored, number of sensor data entries, whether visitors were included, and number of activity types.

[0137] Table 1 Dataset information

[0138]

[0139] Environmental sensors in the home are also distributed across different types. Based on the length of their trigger response time, they can be categorized into: 1) immediate-response sensors, where changes in sensor state are inherently short-term and the response is very sensitive; and 2) delayed-feedback sensors, where changes in sensor state depend on the broader environment, such as temperature and humidity. Table 2 shows sensor types, their descriptions, real-time performance, and status values.

[0140] Table 2 Sensor status statistics

[0141]

[0142]

[0143] 4.2 Comparison with other algorithms

[0144] This paper compares the recognition method with a baseline method. Table 3 summarizes the overall impact of different algorithms on activity recognition after applying data constraints. First, the recognition accuracy of NB, HMM, and CRF is relatively low. This indicates that traditional features are inefficient when representing complex activities. Second, the performance of algorithms based on deep neural networks varies significantly across different datasets. The LSTM algorithm has excellent feature capture capabilities for time series features, and the Inception-time algorithm also designs a wide temporal filter. Both algorithms demonstrate good performance. Unfortunately, due to the small encoding feature space of home environment sensors, complex deep neural networks, such as the TTN algorithm, are prone to overfitting and vanishing gradient problems. By adding an embedding module to the beginning of the neural network, deep encoding of sensor data, such as the uni_LSTM algorithm, further facilitates neural network feature extraction. Finally, the WCNN outperforms the LSTM series, which currently has excellent results in human action recognition. This means that the WCNN can also capture correlations between time periods and obtain correlations between features through its hierarchical structure. HAR-WCNN outperforms all baselines in all datasets. The results show that our proposed WCNN algorithm has excellent activity recognition capabilities, and the denoising method based on CSA and SDM is beneficial to end-to-end neural network feature extraction.

[0145] Table 3 Active recognition accuracy of NB, HMM, CRF, LSTM, WCNN, and HAR-WCNN under different data sets (%)

[0146]

[0147] In summary, from the perspective of classification performance, the present invention decomposes multi-user cross-activity into single-user activities through data preprocessing technology. CSA measures the contribution of different types of sensors and provides quantitative indicators for sensor layout schemes, which helps to save layout costs. The anonymity of environmental sensors makes the monitoring data very disturbing and difficult to distinguish. SDM constrains noisy data through contextual distance information, while reducing activity recognition data based on the temporal and spatial gradient characteristics of human activities. Therefore, the data characteristics are further emphasized. The HAR-WCNN algorithm can make up for the lack of accuracy of cheap sensors in identifying small group activities to a certain extent. However, when the number of groups is large, the relationship between MA users is much more complicated. When there are multiple topics in the context and the relationship between them increases, the complexity level also increases accordingly. Combined with the dynamic programming algorithm, the flexible setting of SDM can improve the recognition performance of MA.

[0148] From a classification perspective, experimental results show that WCNN demonstrates the best recognition performance in a comprehensive evaluation. Human activity data monitored by sensors is a time series data type. The LSTM method's structural design enables it to process time series data. However, due to its LSTM structure, the computational process is slow. By designing wide temporal convolution kernels to capture long-term features and processing data in layers and depth, WCNN also has the ability to capture time-dependent features. Results show that the WCNN method, with its four-layer lightweight network design, is shallower than image processing networks, capable of capturing the characteristics of sensor data while using fewer resources.

[0149] From the perspective of identifying single-user behavior in multi-user smart home scenarios, multi-user activity recognition is also of great interest. With the development of various sensor categories, the state information available in smart homes has become more comprehensive. Due to the more complex context and semantic information, the hierarchical structure of multi-user activities is difficult to determine and identify. These issues can be alleviated to some extent by integrating more types of sensing and monitoring devices, especially combining environmental sensors with visual sensors. However, dataset quality is limited to some extent by data exchange and synchronization issues, as multiple devices must communicate and merge. However, few studies have focused on scenarios where multiple people perform individual or joint actions and activities to achieve a given common goal, and relevant datasets are also scarce. Therefore, future work in this area needs to be further expanded. Furthermore, it is necessary to develop an efficient, real-time, sensor-based HAR system to identify various emergency activities in various real-life scenarios and provide the necessary real-world remedial measures in real time.

[0150] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A method for human activity recognition driven by environmental sensor data, characterized by: The method comprises the following steps: Step 1: Collect environmental sensor data: Collect environmental sensor data, perform sequence synchronization and alignment on the environmental sensor data, and build a data set; Step 2: Feature selection: We use the sensor data contribution significance analysis (CSA) method based on sensor state frequency to calculate the contribution of sensor categories. Based on this, we select the sensor category that is most conducive to identifying the behavior, obtain feature data, and embed it into the state sequence of the object to be identified. Step 3: Data noise constraint: To reduce noise in the activity recognition process, a sensor constraint distance matrix D is constructed based on the sensor layout diagram. This excludes irrelevant data in the state sequence of the object to be identified, forms a new feature dataset, and encodes the sensor data state sequence. Step 4: Activity Recognition: Based on the environmental sensor data, a wide-time-domain convolutional neural network recognition model is constructed. With the goal of minimizing the cross-entropy loss function, the recognition model is trained using the AdaGrad optimizer to perform human activity recognition. Wherein: the sensor data contribution significance analysis method CSA based on sensor state frequency in step 2 is: Assume that the activity type of the behavior to be identified is A={A1,A2,…,A i …,A t }, where A i ={a i1 ,a i2 ,…a ij ,…,a in } represents the i-th activity, a ij ={r i j1 ,r i j2 ,…,r i jk } represents the jth time series record of the i-th activity, r i jk represents the kth state monitoring value of the sensor data window triggered during the behavior; The arranged sensor group is represented as S={s1,s2,…,s i ,…,s m }, where m represents the number of sensor types, s i represents the i-th sensor; Based on the entropy uncertainty theory, the information gain of different sensor types when measuring an activity occurs is calculated using the following formula: i The trigger frequency of the jth sensor when it occurs in Indicates activity A i The total triggering time of the jth sensor when it occurs; In order to determine the contribution of various sensors in behavior recognition, the following formula is used to calculate the contribution of the jth sensor to activity A i The amount of self-information brought by the occurrence of Then, the universality of various sensors in the behavior process is calculated: in the entire activity recognition sensor state dataset, if a certain type of activity usually triggers a specific type of sensor, it means that the sensor is associated with the activity; otherwise, the sensor category that is triggered intermittently can be regarded as noise sensor data; the information of the sensors is balanced and the activity A is calculated using the following formula i The generality of the jth sensor when it occurs: Among them, |A i |Indicates activity A i The total number of occurrences of represents the total number of triggers in the activity occurrence covering the jth sensor; Calculate the information contribution of various sensors to behavior recognition: By integrating the self-information and universality of the sensor, the following formula is used to calculate the contribution D of the jth sensor to the i-th activity ij : Finally, based on the measurement of sensor information contribution, the sensor category that is most conducive to identifying behavior is selected.

2. The method for human activity recognition driven by environmental sensor data according to claim 1, wherein: The status monitoring value includes trigger time, sensor number, and status value.

3. The method for human activity recognition driven by environmental sensor data according to claim 1, wherein: In step 3, a sensor constraint distance matrix D is constructed based on the sensor layout diagram, wherein the distance element D(s i , s j ) is calculated using the following formula: |s i , s j | spatial Represents the spatial distance between two instantaneous sensors, and is calculated by the minimum number of intermediate sensors when connected, that is, the Manhattan distance; otherwise, considering the lag of the delayed sensor state change, the corresponding distance is recorded as 1, that is, all data with high significant contribution monitored by the delayed sensor are regarded as non-noise data.

4. The method for human activity recognition driven by environmental sensor data according to claim 1, wherein: The wide time domain convolutional neural network in step 4 includes an embedding layer, four one-dimensional convolutional layers, three fully connected layers, and a softmax classifier.

5. The method for human activity recognition driven by environmental sensor data according to claim 4, wherein: The embedding layer: the encoding of the input data, using the embedded coding method to map the data from the low-dimensional space to the high-dimensional space. The formula is as follows: x=W E *{s1,s2,...,s i ,...,s n } Among them, s i is the i-th state code value of the sensor data sequence; W E is a randomly initialized learnable embedding matrix W E ∈R V×U , where R is the real domain space, U is the embedding size, and V is the number of environmental sensors deployed; x is the sensor feature vector value.

6. The method for human activity recognition driven by environmental sensor data according to claim 4, wherein: The four one-dimensional convolutional layers and two fully connected layers constitute the feature extraction layer of the wide temporal domain convolutional network: the sensor feature vector value x is input, a small-size convolution kernel is used to extract short-term and detail features, and a large-size convolution kernel is used to extract long-term and context information fusion features. The convolution feature f(x h )The extraction formula is: f(x h )=f(x*w h +b h ) where w h is a one-dimensional convolution kernel, the size of the convolution kernel is h; b h is a random bias value in the real number domain, and the step size is set to 1 to extract accurate information; Then, the features extracted by the convolutional layer are spliced and fused to obtain a feature map f(x c ), where the features are downsampled by the maximum pooling method to obtain low-dimensional vectors with significant features. The formula is as follows: f(x c )=BN{max[f(x1),...,f(x h )]} Among them, BN is standardized normal distribution processing; Then, a fully connected layer is added to manage the weight of each feature, and a ReLU layer is added to perform nonlinear transformation of the features.

7. The method for human activity recognition driven by environmental sensor data according to claim 6, wherein: During the nonlinear transformation of the features, in order to avoid overfitting, a neuron probability learning strategy Drop is adopted.

8. The method for human activity recognition driven by environmental sensor data according to claim 4, wherein: The three fully connected layers and the softmax classifier constitute a feature classification layer: human activities are classified according to the number of activity categories.

9. The method for human activity recognition driven by environmental sensor data according to any one of claims 1 to 8, characterized in that: The environmental sensor data collected in step 1 include motion sensors, door sensors, temperature sensors, power consumption sensors, burner sensors, object sensors, hot water sensors, and cold water sensors.

Images