Human activity recognition method, system, computer device and storage medium

Abstract

The present invention provides a method, system, computer device and storage medium for identifying human activities, wherein the identification method comprises the following steps: obtaining original human activity data; preprocessing the original human activity data to obtain first data; downsampling the time dimension of the first data to obtain a downsampled sequence; refining the downsampled sequence to obtain second data; inputting the second data into an LSTM network for data processing to obtain a first feature map; performing full connection layer processing on the time dimension of the first feature map to obtain a second feature map; multiplying the second feature map with the first feature map to obtain a weighted fusion feature map; adding the weighted fusion feature map with the first feature map to obtain a time fusion feature map; classifying the time fusion feature map using a full connection layer to obtain a classification result of the original human activity data.

Description

Technical Field

[0001] The present invention relates to the field of human activity recognition using artificial intelligence and deep learning technology, and in particular to a human activity recognition method, system, computer equipment and storage medium. Background Art

[0002] With the rapid development of deep learning technology and the widespread use of sensor devices, sensor-based human activity recognition technology (HAR) has received increasing attention. Human activity recognition (HAR) is a broad research field that involves identifying specific movements or actions of people based on sensor data. These movements usually refer to specific activities performed indoors, such as walking, talking, standing, and sitting. They may also be more specific activities, such as certain types of activities performed in the kitchen or on the factory floor.

[0003] Sensor data can be recorded remotely, such as by video, radar or other wireless methods. Or the data can be recorded directly on a portable device, such as by carrying custom hardware or a smartphone with an accelerometer and gyroscope. Compared with sensor-based activity recognition technology and image-based activity recognition technology, sensor-based human activity recognition technology has the following advantages: flexibility, convenience, better privacy, and lower cost. By carrying these sensors on different parts of the body, people can effectively avoid relying on other external devices when collecting data, thereby reducing the cost of data collection and protecting the privacy of users. This convenient way of data collection makes the application scope of sensor-based activity recognition infinitely expanded to close to any location, so the application prospects of sensor-based human activity recognition technology are very broad. At present, sensor-based human activity recognition technology has been widely used in many fields.

[0004] In the field of medical monitoring, by analyzing the data from the sensors worn by the elderly, it is possible to effectively infer whether there are safety hazards in the elderly's current daily activities, and to achieve early detection and early treatment of certain hidden diseases. In the field of sports, coaches can use sensors worn on athletes' clothes to understand the athletes' movement status in real time, analyze the athletes' movements, and facilitate coaches to guide the athletes' movements, thereby improving the athletes' training efficiency. In the field of smart homes, sensors can be conveniently arranged in various locations in the room, and can effectively collect the homeowner's daily activity data, and then use the smart home system to analyze the homeowner's daily life behavior, which can bring a more convenient life experience to the homeowner and make the whole room full of technology.

[0005] In early studies, many classic algorithms in traditional pattern recognition problems have been successfully applied to HAR. For example, researchers currently use multi-layer perceptrons for human activity recognition; some researchers apply hidden Markov models to HAR and explore the impact of sensor position on experimental results; some researchers systematically apply least squares, K nearest neighbor, Bayesian and other machine learning algorithms to the HAR field, and conduct detailed comparative experiments in terms of recognition effect and computational complexity. Traditional machine learning methods usually require manual feature extraction, which often requires specific domain knowledge, so it is difficult to select suitable features for specific tasks, which is very unfavorable to improve the generalization performance of the model. With the development of deep learning technology, more and more researchers have begun to apply deep learning related methods to the field of sensor-based human activity recognition.

[0006] At present, great progress has been made in sensor-based human activity recognition using deep learning methods, and research on sensor-based human activity recognition has become increasingly mature, but the following major problems still exist.

[0007] (1) Lack of research on sensor data segmentation

[0008] In the current research on human activity recognition based on deep learning, most of the research focuses on the design of the model structure, and there is little research on the characteristics of sensor data itself. In addition to the inherent timing characteristics of sensor data itself, data segmentation settings cannot be ignored.

[0009] (2) There is a lack of research on network models that can extract bidirectional features

[0010] Compared with the one-dimensional convolution kernel that can only extract the time dependency of one-dimensional time series, the two-dimensional convolution kernel can not only extract the time dependency of time series, but also can well extract the spatial dependency between different sensors. However, in most studies, most researchers simply consider the convolution form of sensor data, and therefore simply splice the sensor data into a two-dimensional matrix input form. Although this simple input form of arranging sensor data together according to channels to form a two-dimensional matrix has achieved better recognition results than traditional feature extraction based on machine learning and feature extraction based on one-dimensional convolution kernels, it does not make good use of the characteristics of features with different weights of contextual information features. Summary of the invention

[0011] In order to solve the problems existing in the background technology, in a first aspect, the present invention provides a method for identifying human activities, which comprises the following steps: obtaining original human activity data; segmenting the original human activity data to obtain first data; downsampling the time dimension of the first data to obtain a downsampled sequence; refining the downsampled sequence to obtain second data; inputting the second data into an LSTM network for data processing to obtain a first feature map; performing a fully connected layer processing on the time dimension of the first feature map to obtain a second feature map; multiplying the second feature map with the first feature map to obtain a weighted fusion feature map; adding the weighted fusion feature map with the first feature map to obtain a time fusion feature map; classifying the time fusion feature map using a fully connected layer to obtain a classification result of the original human activity data.

[0012] Furthermore, downsampling the time dimension of the first data to obtain a downsampled sequence includes the steps of: performing convolution processing on the first data to obtain a third feature map; performing normalization and activation function processing on the third feature map in sequence to obtain third data; and performing self-attention calculation processing on the third data to obtain the downsampled sequence.

[0013] Furthermore, the convolution kernel size in the convolution processing of the first data is (5,1) and the step size is 1.

[0014] Furthermore, the time dimension of the first data is downsampled four times to obtain a downsampled sequence.

[0015] Furthermore, the original activity data is preprocessed by sliding window processing, the window size of the sliding window processing is 24, and the step size is 12.

[0016] Furthermore, when the LSTM network performs data processing, the LSTM network has 3 layers.

[0017] In a second aspect, the present invention also provides a human activity recognition system, which is used to implement the above-mentioned human activity recognition method, and the human activity recognition system includes: a preprocessing module, which is used to obtain original human activity data and segment the original human activity data to obtain first data; a first data processing module, which is used to downsample the time dimension of the first data to obtain a downsampled sequence; a second data processing module, which is used to refine the downsampled sequence to obtain second data; a third data processing module, which is used to input the second data into an LSTM network for data processing to obtain a first feature map; a fourth data processing module, which is used to perform a fully connected layer processing on the time dimension of the first feature map to obtain a second feature map; a fifth data processing module, which is used to multiply the second feature map with the first feature map to obtain a weighted fusion feature map; a sixth data processing module, which is used to add the weighted fusion feature map and the first feature map to obtain a time fusion feature map; a classification module, which is used to classify the time fusion feature map using a fully connected layer to obtain a classification result of the original human activity data.

[0018] Furthermore, the first data processing module includes: a convolution processing unit, used to perform convolution processing on the first data to obtain a third feature map; a normalization processing unit, used to perform normalization processing on the data in the third feature map; activation function processing, used to perform activation function processing on the data after the normalization processing of the third feature map to obtain third data; self-attention calculation processing, used to perform self-attention calculation processing on the third data to obtain a down-sampling sequence.

[0019] In a third aspect, the present invention also provides a computer device comprising a memory, a processor, and a human activity recognition program with bidirectional feature extraction stored in the memory and executable on the processor, wherein the human activity recognition program with bidirectional feature extraction implements the steps of the above-mentioned human activity recognition method when executed by the processor.

[0020] In a fourth aspect, the present invention further provides a readable storage medium, on which is stored a human activity recognition program with bidirectional feature extraction, which implements the steps of the above-mentioned human activity recognition method when executed by a processor.

[0021] The beneficial effects of the present invention are as follows:

[0022] In the human activity recognition method of the present application, based on the segmentation and downsampling processing of the original activity data, the length of the original data is greatly shortened, which helps to improve the subsequent processing efficiency of the human activity data; at the same time, in the human activity recognition process, model training is performed based on the different weights of the features in the weighted fusion feature map and the time fusion feature map obtained by the first feature map and the second feature map, which can enhance the features that are useful for classification and weaken the features that are not useful for classification, thereby improving the accuracy of the classification results, thereby improving the accuracy of human activity recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flow chart of the method for identifying human activities of the present invention;

[0024] Figure 2 It is a structural schematic diagram of the human activity recognition system of the present invention;

[0025] Figure 3 is a structural schematic diagram of a first data processing module of a human activity recognition system of the present invention;

[0026] Figure 4 It is a schematic diagram of a self-attention structure in a method for recognizing human activities and a system of the present invention;

[0027] Figure 5 It is a diagram of the working principle of self-attention in the method for recognizing human activities and the system of the present invention;

[0028] Figure 6 It is a diagram of the working principle of LSTM in the human activity recognition method and system of the present invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used in the specification of the application are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0031] The human activity recognition method, system, computer device and storage medium of the present application are described in detail below with reference to the accompanying drawings.

[0032] Reference Figure 1 , the human activity recognition method of the present application includes steps S1-S9. Specifically, S1, obtain original activity data; S2, segment the original activity data to obtain first data; S3, downsample the time dimension of the first data to obtain a downsampled sequence; S4, refine the downsampled sequence to obtain second data; S5, input the second data into the LSTM network for data processing to obtain a first feature map; S6, perform full connection layer processing on the time dimension of the first feature map to obtain a second feature map; S7, multiply the second feature map with the first feature map to obtain a weighted fusion feature map; S8, add the weighted fusion feature map and the first feature map to obtain a time fusion feature map; S9, classify the time fusion feature map using a full connection layer to obtain the classification result of the original human activity data.

[0033] In the human activity recognition method of the present application, based on the segmentation and downsampling processing of the original activity data, the length of the original data is greatly shortened, which helps to improve the subsequent processing efficiency of the human activity data; at the same time, in the human activity recognition process, model training is performed based on the different weights of the features in the weighted fusion feature map and the time fusion feature map obtained by the first feature map and the second feature map, which can enhance the features that are useful for classification and weaken the features that are not useful for classification, thereby improving the accuracy of the classification results, thereby improving the accuracy of human activity recognition.

[0034] Since the length of the original activity data collected by the sensor is often tens of thousands and contains multiple activities, these activity data cannot be directly input into the network. Therefore, in step S1, the sliding window method can be used to segment the original activity data, wherein the activity category corresponding to the last moment of the window is used as the data label of the segmented sample, so that the original activity data collected by the sensor can be converted into the first data (i.e., sample data) that can be input into the network. Among them, when using a sliding window to segment data, there are two very important parameters: the window size of the sliding window and the step size of each sliding of the window. Specifically, the sliding window size used in the present invention is 24, and the step size of the window sliding is 12.

[0035] It should be noted that the input data format of the sliding window processing is: batch*1*window length*feature dimension. Taking the Skoda dataset as an example, the sliding window size is 24, and the dimension of the dataset itself is 30. Since the data of 30 sensors are selected for training, the input data format of the sliding window processing is batch_size*1*24*30. The first data obtained after the sliding processing is the data input in the downsampling step.

[0036] In step S2, downsampling is performed on the time dimension of the first data to obtain a downsampling sequence. Figure 3 The downsampling process specifically includes the steps of: S21, performing convolution processing on the first data to obtain a third feature map; S22, performing normalization and activation function processing on the third feature map in sequence to obtain third data; S23, performing self-attention calculation processing on the third data to obtain the downsampling sequence.

[0037] In other words, the so-called downsampling processing is the process of performing convolution processing, normalization processing, activation function processing and self-attention calculation processing on the time dimension of the first data in sequence. In this way, the length of the time series subsequently input into the LSTM network is reduced through downsampling processing.

[0038] Preferably, the convolution kernel size can be (5, 1), and the step size is 1. Then in step S2, 4 downsampling processes are performed to obtain a downsampling sequence. For example, the input structure of the data before the downsampling process is batch_size*1*24*30, and the output structure of the data after the first downsampling process is batch_size*1*20*30, the output structure of the data after the second downsampling process is batch_size*1*16*30, the output structure of the data after the third downsampling process is batch_size*1*12*30, and the output structure of the data after the fourth downsampling process is batch_size*1*8*30. Therefore, the time length of the output data set after each downsampling process is reduced by 4.

[0039] In step S21, the Cov2d layer (two-dimensional convolution function) is used to perform convolution processing on the first data, such as the convolution kernel size can be (5, 1) and the step size is 1. During the convolution operation, each convolution kernel can be regarded as a feature that needs to be extracted. The convolution kernel is continuously slid on the input data (i.e., the first data) for operation, so that the features of different parts of the input data can be easily obtained. When the characteristic value obtained after the input data is convolutionally operated is large, it means that the input data can better match the features represented by the convolution kernel. Among them, compared with the naive multi-layer perceptron, the convolution layer can well extract the local features of the input data. In addition, the parameter sharing mechanism of the convolution layer can effectively reduce the amount of parameters in the network structure, thereby facilitating the acceleration of model training.

[0040] Specifically, the first data is recorded as X, and the data output after the convolution processing of the Cov2d layer is recorded as Z (i.e., the third feature map). Then the element of the nth channel of Z is The calculation formula is shown in formula (1):

[0041]

[0042] Among them, C in is the number of channels of X, K is the two-dimensional convolution kernel, K h and K w Respectively represent the width and height of the convolution kernel; b s is the bias term. Specifically, K can be regarded as a two-dimensional matrix, then It represents the value of the uth row and vth column in the matrix K. The matrix K marked by s is the nth matrix; X can be regarded as a three-dimensional matrix containing C in A two-dimensional matrix (that is, the number of channels of X is C in ), k marks the number of two-dimensional matrices in X.

[0043] In step S22, the third feature map is normalized using a Batchnormal layer (also referred to as a batch normalization module) and activated using a Relu activation function layer, thereby obtaining third data.

[0044] Specifically, when the Batchnormal layer is used for normalization, it is normalized by subtracting the mean and unitizing the variance. After this processing, the data set can be mapped to the vicinity of the origin, and the problem of gradient disappearance can be solved. Therefore, the data training speed can be improved.

[0045] The batchnormal layer uses the min-max normalization idea to normalize the third feature map (a data set). The min-max normalization idea is simple. After the data set Z (that is, the third feature map) is normalized, the data set Z' is obtained. The data samples in the data set Z' are but The calculation formula is shown in formula (2):

[0046]

[0047] in, is the maximum value of all data in the third feature map, is the minimum value among all the data.

[0048] The Relu activation function layer is used for activation function processing because the Relu activation function in the Relu activation function layer is a simple nonlinear activation function, which is more efficient in operation. The main functions are reflected in two aspects: one is that it can overcome the problem of gradient disappearance, and the other is that it can speed up the training speed. The data processed by the activation function is the third data. Specifically, the calculation formula of the Relu activation function layer is shown in the following formula (3) and formula (4):

[0049] R(x)=max(x,0) (3)

[0050]

[0051] Among them, x represents the input data and is the data obtained after the Batchnormal layer performs normalization processing (i.e. ). It can be seen from formula (4) that when the input x is greater than 0, Relu is equivalent to a linear function and its derivative is 1; when x is less than or equal to 0, Relu is an identity function and its derivative is 0.

[0052] In step S23, the self-attention module layer is used to perform self-attention calculation processing on the third data, and the result obtained is the down-sampling sequence. Figure 3-4 As shown in the figure, the main workflow of the self-attention calculation process (i.e., the self-attention module layer) is as follows:

[0053] In the first stage, the vectors Query, Key and Value are obtained according to the third data, and then the key value Key in the vector Query and the vector Key is calculated. i, calculate the similarity or correlation between the two. Among them, the most common methods for calculating the similarity or correlation between the two include: calculating the dot product of the two vectors (Formula 5), ​​calculating the cosline similarity of the two vectors (Formula 6), or evaluating by introducing an additional neural network (Formula 7):

[0054] Dot Product Similarity(Query,Key i )=Query·Key i (5);

[0055] Vector Similarity

[0056] Network Similarity(Query,Key i )=MLP(Query,Key i ) (7);

[0057] The score generated in the first stage has a different value range depending on the specific generation method. The present invention adopts the formula score = Query·Key T Calculate the score.

[0058] In the second stage, a similar Softmax operation is introduced to convert the scores of the first stage into numerical values. On the one hand, it can normalize the original operation scores into a probability distribution in which the sum of the weights of all elements is 1; on the other hand, the weights of important elements can be more highlighted through the inherent mechanism of Softmax. The following formula is generally used for operation:

[0059]

[0060] Among them: C is the number of output nodes, that is, the number of classifications. The probability distribution can be obtained through the Softmax function for classification. The calculation result of the second stage is a i That is the Value in the vector Value i The corresponding weight coefficients are then weighted summed to get the Attention value (i.e., the downsampling sequence):

[0061]

[0062] Source represents the relationship between Key and Value. x The length of the input Value, such as Figure 5 As shown in , the length of Value is 4, then L x =4.

[0063] The self-attention mechanism is a variant of the attention mechanism, which reduces the reliance on external information and is better at capturing the internal correlation of data or features. For more details about self-attention, please refer to the article "Attention Is All You Need".

[0064] In step S4, the downsampled sequence is refined to obtain the second data. Specifically, the refinement process is: first input the downsampled sequence into the channel attention layer (refer to "AttentionIsAllYouNeed"), and the structure of the data output by the channel attention layer is still batch_size*8*32*30; then the data output by the channel attention layer is converted into the second data of a size that conforms to the LSTM network input structure through the reshape function (a function that can re-adjust the number of rows, columns, and dimensions of the matrix), and the data structure of the second data is batch_size*8*960.

[0065] In step S5, the second data is input into the LSTM network for data processing to obtain a first feature map.

[0066] The value of each gate in the recurrent unit of the LSTM network is between (0, 1), indicating the proportion of messages that can pass through this gate. The functions of each gate in the network are as follows: 1) Forget gate f t Can control the internal state c at the previous moment t-1 How much information can be forgotten; 2) Input gate i t Control the candidate state c' at the current moment t How much information can be stored; 3) Output gate o t Control the internal state c at the current moment t How much information can be output to the external state h at the current moment? t .

[0067] Reference Figure 6 As shown in Figure 1, LSTM uses the data at the current moment and the previous moment to calculate each gate as follows: (1) First, according to the input data x at the current moment, t (i.e., the second data) and the external state h at the previous moment t-1 , calculate the control strength of the three doors and the candidate state c' of the current state t ; (2) Then, use the forget gate f t and input gate i t , the memory unit c at the current moment t Update; (3) Finally, use the output gate o t To control how much information in the current internal state can be passed to the external state h t .

[0068] Forget Gate t , input gate i t and output gate o t The calculation process is shown in the following formula:

[0069] f t =σ(W f ·[c t-1 ,h t-1 , x t ]+b f ) (10)

[0070] i t =σ(W i ·[c t-1 ,h t-1 , x t ]+b i ) (11)

[0071] o t =σ(W o ·[c t-1 ,h t-1 , x t ]+b o ) (12)

[0072] Among them, W f , W i , W o is the weight matrix (W f is the forget gate matrix, W i is the input gate matrix, W o is the output gate matrix); b f , b i , b o is the bias vector; c t-1 is the memory unit of the previous moment (internal state in the LSTM layer); σ is the sigmoid activation function. The specific implementation of sigmoid is: the element value output by the sigmoid layer is a real number between 0 and 1, and the element value of this output is the weight of the corresponding information passing through; for example, the output element value of 0 means "no information is allowed to pass", and the output element value of 1 means "all information is allowed to pass". Candidate state c' t The calculation method is as follows:

[0073] c' t =tanh(W c ·[c t-1， h t-1 , x t ]+b c ) (13)

[0074] Among them, Wc Each layer is related to the input data x t (Second data) connection weight matrix; b c is the bias vector.

[0075] Internal state c t and the external state h t The update calculation of is as follows:

[0076] c t =f t ⊙c t-1 +i t ⊙c' t (14)

[0077] h t =o t ⊙tanh(c t ) (15)

[0078] Among them, ⊙ is the multiplication method of corresponding elements in the operation matrix.

[0079] In other words, the first feature map is obtained by calculating the above formulas (10)-(15). In step S6, the time dimension of the first feature map is fully connected to obtain the second feature map. The purpose of step S6 is to obtain the weights of the feature maps at different times through the fully connected layer operation.

[0080] The fully connected layer plays the role of a classifier in the entire deep learning neural network. It maps the continuous feature space to the representation space of discrete sample categories by integrating the distributed abstract features obtained from other layers. The core calculation process is the product of matrix vectors. The formula is as follows:

[0081] y=Wz+b (16)

[0082] Among them, z is the input of the fully connected layer (that is, the data in the first feature map); W is the weight coefficient; b is the bias term; y is the data in the second feature map.

[0083] In step S7, the second feature map is multiplied by the first feature map to obtain a weighted fusion feature map.

[0084] In step S8, the weighted fusion feature map and the first feature map are added to obtain a time fusion feature map, that is, a weight sequence whose sum of weights is 1 is obtained.

[0085] In step S9, the time fusion feature map is classified using a fully connected layer to obtain a classification result of the original activity data, thereby achieving recognition and classification of human activities.

[0086] In summary, in the human activity recognition method of the present application, the application of model technologies such as Cov2d+Batchnormal+relu+self-attention in the deep learning neural network greatly shortens the length of the original data; and based on the application of the reshape function, the data input can be converted into data whose size conforms to the LSTM network input structure, which facilitates subsequent calculations; at the same time, based on the application of LSTM network technology and fully connected layer processing technology, the weights of feature maps at different times (i.e., weighted fusion feature maps and time fusion feature maps) can be obtained, thereby enhancing the features that are useful for classification and weakening the features that are not useful for classification, thereby improving the accuracy of the classification results, thereby improving the accuracy of human activity recognition.

[0087] The human activity recognition system of the present application includes a preprocessing module, a first data processing module, a second data processing module, a third data processing module, a fourth data processing module, a fifth data processing module, a sixth data processing module and a classification module.

[0088] The preprocessing module acquires the original activity data and performs segmentation processing on the original activity data (eg, the original activity data may be segmented using a sliding window method) to obtain first data in a data format that meets the requirements of the first data processing module.

[0089] The first data processing module is used to perform downsampling processing on the first data and obtain a downsampling sequence. Specifically, the first data processing module is a CBRSA module. When it is necessary to perform downsampling processing on the first data four times, then correspondingly, the number of the first data processing modules is four.

[0090] like Figure 2-3 As shown in Figure 1, the first data processing module includes a convolution processing unit (i.e., Cov2d layer), a normalization processing unit (i.e., Batchnormal layer), an activation function processing unit (i.e., relu activation function layer), and a self-attention convolution processing unit (i.e., self-attention module layer). The specific implementation and principle of the self-attention module layer can be referred to Figure 4 and 5 .

[0091] The convolution processing unit is used to perform convolution processing on the first data to obtain a third feature map, the normalization processing unit is used to perform normalization processing on the data in the third feature map, the activation function processing unit is used to perform activation function processing on the data after the normalization processing of the third feature map to obtain third data, and the self-attention calculation processing unit is used to perform self-attention calculation processing on the third data to obtain a down-sampling sequence.

[0092] The second data processing module is used to refine the downsampled sequence to obtain second data. Specifically, the second data processing module is a channel attention module, which converts the downsampled sequence into second data of a size that conforms to the input structure of the LSTM network by using a reshape function (a function that can re-adjust the number of rows, columns, and dimensions of a matrix).

[0093] The third data processing module is used to input the second data into the LSTM network for data processing to obtain a first feature map. Specifically, the third data processing module is a calculation model in the LSTM network layer.

[0094] The fourth data processing module is used to perform full connection layer processing on the time dimension of the first feature map to obtain a second feature map. Specifically, the fourth data processing module is a calculation model in the full connection layer.

[0095] The fifth data processing module is used to perform multiplication operation on the second feature map and the first feature map to obtain a weighted fusion feature map; the sixth data processing module is used to perform addition operation on the weighted fusion feature map and the first feature map to obtain a time fusion feature map; the classification module is used to classify the time fusion feature map using a fully connected layer to obtain the classification result of the original human activity data.

[0096] In the human activity recognition system of the present application, based on the segmentation and downsampling processing of the original activity data, the length of the original data is greatly shortened, which helps to improve the subsequent processing efficiency of the human activity data; at the same time, in the human activity recognition process, model training is performed based on the different weights of the features in the weighted fusion feature map and the time fusion feature map obtained by the first feature map and the second feature map, which can enhance the features that are useful for classification and weaken the features that are not useful for classification, thereby improving the accuracy of the classification results, thereby improving the accuracy of human activity recognition.

[0097] In one embodiment, the recognition system of the present invention includes 1 preprocessing module, 4 first data processing modules (i.e., 4 CBRSA modules), 1 second data processing module (i.e., 1 channel attention layer), 3 third data processing modules (i.e., 3 LSTM network layers), 1 fourth data processing module (i.e., 1 fully connected layer) and 1 fifth data processing module, 1 sixth data processing module and 1 classification module.

[0098] The following is a detailed description of the operation process of the recognition system of the present invention using the Skoda dataset as an example: The Skoda dataset contains 11 activity categories, has obvious pattern differences, sufficient data volume, and occurs under factory production conditions, with standard actions and easy classification. The specific categories are shown in Table 1 below.

[0099] Table 1 Action categories in the Skoda dataset

[0100]

[0101] The Skoda dataset (original human activity data) is processed in the following order to obtain the classification results. Figure 1 :

[0102] The first layer (preprocessing module): The input structure of the current data is batch*1*window length*feature dimension. The sliding window length is 24, the dimension of the data set itself is 30, and the data of 30 sensors are selected for training. Therefore, the input structure of the first data obtained after processing by the preprocessing module is batch_size*1*24*30.

[0103] The second layer (first data processing module): The first data is sequentially input into 4 CBRSA modules, each CBRSA module includes Cov2d+Batchnormal+Relu+self-attention, that is, the first data with a data structure of batch_size*1*24*30 is firstly input into Cov2d, and then sequentially output through Batchnormal, Relu, and self-attention, and finally a downsampled sequence is obtained. Among them, the data structure of the output data of the first data sequentially input into the 4 CBRSA modules is changed to batch_size*32*20*30, batch_size*32*16*30, batch_size*32*12*30, batch_size*32*8*30 (the data structure of the data in the downsampled sequence is batch_size*32*8*30 at this time). The size of the convolution kernel in the Cov2d layer in each CBRSA module is (5, 1). Therefore, the height dimension of the output data set is reduced by 4 after each CBRSA module, that is, the 24-time series is reduced by 4 after each CBRSA module, while the input and output data structures of batchnormal, relu, and self-Attention are consistent.

[0104] The third layer (the second data processing module): The data in the downsampled sequence obtained in the second layer is first input into the channel attention layer (i.e., Channel Attention Layer), and its output data structure remains unchanged, and the data structure is still batch_size*32*8*30; then the data output by the channel attention layer is converted into the second data whose size conforms to the input structure of the LSTM network through the reshape function. At this time, the data structure of the second data is batch_size*8*960.

[0105] The fourth layer (the third data processing module): a three-layer LSTM network layer is used for data processing, where the data structure of the input data of each LSTM network layer is batch_size*8*960, and the data structure of the output data is batch_size*8*128 (i.e., the first feature map). The implementation method of the LSTM network layer can refer to "Learning to Forget: Continual Prediction with LSTM Technical Report IDSIA-01-99January, 1999".

[0106] The fifth layer (the fourth data processing module): Linear (128, 1), the weight of the data at each time point is calculated through the fully connected layer, that is, the second feature map is obtained.

[0107] The sixth layer (fifth data processing module): Sum (dim = 1), multiplying the weight in the second feature map obtained by the fifth layer by the first feature map output by the fourth layer to obtain a weighted fusion feature map.

[0108] The seventh layer (the sixth data processing module): Linear(128, class_label), the weighted fusion feature map obtained by the sixth layer and the first feature map output by the fourth layer are weighted summed to obtain the time fusion feature map.

[0109] The eighth layer (classification module): the fully connected layer is used for classification, that is, the data in the time fusion feature map is input into the softmax layer. The softmax layer selects the position corresponding to the largest data as the category of the human activity data according to the value of the data in the input data. The specific number of output categories is 11 in this case, that is, the output is a vector with 11 data. For specific implementation methods, please refer to "Convolutional Networks and Applications in Vision, Yann LeCun, Koray Kavukcuoglu and Cl′ementFarabet".

[0110] The recognition effect of the human activity recognition system of the present invention is verified by experiments on the Skoda dataset. Taking the Skoda dataset as an example, the human activity recognition method and recognition system proposed in the present invention and the existing classic method and system DeepConvLSTM (deep learning model, see "Convolutional and LSTM Recurrent Section 3.1. DeepConvLSTM") use the same data set division method. Compared with them, the human activity recognition method and recognition system proposed in the present invention can improve the accuracy of human activity recognition. It is verified by experiments that the recognition accuracy can be improved by at least 1%. For specific results, see the experimental results table in Table 2 below.

[0111] Table 2 Experimental results

[0112]

[0113]

[0114] The technical solution of the present invention adopts the self-attention mechanism in the field of natural language, and introduces the network model structure constructed by the self-attention mechanism to perform human activity recognition. The main idea of ​​the self-attention mechanism is to allow the most important part of the model input to be trained according to different weights based on the characteristics of the current context. The self-attention model eliminates the difficulty of RNN (Recurrent Neural Network) learning from long input sequences, and the problem of only attention mechanism based on NLP (natural language processing) model caused by continuous development. In deep learning, the self-attention model is used to extract time-dependent architecture data, which is converted into a 4D tensor input to the network. According to the different weights of the feature map of the network model, the features useful for classification are enhanced, and the features that are useless for classification are weakened. In this way, the accuracy of human activity recognition is improved.

[0115] The recognition device of the present application includes a memory, a processor, and a human activity recognition program with bidirectional feature extraction stored in the memory and executable on the processor, wherein the human activity recognition program with bidirectional feature extraction implements the steps of the human activity recognition method as described above when executed by the processor.

[0116] The readable storage medium of the present application stores a human activity recognition program with bidirectional feature extraction, and when the human activity recognition program with bidirectional feature extraction is executed by a processor, the steps of the above-mentioned human activity recognition method are implemented.

Claims

1. A method for identifying human activities, characterized in that: The steps include: Obtaining raw human activity data; Segmenting the original human activity data to obtain first data; Downsampling the time dimension of the first data to obtain a downsampled sequence; Performing refinement processing on the down-sampling sequence to obtain second data; Inputting the second data into the LSTM network for data processing to obtain a first feature map; Performing a fully connected layer processing on the time dimension of the first feature map to obtain a second feature map; Performing a multiplication operation on the second feature map and the first feature map to obtain a weighted fusion feature map; Performing an addition operation on the weighted fusion feature map and the first feature map to obtain a time fusion feature map; Classifying the time fusion feature map using a fully connected layer to obtain a classification result of the original human activity data; The refinement process is: first input the downsampled sequence into the channel attention layer, and the structure of the data output by the channel attention layer is still batch_size*8*32*30; then convert the data output by the channel attention layer into second data whose size conforms to the LSTM network input structure through the reshape function, and the data structure of the second data is batch_size*8*960.

2. The method for identifying human activities according to claim 1, characterized in that: Downsampling the time dimension of the first data to obtain a downsampled sequence comprises the following steps: Performing convolution processing on the first data to obtain a third feature map; Performing normalization and activation function processing on the third feature map in sequence to obtain third data; Perform self-attention calculation processing on the third data to obtain the down-sampling sequence.

3. The method for identifying human activities according to claim 2, characterized in that: The convolution kernel size in the convolution processing of the first data is (5, 1) and the step size is 1.

4. The method for identifying human activities according to claim 1, characterized in that: The time dimension of the first data is downsampled four times to obtain a downsampled sequence.

5. The method for identifying human activities according to claim 1, characterized in that: The original activity data is preprocessed by sliding window processing, the window size of the sliding window processing is 24, and the step size is 12.

6. The method for identifying human activities according to claim 1, characterized in that: The LSTM network performs data processing, and the LSTM network has 3 layers.

7. A human activity recognition system, characterized in that: A method for realizing the human activity recognition method according to any one of claims 1 to 6, wherein the human activity recognition system comprises: A preprocessing module, used for acquiring original human activity data and performing segmentation processing on the original human activity data to obtain first data; A first data processing module, configured to downsample the time dimension of the first data to obtain a downsampled sequence; A second data processing module, used for performing refinement processing on the down-sampling sequence to obtain second data; A third data processing module, used for inputting the second data into the LSTM network for data processing to obtain a first feature map; a fourth data processing module, configured to perform a fully connected layer processing on the time dimension of the first feature map to obtain a second feature map; a fifth data processing module, configured to perform a multiplication operation on the second feature map and the first feature map to obtain a weighted fusion feature map; a sixth data processing module, configured to perform an addition operation on the weighted fusion feature map and the first feature map to obtain a time fusion feature map; The classification module is used to classify the time fusion feature map using a fully connected layer to obtain a classification result of the original human activity data.

8. The identification system according to claim 7, characterized in that: The first data processing module comprises: A convolution processing unit, configured to perform convolution processing on the first data to obtain a third feature map; A normalization processing unit, used for performing normalization processing on the data in the third feature map; Activation function processing, used for performing activation function processing on the data after the normalization processing of the third feature map to obtain third data; Self-attention calculation processing is used to perform self-attention calculation processing on the third data to obtain a down-sampling sequence.

9. A computer device, characterized in that: The invention comprises a memory, a processor and a human activity recognition program for bidirectional feature extraction stored in the memory and executable on the processor, wherein: When the human activity recognition program based on bidirectional feature extraction is executed by the processor, the steps of the human activity recognition method according to any one of claims 1 to 6 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a human activity recognition program with bidirectional feature extraction, and when the human activity recognition program with bidirectional feature extraction is executed by a processor, the steps of the human activity recognition method according to any one of claims 1 to 6 are implemented.

Images