Human body action recognition method based on single IMU (Inertial Measurement Unit) sensor

Abstract

The invention discloses a human body action recognition method based on a single IMU (Inertial Measurement Unit) sensor. Comprising the following steps: acquiring data by adopting an IMU sensor at a fixed frequency, attaching an action category label, carrying out data noise reduction and normalization processing, delivering the data to a neural network model by utilizing a sliding window, training the neural network model, storing the model and uploading the model to a server, and obtaining a human body action recognition result by a mobile phone App. According to the recognition method, the memory ability of the neural network model to the action data time sequence is improved by using the BiLSTM, and the recognition precision is still very high under the condition that only one IMU sensor exists; meanwhile, an attention mechanism is added, significance features of the action data are extracted, context information of action data time sequences is obtained more comprehensively, the importance degree of local sequence features is learned, the influence of important time sequence features in the model is improved, and therefore the amount of information to be processed is reduced, the calculation load is relieved, and the real-time performance of recognition is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent control and relates to a human motion recognition method, and specifically to a human motion recognition method combining BiLSTM and an attention mechanism based on a single IMU sensor. Background Art

[0002] Human motion recognition technology detects, collects, processes and analyzes human motion data to identify and understand the type, duration, speed, strength and other parameters of human motion. In recent years, with the widespread use of devices such as cameras and sensors, as well as the improvement of computer processing power, human motion recognition technology has made significant progress. Human motion recognition technology has good applications in behavior monitoring, security inspection, game interaction, medical health and motion analysis.

[0003] Human motion recognition technology can generally be divided into two methods based on the type of sensor used: vision-based methods and motion sensor-based methods. An inertial measurement unit (IMU) is a motion sensor that combines an accelerometer, gyroscope, and magnetometer in one sensor. By placing one or more IMU sensors on the human body, sensor signals containing motion information can be collected, and then these signals can be classified into specific actions. Compared with vision-based recognition technology, this method does not have problems with motion detection in complex backgrounds, privacy security, and other issues. At the same time, the data collected by motion sensors can be directly used to identify worker activities. Compared with image data, motion sensor data requires less computing power in data preprocessing, model training, and deployment. However, in daily use, it is neither convenient nor realistic to place multiple sensors on the human body.

[0004] The essence of human motion recognition is the classification of time series. Traditional machine learning is prone to insufficient generalization in this regard, while deep learning has a complex structure and enhanced generalization ability, making it suitable for solving such problems. LSTM, as a representative of these methods, performs well in processing and predicting time series with long intervals. Compared with traditional machine learning methods, it has higher accuracy, but its computational workload also increases exponentially. Therefore, how to reduce the amount of computation and reduce the computational load while ensuring accuracy and ensuring the real-time performance of recognition is a key issue that needs to be solved urgently. Summary of the invention

[0005] In view of the defects of the prior art, the present invention proposes a human motion recognition method based on a single IMU sensor. The purpose of the present invention is to provide a human motion recognition method, which uses a single IMU sensor to obtain information data, combines BiLSTM and attention mechanism, and can ensure the accuracy of human motion recognition, reduce the computational load, and improve the real-time performance of recognition.

[0006] The present invention adopts the following technical solution:

[0007] A human motion recognition method based on a single IMU sensor, the specific steps include:

[0008] Step 1: Use the IMU sensor to collect human motion data and attach motion category labels to the collected data;

[0009] Step 2: preprocess the action data and action labels; the data preprocessing includes filtering, outlier processing, and normalization; the data preprocessing of the action labels is one-hot encoding.

[0010] Step 3: Divide the action data and the corresponding action labels into training samples and test samples according to the ratio of 7:3;

[0011] Step 4: Initialize the parameters in the BiLSTM neural network model, use the sliding window method to send the processed action data and its corresponding action label into the BiLSTM neural network model, and continuously update the parameters of the BiLSTM neural network model until the optimal BiLSTM neural network model is obtained;

[0012] Step 5. Save the optimal BiLSTM neural network model as a pb format file and deploy the file to the server; users can access the server port through the mobile app to obtain the prediction results of the BiLSTM neural network model for the action data.

[0013] The motion data in the method is transmitted from the IMU sensor to the mobile phone via Bluetooth, and then uploaded to the server by the mobile phone via the network.

[0014] In the step 1, the IMU sensor is used to collect the motion data of the human body, and the IMU sensor is placed on the human body to collect the motion data at a fixed frequency; different people need to be collected during the collection process, and each person is required to perform a variety of specified actions; the fixed frequency of the IMU sensor in the step 1 is 50Hz. The collected motion data are the three-axis acceleration, three-axis angular velocity and three-axis angular acceleration of the human body.

[0015] The filtering method in step 2 uses Kalman filtering to remove the noise of the original data; outlier processing refers to removing singular values ​​and default values, and outliers refer to singular values, that is, obviously too large, obviously too small, and default values. Normalization processing uses the Sigmoid activation function, that is, mapping each original value to the interval (0,1).

[0016] The data format fed into the BiLSTM neural network model in step 4 is (N, W, C), where N is the number of data, W is the data graph width, i.e., the sliding window width, and C is the number of data channels. After initializing the parameters of the BiLSTM neural network model, the sliding window width action data is fed into the model in sequence for training, and the various parameters of the BiLSTM neural network model are adjusted to obtain the optimal BiLSTM neural network model.

[0017] The structure of the neural network includes an input layer, a BiLSTM layer, a fully connected_1 layer, an attention layer, a fully connected_2 layer, and an output layer. A dropout operation needs to be performed before the BiLSTM layer, a dropout operation needs to be performed before the fully connected_1 layer, a flatten operation needs to be performed after the attention layer, and a softmax operation needs to be performed after the fully connected_2 layer.

[0018] Furthermore, the BiLSTM layer adds or removes information through a gate structure. Different neural networks can decide which relevant information to remember or forget through the gate structure on the cell state. During the calculation process, the BiLSTM model forgets the information in the cell state and remembers new information so that useful information for subsequent calculations can be transmitted, while useless information is discarded. The hidden state is output at each time step, where forgetting, memory, and output are controlled by the forget gate, memory gate, and output gate calculated by the hidden state of the previous moment and the current input.

[0019] The input raw data is n 9-dimensional vectors collected by the IMU sensor at the same time interval within a certain period of time, that is, [9,n]. Each data is a time step, which constitutes the time series X=[x1,x2...,xn]. Each x is a 9-dimensional vector.

[0020] After being processed by the BiLSTM layer, a corresponding hidden state is generated for each n time step in the input sequence. A hidden state sequence H=[h1,h2...,hn] is obtained. A matrix of [m,n] is obtained, where m is the dimension of the hidden state, which is changed from 9 dimensions to a higher-dimensional state vector. This is determined by the hidden size of the BiLSTM network.

[0021] Furthermore, the specific implementation steps of the attention layer are as follows:

[0022] 1. Calculate the context vector: Read the input data through a Transformer encoder to get a context vector.

[0023] The context vector is used as the query vector q. In the model, the query vector q matches the dimension of the hidden state H. Here, a single query vector is taken as an example, that is, [m, 1].

[0024] 2. Calculate weights: Convert the context vector into attention weights based on the data content.

[0025] The hidden state H[h1,h2...,hn] and the query vector q are used to calculate the relevance score of each hidden state hi and the query vector q using the scoring function f(h,q). The scoring function of this model selects the scaled dot product model, where D is the dimension of the input vector.

[0026]

[0027] 3. Weighted summation: Use the attention weights to perform weighted summation on the input data to obtain the final output result.

[0028] Generate a score vector [n,1] whose length is the number of time steps in the sequence, and each element represents the correlation score between the hidden state of the corresponding time step and the query vector q. Normalize the correlation score vector through the softmax function to obtain a probability distribution [n,1]. This probability distribution represents the correlation score between the hidden state of each time step in the sequence and the query vector q The softmax function ensures that the sum of all weights is 1 and each weight is between 0 and 1.

[0029] The weight coefficient is used to perform a weighted summation of the original hidden state H[h1,h2...,hn] (i.e. [m,n]) and the probability distribution [n,1] to obtain the vector S[m,1] weighted by the attention mechanism. This input weighted m-dimensional hidden state is then input into the fully connected_2 layer, softmax, and output six categories.

[0030] The m hidden states mentioned above are not assigned the same weights but are adjusted based on the query vector. The query vector q is adjusted in each time step based on the internal state of the model and the output results, so as to guide the model to focus on the part of the input sequence that is most relevant to the current task. The training effect is better than that of the unweighted input fully connected layer.

[0031] Compared with the prior art, the method of the present invention has the following outstanding advantages:

[0032] After the original data is preprocessed, the data features of the three-axis acceleration, three-axis angular velocity, and three-axis angular acceleration are unified, and BiLSTM is used to improve the memory ability of the neural network model for the time series of motion data, and the recognition accuracy is still high when there is only one IMU sensor. At the same time, the attention mechanism is added to extract the significant features of the motion data, obtain the context information of the motion data time series more comprehensively, learn the importance of local sequence features, and enhance the influence of important time series features in the model, thereby reducing the amount of information processed, reducing the calculation load, and ensuring the real-time recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 is a flow chart of the method of the present invention;

[0034] Figure 2 This is a neural network structure framework diagram of the present invention;

[0035] Figure 3 This is a schematic diagram of sensor data collection in an embodiment of the present invention;

[0036] Figure 4 is a diagram showing the angle variation curve of the "climbing" action obtained in an embodiment of the present invention;

[0037] Figure 5 It is a comparison of the X-axis acceleration before and after the application of Kalman filtering in an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of a sliding window in data loading according to an embodiment of the present invention;

[0039] Figure 7 is a confusion matrix heat map of the model of the embodiment of the present invention;

[0040] Figure 8 The embodiment of the present invention uses a mobile phone App to obtain and upload data in real time;

[0041] Fig. 9 This is an embodiment of the present invention that obtains human motion prediction information. DETAILED DESCRIPTION

[0042] The present invention is further described below in conjunction with specific implementation methods. The specific implementation methods are further descriptions of the principles of the present invention and do not limit the present invention in any way. Technologies that are the same or similar to the present invention do not exceed the scope of protection of the present invention.

[0043] Figure 1This is the flow chart of the method of the present invention. Human motion data is collected from IMU sensor data. After data preprocessing and model parameter initialization, the human motion data is sent to the BiLSTM neural network model for training. The trained model is deployed to the server side, and the user can obtain the motion recognition result in real time through the mobile phone app.

[0044] Figure 2 This is a neural network structure diagram of the BiLSTM neural network model, including the input layer, BiLSTM layer, fully connected_1 layer, attention layer, fully connected_2 layer, and output layer. A dropout operation is required before the BiLSTM layer, a dropout operation is required before the fully connected_1 layer, a flatten operation is required after the attention layer, and a softmax operation is required after the fully connected_2 layer.

[0045] This embodiment is based on the safety monitoring needs of construction workers on a dam concrete bunker surface, and recognizes the human body movements of construction workers in real time, specifically including:

[0046] Step 1: Select a number of experimenters of different genders, body shapes, and ages, and wear IMU sensors on their waists. Each person performs a specified action for a certain period of time, such as walking, running, climbing, jumping, sitting, lying, standing, etc. Set the IMU sensor to record the three-axis acceleration, three-axis angular velocity, and three-axis angular acceleration data of all people at a frequency of 50Hz, and then attach the corresponding action category labels to these action data;

[0047] To collect the data set, the motion attitude sensor of WIT company, model WT901SDCL, is selected. It has a built-in TDKICM-42670-P inertial measurement unit (IMU), with an accelerometer resolution of 0.0005 (g / LSB), a gyroscope resolution of 0.061 ((° / s) / LSB), and a pitch angle, roll angle, and heading angle resolution of 0.0055 degrees. The sensor data output rate is 0.1HZ~200HZ, the operating temperature is -45℃~85℃, and it is powered by a 200mAh lithium battery, which can run continuously for up to 6 hours. As for the location of the sensor, it should be placed in a position that has little impact on the work of the construction workers and takes into account the upper and lower limbs of the human body. Therefore, this article chooses to use a waist clip to fix the sensor to the waist of the human body.

[0048] Based on the actual construction site conditions, this paper selects three static actions: sitting, standing, and lying, and three moving actions: walking, running, and climbing, from the typical dam casting bin construction scene, a total of 6 actions as target categories.

[0049] For the selection of testers, in order to make the data universal, a total of six people, 3 men and 3 women, were selected to form the testers. Each person wore a MEMS sensor and performed 6 actions in sequence, and the cumulative duration of each action was no less than 10 minutes. Due to the particularity of the "climbing" action, the 3 women did not participate in the collection of this action, and 3 men completed it instead.

[0050] Combined with other action tests, the amount of data obtained in this data collection test is summarized in Table 1. Figure 3 As shown, Figure 4 This is an illustration of the angular velocity of the "climb" action.

[0051] Table 1 below is a summary of the data obtained from the experiment:

[0052]

[0053] Figure 3 It is a schematic diagram of sensor data collection; Figure 4 3 is a graph showing the angle change of the "climbing" action.

[0054] Step 2: Use Kalman filtering to remove noise from the collected three-axis acceleration, three-axis angular velocity, and three-axis angular acceleration data, traverse the entire action data set, delete unreasonable values ​​and default values, and modify the corresponding action category labels. Then use the Sigmoid activation function on the action data to map each original value to the (0,1) interval, and use one-hot encoding to represent the action category labels;

[0055] Taking the X-axis acceleration data of the "walking" action as an example, the Kalman filter algorithm is applied to it, and the visualization data before and after is as follows: Figure 5 As shown, it can be seen that Kalman filtering can effectively reduce the noise in the sensor data, while retaining the data characteristics of the action and reducing the amplitude of the data. Figure 5 Before and after applying Kalman filtering to the X-axis acceleration.

[0056] Step 3: Divide the action data and its corresponding action labels into training set and test set in a ratio of 7:3;

[0057] Step 4: Define and initialize the network parameters in the BiLSTM neural network model. Use a sliding window with a width of 128 and a coverage of 50% to segment the processed action data, such as Figure 6As shown. A three-dimensional tensor is obtained, and its data format is (N, W, C), where N is the number of data, W is the data graph width, that is, the width of the sliding window. In this experiment, the sliding window width is selected as 128, and C is the number of data channels, that is, the three-dimensional data of acceleration, angular velocity, and angular acceleration, which is 9. The above processed data and its corresponding action label are sent as input samples to the BiLSTM neural network model.

[0058] The specific forward calculation process of LSTM is as follows:

[0059] According to the human motion data input by LSTM, t Forget the door for:

[0060]

[0061] in, is the gate activation function (Sigmoid function); is the forget gate weight matrix; is the forget gate data input, yes t The vector input into the network at any moment, the short-term memory state The information of current action data and adjacent action data is saved. is the bias of the forget gate.

[0062] t Time input gate for:

[0063]

[0064] in, is the input gate weight matrix, is the bias term of the input gate.

[0065] t Long-term memory state for:

[0066]

[0067] in, It is a long-term unit state, which stores the relevant information of the action from the beginning to the present; is its weight matrix; To bias it; is element-wise multiplication.

[0068] t Time output gate for:

[0069]

[0070] in, is the output gate weight matrix; Bias it.

[0071] Finally, the LSTM network can be obtained t The output value at this moment is:

[0072]

[0073] BiLSTM fuses the output of 2-layer LSTM through the following formula, and then obtains the recognition result through the Softmax function, namely:

[0074]

[0075] in, is the output of the forward LSTM; is the output of the reverse LSTM.

[0076] Feature extraction is then performed through the attention mechanism to improve computational efficiency.

[0077] The input raw data is 128 9-dimensional vectors collected by the IMU sensor at the same time interval within a certain period of time, that is, [9,128]. Each data is a time step, which constitutes the time series X=[x1,x2...,x128]. Each x is a 9-dimensional vector.

[0078] After being processed by the BiLSTM layer, a corresponding hidden state is generated for each of the 128 time steps in the input sequence. A hidden state sequence H=[h1,h2...,h128] is obtained. A matrix of [64,128] is obtained, where 64 is the dimension of the hidden state, which is converted from 9 dimensions to a higher-dimensional state vector of 64. This is determined by the hidden size of the BiLSTM network, which is set here to 32 LSTM units in the forward direction and 32 LSTM units in the reverse direction, for a total of 64 dimensions.

[0079] The input data is read through a Transformer encoder to obtain a context vector.

[0080] The context vector is used as the query vector q. In the model, the query vector q matches the dimension of the hidden state H. Here, a single query vector is taken as an example, that is, [64,1].

[0081] The hidden state H[h1,h2...,h128] and the query vector q are assigned a scoring function f(h,q) to calculate the relevance score of each hidden state hi and the query vector q. The scoring function of this model selects the scaled dot product model, where D is the dimension of the input vector.

[0082]

[0083] Generate a score vector [128,1] whose length is the number of time steps in the sequence, and each element represents the correlation score between the hidden state of the corresponding time step and the query vector q. Normalize the correlation score vector through the softmax function to obtain a probability distribution [128,1]. This probability distribution represents the correlation score between the hidden state of each time step in the sequence and the query vector q The softmax function ensures that the sum of all weights is 1 and each weight is between 0 and 1.

[0084] The weight coefficient is used to perform weighted summation on the original hidden state H[h1,h2...,h128] (64,128) and the probability distribution [128,1] to obtain the vector S[64,1] weighted by the attention mechanism. The weighted 64-dimensional hidden state is then input into the fully connected 2 layers, softmax, and output six categories.

[0085] Continuously update the BiLSTM neural network parameters until the optimal BiLSTM neural network model is obtained (when the loss value is stable);

[0086] The training results are as follows: The confusion matrix of the trained model is shown in Table 2 below.

[0087]

[0088] The heat map of the confusion matrix is ​​as follows Figure 7 shown.

[0089] The model has an accuracy of 91.89%, a precision of 91.96%, a recall of 91.89%, and an F-1 value of 91.92% on the test set.

[0090] In order to verify the advancement of this model, Sklearn and Keras are used to build other machine learning models on the dataset of this article for comparative experiments. The test results are shown in Table 3 below:

[0091]

[0092] In order to verify the role of bidirectional LSTM and attention mechanism, this paper also conducted an ablation experiment. The ablation experiment results are shown in Table 4 below:

[0093] Group No. Bidirectional attention Accuracy Training time(s) Identification time(s) Weight size (MB) 1 Not used Not used 85.75% 12.366 <0.001 5.82 2 use Not used 88.36% 13.536 <0.001 6.85 3 Not used use 87.48% 12.911 <0.001 6.43 4 use use 91.89% 14.372 <0.001 7.58

[0094] As can be seen from the table, although the model in this paper has improved the training time and weight by 2.006s and 1.76MB respectively compared with LSTM, its accuracy has been significantly improved by 5.9%. More importantly, the recognition time of the two has hardly changed, which is very critical for the actual deployment and application of the model. Comparing Group 1 with Group 2 and Group 3, it can be seen that both BiLSTM and the attention mechanism help improve the accuracy of the model. This is because BiLSTM can use data in both directions to obtain more comprehensive sequence features, and the addition of the attention mechanism increases the weight of important features, making it easier for the model to extract time series features, thereby improving the model performance.

[0095] Step 5. Save the trained optimal BiLSTM neural network model as a pb file, upload the pb file to the server with the configured environment, and the server can recognize human motion. In actual use, the IMU sensor acquires human motion data at a frequency of 50Hz and sends the data to the mobile app via Bluetooth. The mobile app then uploads the data to the server via the network and obtains the recognition result of this human motion data through the server port, thereby realizing real-time human motion recognition. Figure 8 shown.

[0096] After the model is deployed, you can initiate a request through a web page, client, etc. to obtain the current human body motion prediction results in real time. Fig. 9 To obtain the predicted information of human body movements at a certain moment. In the experiment, the accuracy of the model can reach more than 90%, and the delay of each action is about 0.5s. In summary, it can be seen that this deployment scheme can complete the prediction task well, and the feasibility and practicality of the scheme are good.

[0097] This embodiment uses IMU sensor data and a human motion recognition method based on a BiLSTM neural network model and an attention mechanism to recognize six actions: walking, running, climbing, sitting, lying, and standing.

Claims

1. A human motion recognition method based on a single IMU sensor, characterized in that: The following steps are involved: Step 1: Use the IMU sensor to collect human motion data and attach motion category labels to the collected data; Step 2: preprocess the action data and action labels; the data preprocessing includes filtering, outlier processing, and normalization processing; The data preprocessing of action labels is one-hot encoding; Step 3: Divide the action data and the corresponding action labels into training samples and test samples according to the ratio of 7:3; Step 4: Initialize the parameters in the BiLSTM neural network model, use the sliding window method to send the processed action data and its corresponding action label into the BiLSTM neural network model, and continuously update the parameters of the BiLSTM neural network model until the optimal BiLSTM neural network model is obtained; Step 5: Save the optimal BiLSTM neural network model as a pb format file and deploy the file to the server; Users can access the server port through the mobile app to obtain the prediction results of the BiLSTM neural network model for action data.

2. The human motion recognition method based on a single IMU sensor according to claim 1, characterized in that: The step one specifically includes: placing an IMU sensor on a human body to collect motion data at a fixed frequency; including the three-axis acceleration, three-axis angular velocity and three-axis angular acceleration of the human body of different people and different specified movements.

3. The human motion recognition method based on a single IMU sensor according to claim 1, characterized in that: In the step 2, the filtering processing method uses Kalman filtering to remove noise from the original data; Outlier processing is to remove singular values ​​and default values; normalization processing uses the Sigmoid activation function, which maps each original value to the (0,1) interval.

4. The human motion recognition method based on a single IMU sensor according to claim 1, characterized in that: The data format sent to the BiLSTM neural network model in the step 4 is (N, W, C), where N is the number of data, W is the data graph width, that is, the sliding window width, and C is the number of data channels; after initializing the parameters of the BiLSTM neural network model, sliding window width action data are sequentially sent to the model for training, and various parameters of the BiLSTM neural network model are adjusted to obtain the optimal BiLSTM neural network model.

5. The human motion recognition method based on a single IMU sensor according to claim 1, characterized in that: The structure of the neural network includes an input layer, a BiLSTM layer, a fully connected_1 layer, an attention layer, a fully connected_2 layer, and an output layer; a dropout operation needs to be performed before the BiLSTM layer, a dropout operation needs to be performed before the fully connected_1 layer, a flatten operation needs to be performed after the attention layer, and a softmax operation needs to be performed after the fully connected_2 layer.

6. The human motion recognition method based on a single IMU sensor according to claim 5, characterized in that: The BiLSTM layer adds or removes information through a gate structure; each neural network can decide to remember or forget relevant information through the gate structure of the unit state; during the calculation process, the BiLSTM model realizes the transmission of useful information for subsequent calculations by forgetting information in the cell state and memorizing new information, and useless information is discarded; the hidden layer state is output at each time step, where forgetting, memory and output are controlled by the forget gate, memory gate and output gate calculated through the hidden layer state of the previous moment and the current input.

7. The human motion recognition method based on a single IMU sensor according to claim 5, characterized in that: The input raw data is n 9-dimensional vectors collected by the IMU sensor at the same time interval within a certain period of time, that is, [9,n]; each data is a time step, forming a time series X=[x1,x2...,xn]; where each x is a 9-dimensional vector.

8. The human motion recognition method based on a single IMU sensor according to any one of claims 1 to 7, characterized in that: Combining BiLSTM with the attention mechanism, the specific implementation method of the attention layer is as follows: (1) Calculate the context vector: Read the input data through a Transformer encoder to obtain a context vector. Specifically, the context vector is used as the query vector q. In the model, the query vector q matches the dimension of the hidden state H. For example, a single query vector is [m, 1]. (2) Calculate weights: Convert context vectors into attention weights based on data content; Specifically: the hidden state H[h1,h2...,hn] and the query vector q are assigned a scoring function f(h,q) to calculate the correlation score between each hidden state hi and the query vector q; (3) Weighted summation: Use the attention weights to perform weighted summation on the input data to obtain the final output result.

9. The human motion recognition method based on a single IMU sensor according to claim 8, characterized in that: In the method (2), the scoring function selects a scaled dot product model; , where D is the dimension of the input vector.

10. The human motion recognition method based on a single IMU sensor according to claim 8, characterized in that: The method (3) specifically includes: Generate a score vector [n,1] whose length is the number of time steps in the sequence, and each element represents the correlation score between the hidden state of the corresponding time step and the query vector q; normalize the correlation score vector through the softmax function to obtain a probability distribution [n,1]; this probability distribution represents the importance weight of the hidden state of each time step in the sequence to the query vector q; the softmax function ensures that the sum of all weights is 1, and each weight is between 0 and 1; The weight coefficient is used to perform weighted summation on the original hidden state H[h1,h2...,hn], that is, [m,n] and the probability distribution [n,1] to obtain the vector S[m,1] weighted by the attention mechanism; then this input weighted m-dimensional hidden state is input into the fully connected_2 layer, softmax and output six categories.

Images