Real-time behavior recognition and vital sign status monitoring method and system based on Capsule and GRU
Through the deep learning model based on Capsule and GRU, the spatial and temporal feature extraction of multimodal information, combined with the spatial and temporal feature weight allocation, the problem of traditional CNNs neglecting feature spatial relationships is solved, and the accuracy and stability of behavior recognition are improved.
Patent Information
- Application Number
- CN202011621083.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-30
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-30
AI Technical Summary
The traditional spatial feature extraction algorithm is based on CNN, which ignores the spatial relationship between features, resulting in insufficient accuracy in behavior recognition.
The deep learning model based on Capsule and GRU is adopted to extract spatial and temporal features of multimodal motion information and physiological information, and combine spatial and temporal feature weight allocation to improve the accuracy of behavior recognition.
It significantly improves the accuracy of behavior recognition, and ensures the stability and suitability of behavior recognition through real-time monitoring and adaptive adjustment.
Smart Images

Figure CN114694245B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence and pattern recognition, and in particular relates to a real-time behavior recognition and vital sign status monitoring method and system based on Capsule and GRU. Background Art
[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.
[0003] Human Activity Recognition (HAR) is the classification and identification of specific human action types and movement patterns. With the development and maturity of transmission technologies such as Bluetooth, 5G, WiFi, LoRa, artificial intelligence technologies such as machine learning and deep learning, and advanced technologies such as cloud computing and big data, activity recognition has become a hot research direction and has been widely used in the fields of medical care, home, sports, and human-computer interaction.
[0004] There is a close connection between human behavior and kinematics, and it is feasible to use motion sensors to collect motion data to judge human behavior. Motion sensors mainly include acceleration sensors, angular velocity sensors, and magnetic induction sensors. They collect nine-axis multimodal motion data, namely three-axis acceleration data, three-axis angular velocity data, and three-axis magnetic induction data. By fusing and extracting features from the nine-axis data, human behavior can be judged.
[0005] The inventors found that the traditional spatial feature extraction algorithm is mainly based on CNN feature extraction, and CNN-based feature extraction ignores the spatial relationship between features. As long as certain features exist, they will be classified as a certain type of behavior. Summary of the invention
[0006] In order to solve at least one technical problem existing in the above-mentioned background technology, the present invention provides a real-time behavior recognition and vital sign status monitoring method and system based on Capsule and GRU, which greatly improves the accuracy of behavior recognition based on the deep learning model of Capsule and GRU, multimodal information and spatiotemporal multi-features.
[0007] In order to achieve the above object, the present invention adopts the following technical solution:
[0008] The first aspect of the present invention provides a real-time behavior recognition and vital sign status monitoring method based on Capsule and GRU.
[0009] A real-time behavior recognition and physical sign status monitoring method based on Capsule and GRU, comprising:
[0010] Acquire multimodal motion information and multimodal physiological information;
[0011] Fusion of multimodal motion information with nine-axis motion data, fusion of multimodal physiological information with four-axis physiological data;
[0012] The fused multi-modal motion information and the fused physiological information are sequentially normalized and subjected to sliding window segmentation operations to obtain motion data blocks and physiological data blocks respectively;
[0013] The motion data blocks are processed through a deep learning model based on Capsule and GRU spatial and temporal feature extraction to identify the current behavior of the human body;
[0014] Calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold to determine the user's physical state and the corresponding behavior suitability state, provide feedback to adjust the physiological parameter threshold and determine whether to issue an alarm message;
[0015] According to the output behavior category, human behavior information within a set time period is counted and fed back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
[0016] As an implementation method, a deep learning model for spatial and temporal feature extraction based on Capsule and GRU includes a feature extraction layer, a spatiotemporal feature weight allocation layer, a fully connected layer and a Softmax classification layer connected in series in sequence; the feature extraction layer is composed of a parallel spatial feature extraction layer and a temporal feature extraction layer.
[0017] As an implementation method, the process of identifying the current behavior of the human body is:
[0018] The motion data block is input into the spatial feature extraction layer, and the spatial features are extracted through the convolution layer, Capsule 1 layer, and Capsule 2 layer; the motion data block is input into the temporal feature extraction layer, and the temporal features are extracted through the GRU layer and the Attention layer;
[0019] Assign weights to the extracted spatial and temporal features, and fuse the weighted spatiotemporal features into a one-dimensional spatiotemporal feature array;
[0020] The one-dimensional spatiotemporal feature array is input into the fully connected layer and mapped into a one-dimensional array in the sample tag space, and then input into the Softmax classification layer for operation to identify the current behavior of the human body.
[0021] As an implementation method, a Z-score normalization method is used to map the fused multimodal motion information and the fused physiological information to corresponding data with a mean of 0 and a standard deviation of 1.
[0022] As an implementation method, the process of the sliding window segmentation operation is: the continuous normalized fused multimodal motion information and fused physiological information are segmented into data blocks of the same size through a sliding window, and each data block after segmentation represents a complete data cycle.
[0023] As an implementation manner, the multimodal motion data includes three-axis acceleration values, three-axis angular velocity values and three-axis magnetic induction intensity values; the multimodal physiological data includes blood pressure values, heart rate values, blood oxygen values and respiratory rate values.
[0024] The second aspect of the present invention provides a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU.
[0025] In one or more embodiments, a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU includes:
[0026] An information acquisition module, which is used to acquire multimodal motion information and multimodal physiological information;
[0027] An information fusion module, which is used to fuse multimodal motion information with nine-axis motion data, and to fuse multimodal physiological information with four-axis physiological data;
[0028] An information preprocessing module, which is used to perform normalization and sliding window segmentation operations on the fused multimodal motion information and the fused physiological information in sequence, and obtain motion data blocks and physiological data blocks accordingly;
[0029] The behavior recognition module is used to process the motion data blocks through a deep learning model based on the spatial and temporal feature extraction of Capsule and GRU to identify the current behavior of the human body;
[0030] The module for judging the physical state and appropriate behavior state is used to calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold, judge the user's physical state and the corresponding appropriate behavior state, adjust the physiological parameter threshold and judge whether to issue an alarm message;
[0031] The deep learning model optimization module is used to count the human behavior information within a set time period according to the output behavior category and feed it back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
[0032] In one or more embodiments, a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU includes:
[0033] A multimodal information acquisition module, which is used to acquire multimodal motion information and multimodal physiological information;
[0034] A multimodal information transmission module, which is used to transmit the information collected by the multimodal information collection module to the multimodal information processing module;
[0035] The multimodal information processing module is used to fuse the multimodal motion information with the nine-axis motion data, and to fuse the multimodal physiological information with the four-axis physiological data; the fused multimodal motion information and the fused physiological information are normalized and segmented by sliding windows in turn to obtain motion data blocks and physiological data blocks respectively; the motion data blocks are processed by a deep learning model based on the spatial and temporal feature extraction of Capsule and GRU to identify the current behavior of the human body; the mean of each column of data in the physiological data block is calculated and compared with the physiological parameter threshold to determine the user's physical state;
[0036] The multimodal information application module is used to determine whether to issue an alarm message based on the user's behavioral suitability status; based on the output behavior category, the human behavior information within a set time period is counted and fed back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
[0037] A third aspect of the present invention provides a computer-readable storage medium.
[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps in the above-mentioned method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU.
[0039] A fourth aspect of the present invention provides a computer device.
[0040] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU as described above are implemented.
[0041] Compared with the prior art, the present invention has the following beneficial effects:
[0042] The present invention is based on the deep learning framework of Capsule and GRU. It extracts temporal features, spatial features and the relationship between spatial features from behavior information and distributes spatiotemporal feature weights, thereby better judging the behavior category and effectively improving the accuracy of behavior recognition.
[0043] The present invention proposes a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU. By processing and analyzing multimodal information of the human body, statistical analysis of human behavior information and monitoring of vital sign status can be achieved, thereby improving people's quality of life.
[0044] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.
[0046] Figure 1 It is a flow chart of a method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU according to an embodiment of the present invention.
[0047] Figure 2 It is a schematic diagram of the main module structure framework and connection relationship of the real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU in an embodiment of the present invention.
[0048] Figure 3 It is a schematic diagram of a behavior classification module of a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU in an embodiment of the present invention.
[0049] Figure 4 It is a schematic diagram of spatial feature extraction in a behavior classification module of a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU in an embodiment of the present invention.
[0050] Figure 5 It is a schematic diagram of the time feature extraction principle in the behavior classification module of the real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0052] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.
[0053] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.
[0054] Capsule: Capsule network, a concept proposed by Hinton in 2017, can not only identify whether the behavior information has certain characteristics, but also identify the spatial relationship between each feature, which improves the accuracy of classification.
[0055] GRU network: A network proposed by Cho et al. in 2014 to extract temporal features. Through the gating mechanism, it can not only remember past information, but also selectively forget unimportant information and model long-term information. The GRU network is a variant of the LSTM network, which retains the effect of the LSTM network while having a simpler structure, fewer parameters and better convergence.
[0056] Embodiment 1
[0057] like Figure 1 As shown, the real-time behavior recognition and vital sign status monitoring method based on Capsule and GRU in this embodiment includes:
[0058] Step 1: Acquire multimodal motion information and multimodal physiological information.
[0059] based on Figure 1 , step 1 specifically includes S01 a multimodal motion information and multimodal physiological information acquisition step and S02 a multimodal information transmission step.
[0060] In the specific implementation, the multimodal motion and physiological data of the human body are collected in real time through acceleration sensors, angular velocity sensors, magnetic induction sensors, blood pressure sensors, heart rate sensors, blood oxygen sensors and respiratory rate sensors; the multimodal motion data include three-axis acceleration values, three-axis angular velocity values and three-axis magnetic induction values; the multimodal physiological data include blood pressure values, heart rate values, blood oxygen values and respiratory rate values.
[0061] The collected multimodal information of the human body can be transmitted to the corresponding signal processing end through wired or wireless transmission technology.
[0062] Step 2: Fusion of multimodal motion information with nine-axis motion data, fusion of multimodal physiological information with four-axis physiological data.
[0063] Specifically, step 2 corresponds to Figure 1The data fusion steps of S03 in are:
[0064] The collected three-axis acceleration values, three-axis angular velocity values and three-axis magnetic induction values are fused into the same array according to the merging rule that the vertical direction is the time unit and the horizontal direction is the data values collected by each sensor. The fused array has 9 columns, namely X-axis acceleration value, Y-axis acceleration value, Z-axis acceleration value, X-axis angular velocity value, Y-axis angular velocity value, Z-axis angular velocity value, X-axis magnetic induction value, Y-axis magnetic induction value and Z-axis magnetic induction value. This array is the original motion data.
[0065] The collected blood pressure values, heart rate values, blood oxygen values and respiratory rate values are fused into the same array according to the merging rule that the vertical direction is the time unit and the horizontal direction is the data values collected by each sensor. The fused array has 4 columns, namely blood pressure values, heart rate values, blood oxygen values and respiratory rate values. This array is the original physiological data.
[0066] Step 3: The fused multimodal motion information and the fused physiological information are normalized and segmented by sliding windows in turn to obtain motion data blocks and physiological data blocks respectively.
[0067] In a specific implementation, step 3 includes Figure 1 The S04 normalization step and the S05 sliding window segmentation step in .
[0068] Specifically, in this embodiment, the Z-score normalization method is used to map the 9 columns of motion data and the 4 columns of physiological data into new data with a mean of 0 and a standard deviation of 1, respectively. The implementation formula is as follows:
[0069]
[0070] In formula (I), x norm is the data after normalization of a column of data. is the mean of the data column, and σ is the standard deviation of the data column. Through the above operation, the sensor data of different dimensions and dimensional units are normalized to the same level to eliminate the dimensional influence between different data and the influence of prolonged training time caused by singular sample data.
[0071] It should be noted here that, in other embodiments, other normalization methods may also be used to normalize the fused multimodal motion information and the fused physiological information, and the results do not affect the behavior recognition and vital sign status monitoring results of the present invention.
[0072] Since the input data is a continuous signal based on a time period, the motion and physiological data after normalization in step S04 are divided into data blocks of the same size by a sliding window method, and each data block after division represents a complete data cycle.
[0073] It should be noted here that, according to user needs and personal behavior characteristics, users can preset relevant parameters, including sliding window size and sliding step size. The default sliding window size is 4 times the sensor sampling frequency value, and the sliding step size is 50% of the sliding window length.
[0074] Step 4: The motion data blocks are processed through a deep learning model based on Capsule and GRU spatial and temporal feature extraction to identify the current behavior of the human body.
[0075] Specifically, step 4 includes Figure 1 S06 spatial feature extraction step, S07 temporal feature extraction step, S09 spatiotemporal feature weight assignment step and S10 behavior information classification step.
[0076] The deep learning model for spatial and temporal feature extraction based on Capsule and GRU in this embodiment includes a feature extraction layer, a spatiotemporal feature weight allocation layer, a fully connected layer and a Softmax classification layer connected in series in sequence; the feature extraction layer is composed of a parallel spatial feature extraction layer and a temporal feature extraction layer.
[0077] The specific process of S06 spatial feature extraction step is as follows:
[0078] The motion data block is input into the spatial feature extraction layer, and the spatial features are extracted through the convolution layer, Capsule 1 layer and Capsule 2 layer. The parameters set by the user in this step include the size of the input data, the size of the convolution kernel, the step size of the convolution kernel, etc.
[0079] The spatial feature extraction layer includes the convolutional layer, Capsule 1 layer and Capsule 2 layer connected in sequence; Figure 4 As shown, the specific implementation process of the modules used in this embodiment is as follows:
[0080] Set the number of convolution kernels in the convolution layer to 256, the size of each convolution kernel to 1×41, and the step size to 1;
[0081] Set the number of convolution kernel groups in Capsule 1 layer to 34, the number of convolution kernels in each group to 11, the size of each convolution kernel to 1×21, and the stride to 2;
[0082] Set the dimension of the output features in the Capsule 2 layer to 12 dimensions, and the feature of each dimension is a vector of length 16;
[0083] The steps include:
[0084] (1) Behavior information with input dimensions of 10×1×80×3;
[0085] (2) After the input behavior information passes through the convolution layer, the input behavior information is converted from a scalar to a vector using formula (II):
[0086]
[0087] In formula (II), x i It refers to the behavior information of the ith row in the time series after multimodal data fusion, normalization and sliding window segmentation; W ij It means x i The jth weight parameter in the relevant convolutional layer has a default initial value that is a random number generated by a truncated normal distribution; b j is the bias parameter of the convolution layer, and its initial value defaults to 0; j Represents the output of the convolutional layer, which is a vector whose size meets the requirements of the Capsule 1 layer input data;
[0088] The output information dimension after the convolution layer is 10×1×40×256;
[0089] (3) The system encapsulates 34 groups of convolution kernels in the Capsule 1 layer, and inputs the 10×1×40×256-dimensional behavior information output by the convolution layer into the Capsule 1 layer. The input behavior information is converted into behavior information with spatial characteristics through formula (III):
[0090]
[0091] In formula (III), W jk Indicates the Capsule 1 layer and y j The initial value of the kth weight parameter is a random number generated by a truncated normal distribution by default; b k is the bias parameter in Capsule 1 layer, and its initial value defaults to 0; squash(·) is a new nonlinear function, similar to the common nonlinear functions such as tanh(·) and relu(·). The squash(·) function can make the vector modulus not exceed 1 after transformation, and can keep the direction of the vector unchanged; u k Represents the output of Capsule 1 layer, which is a vector that can identify the characteristics of behavioral information and the spatial relationship between different features;
[0092] The output information dimension after Capsule1 layer is 10×340×11×1;
[0093] (4) The behavior information with spatial characteristics is input into Capsule 2 layer, and the behavior information is processed by the dynamic routing protocol and squash (·) function shown in equations (IV) and (V):
[0094]
[0095]
[0096] In formula (IV), b ik is the dynamic routing weight between the i-th neuron in Capsule 1 layer and the k-th neuron in Capsule 2 layer; b ij is the dynamic routing weight between the i-th neuron in Capsule 1 layer and the j-th neuron in Capsule 2 layer; u j|i is the weighted sum of the i-th output vector of Capsule 1 layer; s j is the behavior information processed by the dynamic routing protocol; in formula (V), v j For j The output vector after being processed by the squash(·) function;
[0097] The output information dimension after Capsule 2 layer is 10×12×16×1.
[0098] The specific process of S07 time feature extraction step is:
[0099] The motion data block is input into the temporal feature extraction layer, and the temporal features are extracted through the GRU layer and the Attention layer. The parameters set by the user in this step include the size of the input data, the number of GRU output units, and the attention length of the Attention layer.
[0100] Specifically, the temporal feature extraction layer includes a GRU layer and an Attention layer connected in sequence; Figure 5 As shown, the specific implementation process of the module used in this example is as follows:
[0101] Set the number of GRU units in the GRU layer to 128;
[0102] Set the attention length of the Attention layer to 50;
[0103] The steps include:
[0104] (1) Input the behavior information with the dimension of 10×1×80×3 into the GRU layer;
[0105] (2) The input behavior information is processed by 128 GRU units and converted into behavior information with temporal characteristics;
[0106] The output information dimension after the GRU layer is 10×1×128;
[0107] (3) The behavior information output by the GRU layer is input into the Attention layer. By assigning different weights to the features of different regions, the network can better focus on important information and weaken irrelevant information.
[0108] The output information dimension after the Attention layer is 10×1×128;
[0109] Step S10 includes the following steps:
[0110] Set the output information length of the fully connected layer to 19;
[0111] (1) Input the one-dimensional spatiotemporal feature array into the fully connected layer and map it into a one-dimensional array in the sample label space;
[0112] The output information dimension after the fully connected layer is 10×19×1;
[0113] (2) The output information of the fully connected layer is input into the Softmax classification layer for operation to identify the current behavior.
[0114] The specific process of the S09 spatiotemporal feature weight allocation step is:
[0115] The extracted spatial and temporal features are weighted, and the weighted spatiotemporal features are fused into a one-dimensional spatiotemporal feature array; the initial value of the weight assignment is set to 0.5 for the spatiotemporal features respectively.
[0116] The specific process of the S10 behavior information classification step is as follows:
[0117] The one-dimensional spatiotemporal feature array is input into the fully connected layer and mapped into a one-dimensional array in the sample label space, and then input into the Softmax classification layer for operation to identify the current behavior. The optimal deep learning model is obtained by iteratively training the preset labeled data, and the multimodal information collected in real time is input into this model to perform real-time behavior type discrimination. The parameter set by the user in this step is the output information length of the fully connected layer.
[0118] Step 5: Calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold to determine the user's physical state and the corresponding behavioral suitability state and determine whether to issue an alarm message.
[0119] In this embodiment, step 5 specifically includes: Figure 1 S08 threshold detection step, S11 behavior suitability state judgment step, S12 threshold feedback adjustment step, S15 alarm judgment step and S16 alarm step;
[0120] The specific process of the S08 threshold detection step is:
[0121] Calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold to determine the range of the current user's physiological parameters;
[0122] For example: four physiological parameter values are preset to determine the range of the current user's physiological parameters; the four physiological parameter values divide the physiological data into five ranges, namely extremely below the normal range, below the normal range, normal range, above the normal range and extremely above the normal range.
[0123] The specific process of the S11 behavior appropriate state judgment step is:
[0124] The suitability of the user's behavior is determined based on the range of the user's physiological data; extremely below the normal range and extremely above the normal range correspond to a severely unsuitable state, above the normal range and below the normal range correspond to a slightly unsuitable state, and the normal range corresponds to a suitable state.
[0125] Taking sports behavior as an example, an unsuitable sports behavior state means that it is not suitable to perform the sport at the moment.
[0126] In the threshold feedback adjustment step S12, the user provides feedback to the system based on his / her own situation and the degree of conformity between the output user behavior appropriate state and his / her actual state, and the system adjusts the system preset threshold according to the user feedback;
[0127] In the S15 alarm determination step, it is determined whether the behavior suitability state has reached the alarm value, and whether to issue an alarm message is determined based on whether the output user behavior suitability state is severely unsuitable; if it is severely unsuitable, the S16 alarm step is executed to send an alarm message.
[0128] Step 6: According to the output behavior category, the human behavior information within the set time period is counted and fed back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
[0129] In this embodiment, step 5 specifically includes: Figure 1 The S13 behavior information statistics step and the S14 spatiotemporal feature weight feedback adjustment step.
[0130] In the S13 behavior information statistics step, the output behavior category is output to the user end in real time, and the behavior information statistics are performed according to the time period set by the user; taking exercise behavior as an example, the exercise behavior information includes model accuracy, exercise type, duration and calorie consumption value; the time period for exercise information statistics is set in days, for example: the range is 1 to 7 days.
[0131] In step S14 of spatiotemporal feature weight feedback adjustment, the user provides feedback to the system based on whether the output model accuracy meets their own needs, and the system adjusts the spatiotemporal feature weights based on the user feedback.
[0132] This embodiment improves the accuracy of behavior recognition by fusing the spatial and temporal features extracted by Capsule and GRU.
[0133] Embodiment 2
[0134] This embodiment provides a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU, which specifically includes the following modules:
[0135] An information acquisition module, which is used to acquire multimodal motion information and multimodal physiological information;
[0136] An information fusion module, which is used to fuse multimodal motion information with nine-axis motion data, and to fuse multimodal physiological information with four-axis physiological data;
[0137] An information preprocessing module, which is used to perform normalization and sliding window segmentation operations on the fused multimodal motion information and the fused physiological information in sequence, and obtain motion data blocks and physiological data blocks accordingly;
[0138] The behavior recognition module is used to process the motion data blocks through a deep learning model based on the spatial and temporal feature extraction of Capsule and GRU to identify the current behavior of the human body;
[0139] The module for judging the physical state and appropriate behavior state is used to calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold, judge the user's physical state and the corresponding appropriate behavior state, adjust the physiological parameter threshold and judge whether to issue an alarm message;
[0140] The deep learning model optimization module is used to count the human behavior information within a set time period according to the output behavior category and feed it back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
[0141] It should be noted here that the various modules in the real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU in this embodiment correspond one to one with the various steps in the real-time behavior recognition and vital sign status monitoring method based on Capsule and GRU in Example 1, and the specific implementation process is the same, which will not be repeated here.
[0142] Embodiment 3
[0143] like Figure 2 As shown, this embodiment provides a real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU, which specifically includes the following structure.
[0144] (1) a multimodal information acquisition module, which is used to acquire multimodal motion information and multimodal physiological information;
[0145] Specifically, the multimodal information acquisition module includes a motion data acquisition submodule and a physiological data acquisition submodule.
[0146] The motion data acquisition submodule is used for: real-time acquisition of multimodal motion data, including three-axis acceleration values, three-axis angular velocity values and three-axis magnetic induction values; the three axes are X-axis, Y-axis and Z-axis, the three-axis acceleration values include X-axis acceleration value, Y-axis acceleration value and Z-axis acceleration value, the three-axis angular velocity values include X-axis angular velocity value, Y-axis angular velocity value and Z-axis angular velocity value, and the three-axis magnetic induction values include X-axis magnetic induction value, Y-axis magnetic induction value and Z-axis magnetic induction value;
[0147] The physiological data acquisition submodule is used to: collect multimodal physiological data in real time, including blood pressure, heart rate, blood oxygen and respiratory rate.
[0148] (2) A multimodal information transmission module, which is used to transmit the information collected by the multimodal information collection module to the multimodal information processing module.
[0149] The multimodal information transmission module transmits the collected human multimodal information to the receiving end through wired or wireless transmission technology. The multimodal information transmission module includes a multimodal information sending unit and a multimodal information receiving unit; the multimodal information sending unit is used to send multimodal information; the multimodal information receiving unit is used to receive multimodal information.
[0150] (3) A multimodal information processing module, which is used to fuse the multimodal motion information with the nine-axis motion data, and to fuse the multimodal physiological information with the four-axis physiological data; the fused multimodal motion information and the fused physiological information are normalized and window segmented in turn to obtain motion data blocks and physiological data blocks respectively; the motion data blocks are processed by a deep learning model based on the spatial and temporal feature extraction of Capsule and GRU to identify the current behavior of the human body; the mean of each column of data in the physiological data block is calculated and compared with the physiological parameter threshold to determine the user's physical state.
[0151] Specifically, the multimodal information processing module is used to: process the received multimodal motion and physiological data through the multimodal information preprocessing submodule and the multimodal information classification submodule to obtain the behavior type and physical state of the current user;
[0152] The multimodal information preprocessing submodule is used to: perform multimodal data fusion, normalization and sliding window segmentation operations on the received multimodal motion and physiological data respectively;
[0153] The multimodal information classification submodule is used to: judge the current human behavior through a deep learning algorithm based on the spatial and temporal feature extraction of Capsule and GRU; judge the user's current physical state through the comparison of physiological parameter thresholds;
[0154] The multimodal information preprocessing submodule includes a multimodal data fusion unit, a multimodal data normalization unit and a multimodal data sliding window segmentation unit which are connected in sequence;
[0155] The multimodal data fusion unit includes a motion data fusion subunit and a physiological data fusion subunit;
[0156] The motion data fusion subunit is used to fuse the three-axis acceleration values, three-axis angular velocity values and three-axis magnetic induction values collected by the acceleration sensor, angular velocity sensor and magnetic induction sensor respectively into the same array according to the merging rule that the vertical direction is the time unit and the horizontal direction is the data value collected by each sensor in turn. The fused array has 9 columns, namely, the X-axis acceleration value, the Y-axis acceleration value, the Z-axis acceleration value, the X-axis angular velocity value, the Y-axis angular velocity value, the Z-axis angular velocity value, the X-axis magnetic induction value, the Y-axis magnetic induction value and the Z-axis magnetic induction value, which are used for further data preprocessing;
[0157] The physiological data fusion subunit is used to fuse the blood pressure values, heart rate values, blood oxygen values and respiratory rate values collected by the blood pressure sensor, heart rate sensor, blood oxygen sensor and respiratory rate sensor into the same array according to the merging rule of taking the vertical direction as the time unit and the horizontal direction as the data values collected by each sensor in turn. The fused array has 4 columns, namely the blood pressure value, heart rate value, blood oxygen value and respiratory rate value, which are used for further data preprocessing;
[0158] The multimodal data normalization unit includes a motion data normalization subunit and a physiological data normalization subunit;
[0159] The motion data normalization subunit and the physiological data normalization subunit are used to normalize the sensor data of different dimensions and dimensional units to the same level. The Z-score normalization method adopted in the present invention maps the data to new data with a mean of 0 and a standard deviation of 1 to eliminate the dimensional influence between different data and the influence of prolonged training time caused by singular sample data;
[0160] The multimodal data sliding window segmentation unit includes a motion data sliding window segmentation subunit and a physiological data sliding window segmentation subunit;
[0161] The motion data sliding window segmentation subunit and the physiological data sliding window segmentation subunit are used to segment continuous data into data blocks of the same size through sliding windows, and each data block after segmentation represents a complete data cycle; the window size of the sliding window is the size of the data block, and the sliding step size is the distance the sliding window moves each time;
[0162] The multimodal information classification submodule includes a spatial feature extraction unit, a temporal feature extraction unit, a spatiotemporal feature weight allocation unit, a behavior classification unit, a threshold comparison unit and a body state judgment unit; the motion data sliding window segmentation subunit is respectively connected to the spatial feature extraction unit and the temporal feature extraction unit; the spatial feature extraction unit and the temporal feature extraction unit are both connected to the spatiotemporal feature weight allocation unit; the spatiotemporal feature weight allocation unit is connected to the behavior classification unit; the physiological data sliding window segmentation subunit is connected to the threshold comparison unit; the threshold comparison unit is connected to the body state judgment unit;
[0163] exist Figure 3 In the example, the spatial feature extraction unit includes a convolutional layer, a Capsule 1 layer, and a Capsule 2 layer connected in sequence, which are used to extract the spatial features of the input sensor data;
[0164] The convolution layer is used to initially extract spatial features and convert the features from scalar to vector;
[0165] Capsule 1 layer is used to convert the input behavior information into behavior information with spatial characteristics;
[0166] Capsule 2 layer is used to process behavioral information through dynamic routing protocols;
[0167] exist Figure 3 In the example, the temporal feature extraction unit includes a GRU layer and an Attention layer connected in sequence, which are used to extract the temporal features of the input sensor data;
[0168] The GRU layer is used to extract temporal features from the input behavior information sequence;
[0169] The Attention layer is an information selection mechanism that assigns different weights to features in different regions, allowing the network to better focus on important information and weaken irrelevant information.
[0170] The spatiotemporal feature weight allocation unit is used to allocate weights to the extracted spatial and temporal features, and fuse the allocated spatiotemporal features into a one-dimensional spatiotemporal feature array;
[0171] The behavior classification unit includes a fully connected layer and a Softmax classification layer connected in sequence;
[0172] The fully connected layer is used to map the fused spatiotemporal feature array into the sample tag space to form a one-dimensional array;
[0173] The Softmax classification layer is used to operate on the one-dimensional array output by the fully connected layer to identify the current behavior;
[0174] The threshold comparison unit is used to compare the collected physiological data with the preset physiological parameter threshold to determine whether the current user's physiological parameters are within the normal range;
[0175] The body state judgment unit is used to judge the body state of the current user according to the output of the threshold comparison unit.
[0176] (4) A multimodal information application module, which is used to determine whether to issue an alarm message based on the user's behavioral suitability status; based on the output behavior category, the human behavior information within a set time period is statistically analyzed and fed back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
[0177] Specifically, the multimodal information application module includes a motion suitability judgment unit, an alarm unit, and a behavior information statistics unit connected in sequence;
[0178] The behavior suitability determination unit is used to determine the behavior suitability state of the user according to the output of the body state determination unit;
[0179] The alarm unit is used to send out an alarm message when the user's behavior suitability status reaches a severely inappropriate level;
[0180] The behavior information statistics unit is used to output the current user's behavior category in real time and perform exercise information statistics according to the time period set by the user.
[0181] Embodiment 4
[0182] This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the real-time behavior recognition and vital sign status monitoring method based on Capsule and GRU as described in the above-mentioned embodiment 1 are implemented.
[0183] Embodiment 5
[0184] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU as described in the first embodiment are implemented.
[0185] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.
[0186] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0187] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0188] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0189] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0190] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A real-time behavior recognition and vital sign status monitoring method based on Capsule and GRU, characterized in that: include: Acquire multimodal motion information and multimodal physiological information; Fusion of multimodal motion information with nine-axis motion data, fusion of multimodal physiological information with four-axis physiological data; The fused multi-modal motion information and the fused physiological information are sequentially normalized and subjected to sliding window segmentation operations to obtain motion data blocks and physiological data blocks respectively; The motion data blocks are processed through a deep learning model based on Capsule and GRU spatial and temporal feature extraction to identify the current behavior of the human body; Calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold to determine the user's physical state and the corresponding behavior suitability state, provide feedback to adjust the physiological parameter threshold and determine whether to issue an alarm message; According to the output behavior category, human behavior information within a set time period is counted and fed back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
2. The method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU as claimed in claim 1, characterized in that: The deep learning model for spatial and temporal feature extraction based on Capsule and GRU includes a feature extraction layer, a spatiotemporal feature weight allocation layer, a fully connected layer and a Softmax classification layer connected in series; the feature extraction layer is composed of a parallel spatial feature extraction layer and a temporal feature extraction layer.
3. The method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU as claimed in claim 2, characterized in that: The process of identifying the current behavior of the human body is as follows: The motion data block is input into the spatial feature extraction layer, and the spatial features are extracted through the convolution layer, Capsule 1 layer, and Capsule 2 layer; the motion data block is input into the temporal feature extraction layer, and the temporal features are extracted through the GRU layer and the Attention layer; Assign weights to the extracted spatial and temporal features, and fuse the weighted spatiotemporal features into a one-dimensional spatiotemporal feature array; The one-dimensional spatiotemporal feature array is input into the fully connected layer and mapped into a one-dimensional array in the sample tag space, and then input into the Softmax classification layer for operation to identify the current behavior of the human body.
4. The method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU as claimed in claim 1, characterized in that: The Z-score normalization method is used to map the fused multimodal motion information and the fused physiological information to corresponding data with a mean of 0 and a standard deviation of 1.
5. The method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU as claimed in claim 1, characterized in that: The process of the sliding window segmentation operation is: the continuous normalized fused multimodal motion information and fused physiological information are segmented into data blocks of the same size through a sliding window, and each data block after segmentation represents a complete data cycle.
6. The method for real-time behavior recognition and vital sign status monitoring based on Capsule and GRU as claimed in claim 1, characterized in that: The acquired multimodal motion information includes three-axis acceleration values, three-axis angular velocity values, and three-axis magnetic induction intensity values; the acquired multimodal physiological information includes blood pressure values, heart rate values, blood oxygen values, and respiratory rate values.
7. A real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU, characterized in that: include: An information acquisition module, which is used to acquire multimodal motion information and multimodal physiological information; An information fusion module, which is used to fuse multimodal motion information with nine-axis motion data, and to fuse multimodal physiological information with four-axis physiological data; An information preprocessing module, which is used to perform normalization and sliding window segmentation operations on the fused multi-modal motion information and the fused physiological information in sequence, and obtain motion data blocks and physiological data blocks accordingly; The behavior recognition module is used to process the motion data blocks through a deep learning model based on the spatial and temporal feature extraction of Capsule and GRU to identify the current behavior of the human body; The module for judging the physical state and appropriate behavior state is used to calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold, judge the user's physical state and the corresponding appropriate behavior state, adjust the physiological parameter threshold and judge whether to issue an alarm message; The deep learning model optimization module is used to count the human behavior information within a set time period according to the output behavior category and feed it back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
8. A real-time behavior recognition and vital sign status monitoring system based on Capsule and GRU, characterized in that: include: A multimodal information acquisition module, which is used to acquire multimodal motion information and multimodal physiological information; A multimodal information transmission module, which is used to transmit the information collected by the multimodal information collection module to the multimodal information processing module; A multimodal information processing module, which is used to fuse the multimodal motion information with the nine-axis motion data, and to fuse the multimodal physiological information with the four-axis physiological data; the fused multimodal motion information and the fused physiological information are normalized and segmented by sliding windows in turn, and motion data blocks and physiological data blocks are obtained accordingly; The motion data blocks are processed through a deep learning model based on Capsule and GRU spatial and temporal feature extraction to identify the current behavior of the human body; Calculate the mean of each column of data in the physiological data block and compare it with the physiological parameter threshold to determine the user's physical condition; The multimodal information application module is used to determine whether to issue an alarm message based on the user's behavioral suitability status; based on the output behavior category, the human behavior information within a set time period is counted and fed back to the deep learning model to adjust the spatiotemporal feature weight parameters in the deep learning model to ensure the stability and adaptability of behavior recognition.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps in the real-time behavior recognition and vital sign status monitoring method based on Capsule and GRU as described in any one of claims 1 to 6 are implemented.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the real-time behavior recognition and vital sign status monitoring method based on Capsule and GRU as described in any one of claims 1 to 6 are implemented.
