Building worker action classification method of hierarchical integrated network structure

Abstract

The invention provides a building worker action classification method of a hierarchical integrated network structure, which comprises the following steps: acquiring action data of a plurality of actions executed by a building worker wearing an IMU (Inertial Measurement Unit), preprocessing and establishing an action data set; selecting different IMU numbers and installation positions, sliding time window sizes and feature extraction data combinations to obtain a plurality of IMU configuration schemes; establishing a directed label tree based on the specificity among multiple actions, constructing a layered integrated network based on the directed label tree, and training the layered integrated network by utilizing action data in the action data set to obtain a trained action classification model; and inputting the action data of the complete action cycle of each IMU configuration scheme into the trained action classification model, evaluating the performance of each IMU configuration scheme according to the four accuracy performance indexes and the two real-time performance indexes, and selecting the IMU configuration scheme meeting a preset condition as an optional scheme for building worker action classification. The method has the advantages of being large in number of classified actions, high in precision, high in real-time performance and capable of providing various IMU configuration schemes, and it can be ensured that the wearable exoskeleton equipment accurately assists building workers in completing various operation tasks in real time.

Description

Technical Field

[0001] The invention relates to the technical field of construction worker motion classification, and in particular to a construction worker motion classification method with a hierarchical integrated network structure. Background Art

[0002] Wearable devices offer great potential for preventing work-related musculoskeletal disorders for construction workers. Advanced research has shown that exoskeletons have the potential to reduce muscle action and range of motion in a variety of body movements, such as tasks involving walking (flat and sloped), carrying, lifting (including lifting, putting down and repeated lifting), pushing and pulling, and kneeling on the ground and roof. The widespread use of exoskeletons in a variety of tasks has significantly improved the working environment and efficiency of construction workers. As the application tasks expand, motion classification technology is essential for exoskeletons to provide adaptive assistance. Currently, most wearable robots use rule-based methods to classify motion patterns, such as finite state machines, fuzzy logic, or decision trees. However, the above methods require a detailed description of each phase to design rules. As the number of actions increases, rule design will become more complex, time-consuming, and lack a certain degree of adaptability.

[0003] Machine learning-based methods can perform motion classification tasks because they do not need to rely too much on cumbersome rule design. For example, traditional machine learning based on support vector machines, linear discriminant analysis, Bayesian networks, Markov models, and artificial neural networks (ANN) are widely used for motion classification. However, due to the complexity of the input signal data (including spatial and temporal information), the above methods are difficult to accurately classify body movements. In addition, the above methods rely on feature engineering to improve accuracy, and the feature extraction strategy in feature engineering requires manual intervention, which increases time consumption.

[0004] In contrast, deep learning models have the advantage of automatically extracting complex abstract features from raw data, thereby improving decision-making and enabling classification of data with complex spatial and temporal information. For example. Convolutional Neural Network (CNN), Long Short-Term Memory (LSTM), and further improve the accuracy of action classification through algorithm adjustment and / or model structure. However, the above methods usually classify at a high level of specificity and represent the transition between high-specificity actions as a single transition occurring at a single time point, without considering the hierarchical relationship between multiple actions, that is, multiple outputs are not connected in series to jointly determine the final action category. In addition, while improving the accuracy, the above methods also increase the computational complexity and spatial complexity, which is not conducive to the real-time control of resource-constrained exoskeletons.

[0005] In addition, advanced studies usually provide a single solution based on certain sensor combinations. For example, many studies discuss the impact of feature extraction, IMU configuration, and sliding time window size on classification accuracy. However, the above studies do not provide multiple optional configurations, and a single solution may have indirect effects on different assistive devices (algorithms using foot IMUs are not applicable to bilateral portable hip exoskeletons or unilateral knee exoskeletons).

[0006] The invention patent with application number 201811328308.3 discloses a method for human action recognition based on the TP-STG framework, which includes: taking video information as input, adding prior knowledge to the SVM classifier, and proposing a posteriori discrimination criteria to remove non-personnel targets; segmenting the human targets through the target positioning and detection algorithm, and outputting them in the form of target frames and coordinate information to provide input data for human key point detection; using the improved posture recognition algorithm to locate body parts and analyze the degree of association to extract all human key point information and form a key point sequence; constructing a spatiotemporal graph on the key point sequence through the action recognition algorithm, applying it to multi-layer spatiotemporal graph convolution operations, and using the Soft-max classifier to perform action classification to achieve human action recognition in complex scenes. The above-mentioned invention method is combined with the actual scene of the offshore platform for the first time, and the proposed TP-STG framework attempts to use the methods of target detection, posture recognition and spatiotemporal graph convolution for the first time to identify the actions of workers on the offshore drilling platform. However, the above patent requires converting video and image information into point sequence information. The feature engineering is complex and time-consuming, which is not conducive to the real-time control of resource-constrained exoskeletons. The multi-layer spatiotemporal convolution operation does not consider the hierarchical relationship between multiple actions or calculate the output probability based on multiple Soft-max classifiers, thereby affecting the accuracy of action classification. Summary of the invention

[0007] In view of the technical problem that the existing construction workers' motion classification methods are difficult to meet the requirements of accuracy and real-time at the same time, the present invention proposes a construction workers' motion classification method with a hierarchical integrated network structure, and designs 6 configurations of IMU sensors with high accuracy and strong real-time performance as well as wearing them on different body parts, so as to enable a variety of wearable devices to accurately and real-timely assist construction workers in completing a variety of work tasks.

[0008] In order to achieve the above object, the technical solution of the present invention is implemented as follows: a construction worker action classification method with a hierarchical integrated network structure comprises the following steps:

[0009] S1: Obtain motion data of construction workers wearing IMUs to perform various actions, pre-process the motion data, and establish a motion data set;

[0010] S2: Select different IMU numbers and installation positions, sliding time window sizes and feature extraction data combinations to obtain multiple IMU configuration schemes; establish a directed label tree based on the specificity between multiple actions, build a hierarchical integration network based on the directed label tree, and use the corresponding action data in the action data set to train the hierarchical integration network respectively to obtain a trained action classification model;

[0011] S3: Input the motion data of the complete motion cycle of each IMU configuration scheme into the trained motion classification model, evaluate the performance of each IMU configuration scheme according to four accuracy performance indicators and two real-time performance indicators, and select the IMU configuration scheme that meets the preset conditions as the optional scheme for the construction worker's motion.

[0012] Preferably, the method for obtaining motion data in step S1 is: an IMU is installed at the front of the two thighs, the front of the two calves and the center of the trunk of the construction worker under test; the construction worker under test performs a variety of actions such as walking, going uphill, going downhill, going up stairs, going down stairs, running, standing still, squatting, kneeling still, squatting and kneeling, and the motion data collected by each IMU is wirelessly transmitted to the PC for data preprocessing; the 3D acceleration and 3D angular velocity of the IMU sensor are selected as the signal channel for feature extraction;

[0013] The data preprocessing includes: suppressing the noise of the motion data by Kalman filtering, cleaning the motion data by data visualization method, and obtaining the time series X of the complete motion cycle.

[0014] Preferably, the Z-axes of all IMUs point to the front of the construction worker under test, and the Y-axes are perpendicular to the sagittal plane of the construction worker under test;

[0015] The movements of the construction workers in a complete movement cycle are as follows: 1) standing still for 30 seconds, 2) half-kneeling still for 30 seconds, 3) squatting still for 30 seconds, 4) walking on flat ground at a self-selected speed for 30 seconds, 5) running on flat ground at a self-selected speed for 30 seconds, 6) climbing stairs at a self-selected speed for 30 seconds, 7) descending stairs at a self-selected speed for 30 seconds, 8) climbing uphill for 20 meters at a self-selected speed, 9) descending downhill for 20 meters at a self-selected speed, 10) squatting and getting up five times at a rhythm of 4 seconds, and 11) kneeling and getting up five times at a rhythm of 4 seconds.

[0016] Preferably, the method for constructing a directed label tree is: according to different action attributes, multiple actions are divided into static actions and dynamic actions, the left child node and the right child node of the root node are static actions and dynamic actions respectively, the left child node and the right child node of the static action are bending the knees and standing still respectively, the left child node and the right child node of bending the knees are squatting and kneeling respectively; the left child node and the right child node of the dynamic action are moving around and moving in place respectively, the left child node and the right child node of moving in place are squatting and kneeling respectively, the left child node and the right child node of moving around are flat ground and uneven ground respectively, the left child node and the right child node of flat ground are walking and running respectively, the left child node and the right child node of uneven ground are stairs and slope respectively, the left child node and the right child node of stairs are going up stairs and going down stairs respectively, and the left child node and the right child node of the slope are going uphill and going downhill respectively.

[0017] Preferably, the number of layers N of the directed label tree of the 11 actions including the root node is 6;

[0018] The encoding method of the binary labels of multiple actions in the directed label tree is as follows: in the directed label tree, the left child node and the right child node are marked as "1" and "0" respectively; the child nodes of all actions to be classified are represented as N-1 dimensional vectors; if the child node is not in the Nth layer, the difference M between N and the layer where the current child node is located is calculated, and then the last (NM) bit components are filled with "0".

[0019] Preferably, the method for constructing a hierarchical integrated network based on a directed label tree is as follows: the directed label tree has 10 parent nodes and 11 child nodes, the 11 child nodes represent 11 actions respectively, and the 10 parent nodes include a root node, a static action, a dynamic action, bending the knee, moving around, moving in place, flat ground, uneven ground, stairs and slopes; an ANN network is constructed for each parent node to classify the action, and the ANN classifier of each parent node is trained separately;

[0020] The input of each ANN network is the action data of all child nodes under the parent node as the root node, and the input variables are obtained by preprocessing and signal processing. The output of each ANN network is the action represented by the left child node and the right child node.

[0021] Preferably, each of the ANN networks consists of an input layer, an encoder layer, a decoder layer and an output layer, wherein the size of the input layer is determined by the input vector V, the encoder layer, the decoder layer and the output layer are 8×1, 4×1 and 2×1 respectively, the output layer is followed by a Soft-max layer, and the ANN classifier outputs a binary label vector to represent the action category;

[0022] In action classification, the binary label of standing still is [0 1 0 0 0], the binary label of squatting still is [0 0 0 00], the binary label of kneeling still is [0 0 1 0 0], the binary label of squatting up is [1 1 0 0 0], the binary label of kneeling up is [1 1 1 0 0], the binary label of walking is [1 0 0 0 0], the binary label of running is [1 0 0 1 0], the binary label of climbing stairs is [1 0 1 00], the binary label of descending stairs is [1 0 1 0 1], the binary label of going uphill is [1 0 1 1 1], and the binary label of going downhill is [1 0 1 1 0];

[0023] Action classification starts from the sub-network of the root node. The output action node of the sub-network of the root node will be used as the sub-network of the next level node, and so on, until the output node has no child nodes;

[0024] The loss function for training the hierarchical ensemble network adopts classification cross entropy, and the loss function is:

[0025]

[0026] Where, m represents the number of output actions of a single sub-network in the hierarchical ensemble network, and c m The binary indicator representing the true label of the mth action, p m Represents the classification probability of the sub-network of a single node in the hierarchical ensemble network for the mth action.

[0027] Preferably, the different numbers and installation positions of the IMUs include: there are three situations of installing one IMU, which is installed on the trunk, thighs and calves respectively; there are three situations of installing two IMUs, which are installed on two thighs, two calves and thighs & calves respectively; there are two situations of installing three IMUs, which are installed on the trunk & two thighs and the trunk & two calves respectively; there is one situation of installing four IMUs, which is installed on two thighs and two calves; there is one situation of installing five IMUs, which are installed on the trunk, two thighs and two calves respectively;

[0028] The sliding time window size includes 100ms, 150ms and 200ms;

[0029] The feature extraction data includes 3D acceleration collected by an accelerometer, 3D angular velocity collected by a gyroscope, and 3D acceleration collected by an accelerometer & 3D angular velocity collected by a gyroscope;

[0030] There are 90 IMU configuration solutions in total, with different combinations of the number and installation positions of IMUs, sliding time window sizes, and feature selection data.

[0031] The motion data collected by the multiple IMU configuration schemes and the motion data in the motion data set are subjected to signal processing before being input into the hierarchical integration network. The signal processing method is: using a sliding time window to extract features including the maximum value, minimum value, average value, standard deviation, first signal value, intermediate signal value and last signal value from each signal channel of the feature extraction data to form an input vector V; the input vector V is input into the hierarchical integration network.

[0032] Preferably, a grouped Soft-max method is used to traverse the directed label tree nodes in a breadth-first order, obtain the path encoding label and calculate the probability output of the Soft-max of each node on the path, and then the probability output of the Soft-max of each node is multiplied by the probability output of its parent node to calculate the path probability output;

[0033] By integrating the probability outputs of the sub-networks of multiple parent nodes, the unique path that can represent the child node leading to the current action category is indexed to determine the action category.

[0034] Preferably, each action node on the path corresponds to a classification probability p m , where the classification probabilities of the root node and other nodes filled with “0” are set to 1;

[0035] The label of each action is represented by a path y k =(v1,...,v E ) means that v1 and v E are the root node and the terminal node of the path respectively, and E represents the number of nodes on the path. The grouped Soft-max method is used to calculate the path probability P(y k )for Among them, v j-1 represents the j-1th node on the path;

[0036] The four accuracy performance indicators include: accuracy, precision, recall and F1 score of the action classification model prediction; the two real-time performance indicators include the number of parameters and inference time of the action classification model;

[0037] The preset condition is to use a solution with an accuracy rate higher than 93% and a suitable number of IMUs as an optional solution for classifying construction workers' movements.

[0038] Compared with the prior art, the invention has the following beneficial effects: constructing a directed label tree structure based on specificity to classify multiple actions hierarchically (top-down), that is, first classify low-specificity patterns, and gradually classify higher-specificity patterns as the tree level increases, thereby reducing the reasoning time; associating the correlation between multiple actions, that is, assigning a classifier to each parent node and integrating the outputs of multiple classifiers, jointly indexing the unique path to the leaf node of the current action category, thereby improving the accuracy; developing 6 configurations of wearing IMU sensors on different body parts, including the location and number of IMUs, the size of the sliding time window and feature extraction, which can be applied to different wearable devices. The invention combines artificial intelligence training, integrated artificial neural networks and tree breadth traversal search algorithms to solve the problems of insufficient classification accuracy and weak real-time performance of the existing wearable auxiliary device action classification strategy, and the problem that a single sensor combination solution may have indirect adverse effects on different auxiliary devices. At the same time, the invention has the characteristics of a large number of classified actions, high accuracy, strong real-time performance and the ability to provide a variety of IMU configuration solutions, which can ensure that wearable exoskeleton devices accurately and in real time assist construction workers in completing a variety of tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the present invention.

[0041] Figure 2 This is a diagram of the IMU installation method for motion data collection in the present invention, wherein (a) is IMU configuration and data collection, and (b) is the subject performing various actions.

[0042] Figure 3 A schematic diagram of basic actions for action data collection of the present invention.

[0043] Figure 4 It is a structural diagram of the hierarchical integrated network of the present invention.

[0044] Figure 5 Experimental results of all 90 IMU configurations of the present invention (including 6 optional IMU configurations) on accuracy, recall, precision and F1 score.

[0045] Figure 6 Graph showing the accuracy performance experimental results of the six optional solutions of the present invention.

[0046] Figure 7 Graph showing real-time performance experimental results of six optional solutions of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] As attached Figure 1 As shown, a construction worker action classification method with a hierarchical integrated network structure includes the following steps:

[0049] Step S1: Obtain motion data of construction workers wearing IMUs to perform various actions, transmit the motion data wirelessly to a PC for data preprocessing, and establish a motion data set.

[0050] The installation method of the motion data acquisition IMU is as follows: Figure 2 As shown in (a), the subject is equipped with 5 IMUs (LPMS-B2, Alubi) on the trunk and legs. The motion data collected by the IMUs are wirelessly transmitted to the PC for data preprocessing, which can bring higher data accuracy and comprehensive analysis capabilities. The data preprocessing process specifically includes: using Kalman filtering to effectively suppress motion data noise, thereby obtaining a smoother and more reliable state estimation; using data visualization methods to clean the data, thereby obtaining the time series X of the complete motion cycle.

[0051] The five IMUs are respectively installed on the subject's two thighs (2 cm above the knee joint), two calves (2 cm above the ankle joint) and the center of the trunk. The four IMUs installed on the legs are used to record the motion data of their respective parts, and the one IMU installed on the trunk is used to monitor the subject's overall body movement. The IMU installation method will help to obtain synchronous data and accurate motion analysis. The Z-axis of all IMUs points to the front of the subject, and the Y-axis is perpendicular to the sagittal plane of the subject. Ensuring that all IMUs are in the same direction will help to make data alignment and integration easier when data is fused or analyzed between multiple IMUs. The IMU records data at a sampling rate of 100 Hz and transmits it wirelessly to the PC for data preprocessing. As shown in the attached Figure 2 As shown in (b), the various actions include walking, going uphill, going downhill, going up stairs, going down stairs, running, standing still, squatting still, kneeling still, squatting and kneeling, a total of 11 actions.

[0052] After installing the IMU sensor, the subjects completed various actions of construction workers according to the specified operations and procedures to compile a data set for training the hierarchical integration network. The action data collection protocol is as shown in the attached Figure 3 As shown. The construction workers' movements are: 1) standing still for 30 seconds, 2) half-kneeling still for 30 seconds, 3) squatting still for 30 seconds, 4) walking on flat ground at a self-selected speed for 30 seconds, 5) running on flat ground at a self-selected speed for 30 seconds, 6) climbing stairs at a self-selected speed for 30 seconds, 7) descending stairs at a self-selected speed for 30 seconds, 8) climbing uphill (30° slope) for 20 meters at a self-selected speed, 9) descending downhill (30° slope) for 20 meters at a self-selected speed, 10) squatting and rising 5 times at a 4-second rhythm, 11) kneeling and rising 5 times at a 4-second rhythm. The rest time between static movements is 1 minute, and the rest time between dynamic movements is 5 minutes. The rest time setting between static and dynamic movements will help reduce the impact of subject fatigue on data collection and ensure the stability of data collection for each movement.

[0053] There were 10 subjects, including 5 males and 5 females, with an average age of 21.9±2.3 years, an average height of 172.0±7.0 cm, and an average weight of 69.2±8.3 kg. All subjects gave informed consent and completed the following procedures under the guidance of the safety personnel. Figure 3 The motion data acquisition protocol has been reviewed and approved by the Institutional Review Board of Beijing University of Technology (IRB: xxxb202404-1X).

[0054] Step S2: Select different IMU numbers and installation positions, sliding time window sizes and feature extraction data combinations to obtain multiple IMU configuration schemes; establish a directed label tree based on the specificity between multiple actions, and build a hierarchical integrated network based on the directed label tree. Use the corresponding action data in the action data set of step one to train the parent nodes of the hierarchical integrated network respectively to obtain a trained action classification model.

[0055] The hierarchical integrated network structure is shown in the attached Figure 4 As shown in the figure, the entire structure diagram consists of three parts, namely signal processing, hierarchical integration network and action category.

[0056] Signal processing: Process the action data set obtained in step 1 to obtain the input vector and input label of the hierarchical integration network.

[0057] The signal processing is to process the data signal, that is, to extract features from the signal channel of the action data set in step 1 by using the sliding time window method to reduce the impact of data noise and improve the robustness and accuracy of feature extraction. Since the Euler angle data will drift over time during walking, the 3D acceleration and 3D angular velocity of the IMU sensor are selected as the signal channel for feature extraction.

[0058] Each signal channel extracts 7 features, namely: maximum value, minimum value, average value, standard deviation, first value, middle value and last value, which together constitute the input vector V. The 7 features can more comprehensively and stably describe the basic statistical characteristics of the signal, which helps to improve the effect of feature extraction and improve the performance of subsequent data analysis or machine learning models. Figure 4 In the above, the size of the input vector V needs to be determined according to the specific IMU configuration scheme. In the present invention, by selecting different IMU numbers, sliding time window sizes and feature extraction data, a total of z IMU configuration schemes can be obtained. For example, in the z-th IMU configuration scheme Sz, there are 2 IMUs, the sliding time window size is fixed to 100ms, and the feature extraction data selects 3D acceleration and 3D angular velocity (6 signal channels), that is, the size of the input vector V is 84 (2×1×6×7).

[0059] The IMU configurations described cover all 10 scenarios for capturing lower limb motion data, with the number of IMUs ranging from 1 to 5:

[0060] There are three situations for installing one IMU, which is installed on the torso, thigh, and calf;

[0061] There are three situations for installing two IMUs, namely, installing them on both thighs, both calves, and thighs & calves (one side);

[0062] There are two cases of installing three IMUs, one on the torso & two thighs and one on the torso & two calves;

[0063] There is one case where 4 IMUs are installed, on two thighs and two calves;

[0064] There is one case where five IMUs are installed, one on the torso, two thighs, and two calves.

[0065] The sliding time window size of the present invention selects three cases, namely 100ms, 150ms and 200ms.

[0066] Three cases are selected for feature extraction of the present invention, namely, accelerometer only (3D acceleration), gyroscope only (3D angular velocity), and accelerometer & gyroscope (3D acceleration and 3D angular velocity).

[0067] There are 90 IMU configuration solutions in total, i.e. z=90, by combining the above IMU configurations, sliding time window sizes, and feature selection situations.

[0068] A directed label tree is constructed for hierarchical modeling of multiple action labels. The rules of hierarchical modeling are based on the specificity between different actions, which refers to different action attributes. For example, squatting and standing are static attribute actions, while walking on flat ground and walking on stairs are dynamic attribute actions. More specifically, walking on stairs can be divided into walking up stairs and walking down stairs based on the direction attribute. Figure 4 As shown, the present invention constructs 11 kinds of directed label trees of construction workers' lower limb movements, and the number of layers of the directed label tree is represented by N, and N is set to 6 (including the root node). The left child node and the right child node of the root node are static movement and dynamic movement respectively, and the left child node and the right child node of the static movement are bending knees and standing still respectively, and the left child node and the right child node of bending knees are squatting and kneeling still respectively. The left child node and the right child node of the dynamic movement are moving everywhere and moving in situ respectively, and the left child node and the right child node of moving in situ are squatting and kneeling respectively, and the left child node and the right child node of moving everywhere are flat ground and uneven ground respectively, and the left child node and the right child node of flat ground are walking and running respectively, and the left child node and the right child node of uneven ground are stairs and slope respectively, and the left child node and the right child node of stairs are going up stairs and going down stairs respectively, and the left child node and the right child node of slope are uphill and downhill respectively.

[0069] In the directed label tree, the left and right child nodes are marked as "1" and "0", respectively. All leaf nodes of the action to be classified are represented as N-1 dimensional vectors. If the leaf node is not in the Nth layer, the difference M between N and the layer where the current leaf node is located is calculated, and then the last (NM) bit components are filled with "0". For example, the tree path of the 4th-level leaf node "kneel up" is represented in binary multi-label format as [1 1 1 0 0], where the 4th and 5th bit components are filled with "0"; the tree path of the 6th-level leaf node "going down the stairs" is represented in binary multi-label format as [1 0 1 0 1].

[0070] The paths represented by the binary multi-label format must end with a leaf node, and all paths have the same length and are represented by a 5-dimensional vector. t} for classification, where i = 1, 2…t, and the input label Y = {y1,...,y n}, where k = 1, 2…n. x t represents a sample in the time series, y n represents the path label of the action to be identified, t represents the time step, and n represents the number of actions. The present invention uses a sliding time window of fixed length to classify the time series X. Since the classification must be performed in real time, the sample x i The label prediction must depend on the samples recorded before the current time step, and the window slides over time, updating its content and reclassifying.

[0071] Hierarchical ensemble network: Based on the directed label tree, a hierarchical ensemble network is constructed.

[0072] The directed label tree has 10 parent nodes and 11 leaf nodes, and the 11 leaf nodes represent 11 actions respectively. The 10 parent nodes include the root node, static action, dynamic action, bending knees, moving around, moving in place, flat ground, uneven ground, stairs and slopes. In order to make full use of the hierarchical structure and avoid inconsistent prediction of action labels, the present invention adopts a local classifier method to classify actions, that is, to build an ANN classifier for each parent node and train it separately. The networks of the 10 parent nodes are: the root node is Net0, the static action is Net2, the dynamic action is Net1, the bending knees is Net5, the moving around is Net4, the moving in place is Net3, the flat ground is Net7, the uneven ground is Net6, the stairs is Net9, and the slope is Net8. The input of each ANN network is the action data of all leaf nodes under the root node with the parent node network as the root node, and the output is the action represented by the left and right leaf nodes. The action data are all based on the action data set established in step one. For example, the training data for Net 6 is four sets of action data: "going up stairs", "going down stairs", "going uphill", and "going downhill". The training data for Net 7 is two sets of action data: "walking" and "running".

[0073] Each of the ANN classifiers is composed of an input layer, an encoder layer, a decoder layer, and an output layer. The size of the input layer is determined by the input vector V, and the encoder layer, decoder layer, and output layer are all fixed sizes: 8×1, 4×1, and 2×1, respectively. The fixed-size structure makes the number of parameters and computational complexity of the model controllable, helps to avoid overfitting, and improves the efficiency of training and reasoning. In addition, it also simplifies the design and implementation process of the model. The output layer is followed by a Soft-max layer, and the ANN classifier outputs a vector to represent the action category. Each action is assigned a unique identifier: [1,0] and [0,1] represent actions on the left child node and the right child node, respectively. Therefore, in action classification, the binary label of standing still is [0 1 0 0 0], the binary label of squatting still is [0 0 0 0 0], the binary label of kneeling still is [0 0 1 0 0], the binary label of squatting up is [1 1 0 0 0], the binary label of kneeling up is [1 1 1 0 0], the binary label of walking is [1 0 0 0 0], the binary label of running is [1 0 0 1 0], the binary label of going up stairs is [1 0 1 0 0], the binary label of going down stairs is [1 0 1 0 1], the binary label of going uphill is [10 1 1 1], and the binary label of going downhill is [1 0 1 1 0].

[0074] The training process of action classification aims to minimize the loss function. The loss function of the present invention adopts classification cross entropy, as shown in formula (1):

[0075]

[0076] Where, m represents the number of output actions of a single sub-network in the hierarchical ensemble network, and c m The binary indicator representing the true label of the mth action, p m Represents the predicted probability of the mth action by the subnetwork of a single node in the hierarchical ensemble network.

[0077] Hierarchical classification starts from the sub-network of the root node. The output action node of the sub-network of the root node will be used as the sub-network of the next level node, and so on, until the output node has no child nodes. Consider each action label represented by a path y k =(v1,...,v E ) means that v1 and v E are the starting node (root node) and the ending node of the path respectively, and E represents the number of nodes on the path. In the present invention, E=5. The present invention uses the grouped Soft-max method to calculate the path probability P(y k ). Path probability P(y k ) represents the path calculation probability of the kth action node. Each action node on the path corresponds to a classification probability p m . The classification probabilities p of the root node and other nodes filled with "0" are set to 1 to maintain the consistency of the path calculation, so that the output probability of the final classification node is distributed between 0 and 1, so the path probability P(v1) = 1. The root node is the starting node of the search path and is not a node on the path obtained by the classification calculation probability; the zero-filled node does not really exist, so it is not a node on the path obtained by the classification calculation probability. The path probability calculation is shown in formula (2).

[0078]

[0079] Where j = 1, 2…E, v j-1 Represents the j-1th node on the path.

[0080] The Softmax layers of each sub-network mentioned above are combined and calculated. The grouped Soft-max traverses the directed label tree nodes in breadth-first order, obtains the path encoding label and calculates the probability output of the Soft-max of each node on the path, and then multiplies the grouped Soft-max output of each node by the output of its parent node to calculate the probability output of the path.

[0081] Action category: Based on the hierarchical probability calculation results, output the final classification target.

[0082] The present invention integrates the probability outputs of multiple sub-networks to index the unique path that can represent the leaf node leading to the current action category, thereby determining the action category. The purpose of this is to take into account the hierarchical relationship of multiple nodes on the path, thereby maintaining the consistency of action label prediction and improving accuracy. Figure 4 The red arrow route and the Figure 4 The red box in the figure shows the classification process of “going down stairs”.

[0083] Step S3: After feature extraction, the time series of the complete motion cycle obtained by each IMU configuration scheme is input into the trained motion classification model. The performance of each IMU configuration scheme is evaluated according to four accuracy performance indicators (accuracy, precision, recall and F1-score) and two real-time performance indicators (parameters and inference time). Six IMU configuration schemes that meet the preset conditions are selected as optional schemes for construction workers' motion classification.

[0084] The calculation of the four accuracy performance indicators is shown in formulas (3)-(6):

[0085]

[0086] The four accuracy performance indicators, accuracy, precision, recall and F1-score, respectively represent the accuracy, precision, recall and F1 score predicted by the action classification model. Among them, TP, TN, FP and FN are the four basic elements of the "confusion matrix", which represent true positives, true negatives, false positives and false negatives respectively. True positive TP is the number of samples correctly predicted as positive by the action classification model in the positive class samples. True negative TN is the number of samples correctly predicted as negative by the action classification model in the negative class samples. False positive FP is the number of samples incorrectly predicted as positive by the action classification model in the negative class samples. False negative FN is the number of samples incorrectly predicted as negative by the action classification model in the positive class samples.

[0087] In addition, the present invention also uses the number of parameters and inference time to evaluate the real-time performance of the action classification model.

[0088] The number of parameters reflects the size and complexity of the model, and is usually related to the representational power of the model. Models with larger parameter sizes tend to better fit complex data distributions. However, a large number of parameters may require more computing resources for training and inference, so the number of parameters may become an important consideration in resource-constrained situations.

[0089] The inference time directly affects the real-time performance and efficiency of the model in practical applications. Generally, models with shorter inference time are more suitable for mobile devices, embedded systems or real-time application scenarios. In the present invention, in order to synchronize the ANN classifier with the IMU sensor (sampling rate is 100Hz), the inference time must be less than 10ms. The inference time of the present invention is calculated as follows: first, 10,000 randomly generated input samples are classified to measure the inference time. Then the results are averaged after repeating the measurement 100 times and finally divided by 10,000.

[0090] The preset condition is to select the solution with accuracy higher than 93% and appropriate number of IMUs as the optional solution for construction worker action classification. The experimental results of all 90 IMU configuration solutions on classification accuracy performance are shown in the attached figure. Figure 5 As shown. Among all the configuration schemes, the configuration scheme with 5 IMUs installed has higher results in accuracy, precision, recall and F1-score indicators than other schemes, but it does not have a particularly obvious advantage compared with the configuration scheme with 4 IMUs installed. In addition, the configuration scheme with 1 IMU installed has unsatisfactory results in accuracy, precision, recall and F1-score indicators, and the accuracy rate is far below 93%. Therefore, the present invention does not consider the configuration scheme of installing 5 IMUs or 1 IMU as an optional scheme. Appendix Figure 5 The green dashed box in the middle marks the six optional IMU configuration schemes that meet the preset conditions.

[0091] The six optional IMU configuration schemes that meet the preset conditions are as follows:

[0092] Scheme 1 (S1): This scheme has an accuracy of 95.71%. A total of 4 IMUs are installed, 2 of which are installed on the thighs and the other 2 are installed on the calves. The time window size is set to 200ms. Both the accelerometer and the gyroscope are used for feature extraction.

[0093] Scheme 2 (S2): This scheme has an accuracy of 95.54%. A total of three IMUs are installed, one on the torso and the other two on the thighs. The time window size is set to 200ms. Both the accelerometer and the gyroscope are used for feature extraction.

[0094] Scheme 3 (S3): This scheme has an accuracy of 95.21%. A total of three IMUs are installed, one on the torso and two on the calves. The time window size is set to 150ms. Both the accelerometer and the gyroscope are used for feature extraction.

[0095] Scheme 4 (S4): This scheme has an accuracy of 94.87%. Two IMUs are installed, both on the thighs. The time window size is set to 150ms. Both the accelerometer and gyroscope are used for feature extraction.

[0096] Scheme 5 (S5): The accuracy of this scheme is 93.17%. Two IMUs are installed, both on the calf. The time window size is set to 200ms. Both the accelerometer and the gyroscope are used for feature extraction.

[0097] Scheme 6 (S6): The accuracy of this scheme is 93.34%. Two IMUs are installed, one on the thigh and one on the calf (one side). The time window size is set to 200ms. Both the accelerometer and the gyroscope are used for feature extraction.

[0098] The six optional IMU configurations that meet the preset conditions are compared with a single ANN (baseline model), CNN, and LSTM models in terms of accuracy and real-time performance as follows:

[0099] Accuracy performance: Figure 6 The experimental results of the accuracy performance of 6 optional schemes based on hierarchical integrated network structure and single ANN, CNN and LSTM models are shown. The results show that the model of the present invention is significantly better than the single ANN model in all schemes, and achieves similar performance to the single CNN and LSTM models. In particular, in scheme S3, the model of the present invention shows the highest performance in all indicators. The single ANN model has the lowest performance, with accuracy, precision, recall and F1-score of 0.9052, 0.9240, 0.9052 and 0.8996 respectively. The performance of the single LSTM model is second, with accuracy, precision, recall and F1-score of 0.9428, 0.9505, 0.9428 and 0.9389 respectively. Compared with the single LSTM model, the model of the present invention improves by 0.99%, 1.48%, 0.99% and 1.04% in accuracy, precision, recall and F1-score respectively. In scheme S6, although the accuracy of the model of the present invention is slightly lower than that of the single CNN model, the precision, recall and F1-score are improved by 0.21%, 0.22% and 0.05% respectively compared with the single CNN model.

[0100] Real-time performance: Figure 7 The experimental results of the real-time performance of 6 optional schemes based on hierarchical integrated network structure and single ANN, CNN and LSTM models are shown. The results show that the model of the present invention performs best in terms of the number of parameters and inference time. Among them, the models with the largest number of parameters and the longest inference time are single CNN and LSTM models. The single ANN model performs second best in terms of the number of parameters and inference time. The inference time of single CNN and LSTM models exceeds 10ms, the inference time of single ANN model is between 2-4ms, and the model of the present invention is kept within 2ms. In particular, in S5, the number of parameters of the model of the present invention is 2.733k, which is 73.57%, 93.93% and 72.38% less than that of single ANN, CNN and LSTM models respectively. The inference time is 1.4ms, which is 46.15%, 88.30% and 94.62% less than that of single ANN, CNN and LSTM models respectively.

[0101] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A construction worker motion classification method with a hierarchical integrated network structure, characterized in that: The steps include: S1: Obtain motion data of construction workers wearing IMUs to perform various actions, pre-process the motion data, and establish a motion data set; S2: Select different IMU numbers and installation locations, sliding time window sizes, and feature extraction data combinations to obtain multiple IMU configuration solutions; A directed label tree is established based on the specificity between multiple actions. Based on the directed label tree, a hierarchical integration network is constructed. The hierarchical integration network is trained respectively using the corresponding action data in the action data set to obtain a trained action classification model. S3: Input the motion data of the complete motion cycle of each IMU configuration scheme into the trained motion classification model, evaluate the performance of each IMU configuration scheme according to four accuracy performance indicators and two real-time performance indicators, and select the IMU configuration scheme that meets the preset conditions as the optional scheme for construction workers' motion classification.

2. The construction worker action classification method based on the hierarchical integrated network structure according to claim 1 is characterized in that: The method for obtaining motion data in step S1 is as follows: an IMU is installed at the front of the two thighs, the front of the two calves and the center of the trunk of the construction worker under test; the construction worker under test performs a variety of actions such as walking, going uphill, going downhill, going up stairs, going down stairs, running, standing still, squatting, kneeling still, squatting and kneeling, and the motion data collected by each IMU is wirelessly transmitted to the PC for data preprocessing; the 3D acceleration and 3D angular velocity of the IMU sensor are selected as the signal channel for feature extraction; The data preprocessing includes: suppressing the noise of the motion data by Kalman filtering, cleaning the motion data by data visualization method, and obtaining the time series X of the complete motion cycle.

3. The construction worker action classification method based on the hierarchical integrated network structure according to claim 2 is characterized in that: The Z-axis of all IMUs pointed to the front of the tested construction workers, and the Y-axis was perpendicular to the sagittal plane of the tested construction workers; The movements of the construction workers in a complete movement cycle are as follows: 1) standing still for 30 seconds, 2) half-kneeling still for 30 seconds, 3) squatting still for 30 seconds, 4) walking on flat ground at a self-selected speed for 30 seconds, 5) running on flat ground at a self-selected speed for 30 seconds, 6) climbing stairs at a self-selected speed for 30 seconds, 7) descending stairs at a self-selected speed for 30 seconds, 8) climbing uphill for 20 meters at a self-selected speed, 9) descending downhill for 20 meters at a self-selected speed, 10) squatting and getting up five times at a rhythm of 4 seconds, and 11) kneeling and getting up five times at a rhythm of 4 seconds.

4. The construction worker motion classification method according to any one of claims 1 to 3, characterized in that: The method for constructing a directed label tree is as follows: according to different action attributes, multiple actions are divided into static actions and dynamic actions, the left child node and the right child node of the root node are respectively static actions and dynamic actions, the left child node and the right child node of the static action are respectively bending the knees and standing still, the left child node and the right child node of bending the knees are respectively squatting and kneeling still; the left child node and the right child node of the dynamic action are respectively moving around and moving in situ, the left child node and the right child node of moving in situ are respectively squatting and kneeling, the left child node and the right child node of moving around are respectively flat ground and uneven ground, the left child node and the right child node of flat ground are respectively walking and running, the left child node and the right child node of uneven ground are respectively stairs and slopes, the left child node and the right child node of stairs are respectively going up stairs and going down stairs, and the left child node and the right child node of slope are respectively going uphill and going downhill.

5. The construction worker action classification method based on the hierarchical integrated network structure according to claim 4 is characterized in that: The number of layers N of the directed label tree of 11 actions including the root node is 6; The encoding method of the binary labels of multiple actions in the directed label tree is as follows: in the directed label tree, the left child node and the right child node are marked as "1" and "0" respectively; the child nodes of all actions to be classified are represented as N-1 dimensional vectors; if the child node is not in the Nth layer, the difference M between N and the layer where the current child node is located is calculated, and then the last (NM) bit components are filled with "0".

6. The construction worker action classification method based on the hierarchical integrated network structure according to claim 5 is characterized in that: The method for constructing a hierarchical integrated network based on a directed label tree is as follows: the directed label tree has 10 parent nodes and 11 child nodes, the 11 child nodes represent 11 actions respectively, and the 10 parent nodes include a root node, a static action, a dynamic action, bending the knee, moving around, moving in place, a flat ground, an uneven ground, stairs and a slope; an ANN network is constructed for each parent node to classify the action, and the ANN classifier of each parent node is trained separately; The input of each ANN network is the action data of all child nodes under the parent node as the root node, and the input variables are obtained by preprocessing and signal processing. The output of each ANN network is the action represented by the left child node and the right child node.

7. The construction worker action classification method based on the hierarchical integrated network structure according to claim 6 is characterized in that: Each of the ANN networks consists of an input layer, an encoder layer, a decoder layer, and an output layer, wherein the size of the input layer is determined by the input vector V, the encoder layer, the decoder layer, and the output layer are 8×1, 4×1, and 2×1, respectively, and the output layer is followed by a Soft-max layer. The ANN classifier outputs a binary label vector to represent the action category; In action classification, the binary label of standing still is [0 1 0 0 0], the binary label of squatting still is [0 0 0 0 0], the binary label of kneeling still is [0 0 1 0 0], the binary label of squatting up is [1 1 0 0 0], the binary label of kneeling up is [1 1 1 0 0], the binary label of walking is [1 0 0 0 0], the binary label of running is [1 0 0 1 0], the binary label of climbing stairs is [1 0 1 00], the binary label of descending stairs is [1 0 1 0 1], the binary label of going uphill is [1 0 1 1 1], and the binary label of going downhill is [1 0 1 1 0]; Action classification starts from the sub-network of the root node. The output action node of the sub-network of the root node will be used as the sub-network of the next level node, and so on, until the output node has no child nodes; The loss function for training the hierarchical ensemble network adopts classification cross entropy, and the loss function is: Where m represents the number of output actions of a single sub-network in the hierarchical ensemble network, c m The binary indicator representing the true label of the mth action, p m Represents the classification probability of the sub-network of a single node in the hierarchical ensemble network for the mth action.

8. The construction worker motion classification method according to any one of claims 5 to 7, characterized in that: The different numbers and installation positions of IMUs include: 3 cases of installing 1 IMU, which is installed on the trunk, thighs and calves respectively; 3 cases of installing 2 IMUs, which are installed on two thighs, two calves and thighs & calves respectively; 2 cases of installing 3 IMUs, which are installed on the trunk & two thighs and the trunk & two calves respectively; 1 case of installing 4 IMUs, which is installed on two thighs and two calves; 1 case of installing 5 IMUs, which are installed on the trunk, two thighs and two calves respectively; The sliding time window size includes 100ms, 150ms and 200ms; The feature extraction data includes 3D acceleration collected by an accelerometer, 3D angular velocity collected by a gyroscope, and 3D acceleration collected by an accelerometer & 3D angular velocity collected by a gyroscope; There are 90 IMU configuration solutions in total, with different combinations of the number and installation positions of IMUs, sliding time window sizes, and feature selection data. The motion data collected by the multiple IMU configuration schemes and the motion data in the motion data set are subjected to signal processing before being input into the hierarchical integration network. The signal processing method is: using a sliding time window to extract features including the maximum value, minimum value, average value, standard deviation, first signal value, intermediate signal value and last signal value from each signal channel of the feature extraction data to form an input vector V; the input vector V is input into the hierarchical integration network.

9. The construction worker action classification method based on the hierarchical integrated network structure according to claim 8 is characterized in that: The grouped Soft-max method is used to traverse the directed label tree nodes in breadth-first order, obtain the path encoding label and calculate the probability output of Soft-max of each node on the path, and then multiply the probability output of Soft-max of each node by the probability output of its parent node to calculate the path probability output; By integrating the probability outputs of the sub-networks of multiple parent nodes, the unique path that can represent the child node leading to the current action category is indexed to determine the action category.

10. The construction worker motion classification method based on a hierarchical integrated network structure according to claim 7, characterized in that: Each action node on the path corresponds to a classification probability p m , where the classification probabilities of the root node and other nodes filled with "0" are set to 1; The label of each action is represented by a path y k =(v1,...,v E ) means that v1 and v E are the root node and the terminal node of the path, respectively, and E represents the number of nodes on the path. The grouped Soft-max method is used to calculate the path probability P(y k )for Among them, v j-1 represents the j-1th node on the path; The four accuracy performance indicators include: accuracy, precision, recall and F1 score of the action classification model prediction; the two real-time performance indicators include the number of parameters and inference time of the action classification model; The preset condition is to use the scheme with accuracy higher than 93% and appropriate number of IMUs as the optional scheme for construction worker motion classification.

Images