A multi-task gated fuzzy neural network algorithm and storage medium

Abstract

A multi-task gated fuzzy neural network algorithm and storage medium, the algorithm includes the steps of: constructing a main network and a gated network; obtaining sensor data; obtaining fuzzy rules; calculating the main network membership and the gated network membership; calculating the main network trigger strength and the gated network trigger strength; calculating the main network normalized trigger strength and the gated network normalized trigger strength; calculating the rule output according to the fuzzy rules and the main network normalized trigger strength; calculating the gated network output according to the gated network normalized trigger strength; calculating the gated weighted output according to the gated network output and the rule output; calculating the task classification result output according to the gated weighted output; obtaining an objective function; using the objective function to optimize the task classification result output. The present application can fuse multimodal sensor information and directly and simultaneously identify multiple human lower limb movement intentions, which is simpler and more efficient than other methods.

Description

Technical Field

[0001] The present invention belongs to the technical field of human body motion intention recognition, and in particular relates to a multi-task gated fuzzy neural network algorithm and a storage medium. Background Art

[0002] Wearable robots for lower limbs, including exoskeletons, prostheses, and orthotics, can be used to improve human life or enhance human capabilities, and have broad application prospects in rehabilitation medicine, military, and industry. Accurately identifying human movement intentions is a key technology and major challenge in the research of wearable robots, which can enhance the safety, comfort, and intelligence of wearable robots. At present, in the research of wearable robots for lower limbs, human lower limb movement intentions can be divided into discrete movement intentions and continuous movement intentions. Related application research mainly focuses on the following aspects:

[0003] (1) Movement pattern recognition: Recognizes a person’s movement pattern (horizontal walking, going up and down stairs, going up and down slopes, running, etc.) and movement pattern conversion (such as conversion from “horizontal walking” to “going up stairs”), which is a discrete movement intention;

[0004] (2) Gait phase recognition: Human walking is a periodic process. According to the events of human toe contact and heel lift-off, a gait cycle can usually be divided into eight gait phases, which is a discrete movement intention.

[0005] (3) Joint information estimation: estimating and predicting the angle, angular velocity or torque of the human hip, knee and ankle joints, which is a continuous motion intention;

[0006] (4) Direction or speed estimation: Estimating and predicting the direction or speed of human walking, which is a continuous motion intention.

[0007] In order to accurately identify the user's lower limb movement intention, engineers have placed many sensors on the human body and robot, such as electromyographic signal (EMG) sensors, inertial measurement units (IMUs), angle sensors, and plantar FSR pressure sensors. Sensor fusion technology can fuse information from different sensors from a temporal or spatial perspective, eliminate redundant information, retain complementary information, and more accurately identify the human lower limb movement intention. Considering the level of fusion objects, sensor fusion can be divided into data-level fusion, feature-level fusion, and decision-level fusion according to the fusion level. At present, some studies have proposed three-level sensor fusion methods. The paper "An extended kalman filter to estimate human gait parameters and walking distance" (2013 American Control Conference. IEEE, 2013: 752-757, Bennett T, Jafari R, Gans N) proposed using the extended Kalman filter method to perform data-level fusion on three IMUs to estimate the lower limb joint angles and travel distance when people walk. In the paper "Continuous locomotion-mode identification for prosthetic legs based on neuromuscular–mechanical fusion" (IEEE Transactions on Biomedical Engineering, 2011, 58(10): 2867-287, Huang H, Zhang F, Hargrove LJ, et al), sliding windows were first used to extract features from EMG and mechanical sensor signals respectively, and then support vector machine (SVM) was used for feature-level fusion to achieve recognition of motion mode and motion mode transition, with better results than single-modal sensors. In the paper "A novel HMM distributed classifier for the detection of gait phases by means of a wearable inertial sensor network" (Sensors, 2014, 14(9): 16212-16234, Taborri J, Rossi S, Palermo E, et al), a hidden Markov model (HMM) was used to perform decision-level fusion of three IMU data to achieve recognition of four gait phases.In the paper “A low-cost end-to-end sEMG-based gait sub-phase recognition system” (IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2019, 28(1): 267-276, Luo R, Sun S, Zhang X, et al), a long short-term memory (LSTM) neural network was used to extract features from four EMG sensor signals respectively, and then a multi-layer perceptron (MLP) was used to fuse the features to realize the recognition of the four gait phases.

[0008] However, the methods in the literatures "An extended kalman filter to estimate human gait parameters and walking distance", "A novel HMM distributed classifier for the detection of gait phases by means of a wearable inertial sensor network" and "A low-cost end-to-end sEMG-based gait sub-phase recognition system" all fuse multiple sensors of the same type, without considering multimodal sensor fusion; the method in the literature "Continuous locomotion-mode identification for prosthetic legs based on neuromuscular–mechanical fusion" requires manual extraction of multiple time domain and frequency domain features, which is relatively complex to calculate and needs to solve the problem of feature selection; and the LSTM network used in the literature "A low-cost end-to-end sEMG-based gait sub-phase recognition system" is a deep neural network with high training and computational costs. In complex environments, in order to achieve precise control of lower limb wearable robots, it is often necessary to identify multiple lower limb movement intentions, such as simultaneously identifying a person's gait phase and movement pattern, while the above methods can only identify a single category of lower limb movement intentions and cannot utilize the complementary effects between different movement intentions. Summary of the invention

[0009] In view of the above problems, the present invention provides a multi-task gated fuzzy neural network algorithm and a storage medium that overcome the above problems or at least partially solve the above problems.

[0010] In order to solve the above technical problems, the present invention provides a multi-task gated fuzzy neural network algorithm, which includes the following steps:

[0011] Build the main network and the gating network;

[0012] Get sensor data;

[0013] Get fuzzy rules;

[0014] Calculate the main network membership and the gated network membership;

[0015] Calculate the main network trigger strength and the gated network trigger strength;

[0016] Calculate the normalized trigger intensity of the main network and the normalized trigger intensity of the gated network;

[0017] Output according to the fuzzy rule and the main network normalized trigger strength calculation rule;

[0018] Calculating a gating network output according to the gating network normalized trigger strength;

[0019] Calculating a gated weighted output according to the gated network output and the rule output;

[0020] Calculating task classification result output according to the gated weighted output;

[0021] Get the target function;

[0022] The objective function is used to optimize the task classification result output.

[0023] Preferably, the expression of the primary network membership is:

[0024]

[0025] Among them, μ r,d (x d ) represents the main network membership, x d represents the d-th dimension sensor data, c r,d represents the central value of the Gaussian membership function, σ r,d represents the standard deviation of the Gaussian membership function, c r,d and σ r,d is the antecedent parameter of the fuzzy neural network.

[0026] Preferably, the expression of the primary network trigger strength is:

[0027]

[0028] Among them, f r (x) represents the trigger strength of the rth fuzzy rule in the main network, μ r,d (x d ) represents the main network membership function, x d represents the d-th dimension sensor data, c r,d represents the central value of the Gaussian membership function, σ r,d represents the standard deviation of the Gaussian membership function, and D represents the total dimension of the sensor data.

[0029] Preferably, the expression of the normalized trigger strength of the main network is:

[0030]

[0031] in, represents the normalized trigger strength of the rth fuzzy rule in the main network, f r (x) represents the triggering strength of the rth fuzzy rule in the main network, f i (x) represents the triggering strength of the i-th fuzzy rule, and R represents the number of fuzzy rules.

[0032] Preferably, the expression output by the rule is:

[0033]

[0034] Among them, p r (x) represents the fuzzy rule output, represents the normalized trigger strength of the main network, y(x) represents the consequent output of the fuzzy rule, and a r,o Indicates deviation, a r,d represents the weight, x d represents d-dimensional sensor data, and D represents the dimension of the sensor data.

[0035] Preferably, the expression of the gating network output is:

[0036]

[0037] in, represents the weight of the gating network output, represents the normalized trigger strength of the gating network.

[0038] Preferably, the expression of the gated weighted output is:

[0039]

[0040] in, represents the gated weighted output of the main network, represents the weight of the gating network output, p r (x) represents the fuzzy rule output.

[0041] Preferably, the expression of the task classification result output is:

[0042]

[0043] Among them, y k Represents the final classification result output of the task, represents the gated weighted output of the main network, and R represents the number of fuzzy rules.

[0044] Preferably, the objective function is expressed as:

[0045] O=c1L1+c2L2+…+c k L k ,

[0046] Among them, O represents the objective function, c1, c2, … c k Represent the weights of k tasks, L1, L2, ... L k They represent the cross entropy loss functions of k tasks respectively.

[0047] The present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute any of the aforementioned multi-task gated fuzzy neural network algorithms.

[0048] One or more technical solutions in the embodiments of the present invention have at least the following technical effects or advantages: a multi-task gated fuzzy neural network algorithm and storage medium provided by the present application can fuse multi-modal sensor information and directly identify the movement intention of human lower limbs, which is simpler and more efficient than other methods of first fusion and then identification; a multi-task gated fuzzy neural network is proposed for the first time, which can simultaneously identify two lower limb movement intentions of human lower limb movement pattern and gait phase; the fuzzy neural network can automatically extract fuzzy features through fuzzy rules, without the need to perform feature extraction and feature selection on sensor data in advance, and can be used for data-level multi-modal sensor fusion; compared with deep neural networks, the multi-task gated fuzzy neural network algorithm provided by the present application has fewer network parameters and high training and computing efficiency; it can be applied to other pattern recognition problems and is a machine learning algorithm with strong versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 It is a main network structure diagram in a multi-task gated fuzzy neural network algorithm provided by an embodiment of the present invention;

[0051] Figure 2 It is a gating network structure diagram in a multi-task gating fuzzy neural network algorithm provided by an embodiment of the present invention;

[0052] Figure 3 It is a motion pattern verification schematic diagram of a multi-task gated fuzzy neural network algorithm provided by an embodiment of the present invention;

[0053] Figure 4 It is a schematic diagram of gait phase verification of a multi-task gated fuzzy neural network algorithm provided by an embodiment of the present invention;

[0054] Figure 5 It is a schematic diagram of experimental results of a multi-task gated fuzzy neural network algorithm provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention will be described in detail below in conjunction with specific implementations and examples, and the advantages and various effects of the present invention will be more clearly presented. It should be understood by those skilled in the art that these specific implementations and examples are used to illustrate the present invention, rather than to limit the present invention.

[0056] Throughout the specification, unless otherwise specifically stated, the terms used herein should be understood as meanings commonly used in the art. Therefore, unless otherwise defined, all technical and scientific terms used herein have the same meanings as those generally understood by those skilled in the art to which the present invention belongs. In the event of a conflict, the present specification takes precedence.

[0057] Unless otherwise specified, various raw materials, reagents, instruments and equipment used in the present invention can be purchased from the market or prepared by existing methods.

[0058] In an embodiment of the present application, the present invention provides a multi-task gated fuzzy neural network algorithm, the algorithm comprising the steps of:

[0059] S1: build the main network and gating network;

[0060] In the embodiments of the present application, Figure 1The main network of the present invention is shown, and the main network has 8 layers, wherein the input layer is D-dimensional sensor data, and the output layer is the classification result of the kth task.

[0061] In the embodiments of the present application, Figure 2 The gating network of the present invention is shown, and the gating network has 5 layers, wherein the input layer is the D-dimensional sensor data, and the output layer is the weight vector of the k-th task relative to the fuzzy rules of the main network.

[0062] S2: Get sensor data;

[0063] In the embodiment of the present application, the sensor data can be obtained through various sensor outputs.

[0064] S3: Obtain fuzzy rules;

[0065] In the embodiment of the present application, it is assumed that the rule base has R fuzzy rules, and the expression of the rth fuzzy rule is:

[0066] IF x1 is A r,1 Λ…Λx D is A r,D

[0067]

[0068] Where x=(x1,…x d ,…x D ) is the D-dimensional sensor data.

[0069] S4: Calculate the main network membership and the gated network membership;

[0070] In the embodiment of the present application, the expression of the primary network membership is:

[0071]

[0072] Among them, μ r,d (x d ) represents the main network membership, x d represents the d-th dimension sensor data, c r,d represents the central value of the Gaussian membership function, σ r,d represents the standard deviation of the Gaussian membership function, c r,d and σ r,d is the antecedent parameter of the fuzzy neural network.

[0073] In the embodiment of the present application, the expression of the gated network membership is:

[0074]

[0075] in, represents the kth gating network membership, x d represents the d-th dimension sensor data, represents the central value of the Gaussian membership function of the k-th gated network, represents the standard deviation of the Gaussian membership function of the k-th gating network.

[0076] S5: Calculate the main network trigger strength and the gated network trigger strength;

[0077] In the embodiment of the present application, the expression of the primary network trigger strength is:

[0078]

[0079] Among them, f r (x) represents the trigger strength of the rth fuzzy rule in the main network, μ r,d (x d ) represents the main network membership function, x d represents the d-th dimension sensor data, c r,d represents the central value of the Gaussian membership function, σ r,d represents the standard deviation of the Gaussian membership function, and D represents the total dimension of the sensor data.

[0080] In the embodiment of the present application, the expression of the gated network trigger strength is:

[0081]

[0082] in, represents the triggering strength of the rth fuzzy rule in the kth gating network, represents the membership function of the kth gating network, x d represents the d-th dimension sensor data, represents the central value of the Gaussian membership function of the k-th gated network, represents the standard deviation of the Gaussian membership function of the k-th gating network, and D represents the total dimension of the sensor data.

[0083] S6: Calculate the normalized trigger intensity of the main network and the normalized trigger intensity of the gated network;

[0084] In the embodiment of the present application, the expression of the normalized trigger strength of the main network is:

[0085]

[0086] in, represents the normalized trigger strength of the rth fuzzy rule in the main network, f r (x) represents the triggering strength of the rth fuzzy rule in the main network, f i(x) represents the triggering strength of the i-th fuzzy rule, and R represents the number of fuzzy rules.

[0087] In the embodiment of the present application, the expression of the normalized trigger strength of the gated network is:

[0088]

[0089] in, represents the normalized trigger strength of the rth fuzzy rule in the kth gating network, represents the triggering strength of the rth fuzzy rule in the kth gating network, represents the trigger strength of the i-th fuzzy rule in the k-th gating network, x d represents the d-th dimension sensor data, represents the central value of the Gaussian membership function of the rth fuzzy rule of the kth gating network, represents the central value of the Gaussian membership function of the i-th fuzzy rule of the k-th gating network, represents the standard deviation of the Gaussian membership function of the rth fuzzy rule of the kth gating network, represents the standard deviation of the Gaussian membership function of the i-th fuzzy rule of the k-th gating network, and D represents the total dimension of the sensor data.

[0090] S7: Output according to the fuzzy rule and the main network normalized trigger strength calculation rule;

[0091] In the embodiment of the present application, the expression output by the rule is:

[0092]

[0093] Among them, p r (x) represents the fuzzy rule output, represents the normalized trigger strength of the main network, y(x) represents the consequent output of the fuzzy rule, and a r,o Indicates deviation, a r,d represents the weight, x d represents d-dimensional sensor data, and D represents the dimension of the sensor data.

[0094] S8: Calculating a gated network output according to the normalized trigger strength of the gated network;

[0095] In the embodiment of the present application, the expression of the gating network output is:

[0096]

[0097] in, represents the weight of the gating network output, represents the normalized trigger strength of the gating network.

[0098] S9: Calculating a gated weighted output according to the gated network output and the rule output;

[0099] In the embodiment of the present application, the expression of the gated weighted output is:

[0100]

[0101] in, represents the gated weighted output of the main network, represents the weight of the gating network output, p r (x) represents the fuzzy rule output.

[0102] S10: Outputting a task classification result according to the gated weighted output calculation;

[0103] In the embodiment of the present application, the expression of the task classification result output is:

[0104]

[0105] Among them, y k Represents the final classification result output of the task, represents the gated weighted output of the main network, and R represents the number of fuzzy rules.

[0106] S11: Obtaining the target function;

[0107] In the embodiment of the present application, the expression of the objective function is:

[0108] O=c1L1+c2L2+…+c k L k ,

[0109] Among them, O represents the objective function, c1, c2, … c k Represent the weights of k tasks, L1, L2, ... L k They represent the cross entropy loss functions of k tasks respectively.

[0110] S12: Using the objective function to optimize the task classification result output.

[0111] In the embodiment of the present application, the Adam learner is used to calculate the model parameters of the multi-task fuzzy neural network through the small batch stochastic gradient descent method, and the objective function is used to optimize the output of the task classification result.

[0112] A multi-task gated fuzzy neural network algorithm provided by the present application can fuse multimodal sensor information and directly identify the human lower limb movement intention, which is simpler and more efficient than other methods of first fusion and then recognition. The algorithm can simultaneously recognize multiple movement intentions. Different movement intention recognition tasks have both shared knowledge learned through the main network and individual knowledge learned from the gated network. Therefore, the complementary effects between different movement intentions can be utilized to more accurately identify the lower limb movement intention. In addition, the training and computational efficiency of the algorithm is high, which is convenient for online real-time recognition and can also be applied to other pattern recognition problems.

[0113] A multi-task gated fuzzy neural network algorithm provided by this application is verified below.

[0114] In the embodiment of the present application, a multi-task gated fuzzy neural network algorithm provided by the present application is used to fuse the muscle leg ring sensor, IMU and plantar pressure sensor, and simultaneously identify nine common motion modes (standing, horizontal walking, climbing stairs, descending stairs, uphill, downhill, running, jumping, falling, see attached Figure 3 ) and four gait phases (left leg swing, double support (center of gravity right→left), right leg swing, double support (center of gravity left→right), see attached Figure 4 ).

[0115] like Figure 5 ,The experimental results show that the multi-task gated fuzzy neural network algorithm provided in ,this application has reached the international advanced level in the recognition of ,motion patterns and gait phases.

[0116] The present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute any of the aforementioned multi-task gated fuzzy neural network algorithms.

[0117] The present application provides a multi-task gated fuzzy neural network algorithm and storage medium, which can fuse multi-modal sensor information and directly identify the movement intention of human lower limbs. Compared with other methods of first fusion and then identification, it is simpler and more efficient; a multi-task gated fuzzy neural network is proposed for the first time, which can simultaneously identify two lower limb movement intentions of human lower limb movement pattern and gait phase; the fuzzy neural network can automatically extract fuzzy features through fuzzy rules, without the need to extract and select features from sensor data in advance, and can be used for data-level multi-modal sensor fusion; compared with deep neural networks, the multi-task gated fuzzy neural network algorithm provided by the present application has fewer network parameters and high training and computing efficiency; it can be applied to other pattern recognition problems and is a machine learning algorithm with strong versatility.

[0118] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. The various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features applied for herein.

[0119] In short, the above is only a preferred embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-task gated fuzzy neural network algorithm, characterized in that: The algorithm comprises the steps of: Construct a main network and a gating network, wherein the main network has 8 layers in total, the input layer of the main network is D-dimensional sensor data, and the output layer of the main network is the classification result of the k-th task; the gating network has 5 layers in total, the input layer of the gating network is D-dimensional sensor data, and the output layer of the gating network is the weight vector of the k-th task relative to the fuzzy rules of the main network; Get sensor data; Get fuzzy rules; Calculate the main network membership and the gated network membership; Calculate the main network trigger strength and the gated network trigger strength; Calculate the normalized trigger intensity of the main network and the normalized trigger intensity of the gated network; Outputting the consequent of the fuzzy rule and the normalized trigger strength calculation rule of the main network; Calculating a gating network output according to the gating network normalized trigger strength; Calculating a gated weighted output according to the gated network output and the rule output; Calculating task classification result output according to the gated weighted output; Get the target function; Using the objective function to optimize the task classification result output; Among them, the multi-task gated fuzzy neural network algorithm is used to fuse the myoelectric leg ring sensor, IMU and plantar pressure sensor to simultaneously identify nine movement modes and four gait phases. The nine movement modes include standing, horizontal walking, climbing stairs, descending stairs, going uphill, going downhill, running, jumping and falling; the four gait phases include left leg swinging, double support from right to left center of gravity, right leg swinging, double support from left to right center of gravity.

2. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression of the main network membership is: Among them, μ r,d (x d ) represents the main network membership, x d represents the d-th dimension sensor data, c r,d represents the central value of the Gaussian membership function, σ r,d represents the standard deviation of the Gaussian membership function, c r,d and σ r,d is the antecedent parameter of the fuzzy neural network.

3. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression of the main network trigger strength is: Among them, f r (x) represents the trigger strength of the rth fuzzy rule in the main network, μ r,d (x d ) represents the main network membership function, x d represents the d-th dimension sensor data, c r,d represents the central value of the Gaussian membership function, σ r,d represents the standard deviation of the Gaussian membership function, and D represents the total dimension of the sensor data.

4. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression of the normalized trigger strength of the main network is: in, represents the normalized trigger strength of the rth fuzzy rule in the main network, f r (x) represents the triggering strength of the rth fuzzy rule in the main network, f i (x) represents the triggering strength of the i-th fuzzy rule, and R represents the number of fuzzy rules.

5. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression output by the rule is: Among them, p r (x) represents the fuzzy rule output, represents the normalized trigger strength of the main network, y(x) represents the consequent output of the fuzzy rule, and a r,o Indicates deviation, a r,d represents the weight, x d represents d-dimensional sensor data, and D represents the dimension of the sensor data.

6. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression of the gating network output is: in, represents the weight of the gating network output, Represents the normalized trigger strength of the gating network.

7. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression of the gated weighted output is: in, represents the gated weighted output of the main network, represents the weight of the gating network output, p r (x) represents the fuzzy rule output.

8. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression of the task classification result output is: Among them, y k Represents the final classification result output of the task, represents the gated weighted output of the main network, and R represents the number of fuzzy rules.

9. The multi-task gated fuzzy neural network algorithm according to claim 1, characterized in that: The expression of the objective function is: O=c1L1+c2L2+…+c k L k , Among them, O represents the objective function, c1, c2, … c k Represent the weights of k tasks, L1, L2, ... L k They represent the cross entropy loss functions of k tasks respectively.

10. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the multi-task gated fuzzy neural network algorithm of any one of claims 1 to 9.

Images