Human body gait recognition method and device based on multi-modal fusion

Abstract

The invention discloses a human body gait recognition method and device based on multi-modal fusion, and belongs to the field of human body gait assessment, and the method comprises the steps: collecting electromyographic signals of muscles at different movement positions of lower limbs; collecting near-infrared brain function imaging data of the brain in a gait image and an idle state; collecting pressure information of the planta pedis in the movement process; the electromyographic signals, the near-infrared brain function imaging data and the plantar pressure information are input into a multi-modal fusion model, feature extraction is conducted on the electromyographic signals, the near-infrared brain function imaging data and the plantar pressure information, and different features are fused through a cross attention mechanism; the classification algorithm identifies the fused features and performs gait evaluation, and as the electromyographic signals, the pressure information of the planta pedis and the near-infrared brain function imaging data are synchronously generated with the human body movement, the human body gait identified through the data is low in hysteresis quality and high in accuracy.

Description

Technical Field

[0001] The present invention relates to the field of human gait assessment, and in particular to a human gait recognition method and device based on multimodal fusion. Background Art

[0002] With the development of brain-computer interfaces, motion recognition algorithms for exoskeleton robots are emerging one after another. Most of the current human motion perception technologies use human posture information as the data source, such as joint angles, angular velocities or joint torques, but this data source is generated at the same time as the motion and there is no lead time. There is a certain lag in the recognition of the wearer's motion, which leads to the exoskeleton robot's prediction of human motion being too dependent on the maturity of the prediction algorithm. Therefore, the accuracy of human motion pattern recognition is still not high, the exoskeleton robot's sensor system is not very accurate in collecting data from the wearer, and there is still room for improvement in the analysis and extraction of the wearer's motion information. Summary of the invention

[0003] In order to overcome the deficiencies of the prior art, one of the objectives of the present invention is to provide a human gait recognition method with high recognition accuracy and low hysteresis.

[0004] In order to overcome the deficiencies of the prior art, a second object of the present invention is to provide a human gait recognition device with high recognition accuracy and low hysteresis.

[0005] One of the purposes of the present invention is achieved by the following technical solution:

[0006] A human gait recognition method based on multimodal fusion comprises the following steps:

[0007] Collect electromyographic signals of muscles in different movement positions of lower limbs;

[0008] Collect near-infrared brain functional imaging data of the brain in gait images and idle state;

[0009] Collect pressure information on the sole of the foot during exercise;

[0010] The electromyographic signals, near-infrared brain functional imaging data and plantar pressure information are input into the multimodal fusion model, and features of the electromyographic signals, near-infrared brain functional imaging data and plantar pressure information are extracted respectively. The different features are fused using the cross-attention mechanism. The classification algorithm recognizes the fused features and performs gait evaluation.

[0011] Furthermore, the electromyographic signal is collected by a multi-channel array flexible electromyographic collection sensor, and the multi-channel array flexible electromyographic collection sensor is provided with eight channels.

[0012] Furthermore, the multi-channel array flexible electromyographic acquisition sensor collects electromyographic signals by being attached to leg muscles, and the leg muscles include the tensor fasciae latae, adductor longus, rectus femoris, vastus medialis, tibialis anterior, semitendinosus, gastrocnemius and soleus.

[0013] Furthermore, feature extraction is performed on the near-infrared brain functional imaging data by calculating the average value, peak value and root mean square of hemoglobin based on the near-infrared brain functional imaging data, and taking the average value, peak value, root mean square of hemoglobin and a combination of at least two data as extracted features.

[0014] Furthermore, the classification algorithm is any one of BP neural network, AdaBoost algorithm and SVM algorithm.

[0015] Furthermore, the gait includes standing still, walking, squatting, standing up, climbing stairs, descending stairs, going uphill, going downhill, and joint movement.

[0016] The second object of the present invention is achieved by adopting the following technical solution:

[0017] A human gait recognition device based on multimodal fusion is used to implement any of the above-mentioned human gait recognition methods based on multimodal fusion. The human gait recognition device based on multimodal fusion includes a multi-channel array flexible electromyography acquisition sensor, a near-infrared brain imaging device, a plantar array pressure sensor and a processor. The multi-channel array flexible electromyography acquisition sensor, the near-infrared brain imaging device and the plantar array pressure sensor are all communicatively connected to the processor. The multi-channel array flexible electromyography acquisition sensor collects electromyographic signals of muscles in different movement positions of the lower limbs. The near-infrared brain imaging device collects near-infrared brain function imaging data of the brain in gait images and idle states. The plantar array pressure sensor collects pressure information of the plantar during movement. The processor analyzes the electromyographic signals, near-infrared brain function imaging data and plantar pressure information and determines the gait.

[0018] Furthermore, the multi-channel array flexible electromyography acquisition sensor is provided with eight channels.

[0019] Furthermore, the multi-channel array flexible electromyography acquisition sensor includes an adhesive tape, a plurality of channels arranged on the adhesive tape, and a line connected to each of the channels.

[0020] Furthermore, the circuit is a silver chloride circuit.

[0021] Compared with the prior art, the human gait recognition method based on multimodal fusion of the present invention collects the electromyographic signals of muscles in different movement positions of the lower limbs; collects near-infrared brain function imaging data of the brain in gait images and idle states; collects pressure information of the soles of the feet during movement; inputs the electromyographic signals, near-infrared brain function imaging data and plantar pressure information into a multimodal fusion model, extracts features of the electromyographic signals, near-infrared brain function imaging data and plantar pressure information respectively, and fuses different features using a cross-attention mechanism; a classification algorithm identifies the fused features and performs gait evaluation. Since the electromyographic signals, plantar pressure information and near-infrared brain function imaging data are generated synchronously with human movement, the gait of the human body can be identified through the above data with low lag and high accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of a multi-channel array flexible electromyography acquisition sensor;

[0023] Figure 2 This is a schematic diagram of the attachment position of the multi-channel array flexible electromyography acquisition sensor;

[0024] Figure 3 Collecting distribution locations for near-infrared brain imaging devices;

[0025] Figure 4 This is a schematic diagram of collecting pressure information on the sole of the foot;

[0026] Figure 5 Schematic diagram of the gait recognition process.

[0027] In the figure: 11, tape; 12, hydrogel; 13, line; 14, channel. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] It should be noted that when a component is referred to as being "fixed to" another component, it may be directly on the other component or there may be another intermediate component through which it is fixed. When a component is considered to be "connected to" another component, it may be directly connected to the other component or there may be another intermediate component at the same time. When a component is considered to be "set on" another component, it may be directly set on the other component or there may be another intermediate component at the same time. The terms "vertical", "horizontal", "left", "right" and similar expressions used herein are for illustrative purposes only.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0031] The present invention provides a human gait recognition method based on multimodal fusion, comprising the following steps:

[0032] Collect electromyographic signals of muscles in different movement positions of lower limbs;

[0033] Collect near-infrared brain functional imaging data of the brain in gait images and idle state;

[0034] Collect pressure information on the sole of the foot during exercise;

[0035] The electromyographic signals, near-infrared brain functional imaging data and plantar pressure information are input into the multimodal fusion model, and features of the electromyographic signals, near-infrared brain functional imaging data and plantar pressure information are extracted respectively. The different features are fused using the cross-attention mechanism. The classification algorithm recognizes the fused features and performs gait evaluation.

[0036] Specifically, in the step of collecting the electromyographic signals of muscles at different movement positions of the lower limbs, since the range of muscles at the movement position is often relatively large, it is impossible to accurately detect the movement signal by a single electrode. In this application, a multi-channel array flexible electromyographic acquisition sensor is used for collection. In this embodiment, the multi-channel array flexible electromyographic acquisition sensor is provided with 8 channels. The structure of the multi-channel array flexible electromyographic acquisition sensor is as follows: Figure 1 As shown, the multi-channel array flexible electromyographic acquisition sensor includes an adhesive tape 11, 8 channels 14 arranged on the adhesive tape 11, a hydrogel 12 for fixing, and a line 13 connecting each channel. The adhesive tape 11 is a medical adhesive tape. The 8 channels respectively collect electromyographic signals of the tensor fasciae latae, adductor longus, rectus femoris, vastus medialis, tibialis anterior, semitendinosus, gastrocnemius, and soleus muscles. The attachment position of the multi-channel array flexible electromyographic acquisition sensor is as shown in FIG. Figure 2 shown.

[0037] The near-infrared brain functional imaging information acquisition is specifically: the brain-computer interface driven by functional near-infrared spectroscopy (fNIRS) gait image has potential application prospects in the rehabilitation training of lower limb motor dysfunction. This application collects the fNIRS signals of the subjects in gait images and idle states. The average, peak and root mean square of hemoglobin (HbO) and their combinations are calculated as classification features. For the active brain areas of the lower limbs, the specific acquisition locations are as follows: Figure 3 shown.

[0038] The plantar pressure information collection is specifically as follows: a plantar array pressure sensor is installed on the sole of the subject's foot to detect the foot pressure information of the subject during exercise, and the plantar pressure array image is collected. Figure 4 shown.

[0039] This application adopts a multimodal fusion model to fuse array surface electromyography signals and human chest and abdomen point clouds to form fusion features. The multimodal fusion model includes a data input layer, a feature extraction layer, a feature fusion layer and a decision layer (evaluation layer).

[0040] The specific contents are as follows:

[0041] Data input layer: The first step of multimodal fusion is to input data from multiple sensors into the system. The data includes electromyographic data of lower limb muscles, near-infrared brain functional imaging data, and plantar pressure array information.

[0042] Feature extraction layer: In the feature extraction layer, the data of each sensor passes through the corresponding feature extraction module to extract useful features of the sensor data.

[0043] Feature fusion layer: In the feature fusion layer, the cross-attention mechanism is used to fuse features from different sensors to obtain more comprehensive and consistent environmental perception information.

[0044] Gait estimation: To address the problem of low recognition rate of lower limb gait by a single classifier, a BP neural network, AdaBoost algorithm or SVM algorithm is used for classification based on a parallel multi-source information fusion algorithm structure, and the classification system is used to classify typical knee joint movements. The specific schematic diagram is shown in the figure. Figure 5 The gaits include standing still, walking, squatting, standing up, climbing stairs, descending stairs, going uphill, going downhill, and joint movement, and the gait finally recognized is one of them.

[0045] The present application also discloses a human gait recognition device based on multimodal fusion, which is used to implement a human gait recognition method based on multimodal fusion. The human gait recognition device based on multimodal fusion includes a multi-channel array flexible electromyography acquisition sensor, a near-infrared brain imaging device, a plantar array pressure sensor and a processor. The multi-channel array flexible electromyography acquisition sensor, the near-infrared brain imaging device and the plantar array pressure sensor are all communicatively connected to the processor. The multi-channel array flexible electromyography acquisition sensor collects electromyography signals of muscles in different movement positions of the lower limbs. The near-infrared brain imaging device collects near-infrared brain function imaging data of the brain in gait images and idle states. The plantar array pressure sensor collects pressure information of the plantar during movement. The processor analyzes the electromyography signals, near-infrared brain function imaging data and plantar pressure information and determines the gait.

[0046] For details, see Figure 1 The multi-channel array flexible electromyography acquisition sensor is provided with eight channels 14 .

[0047] The multi-channel array flexible electromyographic acquisition sensor comprises an adhesive tape 11, a plurality of channels 14 arranged on the adhesive tape 11, and a circuit 13 connected to each channel. The circuit is a silver chloride circuit.

[0048] Compared with the prior art, the human gait recognition method based on multimodal fusion of the present invention collects the electromyographic signals of muscles in different movement positions of the lower limbs; collects near-infrared brain function imaging data of the brain in gait images and idle states; collects pressure information of the soles of the feet during movement; inputs the electromyographic signals, near-infrared brain function imaging data and plantar pressure information into a multimodal fusion model, extracts features of the electromyographic signals, near-infrared brain function imaging data and plantar pressure information respectively, and fuses different features using a cross-attention mechanism; a classification algorithm identifies the fused features and performs gait evaluation. Since the electromyographic signals, plantar pressure information and near-infrared brain function imaging data are generated synchronously with human movement, the gait of the human body can be identified through the above data with low lag and high accuracy.

[0049] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which are equivalent modifications and improvements made to the above embodiments based on the essential technology of the present invention, and all of them belong to the protection scope of the present invention.

Claims

1. A human gait recognition method based on multimodal fusion, characterized in that: The following steps are involved: Collect electromyographic signals of muscles in different movement positions of lower limbs; Collect near-infrared brain functional imaging data of the brain in gait images and idle state; Collect pressure information on the sole of the foot during exercise; The electromyographic signals, near-infrared brain functional imaging data and plantar pressure information are input into the multimodal fusion model, and the features of the electromyographic signals, near-infrared brain functional imaging data and plantar pressure information are extracted respectively, and the cross-attention mechanism is used to fuse different features; The classification algorithm recognizes the fused features and performs gait assessment.

2. The method for human gait recognition based on multimodal fusion according to claim 1, characterized in that: The electromyographic signal is collected by a multi-channel array flexible electromyographic collection sensor, and the multi-channel array flexible electromyographic collection sensor is provided with eight channels.

3. The human gait recognition method based on multimodal fusion according to claim 2, characterized in that: The multi-channel array flexible electromyographic acquisition sensor acquires electromyographic signals by being attached to leg muscles, and the leg muscles include the tensor fasciae latae, adductor longus, rectus femoris, vastus medialis, tibialis anterior, semitendinosus, gastrocnemius and soleus.

4. The method for human gait recognition based on multimodal fusion according to claim 1, characterized in that: The feature extraction of the near-infrared brain functional imaging data is specifically as follows: the average value, peak value and root mean square of hemoglobin are calculated according to the near-infrared brain functional imaging data, and the average value, peak value, root mean square of hemoglobin and a combination of at least two data are used as extracted features.

5. The method for human gait recognition based on multimodal fusion according to claim 1, characterized in that: The classification algorithm is any one of BP neural network, AdaBoost algorithm and SVM algorithm.

6. The method for human gait recognition based on multimodal fusion according to claim 1, characterized in that: The gait includes standing still, walking, squatting, standing up, climbing stairs, descending stairs, going uphill, going downhill, and joint movement.

7. A human gait recognition device based on multimodal fusion, used to implement the human gait recognition method based on multimodal fusion as described in any one of claims 1-6, the human gait recognition device based on multimodal fusion comprises a multi-channel array flexible electromyography acquisition sensor, a near-infrared brain imaging device, a plantar array pressure sensor and a processor, the multi-channel array flexible electromyography acquisition sensor, the near-infrared brain imaging device and the plantar array pressure sensor are all communicatively connected to the processor, the multi-channel array flexible electromyography acquisition sensor acquires electromyographic signals of muscles in different movement positions of the lower limbs, the near-infrared brain imaging device acquires near-infrared brain function imaging data of the brain in gait images and idle states, the plantar array pressure sensor acquires pressure information of the plantar during movement, and the processor analyzes the electromyographic signals, near-infrared brain function imaging data and plantar pressure information and determines the gait.

8. The human gait recognition device based on multimodal fusion according to claim 7, characterized in that: The multi-channel array flexible electromyography acquisition sensor is provided with eight channels.

9. The human gait recognition device based on multimodal fusion according to claim 7, characterized in that: The multi-channel array flexible electromyography acquisition sensor comprises an adhesive tape, a plurality of channels arranged on the adhesive tape, and a line connected to each of the channels.

10. The human gait recognition device based on multimodal fusion according to claim 9, characterized in that: The circuit is a silver chloride circuit.

Images