A gait recognition and classification method, device, electronic equipment and storage medium

Abstract

This invention relates to a gait recognition and classification method, apparatus, electronic device, and storage medium. The gait recognition and classification method of this invention includes: preprocessing an image sequence from a gait video to obtain a skeleton energy image corresponding to the gait video; extracting features from the skeleton energy image to obtain a feature vector corresponding to the skeleton energy image; and classifying the feature vector using a trained classification network to obtain a classification result corresponding to the gait video; the classification result includes normal, first hemiplegic gait, second hemiplegic gait, and third hemiplegic gait. The gait recognition and classification method of this invention, based on gait analysis of video, provides automatic and even portable gait detection and classification to assist in assessing stroke recovery progress.

Description

Technical Field

[0001] This invention relates to the field of gait recognition technology, and in particular to a gait recognition and classification method, apparatus, electronic device, and storage medium. Background Technology

[0002] Stroke, caused by obstruction of blood flow or hemorrhage in the brain, can lead to hemiplegia, severely hindering a patient's ability to care for themselves in daily life. Stroke rehabilitation is a lengthy process requiring significant medical and financial resources. To alleviate the burden on patients and their families, it is essential to establish a system capable of automatically assessing stroke severity to aid in evaluating recovery progress. Stroke-induced hemiplegia can cause gait dysfunction (also known as hemiplegic gait), characterized by spasticity and gait instability. Therefore, by observing and analyzing gait abnormalities in patients, this application can obtain information related to rehabilitation progress.

[0003] Patient gait information can be collected in two ways: wearable-based and video-based. Compared to wearable systems that require sensors (such as accelerometers) to be installed on various parts of the body, video-based systems can acquire data through a two-dimensional camera without requiring any patient cooperation and can be implemented conveniently in the home. Considering patient comfort, this application focuses on constructing a video-based automatic assessment system for hemiplegic gait.

[0004] However, state-of-the-art deep learning algorithms for gait analysis typically suffer from high computational complexity, making it difficult to deploy neural network models on mobile or embedded devices with limited computing power. They may even overfit due to the limited data in small-scale pathological datasets. Specifically, vision-based gait analysis algorithms can be broadly categorized into 3D CNN-based and 2D CNN-based frameworks. 3D CNNs have proven effective at capturing spatiotemporal features from videos, but suffer from redundant model parameters and overfitting issues. 2D CNN-based methods alleviate the problem of excessive parameters in 3D CNNs. 2D CNNs are further divided into sequence-based and template-based methods based on the input modality. Clearly, compared to sequence-based methods, template-based methods have the advantage of building efficient algorithms due to their compact input modality. However, most template-based features are always based on silhouettes, such as gait energy images (GEI), which are easily affected by covariant factors such as clothing condition. Summary of the Invention

[0005] Based on this, the purpose of the present invention is to provide a gait recognition and classification method, device, electronic device and storage medium, which provides automatic or even portable gait detection and classification based on video gait analysis to assist in judging the progress of stroke recovery.

[0006] In a first aspect, the present invention provides a gait recognition and classification method, comprising the following steps:

[0007] The image sequence in the gait video is preprocessed to obtain the skeleton energy image corresponding to the gait video;

[0008] Feature extraction is performed on the skeleton energy image to obtain the feature vector corresponding to the skeleton energy image;

[0009] The trained classification network is used to classify the feature vectors to obtain the classification results corresponding to the gait videos; the classification results include normal, first hemiplegic gait, second hemiplegic gait and third hemiplegic gait.

[0010] Furthermore, the image sequences in the gait video are preprocessed, including the following steps:

[0011] Skeleton information is extracted from the image sequence in the gait video to obtain the first single-frame skeleton image;

[0012] The first single-frame skeleton image is cropped and scaled to obtain a second single-frame skeleton image with a preset size and resolution, where the skeleton information is located at the center of the image.

[0013] Several independent gait cycles are extracted through gait cycle detection;

[0014] For each gait cycle, the skeletal energy image is calculated using the following formula:

[0015]

[0016] Among them, I t (x,y) represents the second single-frame skeleton image, and N represents the number of images per gait cycle.

[0017] Furthermore, the Efficient Ghostnet neural network is used to extract features and classify the skeleton energy image;

[0018] The Efficient Ghostnet neural network comprises 5 blocks, each block containing several EG-necks and / or EG-necks with attention.

[0019] Furthermore, the EG-neck includes a first EGM module, a DWConv module, an LHAM module, and a second EGM module connected in sequence.

[0020] Furthermore, the EG-neck includes a first EGM module, an LHAM module, and a second EGM module connected in sequence.

[0021] Furthermore, the LHAM module includes a global average pooling module and a spatial attention module, wherein the spatial attention module includes a pooling operation and a convolution operation.

[0022] Furthermore, the EGM module includes a grouped convolution part, a depthwise convolution part, a linear concatenation part, and an output part.

[0023] Secondly, the present invention also provides a gait recognition and classification device, comprising:

[0024] The preprocessing module is used to preprocess the image sequence in the gait video to obtain the skeleton energy image corresponding to the gait video;

[0025] The feature vector extraction module is used to extract features from the skeleton energy image to obtain the feature vector corresponding to the skeleton energy image.

[0026] The classification module is used to classify the feature vector using a trained classification network to obtain the classification result corresponding to the gait video; the classification result includes normal, first hemiplegic gait, second hemiplegic gait and third hemiplegic gait.

[0027] Thirdly, the present invention also provides an electronic device, comprising:

[0028] At least one memory and at least one processor;

[0029] The memory is used to store one or more programs;

[0030] When the one or more programs are executed by the at least one processor, the at least one processor performs the steps of a gait recognition and classification method as described in any of the first aspects of the present invention.

[0031] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a gait recognition and classification method as described in any of the first aspects of the present invention.

[0032] This invention provides a gait recognition and classification method, device, electronic device, and storage medium. It proposes a novel lightweight network to efficiently extract pathological gait features, thereby accurately mapping the input to stroke gait of relevant severity. Extensive experiments on self-built and publicly available datasets demonstrate that the proposed attention-based lightweight CNN achieves effective and efficient gait analysis for stroke rehabilitation.

[0033] To better understand and implement this invention, the following detailed description is provided in conjunction with the accompanying drawings. Attached Figure Description

[0034] Figure 1 This is a schematic diagram illustrating the steps of a gait recognition and classification method provided by the present invention;

[0035] Figure 2 This is a schematic diagram of the automatic hemiplegic gait assessment process in one embodiment;

[0036] Figure 3 This is a schematic diagram of the SEI generation process in one embodiment;

[0037] Figure 4 A schematic diagram of the Efficient Ghostne structure is shown below.

[0038] Figure 5 This is a schematic diagram of the EG-neck module structure;

[0039] Figure 6 This is a schematic diagram of the LHAM module structure;

[0040] Figure 7 This is a schematic diagram of the EGM module structure;

[0041] Figure 8 This diagram illustrates the relationship between simulated hemiplegic gait of three different degrees of severity and the area of ​​lower limb affected by abnormal synergistic effects, with the affected area represented by a diagonal line.

[0042] Figure 9 Comparison of experimental results between the method provided in this application and the SOTA method;

[0043] Figure 10 Comparison of the ablation results of the attention module;

[0044] Figure 11 Comparison of EGM module ablation experimental results;

[0045] Figure 12 This is a schematic diagram of the structure of a gait recognition and classification device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0047] It should be understood that the described embodiments are merely some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of the embodiments of this application.

[0048] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0049] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims. In the description of this application, it should be understood that the terms "first," "second," "third," etc., are used only to distinguish similar objects and are not necessarily used to describe a specific order or sequence, nor should they be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0050] Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0051] To address the problems in the background technology, this application provides a gait recognition and classification method. In a practical application scenario, this method is executed independently by a user's smartphone, or by a smartphone connected to a remote server / computer via WiFi. Specifically, a relevant app is installed on the user's smartphone, which is responsible for using the smartphone's camera to capture the user's gait video. Then, the video is transmitted via WiFi and the TCP / IP protocol using Socket technology to a desktop device equipped with a preprocessor and gait evaluation model. After processing and analyzing the uploaded video, the desktop device replies to the smartphone app via WiFi, and the app displays the analysis results through a UI interface. In other examples, the smartphone can also directly process and analyze the gait video and display the analysis results on the app's UI interface.

[0052] like Figure 1 and Figure 2 As shown, the method includes the following steps:

[0053] S01: Preprocess the image sequence in the gait video to obtain the skeleton energy image corresponding to the gait video.

[0054] Most template-based features are based on silhouettes, such as gait energy images (GEI), which are easily affected by covariant factors such as clothing condition. Therefore, this application uses skeleton energy images (SEI) as the algorithm input. SEI eliminates the influence of clothing and the input is more concise.

[0055] like Figure 3 As shown, the skeleton is extracted from the input color image sequence using OpenPose (remote system) or Movenet (local system). Then, the images are cropped and scaled to a uniform resolution of 224x224, and the skeleton is centered. Finally, several independent gait cycles are extracted through gait cycle detection, and the SEI is calculated according to the following formula:

[0056]

[0057] Where I t (x,y) corresponds to a single frame image, and N is the number of images per gait cycle.

[0058] S02: Perform feature extraction on the skeleton energy image to obtain the feature vector corresponding to the skeleton energy image.

[0059] S03: Use the trained classification network to classify the feature vector to obtain the classification result corresponding to the gait video; the classification result includes normal, first hemiplegic gait, second hemiplegic gait and third hemiplegic gait.

[0060] Traditional deep learning algorithms for pathological gait analysis, with their small datasets, are often plagued by overfitting due to overly complex models. Therefore, this application explores a lightweight network and further proposes an attention module to help the network focus on more informative spatial and channel features. Among several lightweight CNN algorithms, Ghostnet was chosen as the benchmark because it strikes a good balance between effectiveness and efficiency. The improved model in this application is named Efficient Ghostnet, and its structure is as follows: Figure 4 As shown.

[0061] The entire network mainly consists of 5 blocks, each containing several EG bottleneck structures (Efficient Ghost bottleneck, EG-neck). This application provides two EG-neck structures, the difference being that one of them includes an additional DWConv (depth convolution) to achieve downsampling with stride=2, such as... Figure 5 As shown. EG-neck integrates several shortcuts, the lightweight hybrid attention module LHAM, and the efficient Ghost module (EGM), where EGM is a novel lightweight convolutional module.

[0062] Specifically, such as Figure 6 As shown, the Lightweight Hybrid Attention Module (LHAM) primarily assigns lightweight channel attention weights to features through adaptive one-dimensional convolution, while also allocating spatial attention through simple pooling and convolution operations.

[0063] LHAM is inserted into part of the EG-neck (e.g.) Figure 4 and Figure 5 As shown, EG-neck sections 4, 5, 10, 11, 12, 14, and 16. It was used to replace the SE module in the original Ghostnet. The first step was also to compress features along the channel dimension using Global Average Pooling (GAP). However, the SE module uses two fully connected layers for channel interaction, and both fully connected layers require multiple point-to-point operations related to the number of feature channels, resulting in a large number of parameters. Therefore, this application uses adaptive one-dimensional convolution to replace the fully connected layers. When this application uses a one-dimensional convolution with kernel size k to integrate channel interactions, the number of parameters generated by the channel interaction operation is equal to $k$, which should be proportional to the channel dimension C; therefore, k can be adaptively determined using the following formula.

[0064]

[0065] Here, `odd` indicates that the value of `k` is odd, and `γ` and `b` are two constants that can be set to control the ratio of `C` to `k`. Compared to the method of interacting with channel information through two fully connected layers in the SE module, the use of one-dimensional convolution significantly reduces the number of parameters due to the parameter sharing mechanism of convolution.

[0066] Furthermore, the attention mechanism in Ghostnet only explores attention along the channel dimension, neglecting the spatial dimension, which is crucial for identifying key regions in feature maps. Therefore, LHAM also incorporates spatial attention. Inspired by modules such as CBAM, spatial attention mainly consists of a pooling operation and a convolution operation. It performs max and average pooling operations on the spatial domain features and concatenates the two results before performing a convolution operation to obtain the attention weights. This module can extract attention along the spatial dimension without generating a large number of parameters. Compared to existing attention strategies, the attention module proposed in this application achieves a good balance between effectiveness and lightweight design.

[0067] The novel convolutional module proposed in this application, namely EGM, aims to perform convolution more efficiently. Therefore, its principle is to transform EGM into a completely "divide and conquer" convolutional module. The first step of this module is to compress features through grouped convolution, rather than removing redundant feature maps using traditional convolution. Grouped convolution divides the feature map into several groups by channel (4 groups in this patent's experiment) and performs convolution on the features of each group individually. This allows the kernel size to decrease as the number of groups increases, thereby reducing the number of parameters and operations. However, grouped convolution reduces channel interaction between different groups, or even eliminates it altogether. To address this issue, this application shuffles the convolutional features by channel, allowing features from different groups to influence each other in the next convolution, thereby improving the model's classification accuracy. After obtaining the feature map with redundancy removed through the above operations, depthwise convolution is performed for further convolution. Depthwise convolution is an extreme case of grouped convolution, dividing each channel into an independent group, thus completing the convolution at minimal cost and obtaining a low-cost feature map. Finally, by linearly concatenating the features obtained from grouped convolutions with the features obtained from depthwise convolutions along the channels, the output of the module is obtained, such as... Figure 7 As shown, φ here represents depthwise convolution.

[0068] Based on the above preprocessing methods, lightweight classification module, and LHAM and EGM modules, the input video can be quickly and accurately mapped to the relevant hemiplegic gait level, thereby completing an effective and real-time stroke hemiplegic gait assessment.

[0069] In addition, this application attempts to construct a video dataset for rigorous stroke hemiplegic gait classification, and is supported by relevant medical research papers.

[0070] Typically, sufficiently diverse datasets and effective classification algorithms are fundamental to implementing reliable video-based gait analysis systems. It's worth noting that due to patient privacy concerns, most publicly available pathological gait datasets were completed by healthy subjects using simulation kits. For example, the DAI dataset had five subjects performing one normal gait and seven abnormal gaits with different limb swing amplitudes. Other datasets, INIT, GAIT-IST, and GAIT-IT, simulated one normal gait and four pathological gaits, including hemiplegia, paraplegia, neuropathic gait, and Parkinson's disease. GAIT-IT had the most participants with 21, while the others had fewer than 10. Overall, existing datasets have limited participant numbers, and more importantly, lack detailed severity classification for hemiplegic gait. The two most relevant datasets, GAIT-IST and GAIT-IT, provide two severity levels but have not been used for experiments aimed at severity classification.

[0071] In this invention, a novel self-built dataset is introduced for assessing stroke recovery progress, containing hemiplegic gait with more refined severity. During data collection, color image sequences of the lateral gait were captured by a 1080p resolution camera mounted on a 1.5-meter-high tripod, 4 meters vertically from the center of the track. Note that a green screen background was used next to the track to facilitate foreground segmentation. This application collected data on a normal walking pattern and a simulated hemiplegic gait with three different severity levels. For each gait, a person walked back and forth twice, so four sequences were collected for each person's gait. Prior to data collection, 14 subjects (aged 21 to 25) were instructed on how to simulate the three different severity levels of hemiplegic gait.

[0072] The entire data collection process was completed within two days, ultimately collecting 224 hemiplegic sequences in the proposed dataset. The dataset developed in this application uses two data modes: silhouette and skeleton. The dataset defined the severity of hemiplegic gait based on the degree of spasticity after stroke and the abnormal synergistic patterns associated with spasticity, a concept first proposed in the Brunstrom scale. Abnormal synergism is a motor disorder in post-stroke patients, characterized by loss of independent control over selected muscle groups, leading to highly coupled joint movements. For the lower limbs, abnormal synergism is divided into extensor synergism (hip internal rotation, adduction, and extension; knee extension; ankle extension and eversion) and flexor synergism (hip external rotation, adduction, and flexion and extension; knee flexion and extension; ankle flexion, extension, and eversion). The synergistic principle in the upper limbs is similar to that in the lower limbs. Finally, the relationship between the simulated hemiplegic gait of three different severity levels and the area of ​​the lower limb affected by abnormal synergism is shown in [reference needed]. Figure 8 .

[0073] The gait characteristics under the combined effects of abnormalities of varying severity are described below:

[0074] Lv3: Spasticity is the most severe, with abnormal synergism involving almost every part of the limbs. During this period, the imitator exhibits abnormal synergistic movements of all joints on the affected side of the limb. Abnormal gait manifestations include severe circumduction, limited knee flexion during hip extension, toe-first landing during walking due to ankle eversion, resulting in a shortened double-stance phase in the gait cycle, and increased compensatory tilting movements of the trunk, which corresponds to Brunstrom 3 stage.

[0075] Lv2: Spasticity begins to subside, and the abnormal synergy is broken. During this period, only the highly coupled movements of the hip and knee joints of the affected lower limb are simulated, without deliberately mimicking the abnormal synergy of the ankle joint. Gait abnormalities include a slight circumduction gait and limited knee flexion during hip extension, but do not reach the level of Lv3, equivalent to Brunstrom stages 4-5.

[0076] Lv1: No obvious spasticity or abnormal synergy. During this stage, limb movements on the affected upper and lower limbs are slow and have a small range of motion, with no obvious abnormal synergy, equivalent to Brunstrom stage 6. Patients at this stage can recover to normal through appropriate rehabilitation training.

[0077] Based on the aforementioned self-built dataset and deep learning model, this application conducted extensive experiments on both the publicly available GAIT-IST dataset and its self-created dataset to demonstrate the superiority of the system. Ablation experiments were also performed to showcase the effectiveness of the proposed attention module and improved ghost module. In the experiments, this application used K-fold experiments to verify model robustness, performing a 10-fold cross-subject test on the GAIT-IST dataset and a 7-fold test on the proposed dataset, with a learning rate of 0.001. The difference in the number of folds in the cross-validation set is due to the different number of subjects in the two datasets.

[0078] The model in this application was compared with several state-of-the-art (SOTA) methods, and the experimental results are shown in [link to experimental results]. Figure 9The state-of-the-art (SOTA) classification system for the GAIT-IST dataset utilizes a fine-tuned VGG-19 algorithm, pre-trained on the ImageNet dataset. All layers of the VGG19 algorithm prior to the second block were frozen after pre-training. Furthermore, this application compares the system with several lightweight SOTA methods from recent years. Both the SOTA methods and the model proposed in this application were tested on GAIT-IST and a self-built dataset.

[0079] As can be seen, the method proposed in this application outperforms all state-of-the-art (SOTA) methods on both datasets. Compared to the fine-tuned VGG19 model, the proposed model reduces the number of parameters by approximately 72 times and the total number of floating-point operations (FLOPs) by 186 times, to only 105 million (multiple-adds). Furthermore, the accuracy of the proposed method on the GAIT-IST dataset is still 0.3% higher than that of the fine-tuned VGG-19, and 1.80% higher on the self-built dataset. In addition, in comparison with lightweight SOTA methods, the proposed method also achieves the best accuracy with the lowest number of parameters and computational cost. The number of parameters in the proposed method is only 45% of that in MobilenetV3 and 46% of that in Ghostnet, with FLOPs reduced by 62% and 41%, respectively, while the accuracy is improved by 1.92% and 1.46% on GAIT-IST, and by 1.64% and 0.99% on the dataset of this application. The above results demonstrate that the method presented in this application is superior in both classification accuracy and computational efficiency, and is more suitable for small-scale pathological gait datasets and low-computational-capacity environments.

[0080] To verify the importance of the attention module proposed in this application, some ablation experiments were conducted, and the results are as follows: Figure 10 As shown, two popular state-of-the-art attention modules, namely the SE (Squeeze Excitation) module and the ECA (Efficient Attention Module) module, were used to compare with the attention module of this application.

[0081] Experimental results show that the ECA module does achieve comparable accuracy to the SE module as the number of parameters decreases. However, based on the proposed approach, adding a spatial attention module further improves the performance, achieving a good balance between efficiency and effectiveness. On the GAIT-IST dataset, the LHAM attention module proposed in this application outperforms the SE module by 0.91%, and on our own dataset, it outperforms the SE module by 1.08%. More importantly, the module in this application has 37% fewer parameters than the SE module. Based on these results, we can conclude that our proposed attention module performs better in classification, while it is superior to the SE module in terms of computational effort and superior to the ECA module in terms of accuracy.

[0082] To demonstrate that the improvements to the Ghost module in this application are indeed effective for small-scale datasets, this application compares the experimental results based on the improved steps with those of the original Ghost module, such as... Figure 11 As shown.

[0083] Experimental results show that, in the process of eliminating input feature redundancy, if the EGM module of this application removes the shuffle operation and only uses group convolution (GC), the accuracy decreases by 1.52% on the GAIT-IST dataset and by 1.03% on the dataset proposed in this application compared to the traditional convolution used in the original Ghost module. However, when the shuffle operation adopted by the module of this application is added, the accuracy recovers and reaches a level very close to that of the original Ghost module, with only a 0.13% difference on GAIT-IST and only 0.09% on the dataset proposed in this application, while the number of parameters and FLOPs decrease by 27% and 40%, respectively.

[0084] This application also provides a gait recognition and classification device, such as... Figure 12 As shown, the gait recognition and classification device 400 includes:

[0085] Preprocessing module 401 is used to preprocess the image sequence in the gait video to obtain the skeleton energy image corresponding to the gait video;

[0086] The feature vector extraction module 402 is used to extract features from the skeleton energy image to obtain the feature vector corresponding to the skeleton energy image.

[0087] The classification module 403 is used to classify the feature vector using a trained classification network to obtain the classification result corresponding to the gait video; the classification result includes normal, first hemiplegic gait, second hemiplegic gait and third hemiplegic gait.

[0088] Preferably, the preprocessing module includes:

[0089] The skeleton image extraction unit is used to extract skeleton information from the image sequence in the gait video to obtain a first single-frame skeleton image.

[0090] The image cropping unit is used to crop and scale the first single-frame skeleton image to obtain a second single-frame skeleton image with a preset size and resolution, where the skeleton information is located at the center of the image.

[0091] The gait cycle extraction unit is used to extract several independent gait cycles through gait cycle detection;

[0092] Skeleton energy image calculation is performed for each gait cycle using the following formula:

[0093]

[0094] Among them, I t (x,y) represents the second single-frame skeleton image, and N represents the number of images per gait cycle.

[0095] Preferably, the Efficient Ghostnet neural network is used to extract features and classify the skeleton energy image;

[0096] The Efficient Ghostnet neural network comprises 5 blocks, each block containing several EG-necks and / or EG-necks with attention.

[0097] Preferably, the EG-neck includes a first EGM module, a DWConv module, an LHAM module, and a second EGM module connected in sequence.

[0098] Preferably, the EG-neck includes a first EGM module, an LHAM module, and a second EGM module connected in sequence.

[0099] Preferably, the LHAM module includes a global average pooling module and a spatial attention module, wherein the spatial attention module includes a pooling operation and a convolution operation.

[0100] Preferably, the EGM module includes a grouped convolution part, a depthwise convolution part, a linear concatenation part, and an output part.

[0101] For the apparatus embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The apparatus embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0102] This application also provides an electronic device, including:

[0103] At least one memory and at least one processor;

[0104] The memory is used to store one or more programs;

[0105] When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of the gait recognition and classification method as described above.

[0106] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the gait recognition and classification method described above.

[0108] Computer storage media include permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by computing devices.

[0109] This invention provides a gait recognition and classification method, device, electronic device, and storage medium. It proposes a novel lightweight network to efficiently extract pathological gait features, thereby accurately mapping the input to stroke gait of relevant severity. Extensive experiments on self-built and publicly available datasets demonstrate that the proposed attention-based lightweight CNN achieves effective and efficient gait analysis for stroke rehabilitation.

[0110] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A gait recognition and classification method, characterized in that, Includes the following steps: The image sequence in the gait video is preprocessed to obtain the skeleton energy image corresponding to the gait video; Feature extraction is performed on the skeleton energy image to obtain the feature vector corresponding to the skeleton energy image; The trained Efficient Ghostnet neural network is used to classify the feature vectors to obtain the classification results corresponding to the gait videos; the classification results include normal, first hemiplegic gait, second hemiplegic gait, and third hemiplegic gait. The Efficient Ghostnet neural network comprises 5 blocks, each containing several EG-necks and / or EG-necks with attention. Each EG-neck integrates a lightweight hybrid attention module (LHAM) and a lightweight convolutional module (EGM). The EG-neck with attention further includes an additional depthwise convolutional module (DWConv) on top of the EG-neck. The lightweight hybrid attention module LHAM compresses input features along the channel dimension through global average pooling and assigns lightweight channel attention weights to the features through adaptive one-dimensional convolution. At the same time, it performs max pooling and average pooling operations on the features in the spatial domain, concatenates the two results, performs a convolution operation, and assigns spatial attention weights. The EGM module first performs channel-based group convolution to compress features, then performs depthwise convolution, and finally, linearly concatenates the features after group convolution with the features after depthwise convolution by channels to obtain the module's output.

2. The gait recognition and classification method according to claim 1, characterized in that, Preprocessing the image sequences in gait videos includes the following steps: Skeleton information is extracted from the image sequence in the gait video to obtain the first single-frame skeleton image; The first single-frame skeleton image is cropped and scaled to obtain a second single-frame skeleton image with a preset size and resolution, where the skeleton information is located at the center of the image. Several independent gait cycles are extracted through gait cycle detection; For each gait cycle, the skeletal energy image is calculated using the following formula: in, This is the second single-frame skeleton image. Number of images per gait cycle.

3. A gait recognition and classification device, characterized in that, include: The preprocessing module is used to preprocess the image sequence in the gait video to obtain the skeleton energy image corresponding to the gait video; The feature vector extraction module is used to extract features from the skeleton energy image to obtain the feature vector corresponding to the skeleton energy image. The classification module is used to classify the feature vectors using a trained Efficient Ghostnet neural network to obtain the classification results corresponding to the gait videos; the classification results include normal, first hemiplegic gait, second hemiplegic gait, and third hemiplegic gait; The Efficient Ghostnet neural network comprises 5 blocks, each containing several EG-neck modules and / or EG-neck modules with attention. Each EG-neck integrates a lightweight hybrid attention module (LHAM) and a lightweight convolutional module (EGM). The EG-neck with attention module further includes an additional depthwise convolutional module (DWConv) on top of the EG-neck module. The lightweight hybrid attention module (LHAM) includes a global average pooling module and a spatial attention module. The global average pooling module compresses the input features according to the channel dimension and assigns channel attention weights to the features through adaptive one-dimensional convolution. The spatial attention module first performs max pooling and average pooling operations on the features in the spatial domain, and then concatenates the two results and performs a convolution operation to obtain spatial attention. The EGM module first divides the feature map into several groups by channel through the grouped convolution part, and performs convolution on the features of each group separately to compress the features. Then, it further convolves through the depthwise convolution part. Next, it linearly concatenates the features after grouped convolution and the features after depthwise convolution by channel through the linear concatenation part, and finally outputs the result.

4. An electronic device, characterized in that, include: At least one memory and at least one processor; The memory is used to store one or more programs; When the one or more programs are executed by the at least one processor, the at least one processor implements the steps of a gait recognition and classification method as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of a gait recognition and classification method as described in any one of claims 1-2.

Images