Gait Analysis Method, Device, System, Electronic Device and Storage Medium

Abstract

The present application discloses a gait analysis method, device, system, electronic device and storage medium. The gait analysis method includes: simultaneously acquiring the bioelectrical signals of a target object and dual-view synchronized videos during the movement of the target object; extracting gait parameters from the dual-view synchronized videos through a lightweight pose estimation network model; performing feature extraction on the bioelectrical signals to obtain bioelectrical signal feature parameters; and combining the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result. The gait analysis method provided by the embodiments of the present application acquires relatively comprehensive data, improves the calculation rate and reduces the analysis cost through a lightweight algorithm, and the gait analysis result obtained by combining bioelectrical signals and dual-view synchronized videos is relatively accurate, which can meet the requirements for the comprehensiveness of various types of data in current neurological disease research, realize the quantitative evaluation of neurological diseases, and promote the research progress of neurological diseases.

Description

Technical Field

[0001] The present application relates to the technical field of signal processing, and in particular, to a gait analysis method, apparatus, system, electronic device, and storage medium. Background Art

[0002] At present, neurological diseases pose a huge burden on global healthcare. Animal models are considered important tools for experimental medical research on neurological diseases such as Parkinson's disease (PD), stroke, etc. Since most neurological disorders may lead to abnormal gait patterns, gait analysis of animal models is usually used to evaluate the movement disorders caused by these neurological injuries. In order to quantitatively evaluate neurological dysfunctions, various available animal model gait analysis systems have been proposed. In the related art, gait analysis obtains data related to neurological diseases by analyzing bioelectrical signals, and the obtained data is not comprehensive enough to meet the comprehensive demand for various types of data, which affects the research progress of neurological diseases. Summary of the Invention

[0003] The purpose of the present application is to provide a gait analysis method, apparatus, system, electronic device, and storage medium to solve the problem in the related art that gait analysis obtains data related to neurological diseases by analyzing bioelectrical signals, and the obtained data is not comprehensive enough to meet the comprehensive demand for various types of data, which affects the research progress of neurological diseases. To provide a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary part is not a general review, nor is it intended to identify key / important elements or delineate the protection scope of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the subsequent detailed description.

[0004] According to one aspect of the embodiments of the present application, a gait analysis method is provided, including:

[0005] Simultaneously obtaining the bioelectrical signal of a target object and the dual-view synchronous video when the target object is moving;

[0006] Extracting gait parameters from the dual-view synchronous video through a lightweight pose estimation network model;

[0007] Performing feature extraction on the bioelectrical signal to obtain bioelectrical signal feature parameters;

[0008] Combining the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result.

[0009] In some embodiments of the present application, the step of simultaneously obtaining the bioelectrical signal of a target object and the dual-view synchronous video when the target object is moving includes:

[0010] Collect the bioelectrical signals of the target object through wearable electromyography electrodes or implantable nerve electrodes and simultaneously collect dual-view synchronized videos of the target object during movement from different lateral directions of the target object.

[0011] In some embodiments of the present application, the simultaneous acquisition of the bioelectrical signals of the target object and the dual-view synchronized videos of the target object during movement further includes:

[0012] Filter the bioelectrical signals through a filter with a preset cut-off frequency to remove the irrelevant frequency bands and power frequency noise in the bioelectrical signals, and obtain the filtered bioelectrical signals.

[0013] In some embodiments of the present application, the extraction of gait parameters from the dual-view synchronized videos through a lightweight pose estimation network model includes: extracting gait parameters from the dual-view synchronized videos through a preset lightweight pose estimation network model;

[0014] The preset lightweight pose estimation network model includes a model configured with a preset number of layers of ResNet, a student model, and an imitation loss function;

[0015] The student model consists of an encoder and a decoder, where the encoder extracts feature maps from the images of the combined data through downsampling; the decoder predicts the positions of the target key points in the feature maps through upsampling;

[0016] The imitation loss function is used to train the knowledge in the model configured with the preset number of layers of ResNet to train the student model.

[0017] In some embodiments of the present application, the bioelectrical signal feature parameters include the time-domain features and frequency-domain features of the bioelectrical signal feature parameters; the extraction of feature parameters from the bioelectrical signals to obtain bioelectrical signal feature parameters includes:

[0018] Extract the time-domain features and frequency-domain features from the bioelectrical signals using a motion observation window with a preset length to obtain the bioelectrical signal feature parameters.

[0019] In some embodiments of the present application, the combination of the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result includes:

[0020] Obtain the time-distance parameters of the gait support phase and swing phase of the target object through images;

[0021] Based on the extracted bioelectrical signal feature parameters, use the support vector machine algorithm to train a binary classifier to form a mapping relationship between the bioelectrical signals of the target object and the gait support phase and swing phase, and achieve comprehensive gait analysis.

[0022] According to another aspect of the embodiments of the present application, there is provided a gait analysis device, including:

[0023] An acquisition module, configured to simultaneously acquire the bioelectrical signal of a target object and the dual-view synchronized video when the target object is moving;

[0024] A gait parameter extraction module, configured to extract gait parameters from the dual-view synchronized video through a lightweight pose estimation network model;

[0025] A feature extraction module, configured to perform feature extraction on the bioelectrical signal to obtain bioelectrical signal feature parameters;

[0026] A combination module, configured to combine the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result.

[0027] According to another aspect of the embodiments of the present application, there is provided a gait analysis system, including a data analysis device, a bioelectrical signal recorder, and a left-right dual-view video recorder respectively connected to the data analysis device;

[0028] The bioelectrical signal recorder is configured to acquire the bioelectrical signal of a target object;

[0029] The left-right dual-view video recorder is configured to acquire the dual-view synchronized video when the target object is moving;

[0030] The data analysis device is configured to: extract gait parameters from the dual-view synchronized video through a lightweight pose estimation network model;

[0031] Perform feature extraction on the bioelectrical signal to obtain bioelectrical signal feature parameters;

[0032] Combine the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result.

[0033] According to another aspect of the embodiments of the present application, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the gait analysis method according to any one of the above.

[0034] According to another aspect of the embodiments of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to implement the gait analysis method according to any one of the above.

[0035] One aspect of the technical solution provided by the embodiments of the present application may include the following beneficial effects:

[0036] The gait analysis method provided by the embodiments of the present application simultaneously acquires the bioelectrical signals of the target object and the dual-view synchronized video during the movement of the target object, extracts gait parameters from the dual-view synchronized video through a lightweight pose estimation network model, extracts features from the bioelectrical signals to obtain bioelectrical signal feature parameters, combines the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result. The acquired data is relatively comprehensive. By using a lightweight algorithm, the calculation rate is improved and the analysis cost is reduced. The gait analysis result obtained by combining the bioelectrical signals and the dual-view synchronized video is relatively accurate, which can meet the current requirements for the comprehensiveness of various types of data in the research of neurological diseases, realize the quantitative evaluation of neurological diseases, promote the research progress of neurological diseases, and solve the problems in the related technologies that gait analysis analyzes bioelectrical signals to obtain data related to neurological diseases, the acquired data is not comprehensive enough, it is difficult to meet the requirements for the comprehensiveness of various types of data, and it affects the research progress of neurological diseases.

[0037] Other features and advantages of the present application will be described in the following specification, and, in part, will be obvious from the specification, or can be inferred or definitely determined from the specification without doubt, or can be understood by implementing the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments recorded in the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 The flowchart of the gait analysis method according to an embodiment of the present application is shown.

[0040] Figure 2 The structural block diagram of the multi-mode gait motion analysis system according to an embodiment of the present application is shown.

[0041] Figure 3 The structural schematic diagram of a lightweight pose estimation according to an embodiment of the present application is shown.

[0042] Figure 4 The overall structural schematic diagram of the lightweight rat pose estimation network mainly composed of an encoder and a decoder is shown.

[0043] Figure 5 The normalized diagram of the angle curve of the hind limb joints of a rat with sciatic nerve injury within one gait cycle is shown.

[0044] Figure 6Shows the fusion analysis diagram of the nerve signal and the gait phase detection signal of a normal rat.

[0045] Figure 7 Shows the structural block diagram of the gait analysis device according to an embodiment of the present application.

[0046] Figure 8 Shows the structural block diagram of an electronic device according to an embodiment of the present application.

[0047] Figure 9 Shows the schematic diagram of a computer-readable storage medium according to an embodiment of the present application.

[0048] The realization of the purpose, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. Detailed implementation manners

[0049] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts belong to the scope of protection of the present application.

[0050] Those skilled in the art can understand that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as the general understanding of those of ordinary skill in the art in the field to which the present application belongs. It should also be understood that terms such as those defined in a general dictionary should be understood to have a meaning consistent with the meaning in the context of the prior art, and will not be interpreted in an idealized or overly formal sense unless specifically defined as here.

[0051] The gait motion analysis of animal models plays an important role in experimental medicine and can better study neurological diseases. In order to quantitatively evaluate neurological diseases, the embodiments of the present application propose a gait analysis method, which can synchronously collect nerve signals and dual-view gait video recordings of the test object, and the obtained data is relatively comprehensive. The gait analysis results obtained by combining bioelectrical signals and dual-view synchronous videos are relatively accurate, which can meet the current requirements for the comprehensiveness of various types of data in the research of neurological diseases, realize the quantitative evaluation of neurological diseases, and promote the research progress of neurological diseases.

[0052] As Figure 1 shown, an embodiment of the present application provides a gait analysis method, including steps S10 to S50:

[0053] S10. Simultaneously obtain the bioelectrical signal of the target object and the dual-view synchronous video when the target object is moving.

[0054] In one embodiment, simultaneously obtaining the bioelectrical signal of the target object and the dual-view synchronous video during the movement of the target object includes: collecting the bioelectrical signal of the target object through wearable electromyography electrodes or implantable nerve electrodes and simultaneously collecting the dual-view synchronous video during the movement of the target object from the left and right sides of the target object.

[0055] In one embodiment, simultaneously obtaining the bioelectrical signal of the target object and the dual-view synchronous video during the movement of the target object may further include: filtering the bioelectrical signal through a filter with a preset cut-off frequency to remove the irrelevant frequency bands and power frequency noise in the bioelectrical signal, and obtaining the filtered bioelectrical signal.

[0056] S20. Extract gait parameters from the dual-view synchronous video through a lightweight pose estimation network model.

[0057] In one embodiment, extracting gait parameters from the dual-view synchronous video through a lightweight pose estimation network model includes: extracting gait parameters from the dual-view synchronous video through a preset lightweight pose estimation network model;

[0058] The preset lightweight pose estimation network model includes a model configured with ResNet of a preset number of layers, a student model, and an imitation loss function;

[0059] The student model consists of an encoder and a decoder, where the encoder extracts feature maps from the images of the combined data through downsampling; the decoder predicts the positions of the target key points in the feature maps through upsampling;

[0060] The imitation loss function is used to extract the knowledge in the model configured with ResNet of the preset number of layers to train the student model.

[0061] S30. Extract features from the bioelectrical signal to obtain bioelectrical signal feature parameters.

[0062] In one embodiment, the bioelectrical signal feature parameters include the time-domain features and frequency-domain features of the bioelectrical signal feature parameters; the extracting features from the bioelectrical signal to obtain bioelectrical signal feature parameters includes: extracting the time-domain features and frequency-domain features from the bioelectrical signal by using a motion observation window with a preset length to obtain the bioelectrical signal feature parameters.

[0063] S40. Combine the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result.

[0064] In one embodiment, gait parameters and bioelectrical signal feature parameters are combined to obtain a comprehensive gait analysis result, including:

[0065] Obtain the time-distance parameters of the gait support phase and swing phase of the target object through an image;

[0066] Based on the extracted bioelectrical signal feature parameters, use the support vector machine algorithm to train a binary classifier, so that the bioelectrical signal of the target object forms a mapping relationship with the gait support phase and swing phase, and realize comprehensive gait analysis.

[0067] In one embodiment, the support vector machine can be used to classify the bioelectrical signal into a standing signal and a swaying signal; the standing signal and the swaying signal are respectively connected to the corresponding parameters in the bioelectrical signal feature parameters.

[0068] In one embodiment, the encoder extracts a feature map from the image of the combined data through downsampling; the decoder predicts the positions of the key points of interest in the feature map through upsampling.

[0069] The gait analysis method provided by the embodiments of the present application simultaneously obtains the bioelectrical signal of the target object and the dual-view synchronous video when the target object is moving. The gait parameters are extracted from the dual-view synchronous video through a lightweight pose estimation network model, the bioelectrical signal is feature-extracted to obtain bioelectrical signal feature parameters, the gait parameters and the bioelectrical signal feature parameters are combined to obtain a comprehensive gait analysis result. The obtained data is relatively comprehensive, the calculation rate is improved and the analysis cost is reduced through a lightweight algorithm. The gait analysis result obtained by combining the bioelectrical signal and the dual-view synchronous video is relatively accurate, which can meet the current requirements for the comprehensiveness of various types of data in neurological disease research, realize the quantitative evaluation of neurological diseases, promote the research progress of neurological diseases, and solve the problem that in the related technology, gait analysis analyzes the bioelectrical signal to obtain data related to neurological diseases, the obtained data is not comprehensive enough, it is difficult to meet the requirements for the comprehensiveness of various types of data, and it affects the research progress of neurological diseases.

[0070] Another embodiment of the present application provides a gait analysis system, including a data analysis device, a bioelectrical signal recorder, and a left and right dual-view video recorder respectively connected to the data analysis device; the bioelectrical signal recorder is used to obtain the bioelectrical signal of the target object; the left and right dual-view video recorder is used to obtain the dual-view synchronous video when the target object is moving; the data analysis device is used to: extract gait parameters from the dual-view synchronous video through a lightweight pose estimation network model; perform feature extraction on the bioelectrical signal to obtain bioelectrical signal feature parameters, and combine the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result.

[0071] Another embodiment of the present application provides a multi-modal markerless gait motion analysis system based on a lightweight pose estimation network.

[0072] This system simultaneously collects videos from the left and right sides of the test subject and collects signals through wearable electromyography (EMG) electrodes or implanted electroencephalogram (ENG) electrodes.

[0073] In this embodiment, a real-time kinematic gait analysis method based on a lightweight deep learning model is adopted. Through the lightweight pose estimation network, the positions of the key points of the body structure features related to the gait motion of the experimental subject are identified with low latency and cost. The estimated features are connected to obtain kinematic gait parameters, and the kinematic gait parameters are combined with the bioelectrical potential signals and mapped into the gait pattern evaluation.

[0074] In this embodiment, in order to achieve the goal of real-time online data analysis and implementation on the device, the design of a lightweight pose estimation network is proposed, and nine key points of body structure features related to the gait motion pattern are output. Through multi-modal markerless gait motion analysis, the multi-modal information of the collected gait parameters and the EMG / ENG signals are fused and analyzed. Through experiments on two groups of rats, normal rats and nerve-injured rats, the working process of the proposed gait motion analysis system is verified.

[0075] In one implementation, a deep learning method is adopted. The intermediate feature maps generated by the mainstream convolutional layers usually contain redundant information, and this redundant information can be calculated through cheaper operations.

[0076] Inspired by this, a lightweight neural network for rat model pose estimation with simple structure, few parameters and low computational cost is proposed.

[0077] For a given input data X ∈ R (c×h0×w0) with c channels, height h0 and width w0, the ordinary convolutional layer that generates n feature maps can be expressed as:

[0078] Y = X * f + b (1),

[0079] where Y ∈ R (h1×w1×n) is the output of the convolutional layer with n channels, height h1 and width w1.

[0080] f ∈ R (c×k×k×n) is a convolutional filter with a kernel size of k × k.

[0081] To reduce FLOP, the feature map can be generated by connecting the output Y′ of the primary convolution and the output of the cheap linear operation on Y.

[0082] This module can be expressed as:

[0083] Y' = X * f1

[0084] Y = Y' + Y' * f2(2)

[0085] where f1 ∈ R (c×k×k×m) Generate m < n feature maps. f2 represents a linear and inexpensive operation on each channel to generate redundant feature maps, which can be regarded as grouped convolution with the number of groups being m. Therefore, the compression ratio of parameters and FLOP is approximately equal to m / n.

[0086] According to the above method, the design of the lightweight neural network module (abbreviated as the lightweight module) is as Figure 3 shown. Similar to the basic residual block in ResNet, the middle part of the lightweight model consists of two stacked lightweight modules, which respectively expand and reduce the number of channels of the feature maps.

[0087] Batch normalization (BN) is applied after each layer, and the ReLU non-linear transformation is applied after the first lightweight module. The middle part described above is the case where the stride = 1. For the case where the stride = 2, depthwise convolution is inserted between the two modules.

[0088] The pose estimation algorithm can usually be described by the following combination: the encoder is used for downsampling to extract features from the input image, and the decoder is used for upsampling to predict the positions of the key points of interest. The lightweight pose estimation network for the rat model is as Figure 4 shown. The encoder part consists of the two stacked lightweight modules mentioned above, and basically follows the architecture of MobileNetV3. In the decoder part, three groups of transposed convolutional layers are used for upsampling. The mean squared error (MSE) is used as the loss function between the final predicted heatmap and the target heatmap.

[0089] Figure 3 Shows a schematic structural diagram of lightweight pose estimation. Figure 3 Includes the lightweight module and the structure of the middle part of the network. Figure 4 Shows the overall structure of the lightweight rat pose estimation network mainly composed of an encoder and a decoder.

[0090] Knowledge extraction is an effective method to balance the performance and size of the model. In some embodiments, a large network is used to teach a small network, thereby improving the performance of the small model.

[0091] In a specific example, a model configured with a 50-layer ResNet is used as the teacher network, and the lightweight model is used to simulate the teacher network.

[0092] The imitation loss function is designed to extract the knowledge of the teacher model for training the student model:

[0093]

[0094] where \(m\) k represents the Gaussian heatmap of the \(k\)-th joint. \(m\) s represents the heatmap of the student model, \(m\) t represents the heatmap of the teacher model, \(m\) g represents the thermogram of the real data. The α value can be defined as the confidence of the teacher model for the prediction result.

[0095] A multi-modal gait motion analysis system that can capture videos of both sides of a test subject and synchronously acquire its EMG / ENG signals, as Figure 2 shown. This gait analysis system mainly consists of four parts: an EMG / ENG signal wireless bio-potential recorder, a Bluetooth Low Energy (BLE) dongle, a left and right dual-view video recorder, and a data processing and analysis module. Two integrated 120fps 720p RGB cameras are used to record the rapid movement of the experimental subject and avoid video frame blur. The two cameras are respectively set on the left and right sides of the experimental subject to collect videos from the left and right sides of the experimental subject. The wireless bio-potential recorder can record signals of up to 16 channels with a sampling frequency of up to 20 kHz. The BLE dongle is connected to the host through a USB connector to achieve real-time data transmission with a sampling frequency of 20 kHz and data synchronization between the left and right dual-view videos and the EMG / ENG signals.

[0096] From the dual-view synchronized videos, the X and Y coordinates of the body structure parts related to gait analysis are extracted through a lightweight pose estimation network. After preprocessing the coordinate data and the collected bioelectrical signals, spatio-temporal gait and kinematic gait parameters are modeled and calculated. Combining gait parameters with electromyographic or neural signals can provide a more comprehensive gait analysis. Based on the pose estimation model, the X and Y coordinates of the selected body parts are obtained in pixels, and the X and Y coordinates of the selected body parts can be tracked frame by frame for dual-view recording of gait videos, and then the pixel values are converted into centimeter (cm) values to reduce the position differences and angle differences of the two cameras relative to the treadmill in each experiment.

[0097] After using a low-pass filter and a moving average filter to reduce high-frequency noise and outliers caused by a pair of inaccurate or untraceable identified body parts, quantitative gait parameters are calculated from the preprocessed coordinates. To evaluate the synergy and patterns of movement, gait parameter calculation models based on spatio-temporal trajectory analysis and based on movement angle analysis are established. The gait cycle is defined by the first two consecutive contacts (IC) of the toes with the ground and can be divided into a swing phase and a stance phase. The swing phase refers to the continuous phase from the toes leaving the ground to contacting the ground, and the stance phase refers to the continuous phase from the toes contacting the ground to the toes leaving the ground next time.

[0098] For spatio-temporal trajectory-based analysis, the swing phase and the stance phase of the gait cycle are detected by the change in the X coordinate of the identified toe points. Additionally, the toe height, as another gait feature, is measured by the Y coordinate of the toe point, indicating the distance that the foot can be lifted from the ground.

[0099] For motion angle-based analysis, the joint angles of the hind limbs are mainly concerned, including the hip joint, the knee joint, and the ankle joint, and the average angle, the minimum angle value, the maximum angle value, and the range of motion of the joint are calculated. The average angle is used to characterize the average angle value of joint flexion and joint extension during walking. The range of motion of the joint characterizes the gap between the maximum extension and the maximum flexion, where the maximum extension and the maximum flexion represent the maximum angle and the minimum angle reached during the gait cycle, respectively.

[0100] The raw EMG / ENG signals collected by the electrodes are filtered by a third-order Butterworth filter with cut-off frequencies of 5 Hz and 95 Hz and a 50 Hz notch filter to remove the irrelevant frequency bands and power frequency noise in the signals. Subsequently, motion observation windows with a length of 25 ms are used to extract features from the electromyography signals or nerve signals. Time-domain features and frequency-domain features are calculated in each observation window, including the mean absolute value, variance, Willison amplitude, zero crossing count, wavelength, and energy based on the discrete Fourier transform.

[0101] To connect the EMG / ENG signals with the above-mentioned gait parameters, based on these features, a support vector machine (SVM) is used to classify the bioelectrical signals into two categories: stance signals and swing signals.

[0102] Normal rats and rats with nerve injuries are used to further exhibit movement disorders compared with the normal gait pattern and to verify the performance of the gait analysis system. The subjects under test walk at a constant speed on a treadmill, and two cameras record the walking videos in the right-view direction and the left-view direction in the sagittal plane at a speed of 120 frames per second. At the same time, the nerve signals of the subjects under test walking are recorded synchronously. 556 gait video records are collected as the rat dataset. Nine key points of body structure features related to gait motor performance are selected, including toe 1, ankle 2, knee 3, hip 4, iliac crest 5, back 6, shoulder 7, elbow 8, and wrist 9, and they are marked on each video frame as annotation information taught to the pose estimation network. This dataset is divided into three non-overlapping parts: 72% for training, 8% for validation, and 20% for testing.

[0103] The percentage of correct key points (PCK) is one of the commonly used performance metrics. The percentage of correct key points (PCK) refers to the percentage of detections where the normalized distance to the ground truth is within the error threshold. For the evaluation of the rat dataset, the normalized distance is defined by the distance between the iliac crest 5 and the back 6 key points as the normalization reference, and PCKb (the body-normalized percentage of correct key points) is used as the evaluation metric, as shown in Table 1. Among them, the error thresholds are set to 0.2 and 0.1 respectively.

[0104] Table 1 Evaluation of the rat experimental dataset (PCK)

[0105]

[0106] A comparison was made between normal rats and rats with sciatic nerve injury. According to the gait analysis process mentioned above, quantitative gait characteristics were measured and evaluated. The quantitative gait characteristics include the time and proportion of the stance phase and swing phase, the average value and range of motion of the hindlimb joint angles, and the change in toe height. Five normal rats were tested, and the gait parameters were calculated as baseline values.

[0107] To further verify the effectiveness of the system, the gait analysis of rats with sciatic nerve injury was evaluated and compared with the normal baseline. The synchronous acquisition of videos of rats walking from two perspectives provided the possibility to measure the asymmetry between the left and right sides of abnormal gaits.

[0108] Figure 3 The double-view joint angle curves of rats with sciatic nerve injury in a series of gait cycles and detected gait phases are shown.

[0109] Fusion analysis between spatio-temporal gait characteristics and nerve signals Figure 4 The fusion results between the nerve signals and spatio-temporal gait characteristics of normal rat samples tested on this system are shown.

[0110] By extracting the features of nerve signals as input, the trained SVM classifier uses the Gaussian radial basis function kernel to predict the swing phase and compares it with the actual swing phase detected and calculated from the walking video. In this case, the prediction accuracy of SVM classification is 0.81.

[0111] The multi-modal markerless gait motion analysis system for rat models proposed in this embodiment can synchronously acquire and fuse and analyze nerve signals and gait characteristics extracted from dual-view video recordings.

[0112] To achieve the goals of low latency and low computation, a lightweight pose estimation network was designed for a rat model. In a laboratory environment, the model was trained using a self-made dataset from dual-view videos recorded from the side among different normal rats and nerve-injured rats. Nine key points of body structure features related to the gait movement analysis of the rat model were selected and marked as annotations on the video frames. To further verify the performance of the gait analysis system, experiments were conducted on normal rats and rats with sciatic nerve injury. According to the proposed gait analysis process, the quantitative gait features of the rats were calculated and evaluated.

[0113] As Figure 7 shown, another embodiment of the present application provides a gait analysis device, including:

[0114] An acquisition module, configured to simultaneously acquire the bioelectrical signals of a target object and dual-view synchronized videos of the target object during movement;

[0115] A gait parameter extraction module, configured to extract gait parameters from the dual-view synchronized videos through a lightweight pose estimation network model;

[0116] A feature extraction module, configured to perform feature extraction on the bioelectrical signals to obtain bioelectrical signal feature parameters;

[0117] A combination module, configured to combine the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result.

[0118] In one implementation, the acquisition module is further specifically configured to: collect the bioelectrical signals of the target object through wearable electromyography electrodes or implantable nerve electrodes and simultaneously collect dual-view synchronized videos of the target object during movement from the left and right sides of the target object.

[0119] In one implementation, the acquisition module can further specifically be configured to: filter the bioelectrical signals through a filter with a preset cut-off frequency to remove irrelevant frequency bands and power frequency noise in the bioelectrical signals, obtaining filtered bioelectrical signals.

[0120] In one implementation, the extracting gait parameters from the dual-view synchronized videos through a lightweight pose estimation network model includes: extracting gait parameters from the dual-view synchronized videos through a preset lightweight pose estimation network model; the preset lightweight pose estimation network model includes a model configured with a preset number of layers of ResNet, a student model, and an imitation loss function; the student model consists of an encoder and a decoder, where the encoder extracts feature maps from the images of the combined data through downsampling; the decoder predicts the positions of target key points in the feature maps through upsampling; the imitation loss function is used to train the student model by extracting the knowledge in the model configured with the preset number of layers of ResNet.

[0121] In one embodiment, the bioelectrical signal characteristic parameters include the time-domain characteristics and frequency-domain characteristics of the bioelectrical signal characteristic parameters; the extracting the characteristic parameters of the bioelectrical signal to obtain the bioelectrical signal characteristic parameters includes: extracting the time-domain characteristics and frequency-domain characteristics from the bioelectrical signal by using a motion observation window with a preset length to obtain the bioelectrical signal characteristic parameters.

[0122] In one embodiment, combining the gait parameters and the bioelectrical signal characteristic parameters to obtain a comprehensive gait analysis result includes:

[0123] Obtaining the time-distance parameters of the gait support phase and swing phase of the target object through an image;

[0124] Based on the extracted bioelectrical signal characteristic parameters, training a binary classifier using a support vector machine algorithm to form a mapping relationship between the bioelectrical signal of the target object and the gait support phase and swing phase, so as to achieve comprehensive gait analysis.

[0125] In one embodiment, the preset gait analysis model includes an encoder and a decoder connected to each other; the encoder extracts a feature map from the image of the combined data through downsampling; the decoder predicts the positions of the key points of interest in the feature map through upsampling.

[0126] The gait analysis device provided by the embodiments of the present application simultaneously acquires the bioelectrical signal of the target object and the dual-view synchronous video during the movement of the target object, extracts the gait parameters from the dual-view synchronous video through a lightweight pose estimation network model, extracts the characteristic parameters of the bioelectrical signal to obtain the bioelectrical signal characteristic parameters, combines the gait parameters and the bioelectrical signal characteristic parameters to obtain a comprehensive gait analysis result. The acquired data is relatively comprehensive. The calculation rate is improved and the analysis cost is reduced through a lightweight algorithm. The gait analysis result obtained by combining the bioelectrical signal and the dual-view synchronous video is relatively accurate, which can meet the requirements for the comprehensiveness of various types of data in current neurological disease research, realize the quantitative evaluation of neurological diseases, promote the research progress of neurological diseases, and solve the problems existing in the related technology that gait analysis analyzes the bioelectrical signal to obtain data related to neurological diseases, the acquired data is not comprehensive enough, it is difficult to meet the requirements for the comprehensiveness of various types of data, and it affects the research progress of neurological diseases.

[0127] Another embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the gait analysis method described in any of the above embodiments.

[0128] Such as Figure 8As shown, the electronic device 10 may include: a processor 100, a memory 101, a bus 102, and a communication interface 103. The processor 100, the communication interface 103, and the memory 101 are connected via the bus 102. A computer program that can run on the processor 100 is stored in the memory 101. When the processor 100 runs this computer program, it executes the method provided in any of the foregoing embodiments of the present application.

[0129] Among them, the memory 101 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 103 (which can be wired or wireless), a communication connection is realized between this system network element and at least one other network element, and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used.

[0130] The bus 102 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 101 is used to store a program. After receiving an execution instruction, the processor 100 executes this program. Any method disclosed in any of the foregoing embodiments of the present application can be applied to the processor 100 or implemented by the processor 100.

[0131] The processor 100 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuit in the hardware of the processor 100 or by an instruction in software form. The above-mentioned processor 100 may be a general-purpose processor, which may include a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or this processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 101, and the processor 100 reads the information in the memory 101 and combines its hardware to complete the steps of the above method.

[0132] The electronic device provided by the embodiment of the present application and the method provided by the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by it.

[0133] Another embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. The program is executed by a processor to implement the gait analysis method described in any of the above embodiments. Refer to Figure 9 As shown, the computer-readable storage medium shown is an optical disc 20, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the method provided by any of the foregoing embodiments.

[0134] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here one by one.

[0135] The computer-readable storage medium provided by the above embodiment of the present application and the method provided by the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0136] It should be noted that:

[0137] The term "module" is not intended to be limited to a specific physical form. Depending on the specific application, a module can be implemented as hardware, firmware, software, and / or a combination thereof. In addition, different modules can share common components or even be implemented by the same components. There may or may not be clear boundaries between different modules.

[0138] The algorithms and displays provided herein are not inherently related to any specific computer, virtual device, or other equipment. Various general-purpose devices can also be used in conjunction with the examples based herein. Based on the above description, the structure required to construct such a device is obvious. In addition, the present application is not directed to any specific programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the description of a specific language above is to disclose the best implementation mode of the present application.

[0139] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially according to the indications of the arrows, these steps are not necessarily executed sequentially in the order indicated by the arrows. Unless there is a clear indication in the embodiments of the present application, there is no strict order restriction for the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily completed at the same moment, but can be executed at different moments, and their execution order is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or sub-steps or stages of other steps.

[0140] The above embodiments only express the implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A gait analysis method, characterized in that, including: simultaneously acquiring the bioelectrical signal of the target object and the dual-view synchronous video during the movement of the target object; extracting gait parameters from the dual-view synchronous video through a lightweight pose estimation network model; performing feature extraction on the bioelectrical signal to obtain bioelectrical signal feature parameters; combining the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result; the extracting gait parameters from the dual-view synchronous video through a lightweight pose estimation network model includes: extracting gait parameters from the dual-view synchronous video through a preset lightweight pose estimation network model; For a given input data \(X\in\mathbb{R}\) (c×h0×w0) The ordinary convolutional layer with \(c\) channels, \(h_0\) height and \(w_0\) width, generating \(n\) feature maps, is represented as: \(Y = X * f + b\), where Y ∈ R (h1×w1×n) is the output of a convolutional layer with n channels, height h1, and width w1; f ∈ R (c×k×k×n) is a convolution filter with a kernel size of k × k; generating a feature map by connecting the output Y′ of the primary convolution and the output of the cheap linear operation on Y; Y′ = X * f1, Y = Y′ + Y′ * f2, f1∈R (c×k×k×m) Generate m < n feature maps. f2 represents performing linear inexpensive operations on each channel to generate redundant feature maps, which is grouped convolution with the number of groups being m; the preset lightweight pose estimation network model includes a model configured with a preset number of layers of ResNet, a student model, and an imitation loss function; the student model consists of an encoder and a decoder, where the encoder extracts a feature map from the image of the combined data through downsampling; the decoder predicts the positions of the target key points in the feature map through upsampling; the imitation loss function is used to train the knowledge in the model configured with the preset number of layers of ResNet to train the student model; the combining the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result includes: acquiring the time-distance parameters of the gait support phase and the swing phase of the target object through an image; based on the extracted bioelectrical signal feature parameters, training a binary classifier using a support vector machine algorithm to establish a mapping relationship between the bioelectrical signal of the target object and the gait support phase and the swing phase, so as to achieve comprehensive gait analysis.

2. The gait analysis method according to claim 1, wherein the simultaneously acquiring the bioelectrical signal of the target object and the dual-view synchronous video during the movement of the target object includes: collecting the bioelectrical signal of the target object through wearable electromyography electrodes or implanted neural electrodes and simultaneously collecting the dual-view synchronous video during the movement of the target object from different side directions of the target object.

3. The gait analysis method according to claim 2, characterized in that the simultaneously acquiring the bioelectrical signal of the target object and the dual-view synchronous video during the movement of the target object further includes: filtering the bioelectrical signal through a filter with a preset cut-off frequency to remove the irrelevant frequency bands and power frequency noise in the bioelectrical signal, and obtaining the filtered bioelectrical signal.

4. The gait analysis method according to claim 1, wherein the bioelectrical signal feature parameters include the time-domain feature and the frequency-domain feature of the bioelectrical signal feature parameters; the performing feature extraction on the bioelectrical signal to obtain bioelectrical signal feature parameters includes: extracting the time-domain feature and the frequency-domain feature from the bioelectrical signal by using a motion observation window with a preset length to obtain the bioelectrical signal feature parameters.

5. A gait analysis device, characterized in that, including: an acquisition module, configured to simultaneously acquire the bioelectrical signal of the target object and the dual-view synchronous video during the movement of the target object; a gait parameter extraction module, configured to extract gait parameters from the dual-view synchronous video through a lightweight pose estimation network model; a feature extraction module, configured to perform feature extraction on the bioelectrical signal to obtain bioelectrical signal feature parameters; A combination module, configured to combine gait parameters and bioelectrical signal feature parameters to obtain a comprehensive gait analysis result; The gait parameter extraction module is further configured to: Extract gait parameters from the dual-view synchronized video through a preset lightweight pose estimation network model; For a given input data \(X\in\mathbb{R}\) (c×h0×w0) A common convolutional layer with \(c\) channels, height \(h_0\) and width \(w_0\), generating \(n\) feature maps is represented as: \(Y = X*f + b\), where Y ∈ R (h1×w1×n) is the output of a convolutional layer with n channels, height h1, and width w1; f ∈ ℝ (c×k×k×n) is a convolutional filter with a kernel size of k × k; Generate a feature map by connecting the output Y' of the primary convolution and the output of the inexpensive linear operation on Y; Y' = X * f1, Y = Y' + Y' * f2, f1 ∈ R (c×k×k×m) Generate m < n feature maps. f2 represents a linear and inexpensive operation on each channel to generate redundant feature maps, which is grouped convolution with the number of groups being m; The preset lightweight pose estimation network model includes a model configured with ResNet of a preset number of layers, a student model, and an imitation loss function; The student model consists of an encoder and a decoder, wherein the encoder extracts a feature map from the image of the combined data through downsampling; The decoder predicts the positions of the target key points in the feature map through upsampling; The imitation loss function is used to train the knowledge in the model configured with ResNet of a preset number of layers to train the student model; The combining of the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result includes: Obtaining the time-distance parameters of the gait support phase and the swing phase of the target object through an image; Based on the extracted bioelectrical signal feature parameters, using a support vector machine algorithm to train a binary classifier, so that the bioelectrical signal of the target object forms a mapping relationship with the gait support phase and the swing phase, realizing comprehensive gait analysis.

6. A gait analysis system, characterized in that, It includes a data analysis device, a bioelectrical signal recorder, and a left-right dual-view video recorder respectively connected to the data analysis device; The bioelectrical signal recorder is configured to acquire the bioelectrical signal of the target object; The left-right dual-view video recorder is configured to acquire the dual-view synchronized video of the target object during movement; The data analysis device is configured to: extract gait parameters from the dual-view synchronized video through a lightweight pose estimation network model; Extract the feature of the bioelectrical signal to obtain bioelectrical signal feature parameters; Combine the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result; The extracting of the gait parameters from the dual-view synchronized video through a lightweight pose estimation network model includes: extracting gait parameters from the dual-view synchronized video through a preset lightweight pose estimation network model; For a given input data \(X\in\mathbb{R}\) (c×h0×w0) A common convolutional layer with \(c\) channels, height \(h_0\) and width \(w_0\), generating \(n\) feature maps is expressed as: \(Y = X * f + b\), where Y ∈ R (h1×w1×n) is the output of a convolutional layer with n channels, a height of h1, and a width of w1; f ∈ R (c×k×k×n) is a convolution filter with a kernel size of k × k; Generate a feature map by connecting the output Y' of the primary convolution and the output of the inexpensive linear operation on Y; Y' = X * f1, Y = Y' + Y' * f2, f1∈R (c×k×k×m) Generate m < n feature maps. f2 represents performing a linear and inexpensive operation on each channel to generate redundant feature maps, which is grouped convolution with the number of groups being m; The preset lightweight pose estimation network model includes a model configured with ResNet of a preset number of layers, a student model, and an imitation loss function; The student model consists of an encoder and a decoder, wherein the encoder extracts a feature map from the image of the combined data through downsampling; the decoder predicts the positions of the target key points in the feature map through upsampling; The imitation loss function is used to train the knowledge in the model configured with ResNet of a preset number of layers to train the student model; The combining of the gait parameters and the bioelectrical signal feature parameters to obtain a comprehensive gait analysis result includes: Obtaining the time-distance parameters of the gait support phase and the swing phase of the target object through an image; Based on the extracted bioelectrical signal characteristic parameters, a binary classifier is trained using the support vector machine algorithm to establish a mapping relationship between the bioelectrical signals of the target object and the stance phase and swing phase of the gait, thereby realizing comprehensive gait analysis.

7. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor executes the program to implement the method according to any one of claims 1-4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the method according to any one of claims 1-4.

Images