Gait analysis method and system based on multi-modal features and readable storage medium

Abstract

The invention provides a multimodal feature-based gait analysis method and system and a readable storage medium. The method comprises the following steps of: constructing a stimulation signal time sequence rule of each target under a reference clock; generating a stimulation signal, distributing the stimulation signal based on a preset channel distribution matrix, and sending the distributed stimulation signal to each target point along the corresponding stimulation channel according to a stimulation signal time sequence rule; obtaining a motion image of a mark point in the to-be-analyzed target, and obtaining a kinematics signal of the mark point according to a pre-trained target recognition model; collecting electric signals fed back by each target spot in response to the stimulation signal, performing space-time alignment on various signals, and extracting feature information from each electric signal and kinematics signal after space-time alignment; and performing fusion processing on various feature information to obtain fusion features, inputting the fusion features into a pre-trained motion intention recognition model, and outputting a gait recognition result. The accuracy of the gait analysis result can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of gait analysis, and in particular to a gait analysis method, system and readable storage medium based on multimodal features. Background Art

[0002] Motion intention recognition refers to predicting or decoding information such as the type, direction, and strength of the action that an organism is about to perform by analyzing the characteristics of nervous system activity. It is a core technology in the fields of brain-computer interface and neurorehabilitation, and is widely used in medical rehabilitation, sports science, biometrics and other fields.

[0003] Traditional gait analysis methods mainly rely on single-modal data, such as relying on visual cameras to capture motion trajectories, inertial sensors to record motion data, or analyzing plantar forces through pressure plates. Although they can obtain certain gait information, they have significant limitations: single-modal technology is easily affected by environmental interference (such as lighting changes, occlusions affecting visual analysis, and sensor wearing position deviations causing data distortion), and the feature dimension is single, making it difficult to fully characterize the spatiotemporal dynamics, mechanical characteristics and individual differences of gait, resulting in insufficient accuracy and poor robustness of the analysis results, making it difficult to meet the needs of high-precision gait assessment in complex scenarios. Summary of the Invention

[0004] The purpose of the present invention is to provide a gait analysis method, system and readable storage medium based on multimodal features, aiming to solve the problems of insufficient accuracy and poor robustness of analysis results caused by traditional gait analysis relying on single modal data.

[0005] In a first aspect, the present invention provides a gait analysis method based on multimodal features, the method comprising:

[0006] Defining a reference clock and constructing stimulation signal timing rules for each target point under the reference clock, wherein each target point corresponds to at least one stimulation channel, and the target points include at least the brain region, spinal cord segment, and muscle to be analyzed;

[0007] Generate stimulation signals, distribute the stimulation signals based on a preset channel distribution matrix, and send the distributed stimulation signals to each target along the corresponding stimulation channel according to the stimulation signal timing rule;

[0008] Obtain motion images of the marker points in the target to be analyzed, and obtain kinematic signals of the marker points based on the pre-trained target recognition model;

[0009] collecting spinal cord electrical signals, electromyographic signals, and electroencephalographic signals fed back by each target point in response to the stimulation signal, performing spatiotemporal alignment on the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, and extracting first feature information, second feature information, third feature information, and fourth feature information corresponding to the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, respectively, from each of the spatiotemporal aligned electrical and kinematic signals;

[0010] The first feature information, the second feature information, the third feature information, and the fourth feature information are fused to obtain fused features, and the fused features are input into a pre-trained motion intention recognition model to output a gait recognition result.

[0011] In a second aspect, the present invention provides a gait analysis system based on multimodal features, the system comprising:

[0012] A timing rule construction module is used to define a reference clock and construct stimulation signal timing rules for each target point under the reference clock. Each target point corresponds to at least one stimulation channel. The target points include at least the brain area, spinal cord segment, and muscle to be analyzed.

[0013] a stimulation signal distribution module, configured to generate stimulation signals, distribute the stimulation signals based on a preset channel distribution matrix, and send the distributed stimulation signals to each target along the corresponding stimulation channel according to the stimulation signal timing rule;

[0014] A kinematic signal acquisition module is used to obtain motion images of the marker points in the target to be analyzed and obtain kinematic signals of the marker points based on a pre-trained target recognition model;

[0015] a feature extraction module for collecting spinal cord electrical signals, electromyographic signals, and electroencephalographic signals fed back by each target point in response to the stimulation signal, performing spatiotemporal alignment on the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, and extracting first feature information, second feature information, third feature information, and fourth feature information corresponding to the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, respectively, from each of the spatiotemporal aligned electrical and kinematic signals;

[0016] The feature fusion module is used to fuse the first feature information, the second feature information, the third feature information, and the fourth feature information to obtain a fused feature, and input the fused feature into a pre-trained motion intention recognition model to output a gait recognition result.

[0017] In a third aspect, the present invention provides a readable storage medium, wherein the readable storage medium stores one or more programs, which, when executed by a processor, implement the above-mentioned gait analysis method based on multimodal features.

[0018] In a fourth aspect, the present invention provides an electronic device, comprising a memory and a processor, wherein:

[0019] The memory is used to store computer programs;

[0020] When the processor is used to execute the computer program stored in the memory, the above-mentioned gait analysis method based on multimodal features is implemented.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] 1. By defining a reference clock to construct stimulation signal timing rules, obtaining feedback signals from multiple targets (brain regions, spinal cord segments, muscles), and integrating multiple electrical signals with kinematic signals, a multi-dimensional and comprehensive analysis of gait is achieved, laying a solid foundation for accurate gait recognition and effectively improving the comprehensiveness and accuracy of gait analysis.

[0023] 2. The stimulation signal timing rules constructed based on specific formulas can accurately control the triggering time of each stimulation signal, ensuring that the stimulation signal accurately acts on each target, thereby improving the accuracy and effectiveness of the stimulation. Specifically, by rationally allocating the stimulation signals to the corresponding stimulation channels and taking into account the delay accuracy between the stimulation channels, the stimulation signals are converted into analog signals and accurately transmitted to the target, ensuring the stability and accuracy of the stimulation signals and creating conditions for the target to generate effective feedback signals.

[0024] 3. By performing spatiotemporal alignment on multiple signals, we precisely match the different signals in both time and space through operations such as spatial coordinate mapping, signal interpolation and downsampling, and data association, thereby forming spatiotemporal data with time series characteristics. This provides a unified data foundation for subsequent feature extraction and fusion, ensuring the accuracy and consistency of subsequent analysis. In addition, the first to fourth feature information corresponding to the different signals are extracted from the spatiotemporal aligned signals, covering key features such as field potential oscillation energy, conduction velocity, root mean square value, median frequency, μ / β rhythm power, event-related potential, joint angle, and trajectory curvature. These feature information comprehensively reflects the neuroelectrophysiological and kinematic characteristics of gait, providing a rich feature basis for gait recognition. Finally, the multiple feature information at the same time is fused, and the influence of different features on the fusion result is adjusted by weight coefficients. This allows the fused feature to comprehensively reflect multiple aspects of gait information, enhance the representativeness and reliability of the feature, and provide more discriminative feature input for subsequent gait recognition, thus enabling the model to accurately identify the movement intention of the target to be analyzed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of a gait analysis method based on multimodal features proposed in one embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of signal acquisition according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of a gait analysis system based on multimodal features proposed in one embodiment of the present invention.

[0028] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Unless otherwise defined, the technical terms or scientific terms used herein should be the common meanings understood by people with ordinary skills in the field to which the invention belongs. The words "including" and similar words used in this article mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0030] like Figure 1As shown, an embodiment of the present invention provides a gait analysis method based on multimodal features, the method comprising steps S101 to S105, wherein:

[0031] Step S101: defining a reference clock and constructing stimulation signal timing rules for each target point under the reference clock. Each target point corresponds to at least one stimulation channel. The target points include at least the brain region, spinal cord segment, and muscle to be analyzed.

[0032] It should be noted that in this step, since the focus of the present invention is the fusion of multimodal signals, but each modal signal is actually collected by different devices or electrodes, if the time between the modal signals does not correspond, the final feature fusion will be difficult to represent the actual gait behavior of the target to be analyzed, which will lead to large errors in the intention recognition results. Based on this, before performing gait analysis, a reference clock will be defined, that is, all acquisition devices or acquisition electrodes are required to be based on this reference clock.

[0033] Furthermore, in some embodiments, the stimulation signal timing rule is constructed according to the following formula:

[0034] t trigger,i =n×T sync +(i―1)×Δt slot ;

[0035] Among them, T sync is the synchronization signal period, Δt slot is the time slice allocated to each stimulus signal, n is the synchronization cycle count, t trigger,i is the triggering time of the i-th stimulus signal.

[0036] The stimulation signal timing rules constructed based on the above-mentioned specific formula can accurately control the triggering time of each stimulation signal, ensuring that each device responds to the trigger signal within its own time slice to avoid conflicts. At the same time, it ensures that the stimulation signal accurately acts on each target, thereby improving the accuracy and effectiveness of the stimulation, providing reliable guarantees for subsequent signal acquisition and analysis, and helping to obtain more accurate gait feature information.

[0037] Step S102: generating stimulation signals, distributing the stimulation signals based on a preset channel distribution matrix, and sending the distributed stimulation signals to each target along the corresponding stimulation channel according to the stimulation signal timing rule;

[0038] It should be noted that, in some embodiments, a discrete phase signal is generated by a numerically controlled oscillator, and a continuous waveform stimulation signal is synthesized according to the discrete phase signal. The frequency expression of the stimulation signal is:

[0039]

[0040] Among them, f out is the output frequency of the stimulus signal, f clk is the clock frequency, N p is the number of bits of the phase accumulator, K f By changing K f The value of can realize the generation of arbitrary frequency signals in the range of 0.1Hz-10kHz, and supports single-phase, bi-phase, sinusoidal waveform and noise modulation waveform output.

[0041] In addition, the delay accuracy threshold between the stimulation channels is defined. For any two stimulation channels, the following conditions are satisfied:

[0042] Δt k,m =t start,m ―t start,k =ΔT0;

[0043] Where Δt k,m is the delay accuracy between the kth stimulation channel and the mth stimulation channel, t start,m , t start,k are the start time of the kth stimulation channel and the mth stimulation channel respectively, and ΔT0 is the delay accuracy threshold, which can be ±0.1 μs, for example.

[0044] Furthermore, in some embodiments, the generated stimulation signal is converted into an analog voltage or current signal:

[0045]

[0046] Among them, V out (n) is the converted analog voltage or current signal, V ref is the reference voltage, D(n) is the value of the stimulus signal, ∈ dac (n) is the conversion error, and N is the number of bits of the digital-to-analog converter;

[0047] Furthermore, in some embodiments, the converted analog voltage or current signal is allocated to a channel according to the following formula for transmission to the target:

[0048] C out (t) = M switch (t)×C in (t);

[0049] Among them, C out (t) is the stimulation signal output to the target, C in (t) is the converted analog voltage or current signal of the input, M switch(t) is the preset channel allocation matrix. The preset channel allocation matrix flexibly and accurately allocates multiple stimulation signals to different biological targets (such as brain regions, spinal cord segments, muscles), which can realize channel multiplexing. switch (t), which enables channel multiplexing and dynamic routing to meet stimulation requirements such as expanding a 256-channel DAC to 1024 electrodes, and the switching time of any input-output channel is less than 100ns.

[0050] M switch The dimension of (t) is M×N, where M is the number of output channels and N is the number of input channels. For example, if the input has 2 channels and the output has 3 channels, then M switch (t) is a 3×2 matrix. M switch (t) ij Indicates how the signal of input channel j is distributed to output channel i. The elements can be: Binary value (0 or 1): Indicates whether the signal of input channel j is passed to output channel i. Weight value: Indicates the weight of the signal of input channel j in output channel i.

[0051] Matrices can be static or dynamic. Static matrices are defined when the system is initialized and remain unchanged throughout the experiment. For example:

[0052]

[0053] It means that the signal of input channel 1 is assigned to output channel 1, the signal of input channel 2 is assigned to output channel 2, and output channel 3 does not receive any signal.

[0054] The dynamic matrix changes over time and can be adjusted in real time based on external conditions (such as user input and signal characteristics). For example:

[0055]

[0056] This means that before time t0, input channels 1 and 2 are assigned to output channels 1 and 2 respectively; after t0, input channel 1 is assigned to output channel 2, and input channel 2 is assigned to output channel 1.

[0057] Application scenarios: 1) In brain-computer interfaces, EEG signals are distributed to different stimulators or controllers. For example, left-brain signals are distributed to the left leg stimulator, and right-brain signals are distributed to the right leg stimulator. 2) In neurostimulators, analog signals are distributed to different electrodes for stimulating specific nerves. For example, in spinal cord stimulation therapy, signals are distributed to different spinal cord regions. 3) In multi-channel signal processing, signals are routed to different processing modules. For example, high-frequency signals are distributed to filters, and low-frequency signals are distributed to recorders.

[0058] In summary, the stimulation signal is generated by a numerically controlled oscillator and reasonably distributed to the corresponding stimulation channels. At the same time, the delay accuracy between the stimulation channels is taken into consideration to achieve the timing control of each stimulation signal. The stimulation signal is then converted into an analog signal and accurately transmitted to the target, ensuring the stability and accuracy of the stimulation signal and creating conditions for the target to generate an effective feedback signal.

[0059] Step S103: obtaining a motion image of a marker point in the target to be analyzed, and obtaining a kinematic signal of the marker point based on a pre-trained target recognition model;

[0060] It should be pointed out that the marker points include shoulder joints, elbow joints, wrist joints, heels, etc. The target recognition model is obtained by training based on a large number of images marked with known marker points. Since the acquisition of kinematic signals is not the focus of the present invention, it is not described in detail in this embodiment.

[0061] Furthermore, in some embodiments, the kinematic signal is the spatial coordinates of each marking point and the time corresponding to each spatial coordinate.

[0062] Step S104: collecting spinal cord electrical signals, electromyographic signals, and electroencephalographic signals fed back by each target point in response to the stimulation signal, performing spatiotemporal alignment on the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, and extracting first feature information, second feature information, third feature information, and fourth feature information corresponding to the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, respectively, from each of the spatiotemporal aligned electrical and kinematic signals;

[0063] In some embodiments, regarding the collection of various signals, the original EEG signal fed back by the target is first obtained, a noise signal is introduced into the original EEG signal, and the original EEG signal with the introduced noise signal is amplified:

[0064] y(t)=G·(x(t)+n(t));

[0065] Where y(t) is the amplified original EEG signal, G is the amplification factor, x(t) is the original EEG signal, and n(t) is the noise signal;

[0066] The amplified EEG original signal is filtered to obtain the EEG signal:

[0067] Y1(f)=y(t)·H1(f);

[0068] Where H1(f) is the frequency response function of the filter, and Y1(f) is the frequency domain representation of the filtered EEG signal;

[0069] Furthermore, in some embodiments, the process of collecting spinal cord electrical signals is:

[0070] The first spinal cord original electrical signal and the second spinal cord original electrical signal are collected using a differential amplifier, and the first spinal cord original electrical signal and the second spinal cord original electrical signal are amplified according to the following formula:

[0071] V out =L·(V in+ ―V in― )+V os ;

[0072] Among them, V out is the original electrical signal of the spinal cord after differential amplification, V in+ 、V in― are the original electrical signals of the first and second spinal cords, respectively; L is the amplification factor, V is the os is the offset voltage of the differential amplifier;

[0073] The original spinal cord electrical signal after differential amplification is filtered according to the following formula:

[0074]

[0075] Among them, Y2(f) is the frequency domain representation of the spinal cord electrical signal after filtering, X(f) is the frequency domain representation of the original spinal cord electrical signal after differential amplification, j is an imaginary number, f is the frequency of the original spinal cord electrical signal after differential amplification, and f c is the cutoff frequency of the filter;

[0076] In addition, in some embodiments, the equipment required for myoelectric signal acquisition includes myoelectric electrodes, myoelectric acquisition AFE, and myoelectric acquisition ADC. The myoelectric electrodes use a conductive hydrogel substrate (stretching rate > 500%), embedded with Ag / AgCl nanoparticles to adapt to dynamic deformation of the skin, and use 8×8 matrix electrodes to support high-density acquisition of signals from adjacent muscle groups. The myoelectric acquisition AFE uses a dynamic gain amplifier, configured with needle electrodes or implanted electrodes (bandwidth 20-2kHz), and a programmable filter group using adaptive high-pass filtering to attenuate low-frequency jitter (<5Hz) and suppress motion artifacts. The myoelectric acquisition ADC has a sampling rate of 4kHz and supports 16-channel synchronous sampling.

[0077] In addition, in some embodiments, multi-channel sampling is performed on spinal cord electrical signals, brain electrical signals, and myoelectric signals according to the following formula:

[0078] x i [n] = x i [t i +n·T s ];

[0079] Among them, x i [n] is the discrete signal of the i-th channel at the n-th sampling point, xi [t] is the continuous time signal of the i-th channel, t i is the starting sampling time of the i-th channel, n is the sampling point index, T s is the sampling interval.

[0080] In summary, when collecting various electrical signals fed back by the target, technical means such as introducing noise, amplification, filtering, and multi-channel sampling were used to effectively extract high-quality spinal cord electrical signals, electromyographic signals, and electroencephalographic signals, providing an accurate data source for subsequent spatiotemporal alignment and feature extraction, and ensuring the integrity and availability of the signals.

[0081] In addition, in some embodiments, in order to achieve accurate synchronization of multiple modal signals to avoid affecting the accuracy of subsequent intent recognition, the various collected signals are also temporally and spatially aligned, as follows:

[0082] Get the original space coordinates P of each marker point / electrode orig , and mapping the original space coordinates to the standard anatomical space, thereby achieving the unification of the coordinate space of all markers and electrodes:

[0083]

[0084] Among them, P std is the target space coordinate mapped to the standard anatomical space, s is the scale factor, R∈R 3×3 is a rotation matrix that satisfies R T R=I, I is the unit matrix, T is the transpose, and U is the translation vector used to compensate for the offset of the coordinate system origin. is the deformation correction term, α j is the weight coefficient corresponding to the j-th control point, C j is the original spatial coordinate of the jth control point, which is selected from the brain area or spinal cord area covered by the electrode, and M is the total number of control points. For large animals or pathological brains, local deformation may exist, so a correction term is introduced for correction.

[0085] A first low-frequency signal and a first high-frequency signal are screened out from the spinal cord electrical signal, the myoelectric signal, and the brain electrical signal according to a first preset frequency threshold and a second preset frequency threshold, and the first low-frequency signal is interpolated according to the following formula:

[0086]

[0087] Among them, S H (t n ) is the first low-frequency signal after interpolation, S L (t m ) is the first low-frequency signal, D is the total number of the first low-frequency signals involved in interpolation, tn is the time point at which the first low-frequency signal after interpolation needs to be calculated, t m is the time position of the first low-frequency signal, ΔT L is the sampling interval of the first low-frequency signal, w is the window function used to limit the influence range of the interpolation kernel, and sinc is used to calculate the interpolation point t n and the original sampling point t m The weight between:

[0088]

[0089] In addition, the first preset frequency threshold is lower than the second preset frequency threshold. For example, the first preset frequency threshold is 20 Hz and the second preset frequency threshold is 80 Hz. In this embodiment, for ease of distinction, all spinal cord electrical signals, electromyographic signals, and electroencephalographic signals with frequencies below the first preset frequency threshold are collectively referred to as first low-frequency signals. All spinal cord electrical signals, electromyographic signals, and electroencephalographic signals with frequencies above the second preset frequency threshold are collectively referred to as first high-frequency signals.

[0090] The first high-frequency signal is downsampled according to the following formula:

[0091]

[0092] Among them, S L (t m ) is the first high-frequency signal after downsampling, E is the downsampling factor, f H is the sampling frequency of the first high-frequency signal, is the impulse response of the filter used for anti-aliasing filtering, is the first high-frequency signal at time point The value at

[0093] The target space coordinates of the same sampling point at the same time are associated with the interpolated / downsampled signal to obtain the spatiotemporal data:

[0094]

[0095] Among them, O(t n ) represents the tth n The spatiotemporal data of a moment, Indicates the tth n The target space coordinates at the moment, F1(t n )、F2(t n )、F3(t n ) are the spinal cord electrical signals, myoelectric signals, and EEG signals that have been processed by interpolation / downsampling or are normal. Normal spinal cord electrical signals, myoelectric signals, and EEG signals refer to signals that have not been processed by interpolation / downsampling.

[0096] In summary, by performing spatiotemporal alignment on multiple signals, through operations such as spatial coordinate mapping, signal interpolation and downsampling, and spatiotemporal data association, the different signals are precisely matched in both time and space. This provides a unified data foundation for subsequent feature extraction and fusion, ensuring the accuracy and consistency of subsequent analysis. Furthermore, since this spatiotemporal data carries moments, it has time series properties. This facilitates the model's recognition of fused features with time series properties, thereby ensuring the reliability of the recognition results.

[0097] Step S105: Fusing the first feature information, the second feature information, the third feature information, and the fourth feature information to obtain fused features, and inputting the fused features into a pre-trained motion intention recognition model to output a gait recognition result.

[0098] It should be noted that, in some embodiments, the first characteristic information includes field potential oscillation energy and conduction velocity, the second characteristic information includes root mean square value and median frequency, the third characteristic information includes μ / β rhythm power and event-related potential, and the fourth characteristic information includes joint angle and trajectory curvature;

[0099] Specifically, the field potential oscillation energy is obtained according to the following formula:

[0100]

[0101] Among them, W γ (t,k) is the signal F1(t n ) is the Morlet wavelet transform coefficient at time t and frequency k, where K is the upper limit of the total frequency range of the wavelet transform, E γ (t) is the field potential oscillation energy;

[0102] The conduction velocity is given by the signal F1(t n ) is obtained by dividing the propagation distance on the channel by the propagation time;

[0103] The root mean square value and the median frequency refer to relevant features of the electromyographic signal, and are calculated using conventional techniques and will not be described in detail in this embodiment.

[0104] The μ / β rhythm power is obtained according to the following formula:

[0105]

[0106] Among them, P μ / β is the μ / β rhythm power, X3(f) is the signal F3(t n), f1 is the lower and upper limits of integration, corresponding to the frequency range of μ rhythm or β rhythm, respectively. The frequency range of μ rhythm is 8-12, and the frequency range of β rhythm is 13-30;

[0107] The event-related potential is obtained according to the following formula:

[0108]

[0109] Where ERP stands for event-related potential, F is the number of stimulus events used to calculate ERP, S(t-t m ) is the signal F3(t n ) at the stimulus event timestamp t m The surrounding segments, S(t0) is the signal F3(t n ) in the time period t0;

[0110] The joint angle is obtained according to the following formula:

[0111]

[0112] A is the shoulder joint, B is the elbow joint, C is the wrist joint, and θ is the joint angle;

[0113] The time interval from the first heel touchdown to the next heel touchdown at one of the marker points is defined as the gait cycle.

[0114] By extracting the first to fourth feature information corresponding to different signals from the signals after spatiotemporal alignment, key features such as field potential oscillation energy, conduction velocity, root mean square value, median frequency, μ / β rhythm power, event-related potential, joint angle, trajectory curvature, etc. are covered. These feature information comprehensively reflects the neuroelectrophysiological and kinematic characteristics of gait, providing rich feature basis for gait recognition.

[0115] In addition, in some embodiments, after obtaining the fusion features corresponding to each spatiotemporal data, the first feature information, the second feature information, the third feature information, and the fourth feature information at the same time are fused:

[0116]

[0117] Among them, Q is the fusion feature, α1, α2, β1, and β2 are weight coefficients used to adjust the influence of different features on the fusion results, ε is a constant, σ is a scale parameter, RMS is the root mean square root, MDF is the median frequency, and T t is the gait period, γ and μ are adjustment coefficients used to control the sensitivity of weight adjustment, θ0 and T0 are the reference joint angle and reference gait period, respectively.

[0118] It should be pointed out that the reference joint angle refers to the ideal angle value under a standard motion state (such as normal gait), and the reference gait cycle refers to the standard time range of a healthy individual during normal walking.

[0119] In addition, α1, α2, β1, β2, σ, γ, and μ are all determined through the training of the motion intention recognition model. Specifically, multiple samples of known motion intentions are obtained, and each sample is a time series data. The time series data includes historical fusion features at continuous moments. Each time series data is marked, and the marking result is the motion intention; then all the marked time series data are input into the motion intention recognition model for training to obtain a pre-trained motion intention recognition model, and then various parameters are determined. By constructing a fusion feature formula, each feature is linearly and weightedly fused under the action of different parameters, so that the fusion feature can comprehensively reflect the various aspects of gait information, enhance the representativeness and reliability of the feature, and provide a more discriminative feature input for subsequent gait recognition.

[0120] It should also be noted that the known motion intention may be running, walking, jumping, and the like.

[0121] In summary, according to the above-mentioned gait analysis method based on multimodal features, by constructing precise stimulation signal timing rules, the stimulation signals are accurately applied to multiple targets and a variety of electrical signals and kinematic signals are collected. After the collected signals are aligned in time and space, rich and comprehensive feature information is extracted, and these feature information is effectively integrated. Finally, the pre-trained model is input to achieve accurate gait recognition, which can greatly improve the accuracy of the gait analysis results, has high robustness, and can adapt to the needs of high-precision gait evaluation in complex scenarios.

[0122] like Figure 3 As shown, an embodiment of the present invention provides a gait analysis system based on multimodal features, the system comprising:

[0123] A timing rule construction module 10 is used to define a reference clock and construct stimulation signal timing rules for each target point under the reference clock. Each target point corresponds to at least one stimulation channel. The target points include at least the brain area, spinal cord segment, and muscle to be analyzed.

[0124] a stimulation signal distribution module 20 for generating stimulation signals, distributing the stimulation signals based on a preset channel distribution matrix, and sending the distributed stimulation signals to each target along the corresponding stimulation channel according to the stimulation signal timing rule;

[0125] The kinematic signal acquisition module 30 is used to obtain the motion image of the marker points in the target to be analyzed, and obtain the kinematic signals of the marker points according to the pre-trained target recognition model;

[0126] a feature extraction module 40 for collecting spinal cord electrical signals, electromyographic signals, and electroencephalographic signals fed back by each target point in response to the stimulation signal, performing spatiotemporal alignment on the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, and extracting first feature information, second feature information, third feature information, and fourth feature information corresponding to the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, respectively, from each of the spatiotemporal aligned electrical and kinematic signals;

[0127] The feature fusion module 50 is used to fuse the first feature information, the second feature information, the third feature information, and the fourth feature information to obtain a fused feature, and input the fused feature into a pre-trained motion intention recognition model to output a gait recognition result.

[0128] On the other hand, the present invention further provides a readable storage medium having one or more programs stored thereon, which implement the above-mentioned gait analysis method based on multimodal features when executed by a processor.

[0129] On the other hand, the present invention further proposes an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-mentioned gait analysis method based on multimodal features.

[0130] Those skilled in the art will appreciate that the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device), or in conjunction with such instruction execution system, apparatus, or device. For purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transmit a program for use by an instruction execution system, apparatus, or device, or in conjunction with such instruction execution system, apparatus, or device.

[0131] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0132] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement the hardware: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0133] While the embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations of these embodiments are possible. However, it should be understood that such modifications and variations are within the scope and spirit of the present invention as set forth in the claims. Furthermore, the invention described herein is susceptible to other embodiments and may be practiced or implemented in a variety of ways.

Claims

1. A gait analysis method based on multimodal features, characterized in that: The method comprises: Defining a reference clock and constructing stimulation signal timing rules for each target point under the reference clock, wherein each target point corresponds to at least one stimulation channel, and the target points include at least the brain region, spinal cord segment, and muscle to be analyzed; Generate stimulation signals, distribute the stimulation signals based on a preset channel distribution matrix, and send the distributed stimulation signals to each target along the corresponding stimulation channel according to the stimulation signal timing rule; Obtain motion images of the marker points in the target to be analyzed, and obtain kinematic signals of the marker points based on the pre-trained target recognition model; collecting spinal cord electrical signals, electromyographic signals, and electroencephalographic signals fed back by each target point in response to the stimulation signal, performing spatiotemporal alignment on the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, and extracting first feature information, second feature information, third feature information, and fourth feature information corresponding to the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, respectively, from each of the spatiotemporal aligned electrical and kinematic signals; The first feature information, the second feature information, the third feature information, and the fourth feature information are fused to obtain fused features, and the fused features are input into a pre-trained motion intention recognition model to output a gait recognition result.

2. The gait analysis method based on multimodal features according to claim 1, characterized in that: The steps of defining a reference clock and constructing stimulation signal timing rules for each target point under the reference clock include: The stimulus signal timing rule is constructed according to the following formula: t trigger,i =n×T sync +(i―1)×Δt slot ; Among them, T sync is the synchronization signal period, Δt slot is the time slice allocated to each stimulus signal, n is the synchronization cycle count, t trigger,i is the triggering time of the i-th stimulus signal.

3. The gait analysis method based on multimodal features according to claim 2, characterized in that: The steps of generating stimulation signals, distributing the stimulation signals based on a preset channel distribution matrix, and sending the distributed stimulation signals to each target along the corresponding stimulation channel according to the stimulation signal timing rule include: A discrete phase signal is generated by a numerically controlled oscillator, and a continuous waveform stimulation signal is synthesized according to the discrete phase signal. The frequency expression of the stimulation signal is: Among them, f out is the output frequency of the stimulus signal, f clk is the clock frequency, N p is the number of bits of the phase accumulator, K f is the frequency control word; Define the delay accuracy threshold between stimulation channels. For any two stimulation channels, the following conditions must be met: Δt k,m =t start,m ―t start,k =ΔT0; Where Δt k,m is the delay accuracy between the kth stimulation channel and the mth stimulation channel, t start,m , t start,k are the starting time of the kth stimulation channel and the mth stimulation channel transmitting the stimulation channel, respectively, and ΔT0 is the delay accuracy threshold; Convert the generated stimulus signal into an analog voltage or current signal: Among them, V out (n) is the converted analog voltage or current signal, V ref is the reference voltage, D(n) is the value of the stimulus signal, ∈ dac (n) is the conversion error, and N is the number of bits of the digital-to-analog converter; The converted analog voltage or current signal is allocated to the channel according to the following formula for transmission to the target: C out (t)=M switch (t)×C in (t); Among them, C out (t) is the stimulation signal output to the target, C in (t) is the converted analog voltage or current signal of the input, M switch (t) is the preset channel allocation matrix.

4. The gait analysis method based on multimodal features according to claim 3, characterized in that: The step of collecting the spinal cord electrical signals, myoelectric signals, and brain electrical signals fed back by each target point in response to the stimulation signal comprises: Obtain the original EEG signal fed back by the target, introduce a noise signal into the original EEG signal, and amplify the original EEG signal with the introduced noise signal: y(t)=G·(x(t)+n(t)); Where y(t) is the amplified original EEG signal, G is the amplification factor, x(t) is the original EEG signal, and n(t) is the noise signal; The amplified EEG original signal is filtered to obtain the EEG signal: Y1(f)=y(t)·H1(f); Where H1(f) is the frequency response function of the filter, and Y1(f) is the frequency domain representation of the filtered EEG signal; The first spinal cord original electrical signal and the second spinal cord original electrical signal are collected using a differential amplifier, and the first spinal cord original electrical signal and the second spinal cord original electrical signal are amplified according to the following formula: V out =L·(V in+ ―V in― )+V os ; Among them, V out is the original electrical signal of the spinal cord after differential amplification, V in+ 、V in― are the original electrical signals of the first and second spinal cords, respectively; L is the amplification factor, V is the os is the offset voltage of the differential amplifier; The original spinal cord electrical signal after differential amplification is filtered according to the following formula: Among them, Y2(f) is the frequency domain representation of the spinal cord electrical signal after filtering, X(f) is the frequency domain representation of the original spinal cord electrical signal after differential amplification, j is an imaginary number, f is the frequency of the original spinal cord electrical signal after differential amplification, and f c is the cutoff frequency of the filter; For spinal cord electrical signals, EEG signals, and EMG signals, multi-channel sampling is performed according to the following formula: x i [n]=x i [t i +n·T s ]; Among them, x i [n] is the discrete signal of the i-th channel at the n-th sampling point, x i [t] is the continuous time signal of the i-th channel, t i is the starting sampling time of the i-th channel, n is the sampling point index, T s is the sampling interval.

5. The gait analysis method based on multimodal features according to claim 4, characterized in that: The step of performing spatiotemporal alignment on the spinal cord electrical signal, the myoelectric signal, the EEG signal, and the kinematic signal comprises: The kinematic signal is the spatial coordinates of each marking point and the time corresponding to each spatial coordinate; Get the original space coordinates P of each marker point / electrode orig , and map the original space coordinates to the standard anatomical space: Among them, P std is the target space coordinate mapped to the standard anatomical space, s is the scale factor, R∈R 3×3 is a rotation matrix that satisfies R T R=I, I is the unit matrix, T is the transpose, and U is the translation vector used to compensate for the offset of the coordinate system origin. is the deformation correction term, α j is the weight coefficient corresponding to the j-th control point, C j is the original spatial coordinate of the jth control point, the control point is selected from the brain area or spinal cord area covered by the electrode, and M is the total number of control points; A first low-frequency signal and a first high-frequency signal are screened out from the spinal cord electrical signal, the myoelectric signal, and the brain electrical signal according to a first preset frequency threshold and a second preset frequency threshold, and the first low-frequency signal is interpolated according to the following formula: Among them, S H (t n ) is the first low-frequency signal after interpolation, S L (t m ) is the first low-frequency signal, D is the total number of the first low-frequency signals involved in interpolation, t n is the time point at which the first low-frequency signal after interpolation needs to be calculated, t m is the time position of the first low-frequency signal, ΔT L is the sampling interval of the first low-frequency signal, w is the window function used to limit the influence range of the interpolation kernel, and sinc is used to calculate the interpolation point t n and the original sampling point t m The weight between: The first high-frequency signal is downsampled according to the following formula: Among them, S L (t m ) is the first high-frequency signal after downsampling, E is the downsampling factor, f H is the sampling frequency of the first high-frequency signal, is the impulse response of the filter used for anti-aliasing filtering, is the first high-frequency signal at time point The value at The target space coordinates of the same sampling point at the same time are associated with the interpolated / downsampled signal to obtain the spatiotemporal data: O(t n )=(t n ,P tn ,F1(t n ),F2(t n ),F3(t n )); Among them, O(t n ) represents the tth n The spatiotemporal data of a moment, Indicates the tth n The target space coordinates at the moment, F1(t n )、F2(t n )、F3(t n ) are the spinal cord electrical signals, myoelectric signals and EEG signals after interpolation / downsampling or normal respectively.

6. The gait analysis method based on multimodal features according to claim 5, characterized in that: The step of extracting first feature information, second feature information, third feature information, and fourth feature information corresponding to the spinal cord electrical signal, the electromyographic signal, the electroencephalographic signal, and the kinematic signal, respectively, from the electrical signals and kinematic signals after spatiotemporal alignment includes: The first characteristic information includes field potential oscillation energy and conduction velocity, the second characteristic information includes root mean square value and median frequency, the third characteristic information includes μ / β rhythm power and event-related potential, and the fourth characteristic information includes joint angle and trajectory curvature; The field potential oscillation energy is obtained according to the following formula: Among them, W γ (t,k) is the signal F1(t n ) is the Morlet wavelet transform coefficient at time t and frequency k, where K is the upper limit of the total frequency range of the wavelet transform, and E γ (t) is the field potential oscillation energy; The conduction velocity is given by the signal F1(t n ) is obtained by dividing the propagation distance on the channel by the propagation time; The μ / β rhythm power is obtained according to the following formula: Among them, P μ / β is the μ / β rhythm power, X3(f) is the signal F3(t n ), f1 is the lower and upper limits of integration, corresponding to the frequency range of μ rhythm or β rhythm, respectively. The frequency range of μ rhythm is 8-12, and the frequency range of β rhythm is 13-30; The event-related potential is obtained according to the following formula: Where ERP stands for event-related potential, F is the number of stimulus events used to calculate ERP, S(t-t m ) is the signal F3(t n ) at the stimulus event timestamp t m The surrounding segments, S(t0) is the signal F3(t n ) in the time period t0; The joint angle is obtained according to the following formula: A is the shoulder joint, B is the elbow joint, C is the wrist joint, and θ is the joint angle; The time interval from the first heel touchdown to the next heel touchdown at one of the marker points is defined as the gait cycle.

7. The gait analysis method based on multimodal features according to claim 6, characterized in that: The step of fusing the first feature information, the second feature information, the third feature information, and the fourth feature information to obtain a fused feature includes: The first feature information, the second feature information, the third feature information, and the fourth feature information at the same time are integrated: Among them, Q is the fusion feature, α1, α2, β1, and β2 are weight coefficients used to adjust the influence of different features on the fusion results, ε is a constant, σ is a scale parameter, RMS is the root mean square root, MDF is the median frequency, and T t is the gait period, γ and μ are adjustment coefficients used to control the sensitivity of weight adjustment, θ0 and T0 are the reference joint angle and reference gait period, respectively.

8. The gait analysis method based on multimodal features according to claim 6, characterized in that: The step of inputting the fusion feature into a pre-trained motion intention recognition model and outputting a gait recognition result comprises: Acquire multiple samples of known motion intentions, each sample being a time series data including historical fusion features at consecutive moments, and label each time series data, with the labeling result being the motion intention; Input all labeled time series data into the motion intention recognition model for training to obtain a pre-trained motion intention recognition model; The fused features generated by spatiotemporal data are input into the pre-trained motion intention recognition model to obtain the motion intention.

9. A gait analysis system based on multimodal features, characterized in that: The system comprises: A timing rule construction module is used to define a reference clock and construct stimulation signal timing rules for each target point under the reference clock. Each target point corresponds to at least one stimulation channel. The target points include at least the brain area, spinal cord segment, and muscle to be analyzed. a stimulation signal distribution module, configured to generate stimulation signals, distribute the stimulation signals based on a preset channel distribution matrix, and send the distributed stimulation signals to each target along the corresponding stimulation channel according to the stimulation signal timing rule; A kinematic signal acquisition module is used to obtain motion images of the marker points in the target to be analyzed and obtain kinematic signals of the marker points based on a pre-trained target recognition model; a feature extraction module for collecting spinal cord electrical signals, electromyographic signals, and electroencephalographic signals fed back by each target point in response to the stimulation signal, performing spatiotemporal alignment on the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, and extracting first feature information, second feature information, third feature information, and fourth feature information corresponding to the spinal cord electrical signals, the electromyographic signals, the electroencephalographic signals, and the kinematic signals, respectively, from each of the spatiotemporal aligned electrical and kinematic signals; The feature fusion module is used to fuse the first feature information, the second feature information, the third feature information, and the fourth feature information to obtain a fused feature, and input the fused feature into a pre-trained motion intention recognition model to output a gait recognition result.

10. A readable storage medium, characterized in that: The readable storage medium stores one or more programs, which, when executed by a processor, implement the gait analysis method based on multimodal features according to any one of claims 1 to 8.

Images