Artificial limb driving method, device and equipment based on spiking neural network and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an artificial limb driving method based on a spiking neural network, which comprises the following steps: acquiring a multi-modal signal, and preprocessing the multi-modal signal; converting the preprocessed multi-mode signal into a pulse signal; inputting the pulse signal into a pulse neural network for sensing fusion processing, and outputting a control strategy; dynamically adjusting a control strategy based on pulse reinforcement learning according to the behavior feedback information through a reward function, and generating a learning result; and generating a bionic motion instruction according to the adjusted control strategy and the learning result, and driving an artificial limb execution mechanism to act according to the bionic motion instruction. According to the method, the electromyographic signals, the tactile pressure signals and the inertial measurement data are uniformly coded into the pulse sequence, the pulse neural network is utilized to realize multi-modal sensing fusion, the pulse reinforcement learning algorithm is introduced, the artificial limb control strategy is optimized in real time according to user behavior feedback, and the accurate control of the artificial limb joint is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a prosthetic limb driving method, device, equipment, and storage medium based on a spiking neural network. Background Technology

[0002] Traditional myoelectric prostheses have significant limitations in terms of technological application and practical adaptability. Their core problems lie in three key dimensions: signal dependence, movement adaptation, and intent understanding. First, these prostheses rely solely on surface electromyography (EMG) signals for control. Affected by factors such as signal acquisition stability and environmental interference, the misrecognition rate of movements typically exceeds 30%. This not only reduces the accuracy of prosthetic operation but may also lead to erroneous movements in amputees, increasing operational risks and user frustration. Second, their movement execution depends on a pre-programmed, fixed action library. This model cannot flexibly adapt to complex and changing real-world scenarios. For example, when faced with two actions with drastically different requirements for force and angle—holding an egg (requiring gentle force to avoid breakage) and holding a key (requiring precise force to control rotation)—the pre-programmed action library struggles to dynamically adjust according to object characteristics and operational needs, significantly reducing the prosthesis's practicality in real-life situations. Finally, traditional myoelectric prostheses lack the ability to understand the amputee's intentions. Users must consciously imagine preset fixed movement patterns to trigger corresponding operations, rather than naturally generating movements according to needs. This passive and rigid control method not only increases the user's cognitive burden, but also makes it difficult to achieve natural coordination with body movements, further limiting its widespread application in daily life. Summary of the Invention

[0003] The main objective of this invention is to provide a prosthesis driving method, device, equipment, and storage medium based on a spiking neural network, aiming to solve the problems of existing prostheses having a single perception dimension, rigid movement, and lack of cognition.

[0004] To achieve the above objectives, the present invention provides a prosthesis actuation method based on a spiking neural network, comprising: Acquire multimodal signals and preprocess the multimodal signals; The preprocessed multimodal signal is converted into a pulse signal; The pulse signal is input into a spiking neural network for perception fusion processing, and a control strategy is output. By using a reward function, the control strategy is dynamically adjusted based on behavioral feedback information and impulse reinforcement learning to generate learning results. Based on the adjusted control strategy and learning results, bionic motion commands are generated, and the prosthetic actuator is driven to move according to the bionic motion commands.

[0005] Furthermore, to achieve the above objectives, the present invention provides a prosthetic limb actuation device based on a spiking neural network, comprising: The signal processing module is used to acquire multimodal signals and preprocess the multimodal signals; The signal conversion module is used to convert the preprocessed multimodal signal into a pulse signal; The control strategy module is used to input the pulse signal into the pulse neural network for perception fusion processing and output a control strategy; The learning outcome module is used to dynamically adjust the control strategy based on behavioral feedback information using a reward function and impulse reinforcement learning to generate learning outcomes. The motion execution module is used to generate bionic motion commands based on the adjusted control strategy and learning results, and drive the prosthetic actuator to move according to the bionic motion commands.

[0006] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and a spiking neural network-based prosthesis driver program stored in the memory and executable on the processor, wherein when the spiking neural network-based prosthesis driver program is executed by the processor, it implements the steps of the spiking neural network-based prosthesis driving method described above.

[0007] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a prosthesis driver program based on a spiking neural network, wherein when the spiking neural network-based prosthesis driver program is executed by a processor, it implements the steps of the spiking neural network-based prosthesis driving method as described above.

[0008] Beneficial Effects: This invention relates to the field of artificial intelligence technology and discloses a prosthetic limb actuation method based on a spiking neural network, comprising: acquiring multimodal signals and preprocessing the multimodal signals; converting the preprocessed multimodal signals into pulse signals; inputting the pulse signals into a spiking neural network for perception fusion processing and outputting a control strategy; dynamically adjusting the control strategy based on behavioral feedback information using a reward function and spiking reinforcement learning to generate learning results; generating bionic motion commands based on the adjusted control strategy and learning results, and driving the prosthetic limb actuator to move according to the bionic motion commands. This invention unifies electromyographic signals, tactile pressure signals, and inertial measurement data into pulse sequences, utilizes a spiking neural network to achieve multimodal perception fusion, and introduces a spiking reinforcement learning algorithm to optimize the prosthetic limb control strategy in real time based on user behavior feedback, thereby improving the precise control of the prosthetic joint. Attached Figure Description

[0009] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for a prosthetic limb driving method based on a spiking neural network according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the prosthetic limb actuation method based on a spiking neural network according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the prosthetic drive device based on a spiking neural network of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0010] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0011] The prosthetic actuation method based on a spiking neural network provided in this invention can be applied to, for example... Figure 1 In this application environment, the user terminal communicates with the server via a network. The server can collect multimodal signals from the user terminal and preprocess them; convert the preprocessed multimodal signals into pulse signals; input the pulse signals into a spiking neural network for perception fusion processing and output a control strategy; dynamically adjust the control strategy based on behavioral feedback information using a reward function and spiking reinforcement learning to generate learning results; generate bionic motion commands based on the adjusted control strategy and learning results, and drive the prosthetic actuator to move according to the bionic motion commands. This invention encodes electromyographic signals, tactile pressure signals, and inertial measurement data into pulse sequences, uses a spiking neural network to achieve multimodal perception fusion, and introduces a spiking reinforcement learning algorithm to optimize the prosthetic control strategy in real time based on user behavior feedback, thereby improving the precise control of the prosthetic joint. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0012] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the prosthetic limb actuation method based on a spiking neural network provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0013] like Figure 2As shown, the prosthetic limb actuation method based on spiking neural networks proposed in this invention includes the following steps: S100. Acquire multimodal signals and preprocess the multimodal signals; S200: Convert the preprocessed multimodal signal into a pulse signal; S300: Input the pulse signal into a pulse neural network for perception fusion processing, and output a control strategy; S400: Through the reward function, the control strategy is dynamically adjusted based on the behavioral feedback information and impulse reinforcement learning to generate learning results; S500: Generate bionic motion commands based on the adjusted control strategy and learning results, and drive the prosthetic actuator to move according to the bionic motion commands.

[0014] In this embodiment, multimodal signal acquisition is the foundation for the system to obtain user action intentions and environmental interaction information. Dedicated sensors must be selected for different signal types, and differentiated preprocessing must be performed to ensure signal quality meets subsequent processing requirements. Specifically, electromyography (EMG) signals are acquired using high-density electromyography (HD-EMG), which captures the weak electrical signals generated during muscle contraction. Each channel has a sampling frequency of 2000 Hz (2 kHz), with a total of 256 channels, comprehensively covering the major motor muscle groups of the amputation stump. Tactile pressure signals are acquired through electronic skin sensors, which can sense environmental information such as pressure, vibration, and temperature. These sensors have a sampling frequency of 1000 Hz (1 kHz) and can monitor force feedback when the prosthesis contacts an object in real time. Inertial measurement data is acquired by an inertial measurement unit (IMU). The selected IMU device has a sampling frequency of 500 Hz and can record the motion posture and acceleration changes of the prosthetic joints. The preprocessing stage requires eliminating signal noise and standardizing data format: For electromyography (EMG) signals, a 50 Hz notch filter is first used to remove mains interference, followed by a 20-450 Hz bandpass filter to retain effective signals related to muscle contraction. Finally, full-wave rectification is performed to convert bidirectional electrical signals into unidirectional signals, highlighting changes in muscle contraction intensity. For tactile pressure signals, a 50 ms sliding average filter is used to smooth transient impact noise, followed by dynamic range normalization according to a 0-100 N range to ensure horizontal comparison of different pressure levels. Inertial measurement data is filtered with a 20 Hz low-pass filter to suppress vibration noise, and then a quaternion attitude calculation algorithm is used to calculate the precise attitude angles of the prosthesis joints (such as elbow flexion angle and knee extension angle) to eliminate equipment zero-drift error and ensure the accuracy of attitude data.

[0015] Converting preprocessed continuous signals into pulse signals is to adapt to the biological characteristics of spiking neural networks (SNNs) and achieve a unified representation of multimodal signals. The conversion process follows two core rules: time encoding and intensity encoding. Time encoding allocates delays based on the sensor channel number (channel_id), allowing signals from different channels to be distinguished by time differences. For example, eight channels are grouped together, and the pulse trigger time of each channel is the product of the signal event time (t_event) plus a fixed coefficient and the channel number modulo the result, simulating the spatiotemporal encoding logic of biological neurons. Intensity encoding normalizes the signal intensity and maps it to pulse frequency. The higher the signal intensity, the higher the pulse frequency. For example, when the muscle contraction intensity is 50%, the corresponding pulse frequency is about 85 Hz; when the pressure is 30 Newtons, the corresponding channel's pulse frequency is about 120 Hz; and when the wrist is flexed at 30°, a pulse sequence of a specific frequency is generated through phase encoding.

[0016] The final generated pulse signal contains complete spatiotemporal information and is uniformly formatted as a data packet containing a timestamp, channel ID, and instantaneous frequency: the timestamp accuracy reaches the microsecond (μs) level to ensure time synchronization of signals of different modalities; the channel ID is used to identify the signal source; and the instantaneous frequency reflects the intensity characteristics of the original signal, providing a clear input dimension for subsequent neural network processing, allowing electromyography, tactile, and inertial measurement signals to be parsed by the SNN within the same framework.

[0017] The perceptual fusion process of a spiking neural network simulates the information processing logic of the human cerebral cortex, achieving the integration of multimodal signals and intention decision-making through a multi-level network structure. First, based on a global time reference, a unified timestamp is added to all pulse signals. Then, electromyographic (EMG), tactile, and inertial measurement (IMT) pulse signals are grouped according to a preset fusion time window of 10 milliseconds, ensuring that signals within the same time window correspond to the same action stage. In the fusion layer, the network adjusts synaptic weights using the Spike-Timing-Dependent Plasticity (STDP) algorithm: when the time difference between EMG and tactile pulses is less than 5 milliseconds, the synaptic weights between the two types of signals are strengthened, gradually forming neural pathways for specific actions. Simultaneously, initial weights are assigned to different modalities based on signal reliability. For example, EMG signals have the highest recognition accuracy, so the initial weight is set to 0.4, while the initial weights for tactile and IMT signals are each set to 0.3, balancing the contribution of multimodal information. Finally, a control strategy is generated through pulse density voting: the number of pulses (pulse density) of each mode within each time window is counted, and the total decision value is calculated by combining the synaptic weights and the initial weights. When the decision value exceeds the preset threshold, a clear action control strategy is output, such as "stable grip" (corresponding to the low-frequency continuous pulse mode) or "rapid grasp" (corresponding to the high-frequency burst pulse mode). At the same time, environmental context information is considered. For example, when the electronic skin detects that the cup is too hot, the adjustment instruction of "reducing grip strength to prevent burns" is automatically added to the control strategy to make the strategy more in line with the actual use scenario.

[0018] Pulse reinforcement learning optimizes control strategies through user behavior feedback, ensuring that prosthetic movements conform to user habits and adapt to long-term changes (such as muscle atrophy). First, three types of core feedback information are collected in real time: physiological feedback (a decrease in electromyographic signal frequency exceeding 15% indicates muscle fatigue), behavioral feedback (user ratings from 1 to 5 are converted into quantitative data using speech recognition technology, with 1 indicating "laborious" and 5 indicating "comfortable"), and safety feedback (the user triggering an emergency stop button indicates an abnormal movement, and a tactile sensor detecting a stable grip on an object indicates a successful movement).

[0019] The reward function acts as a bridge between feedback and strategy adjustment, calculating the reward value by comprehensively considering task success, energy consumption, and user experience: task success (such as successfully gripping a cup), low energy consumption (low electromyographic signal intensity, saving user effort), and high user ratings result in positive rewards, while the opposite results in negative rewards. Based on the reward value, the system optimizes the control strategy by adjusting the synaptic weights of the SNN. For example, when a user reports "difficulty gripping a pen," the negative reward triggers synaptic weight adjustment, reducing the weight of grip strength-related pathways and decreasing grip strength. The weights are updated after each action, and the strategy typically converges after about 12 iterations (e.g., optimizing the grip strength from 5 Newtons to 2 Newtons, improving the user rating from 2 to 4 points). The optimized strategy is stored as the learning result and directly invoked when performing the same action again, improving response speed.

[0020] Based on the adjusted control strategy and learning results, bionic motion commands are generated. This requires establishing a mapping relationship between pulse signals and prosthetic motion parameters to simulate the rhythm and precision of natural human movements. First, the mapping rules for core motion parameters are clarified: grip strength is positively correlated with pulse density—the higher the pulse density, the greater the grip strength; movement speed is positively correlated with pulse frequency—the higher the pulse frequency, the faster the joint movement; joint angles are determined by both pulse phase and pulse density—for example, pulse density determines the amplitude of movement, while pulse phase determines the direction of movement.

[0021] Building upon this foundation, a human-like movement rhythm is generated using a Central Pattern Generator (CPG). This involves extracting the pulse density and instantaneous frequency of approximately 10 milliseconds from an electromyographic pulse sequence, adjusting the CPG's amplitude (to control the range of motion) and frequency (to control the speed of movement), and generating smooth joint angle trajectories (such as the curve of the elbow joint flexing from 0° to 90°), avoiding the stiffness of mechanical movement. These joint angle trajectories are then converted into standardized movement commands, which are combined with contextual adjustments in the control strategy (such as reducing grip strength when a hot cup is being used, and increasing grip strength when an object slips) to generate the final biomimetic movement commands.

[0022] The prosthetic actuator uses hydraulic artificial muscles combined with pulse width modulation (PWM) drive technology. After receiving bionic motion commands, it controls the extension and contraction of the hydraulic artificial muscles through PWM signals to drive the joint to move along a preset trajectory. The response delay is less than 5 milliseconds, and the angle control accuracy reaches 0.1°, ensuring the real-time performance and precision of the movement. At the same time, a closed-loop feedback is formed through joint torque sensors and electronic skin to monitor motion errors in real time (such as the deviation between the actual angle and the target angle exceeding 1°), dynamically adjust the PWM signal, and correct the movement trajectory, so that the prosthetic movement can not only conform to the user's intention, but also adapt to changes in the environment (such as grasping objects of different shapes and weights).

[0023] In one embodiment, step S100 includes: S101, Preset the sampling frequency for each signal; S102. Collect electromyographic signals, tactile pressure signals, and inertial measurement data according to the preset sampling frequency; S103. Perform notch filtering, bandpass filtering and full-wave rectification on the electromyographic signal; S104. Perform sliding mean filtering and dynamic range normalization on the tactile pressure signal. S105. Perform low-pass filtering and quaternion attitude calculation on the inertial measurement data.

[0024] In this embodiment, in the multimodal perception-cognition fusion intelligent prosthetic system driven by a spiking neural network, the acquisition and preprocessing of multimodal signals are the fundamental steps for obtaining high-quality input data and ensuring the accuracy of subsequent pulse coding and neural network fusion.

[0025] The sampling frequency for each signal is pre-set, a dedicated acquisition rate determined based on the characteristics of different signals and system processing requirements. This ensures that the signals can fully retain key information without increasing the system burden due to redundant data. Specifically, electromyography (EMG) signals, which need to capture the subtle electrical activity of muscle contraction, are sampled at a frequency of 2000 Hz (2 kHz); tactile pressure signals, which need to reflect the dynamic changes in contact force in real time, are sampled at a frequency of 1000 Hz (1 kHz); and inertial measurement data, used to calculate prosthetic joint posture, are sampled at a frequency of 500 Hz (500 Hz). The sampling frequency settings for these three types of signals are matched to the performance parameters of the corresponding sensors, providing accurate raw data for subsequent processing.

[0026] When collecting electromyography (EMG), tactile pressure, and inertial measurement data at a preset sampling frequency, specialized sensors are required for accurate signal capture. EMG signals are acquired using a high-density electromyography (HD-EMG) sensor with 256 acquisition channels, comprehensively covering the major motor muscle groups of the amputation stump and capturing the weak electrical signals generated during muscle contraction. Tactile pressure signals are acquired by an electronic skin sensor, which can simultaneously sense pressure, vibration, and temperature. Deployed at contact points such as prosthetic fingers and feet, it obtains real-time force feedback information from interactions with objects. Inertial measurement data is acquired through an inertial measurement unit (IMU), a device with a built-in accelerometer and gyroscope installed at prosthetic joints (such as the elbow and knee joints) to record the acceleration and angular changes of joint movement, providing data support for posture calculation.

[0027] Notch filtering, bandpass filtering, and full-wave rectification are performed on electromyographic (EMG) signals to eliminate noise interference and highlight effective signal characteristics. Notch filtering primarily targets 50 Hz mains interference, the most common noise source during EMG signal acquisition. This process filters out noise at the mains frequency, preventing interference with muscle electrical activity signals. Bandpass filtering limits the signal frequency range to 20-450 Hz, a frequency band that concentrates effective EMG signals related to muscle contraction. It filters out low-frequency motion artifacts (such as interference from residual limb movement) and high-frequency electronic noise. Full-wave rectification converts the originally bidirectional fluctuating EMG signal into a unidirectional signal, making the signal intensity change trend clearer and facilitating the subsequent extraction of muscle contraction intensity characteristics, providing a clear intensity basis for pulse coding.

[0028] The core of applying sliding mean filtering and dynamic range normalization to tactile pressure signals is to smooth signal fluctuations and standardize the data range. Sliding mean filtering uses a 50-millisecond window and calculates the pressure signal in 10-millisecond steps, effectively smoothing instantaneous impact noise when contacting objects (such as sudden pressure changes upon grasping an object) while maintaining real-time signal quality and avoiding the loss of crucial force change information due to excessive smoothing. Dynamic range normalization standardizes the pressure signal to a range of 0-100 Newtons (N), covering most pressure scenarios in daily prosthetic use (such as grasping a cup or pressing a button). This allows for horizontal comparison of signals at different pressure levels, eliminating range deviations caused by individual sensor differences or environmental factors, and ensuring consistency in subsequent pulse coding.

[0029] Low-pass filtering and quaternion attitude calculation are performed on inertial measurement data to suppress noise and calculate accurate joint postures. The low-pass filtering uses a 20 Hz cutoff frequency to filter out high-frequency noise generated by IMU vibrations (such as joint jitter during walking), preserving stable joint posture signals. Quaternion attitude calculation, by fusing data from the gyroscope and accelerometer, uses a quaternion algorithm to calculate precise posture angles of the prosthetic joints (such as elbow flexion and knee extension). This algorithm effectively avoids the gimbal lock problem in traditional Euler angle calculation, ensuring the continuity and accuracy of posture calculations. The calculation error can be controlled within 0.5°, providing accurate joint posture references for subsequent generation of bionic motion commands and ensuring consistency between prosthetic movements and user intentions.

[0030] In one embodiment, step S200 includes: S201. Predefine time encoding rules based on the channel array; S202. Map the signal strength to a pulse frequency, and predefine the strength encoding rules according to the pulse frequency; S203. Convert the preprocessed multimodal signal into a pulse signal with a timestamp according to the time encoding rule and intensity encoding rule.

[0031] In this embodiment, in the multimodal perception-cognition fusion intelligent prosthetic system driven by a spiking neural network, converting the preprocessed multimodal signal into a pulse signal is a key link connecting signal acquisition and neural network processing, which is achieved through time coding rules and intensity coding rules.

[0032] Based on a predefined time encoding rule for the channel array, the core principle is to distinguish signals from different sources using sensor channel numbers (channel_id), simulating the spatiotemporal encoding characteristics of biological neurons. The channel array covers all acquisition channels for electromyography (EMG) signals, tactile pressure signals, and inertial measurement data (e.g., 256 channels for EMG signals and multiple channels for tactile signals divided by contact area). The system assigns a unique channel ID to each channel. The time encoding rule is set as follows: the pulse trigger time (t_spike) of each channel is based on the signal event time (t_event, i.e., the moment when the signal reaches an effective threshold, such as the peak moment of EMG signals or the trigger moment of tactile pressure changes), plus a fixed delay coefficient multiplied by the modulo of the channel ID. For example, with 8 channels as a group and a delay coefficient set to 0.2 milliseconds (ms), when a certain channel ID is 5, its pulse delay is 5 × 0.2 ms = 1 ms. Ultimately, the pulse trigger time of this channel is the signal event time plus 1 ms. This rule allows signals from different channels to be clearly distinguished by time difference. Even if different modal signals are triggered at similar times, the source of the signal can be accurately identified by the corresponding delay difference of the channel, providing clear spatiotemporal dimension information for the subsequent multimodal fusion of the spiking neural network.

[0033] Mapping signal intensity to pulse frequency and predefining intensity encoding rules aims to quantify the intensity characteristics of the original signal using pulse frequency, allowing the intensity of different modal signals to be represented through a unified pulse frequency dimension. First, the system normalizes the preprocessed multimodal signals (e.g., electromyographic signals are normalized to the 0-1 range, and tactile pressure signals are normalized to the 0-100 Newton range). Then, a mapping relationship is established between the normalized signal intensity and the pulse frequency: the higher the signal intensity, the higher the corresponding pulse frequency, and vice versa. For example, when the normalized intensity of an electromyographic signal is 50%, the corresponding pulse frequency is approximately 85 Hz; when the tactile pressure signal is 30 Newtons, the corresponding channel's pulse frequency is approximately 120 Hz; and when the joint angle change rate in inertial measurement data is at a moderate level, the corresponding pulse frequency is approximately 50 Hz. At the same time, the system sets upper and lower limits for pulse frequency (such as a minimum of 10 Hz and a maximum of 200 Hz) to prevent extreme signal strength from causing the frequency to exceed the processing range of the neural network, ensuring that the intensity mapping logic of different modal signals is consistent, and laying the foundation for the unified analysis of multimodal signals in the spiking neural network.

[0034] Converting preprocessed multimodal signals into timestamped pulse signals according to time-coding and intensity-coding rules is the final step in the encoding process, requiring the integration of rules from the first two steps to generate a pulse data packet with a unified structure. First, for each preprocessed signal (e.g., a specific channel of an EMG signal or a specific contact area of ​​a tactile signal), the system captures the signal's event time (t_event), then calculates the pulse delay corresponding to that channel based on the time-coding rules, determining the precise timestamp of the pulse trigger (with microsecond-level accuracy, μs). Simultaneously, based on the signal's normalized intensity and combined with intensity-coding rules, the instantaneous frequency of the pulse is determined. The final generated pulse signal is uniformly formatted as a data packet containing "timestamp, channel ID, and instantaneous frequency": the timestamp identifies the precise moment of pulse triggering, ensuring time synchronization of multimodal signals; the channel ID identifies the signal source (e.g., EMG channel, tactile channel), further clarifying the signal category based on the delay differences from time-coding; and the instantaneous frequency reflects the intensity characteristics of the original signal, quantifying the effective information of the signal. For example, if an event is triggered in a certain channel of an electromyography (EMG) signal at 1000 μs, with channel ID 3 and a normalized intensity corresponding to a frequency of 85 Hz, the resulting pulse signal will be "timestamp: 1000 + 3 × 0.2 × 1000 = 1600 μs, channel ID: 3, instantaneous frequency: 85 Hz". Through this conversion, continuous signals from three different modalities—EMG, tactile, and inertial measurement—are all converted into structurally unified and information-complete pulse signals, which can be directly input into a spiking neural network (SNN) for subsequent sensory fusion processing.

[0035] In one embodiment, step S300 includes: S301. Based on a global time reference, add a global timestamp to each pulse signal; S302. Group the electromyographic pulse signal, tactile pressure pulse signal, and inertial measurement pulse signal according to the preset fusion time window; S303. Obtain the timestamps of pulse signals of different modalities within the same time group, and calculate the time difference between electromyographic pulse signals and tactile pressure pulse signals, and the time difference between electromyographic pulse signals and inertial measurement pulse signals. S304. Compare the time difference with the difference threshold. When the time difference is less than the difference threshold, enhance the synaptic weights between the corresponding modalities according to the STDP algorithm to generate a specific action neural pathway. S305. Assign initial weights to each pulse signal based on signal reliability; S306. Based on synaptic weights and initial weights, the action intention decision is made by voting through pulse density, and the control strategy is output.

[0036] In this embodiment, in the multimodal perception-cognition fusion intelligent prosthetic system driven by a spiking neural network (SNN), inputting pulse signals into the SNN for perception fusion and outputting control strategies is the core link to realize the "perception-cognition" transformation, which needs to be completed through three steps: time synchronization, cross-modal weight adjustment, and intention decision-making.

[0037] Adding a global timestamp to each pulse signal based on a global time reference is fundamental to ensuring time synchronization of multimodal pulse signals. The system establishes a unified global time reference (with microsecond-level accuracy, μs), provided by a hardware clock (such as a high-precision clock inside an FPGA), ensuring consistent time measurement standards for all pulse signals. Whether it's an electromyographic pulse, a tactile pressure pulse, or an inertial measurement pulse, it is immediately assigned a global timestamp after generation. This timestamp records the precise moment of pulse triggering, eliminating time deviations caused by hardware delay differences between different sensors. For example, if an electromyographic signal channel generates a pulse at a local time of 1000 μs, after global time calibration, if the hardware delay of that channel is 200 μs, the global timestamp will be recorded as 1200 μs; similarly, if a tactile signal channel generates a pulse at a local time of 800 μs with a hardware delay of 100 μs, the global timestamp will be recorded as 900 μs. Through the global timestamp, pulse signals from different modalities and channels can be correlated in the same time dimension, providing accurate timing data for subsequent time-grouped fusion.

[0038] Grouping the three types of pulse signals according to a preset fusion time window is key to integrating multimodal signals by action stage. The system's preset fusion time window width is 10 milliseconds (ms). This width is determined based on the temporal characteristics of human actions—the key stages of a single fine human action (such as grasping or extending) are typically completed within 10 ms. Using this window width ensures that pulse signals within the same time window correspond to the same action stage. During grouping, the system iterates through all pulse signals with global timestamps, grouping electromyographic, tactile, and inertial measurement pulses whose timestamps fall within the same 10 ms interval. For example, electromyographic, tactile, and inertial measurement pulses with global timestamps in the 1000-1010 μs interval are grouped together, corresponding to the "initial contact with the object during grasping" stage; pulses in the 1010-1020 μs interval are grouped into the next group, corresponding to the "increased grasping force" stage. Through grouping, the originally scattered multimodal pulses are associated according to the action sequence, forming structured pulse groups, laying the foundation for cross-modal fusion.

[0039] Obtaining the time difference between different modal pulses within the same group and adjusting synaptic weights based on the difference threshold is the core of constructing specific motor neural pathways using the Spike-Timing-Dependent Plasticity (STDP) algorithm. First, the system extracts global timestamps from electromyographic (EMG), tactile pressure (TP) pulses, and inertial measurement (IGM) pulses within the same group. It then calculates the time difference between EMG and tactile pulses (e.g., EMG pulse timestamp 1005μs, tactile pulse 1008μs, difference 3μs) and between EMG and IGM pulses (e.g., EMG pulse 1005μs, IGM pulse 1006μs, difference 1μs). For example, with a preset difference threshold of 5ms, when the calculated time difference is less than 5ms, it indicates that the two types of signals are highly synchronized in time, corresponding to the same motor intention (e.g., EMG signals reflect muscle contraction, tactile signals reflect object contact; synchronous triggering indicates the user is performing a grasping action). In this case, the system uses the STDP algorithm to enhance the synaptic weights between the neurons corresponding to these two types of signals. For example, if the time difference between electromyographic (EMG) pulses and tactile pulses is 3 μs < 5 ms, the synaptic weight between them will increase. As the same action is repeatedly performed, this synaptic weight will continue to increase, gradually forming a "grasping action neural pathway". If the time difference is greater than 5 ms (e.g., EMG pulse 1005 μs, tactile pulse 1012 μs, difference 7 μs), it indicates that the signals are not synchronized. The synaptic weight is not adjusted to avoid forming an ineffective neural pathway.

[0040] Assigning initial weights to each pulse signal based on its reliability is a prerequisite for balancing the contributions of multimodal signals and ensuring decision-making accuracy. Signal reliability is determined by both historical data and sensor performance: Electromyography (EMG) signals are acquired by high-density electromyography (HD-EMG) sensors, which have a large number of channels and high recognition accuracy (e.g., grasp recognition accuracy > 98%), thus having the highest reliability and an initial weight of 0.4; tactile pressure signals are acquired by electronic skin sensors, which can provide real-time feedback on contact status, but are susceptible to environmental interference (such as humidity affecting sensitivity), thus having the second highest reliability and an initial weight of 0.3; inertial measurement data is acquired by inertial measurement units (IMUs), which mainly reflect posture changes and are greatly affected by vibration, resulting in relatively low reliability, also with an initial weight of 0.3. The initial weights are not fixed. If the recognition accuracy of a certain type of signal continues to decline during subsequent use (e.g., EMG signal accuracy decreases due to muscle atrophy), the system will dynamically adjust its initial weight to ensure that the most reliable signal always dominates the decision-making process.

[0041] The ultimate goal of the fusion process is to generate action intent decisions and output control strategies based on pulse density voting using synaptic weights and initial weights. First, the system counts the number of pulses of each type within the same pulse group (i.e., pulse density, e.g., 8 EMG pulses, 5 tactile pulses, and 6 inertial measurement pulses in a group), and calculates the "decision contribution value" of each type of signal by combining the two weights—contribution value = pulse density × synaptic weight × initial weight. For example, with an EMG pulse density of 8, a synaptic weight of 1.2 (after STDP enhancement), and an initial weight of 0.4, the contribution value is 8 × 1.2 × 0.4 = 3.84; with a tactile signal pulse density of 5, a synaptic weight of 1.1, and an initial weight of 0.3, the contribution value is 5 × 1.1 × 0.3 = 1.65; and with an inertial measurement signal pulse density of 6, a synaptic weight of 1.0, and an initial weight of 0.3, the contribution value is 6 × 1.0 × 0.3 = 1.8. Subsequently, the system sums the contribution values ​​of the three types of signals to obtain a total decision value (3.84 + 1.65 + 1.8 = 7.29), and compares it with a preset decision threshold (e.g., 5.0). If the total decision value exceeds the threshold, the intention of the action is determined based on the contribution ratio of each type of signal—the electromyographic signal has the highest contribution ratio (3.84 / 7.29 ≈ 52.7%), and combined with its pulse pattern (e.g., low-frequency continuous pulse), the intention is determined to be "stable grip," and the control strategy of "controlling the prosthetic finger to hold the object steadily with a force of 2.8N" is finally output. If the total decision value does not reach the threshold, it is determined that the current signal is insufficient to clarify the intention, and no control strategy is output to avoid erroneous actions.

[0042] In one embodiment, step S400 includes: S401. Obtain user behavior feedback information; S402. Calculate the reward value of behavioral feedback information using a reward function; S403. Based on the reward value, dynamically adjust the control strategy using pulse reinforcement learning to generate learning results.

[0043] In this embodiment, in the multimodal perception-cognition fusion intelligent prosthetic system driven by a spiking neural network (SNN), optimizing the control strategy through user behavior feedback is the core link to achieve adaptive evolution of the system, which is specifically carried out in three steps: "acquiring feedback information, calculating reward value, and dynamically adjusting strategy".

[0044] Obtaining user behavioral feedback is fundamental for the system to perceive user experience and action effects. Feedback is mainly divided into three categories: physiological feedback, behavioral feedback, and safety feedback, covering the core states and needs of users during use. Physiological feedback is monitored in real-time by electromyography (EMG) sensors—when a user's muscles are fatigued, the frequency of EMG signals will decrease significantly; if the frequency decrease rate exceeds 15%, it is judged as "excessive muscle load." Behavioral feedback is collected through a voice interaction module. The system allows users to rate the action effect in real-time via voice (1-5 points, where 1 point represents "laborious and inflexible movement," and 5 points represent "natural and comfortable movement"). The voice rating is then converted into quantifiable data that the system can process using Automatic Speech Recognition (ASR) technology. Safety feedback is obtained through an emergency stop button and tactile sensors—when a user notices abnormal movement (such as excessive force from a prosthesis that may damage an object, or a movement trajectory deviating from the expected path), the emergency stop button can be triggered, and the system judges this as "high-risk negative feedback." If the tactile sensor detects a stable grip on an object (without signs of slippage), it is judged as "successful action positive feedback." The three types of feedback information are transmitted to the system processing module in real time, providing a comprehensive basis for subsequent reward calculations.

[0045] Calculating reward values ​​for behavioral feedback information using reward functions is key to transforming qualitative feedback and quantitative data into unified optimization metrics. The reward function comprehensively considers three core factors: task success rate, energy consumption, and user experience, ensuring that the reward value objectively reflects the action's effectiveness. Task success rate is judged by whether the expected action was completed. For example, successfully grasping a cup or smoothly placing an object is considered a "task success," while the object slipping or the action failing to reach the target position is considered a "task failure." Energy consumption is referenced to electromyography (EMG) signal intensity; low EMG signal intensity (indicating less effort from the user) results in low energy consumption, while high intensity (indicating more effort from the user) results in high energy consumption. User experience is directly related to voice ratings; higher ratings indicate a better user experience. The reward function generates the final reward value through weighted calculation: task success, low energy consumption, and high user ratings result in positive rewards, while the opposite results in negative rewards. For example, "successfully grasping the cup + user rating of 5 + low EMG signal intensity" will receive a high positive reward, while "object slipping + user rating of 1 + high EMG signal intensity" will receive a significant negative reward, clarifying the optimization direction for subsequent strategy adjustments.

[0046] Dynamically adjusting the control strategy based on reward values ​​using spiking reinforcement learning and generating learning results is the core execution step for the system to adapt to user habits and improve the user experience. Spiking reinforcement learning is reward-oriented, optimizing the control strategy by adjusting the synaptic weights in the spiking neural network. When a positive reward is obtained, the system strengthens the synaptic weights of neural pathways related to the current action (such as the electromyographic-tactile synaptic weights corresponding to the "smooth grip" action), making subsequent executions of the same action more consistent with the current successful pattern. When a negative reward is obtained, the system reduces the synaptic weights of related neural pathways and explores better synaptic connection methods (such as reducing the weights of pathways related to gripping force and adjusting parameters related to joint movement speed). For example, when a user reports "difficulty gripping a pen" (negative reward), the system reduces the synaptic weights of the gripping force control pathway, reducing the gripping force of the prosthetic limb. After 3-5 action iterations, if the user's score improves to 4 points (positive reward), the adjusted synaptic weights are fixed, generating the learning result of "optimized strategy for pen gripping action," which is stored in the system's local cache. When the user performs the pen-holding action later, the system can directly call the learning results without iterating and optimizing again. At the same time, after each new action is completed, the system will repeat the "feedback-reward-adjustment" process to continuously update the learning results, so that the control strategy gradually adapts to the user's long-term behavioral changes (such as the weakening of electromyographic signals caused by muscle atrophy), and realizes the dynamic evolution of prosthetic performance.

[0047] In one embodiment, step S500 includes: S5011. Establish a motion-pulse mapping relationship based on motion parameters and pulse signals; S5012. By extracting the electromyographic pulse density and electromyographic frequency characteristics from the pulse signal, a joint angle trajectory is generated. S5013. Generate standardized motion commands based on joint angle trajectories and motion-pulse mapping relationships; S5014. Generate bionic motion instructions based on standardized motion commands, adjusted control strategies, and learning results, and drive the prosthetic actuator to move according to the bionic motion instructions.

[0048] In this embodiment, in the multimodal perception-cognition fusion intelligent prosthetic system driven by the Spiking Neural Network (SNN), the conversion from pulse signals to prosthetic movements needs to be achieved through four steps: "motion-pulse mapping establishment - joint angle trajectory generation - standardized motion command output - bionic motion command driving", ultimately enabling the prosthetic to perform natural movements that conform to the user's intentions.

[0049] Establishing a motion-pulse mapping relationship based on motion parameters and pulse signals is fundamental to achieving precise correlation between pulse signals and prosthetic movements. The core is to map the key features of the pulse signals to the core motion parameters of the prosthesis, forming a quantifiable mapping logic. Motion parameters mainly include grip strength (unit: Newtons, N), movement speed (unit: degrees / second, ° / s), and joint angle (unit: degrees, °), which directly determine the prosthetic's movement effect. Pulse signal features focus on electromyographic pulse density (number of pulses per unit time), electromyographic pulse frequency (unit: Hertz, Hz), and tactile pulse intensity, which are the core basis for reflecting the user's movement intentions. For example, grip strength is positively correlated with electromyographic pulse density—higher pulse density indicates greater muscle contraction strength, corresponding to a stronger grip on the prosthesis; movement speed is positively correlated with electromyographic pulse frequency—higher pulse frequency indicates a more rapid user movement intention, corresponding to a faster joint movement speed in the prosthesis; joint angle is mapped through electromyographic pulse phase and inertial measurement pulse characteristics to ensure that the joint rotation amplitude matches the user's expectations. This mapping relationship is calibrated in conjunction with clinical test data. For example, the corresponding values ​​of gripping force and pulse density are calibrated by using standard weights (1N, 5N), and the corresponding values ​​of movement speed and pulse frequency are calibrated by recording joint movement trajectories with high-speed cameras. Finally, a stable and accurate mapping rule is formed, which provides a basis for subsequent instruction generation.

[0050] Generating joint angle trajectories by extracting the electromyographic pulse density and frequency features from the pulse signal is a key step in transforming abstract pulse characteristics into concrete motion trajectories, simulating the smoothness and continuity of human joint movements. First, the system uses 10-millisecond (ms) time windows to count the number of electromyographic pulses within each window, obtaining the electromyographic pulse density (e.g., if there are 8 electromyographic pulses in a window, the pulse density is 8 / 10ms). Simultaneously, the system calculates the time interval between adjacent electromyographic pulses, converting it to the electromyographic pulse frequency (e.g., if the interval between adjacent pulses is 100 microseconds, the frequency is 10000 Hz). Subsequently, based on the preset motion-pulse mapping relationship, the pulse density is converted into joint motion amplitude (e.g., a pulse density of 8 pulses / 10ms corresponds to a joint flexion of 45°), and the pulse frequency is converted into joint motion speed (e.g., a frequency of 10000 Hz corresponds to a motion speed of 90° / s). Then, a smooth joint angle trajectory is generated through the Central Pattern Generator (CPG). The CPG simulates the rhythm generation function of the human central nervous system and can dynamically adjust the trajectory parameters according to the pulse characteristics. For example, when the electromyographic pulse density gradually increases, the joint angle trajectory will show a curve of "slow increase - stable maintenance", avoiding the stiffness of mechanical movement and ensuring that the movement of the prosthetic joint is consistent with the rhythm of human natural movements. For example, in the process of the elbow joint flexing from 0° to 90°, the trajectory will follow the natural rhythm of human arm muscle exertion and there will be no abrupt angle changes.

[0051] Generating standardized motion commands based on joint angle trajectories and motion-pulse mapping is a crucial step in unifying command formats and ensuring the coordination of various system modules. Joint angle trajectories provide the target angles of the prosthetic joints at different time points (e.g., 0° for the elbow at 0ms, 45° at 500ms, and 90° at 1000ms). The motion-pulse mapping specifies the motion parameters corresponding to the pulse characteristics required to achieve this trajectory (e.g., a gripping force of 2N and a movement speed of 45° / s are needed to reach 45°). The system integrates this information into a standardized command format, which includes key parameters such as timestamp, target joint, target angle, corresponding gripping force, and movement speed. For example, "Timestamp 1000ms, Target Joint: Elbow, Target Angle: 90°, Gripping Force: 2N, Movement Speed: 45° / s". This standardized format can eliminate format differences between different modal signals and different modules, ensuring that the instruction information can be accurately parsed when the control strategy is adjusted and the actuator is driven, avoiding action deviations caused by inconsistent formats. At the same time, standardized instructions are also easy for the system to store and recall. For example, standardized instructions for common actions (such as grasping a cup or holding a pen) can be stored as templates and quickly retrieved when the same action is executed later, improving response speed.

[0052] Based on standardized motion commands, adjusted control strategies, and learning results, bionic motion commands are generated to drive the prosthetic actuator. This is the final execution stage of the process, achieving precise conversion between "command" and "action," ensuring that the prosthetic movements not only conform to the user's intentions but also adapt to changes in the environment and user habits. The adjusted control strategy includes synaptic weights and motion parameters optimized through pulse reinforcement learning (such as grip force parameters optimized for user feedback on "difficulty gripping a pen"). The learning results are adaptation schemes formed by historical optimizations of the system (such as a user-specific "pen grip parameter template"). The system integrates standardized motion commands with these two types of information to generate bionic motion commands—for example, the standardized command "grip force 2N," combined with the optimized parameters in the control strategy ("grip force should be reduced by 10%) and the template in the learning results ("optimal grip force for the user to grip a pen 1.8N"), ultimately determining the grip force of the bionic motion command to be 1.8N, while adjusting the joint movement speed to the user's habitual 30° / s to ensure natural and effortless movement. The prosthetic actuator employs hydraulic artificial muscles combined with Pulse Width Modulation (PWM) drive technology. Upon receiving bionic motion commands, the PWM module converts the command parameters into pulse drive signals, controlling the amount and speed of extension and contraction of the hydraulic artificial muscles. This, in turn, drives the prosthetic joint to move according to the bionic motion commands—for example, driving the elbow joint muscles to contract, allowing the joint to smoothly flex from 0° to 90°, while simultaneously controlling the finger muscles to maintain a grip force of 1.8N. During execution, joint torque sensors and tactile sensors provide real-time feedback on the movement status. If the detected deviation between the actual angle and the target angle exceeds 0.1°, or the grip force deviation exceeds 0.1N, the system dynamically adjusts the drive signals to ensure motion accuracy, ultimately achieving bionic movements that conform to the user's intentions.

[0053] In one embodiment, step S500 further includes: S5021. During the movement of the prosthetic actuator, monitor the changing trend of tactile pressure signals and the stability of the moving joints in real time; S5022. Determine whether an execution abnormality has occurred based on the changing trend of the tactile pressure signal and the stability of the moving joint; S5023. When an abnormal situation occurs, the emergency response mechanism is triggered, and the emergency response plan is retrieved based on the current actions of the organization.

[0054] In this embodiment, in the multimodal perception-cognition fusion intelligent prosthetic system driven by a spiking neural network (SNN), real-time monitoring and emergency response during the movement of the prosthetic actuator are key to ensuring the safety and stability of the movement, and can effectively deal with emergencies such as object slippage and posture deviation.

[0055] During the movement of the prosthetic actuator, real-time monitoring of the changing trends of tactile pressure signals and the stability of the joints requires comprehensive data acquisition using dedicated sensors deployed throughout the system to ensure that abnormal situations are captured promptly. The monitoring of tactile pressure signals is accomplished through electronic skin sensors installed on contact points such as the prosthetic fingers and soles. These sensors, with a sampling frequency of 1000 Hz (1 kHz), can record changes in pressure values ​​in real time. For example, when grasping a cup, the sensor continuously monitors the complete trend of pressure rising from 0 Newtons (N) to a stable gripping pressure (e.g., 3 N), and then falling back to 0 N when lowering the cup. The stability monitoring of the joints relies on inertial measurement units (IMUs) and joint torque sensors installed at the prosthetic joints (e.g., elbow and knee joints). The IMU collects the joint's posture angles and angular velocity data in real time (sampling frequency 500 Hz) to determine whether the joint movement follows a preset trajectory. The joint torque sensors monitor the forces acting on the joints to prevent joint jamming or excessive rotation due to abnormal torque. The two types of monitoring data are transmitted to the system processing module in real time, forming a dynamic perception of the prosthesis's movement status and providing a continuous and accurate data source for anomaly detection.

[0056] To determine whether abnormal movement has occurred, the trend of tactile pressure signal changes and the stability of joints must be considered. This requires combining pre-defined anomaly detection rules with the monitoring data to compare with normal movement characteristics and identify the type and risk level of the anomaly. Regarding the trend of tactile pressure signal changes, if the pressure value decreases by more than 30% within 500 milliseconds (ms) during a stable grip phase (e.g., when holding a cup), it is considered a "risk of object slippage." This usually stems from a smooth object surface, insufficient initial grip strength, or changes in the contact area due to minor adjustments in the user's hand posture. Regarding joint stability, if the IMU detects a deviation of more than 10° between the actual joint movement trajectory and the standard trajectory generated based on historical normal movements (e.g., the elbow joint targets 90° flexion but only flexes 80°), or if the joint torque sensor detects an instantaneous fluctuation in torque exceeding the preset safety range (e.g., a normal grip force versus torque of 2 N·m, with an instantaneous fluctuation exceeding 3.5 N·m), it is considered an "abnormal movement posture." This may be caused by temporary sensor interference, uneven force on the actuator, or the user's intention to temporarily adjust their movements not being captured in time. The system will analyze the two types of monitoring data in real time. Once the above judgment conditions are triggered, it will be immediately marked as an abnormal movement, thus buying time for subsequent emergency response.

[0057] In the event of an anomaly, the system triggers an emergency response mechanism and retrieves an emergency handling plan based on the current organizational actions. This is the core execution step for the system to quickly mitigate risks and ensure operational safety. The entire process relies on a pre-set emergency plan library and the rapid decision-making capability of a spiking neural network. First, after the system triggers the emergency response mechanism, it pauses the current conventional control strategy and prioritizes activating the "Amygdala SNN Emergency Response Module" in the spiking neural network. This module simulates the emergency processing function of the human brain's amygdala and can complete anomaly type identification and plan matching within 300 milliseconds (ms), preventing the risk from escalating due to decision-making delays. Subsequently, the system retrieves an appropriate solution from the emergency response plan library based on the current mechanism's action type (e.g., grasping, extending, walking). For "object slippage risk," the core of the solution is to temporarily increase the grasping force. For example, based on the current pressure drop, the system adjusts the drive signal of the hydraulic artificial muscle according to a preset ratio (e.g., a 20% increase in grip force if the pressure drops by 30%), using pulse width modulation (PWM) technology to quickly increase grip force and stabilize object contact. For "abnormal movement posture," the core of the solution is to correct the joint movement trajectory. For example, based on the deviation angle fed back by the IMU, the system calculates the required joint adjustment and drives the actuator to make reverse fine adjustments (e.g., if the elbow joint is actually flexed at 80°, additional muscle contraction is needed to increase the angle by 10°), allowing the movement to return to the standard trajectory. After the emergency response plan is executed, the system continuously monitors the tactile pressure signal and joint stability until the abnormality is resolved, then restores the normal control strategy. Simultaneously, the system records the signal characteristics, handling measures, and effects of this abnormality, updates the emergency response plan library, and optimizes the response efficiency and accuracy for subsequent similar abnormalities, continuously improving the movement safety and adaptability of the prosthetic actuator.

[0058] In one embodiment, a prosthetic limb actuation device based on a spiking neural network is provided, which corresponds one-to-one with the prosthetic limb actuation method based on a spiking neural network described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a functional block diagram of a preferred embodiment of the prosthetic limb actuation device based on a spiking neural network of the present invention. The module includes a signal processing module 10, a signal conversion module 20, a control strategy module 30, a learning result module 40, and an action execution module 50. Detailed descriptions of each functional module are as follows: The signal processing module 10 is used to acquire multimodal signals and preprocess the multimodal signals; Signal conversion module 20 is used to convert the preprocessed multimodal signal into a pulse signal; Control strategy module 30 is used to input the pulse signal into a pulse neural network for perception fusion processing and output a control strategy; The learning outcome module 40 is used to dynamically adjust the control strategy based on behavioral feedback information using a reward function to generate learning outcomes. The motion execution module 50 is used to generate bionic motion commands based on the adjusted control strategy and learning results, and drive the prosthetic actuator to move according to the bionic motion commands.

[0059] In one embodiment, the signal processing module 10 includes: The sampling frequency unit is used to preset the sampling frequency for each signal; The signal acquisition unit is used to acquire electromyographic signals, tactile pressure signals, and inertial measurement data according to a preset sampling frequency; An electromyography (EMG) processing unit is used to perform notch filtering, bandpass filtering, and full-wave rectification processing on the EMG signal. The tactile processing unit is used to perform sliding mean filtering and dynamic range normalization on the tactile pressure signal. An inertial processing unit is used to perform low-pass filtering and quaternion attitude calculation on the inertial measurement data.

[0060] In one embodiment, the signal conversion module 20 includes: The time encoding rule unit is used to predefine time encoding rules based on the channel array; The strength coding rule unit is used to map the signal strength to the pulse frequency and predefine the strength coding rule according to the pulse frequency; The signal encoding unit is used to convert the preprocessed multimodal signal into a pulse signal with a timestamp according to the time encoding rule and the intensity encoding rule.

[0061] In one embodiment, the control strategy module 30 includes: The global timestamp unit is used to add a global timestamp to each pulse signal based on a global time reference. The signal grouping unit is used to group electromyographic pulse signals, tactile pressure pulse signals, and inertial measurement pulse signals according to a preset fusion time window; The time difference unit is used to acquire the timestamps of pulse signals of different modalities within the same time group, and to calculate the time difference between the electromyographic pulse signal and the tactile pressure pulse signal, and the time difference between the electromyographic pulse signal and the inertial measurement pulse signal. The weight adjustment unit is used to compare the time difference with the difference threshold. When the time difference is less than the difference threshold, the synaptic weights between the corresponding modalities are enhanced according to the STDP algorithm to generate a specific action neural pathway. The initial weighting unit is used to assign an initial weight to each pulse signal based on the signal reliability. The control strategy unit is used to make action intention decisions based on synaptic weights and initial weights, through pulse density voting, and output the control strategy.

[0062] In one embodiment, the learning result module 40 includes: The information acquisition unit is used to acquire user behavior feedback information; The reward value unit is used to calculate the reward value for behavioral feedback information through a reward function. The learning outcome unit is used to dynamically adjust the control strategy based on the reward value using pulse reinforcement learning to generate learning outcomes.

[0063] In one embodiment, the action execution module 50 includes: The mapping relationship unit is used to establish a motion-pulse mapping relationship based on motion parameters and pulse signals; The joint angle trajectory unit is used to generate joint angle trajectories by extracting the electromyographic pulse density and electromyographic frequency characteristics from the pulse signal; Standardized motion command unit, used to generate standardized motion commands based on joint angle trajectory and motion-pulse mapping relationship; The motion execution unit is used to generate bionic motion commands based on standardized motion instructions, adjusted control strategies, and learning results, and to drive the prosthetic actuator to move according to the bionic motion commands.

[0064] In one embodiment, the action execution module 50 further includes: The execution monitoring unit is used to monitor the changing trend of tactile pressure signals and the stability of the moving joints in real time when the prosthetic actuator is in motion; An anomaly detection unit is used to determine whether an execution anomaly has occurred based on the changing trend of the tactile pressure signal and the stability of the moving joint. The anomaly handling unit is used to trigger the emergency response mechanism and retrieve the emergency response plan based on the current organizational actions when an anomaly occurs.

[0065] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side prosthetic actuation method based on a spiking neural network.

[0066] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the user side of a prosthetic limb actuation method based on a spiking neural network. In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Acquire multimodal signals and preprocess the multimodal signals; The preprocessed multimodal signal is converted into a pulse signal; The pulse signal is input into a spiking neural network for perception fusion processing, and a control strategy is output. By using a reward function, the control strategy is dynamically adjusted based on behavioral feedback information and impulse reinforcement learning to generate learning results. Based on the adjusted control strategy and learning results, bionic motion commands are generated, and the prosthetic actuator is driven to move according to the bionic motion commands.

[0067] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire multimodal signals and preprocess the multimodal signals; The preprocessed multimodal signal is converted into a pulse signal; The pulse signal is input into a spiking neural network for perception fusion processing, and a control strategy is output. By using a reward function, the control strategy is dynamically adjusted based on behavioral feedback information and impulse reinforcement learning to generate learning results. Based on the adjusted control strategy and learning results, bionic motion commands are generated, and the prosthetic actuator is driven to move according to the bionic motion commands.

[0068] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0070] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0071] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A prosthetic limb actuation method based on a spiking neural network, characterized in that, Includes the following steps: Acquire multimodal signals and preprocess the multimodal signals; The preprocessed multimodal signal is converted into a pulse signal; The pulse signal is input into a spiking neural network for perception fusion processing, and a control strategy is output. By using a reward function, the control strategy is dynamically adjusted based on behavioral feedback information and impulse reinforcement learning to generate learning results. Based on the adjusted control strategy and learning results, bionic motion commands are generated, and the prosthetic actuator is driven to move according to the bionic motion commands.

2. The prosthetic limb actuation method based on spiking neural network as described in claim 1, characterized in that, The acquisition of multimodal signals and the preprocessing of the multimodal signals include: The sampling frequency for each signal is preset; Electromyographic signals, tactile pressure signals, and inertial measurement data are collected according to a preset sampling frequency. The electromyographic signal is subjected to notch filtering, bandpass filtering and full-wave rectification; The tactile pressure signal is subjected to sliding mean filtering and dynamic range normalization. The inertial measurement data is subjected to low-pass filtering and quaternion attitude calculation.

3. The prosthetic limb actuation method based on spiking neural network as described in claim 1, characterized in that, The step of converting the preprocessed multimodal signal into a pulse signal includes: The time encoding rules are predefined based on the channel array; The signal strength is mapped to the pulse frequency, and the strength encoding rules are predefined according to the pulse frequency. The preprocessed multimodal signal is converted into a pulse signal with a timestamp according to the time encoding rule and intensity encoding rule.

4. The prosthetic limb actuation method based on spiking neural network as described in claim 1, characterized in that, The step of inputting the pulse signal into a spiking neural network for perception fusion processing and outputting a control strategy includes: Based on a global time base, a global timestamp is added to each pulse signal; The electromyographic pulse signals, tactile pressure pulse signals, and inertial measurement pulse signals are grouped according to a preset fusion time window; The timestamps of pulse signals of different modalities within the same time group are obtained, and the time difference between electromyographic pulse signals and tactile pressure pulse signals and the time difference between electromyographic pulse signals and inertial measurement pulse signals are calculated. The time difference is compared with a difference threshold. When the time difference is less than the difference threshold, the synaptic weights between the corresponding modalities are enhanced according to the STDP algorithm to generate a specific action neural pathway. Assign initial weights to each pulse signal based on signal reliability; Based on synaptic weights and initial weights, action intention decisions are made through pulse density voting, and a control strategy is output.

5. The prosthetic limb actuation method based on spiking neural network as described in claim 1, characterized in that, The control strategy is dynamically adjusted based on behavioral feedback information using a reward function and impulsive reinforcement learning to generate learning results, including: Obtain user behavior feedback information; The reward value for behavioral feedback information is calculated using a reward function. Based on the reward value, the control strategy is dynamically adjusted using impulse reinforcement learning to generate learning results.

6. The prosthetic limb actuation method based on spiking neural network as described in claim 1, characterized in that, Based on the adjusted control strategy and learning results, bionic motion commands are generated, and the prosthetic actuator is driven to move according to the bionic motion commands, including: Establish a motion-pulse mapping relationship based on motion parameters and pulse signals; By extracting the electromyographic pulse density and frequency characteristics from the pulse signal, joint angle trajectory is generated. Standardized motion commands are generated based on joint angle trajectories and motion-pulse mapping relationships; Based on standardized motion commands, adjusted control strategies, and learning results, bionic motion commands are generated, and the prosthetic actuator is driven to move according to the bionic motion commands.

7. The prosthetic limb actuation method based on spiking neural network as described in claim 1, characterized in that, The step of generating biomimetic motion commands based on the adjusted control strategy and learning results, and driving the prosthetic actuator to move according to the biomimetic motion commands, further includes: During the movement of the prosthetic actuator, the changing trend of tactile pressure signals and the stability of the moving joints are monitored in real time; The occurrence of execution abnormalities is determined based on the changing trend of the tactile pressure signal and the stability of the moving joint. When an abnormal situation occurs, the emergency response mechanism is triggered, and the emergency response plan is retrieved based on the current actions of the organization.

8. A prosthetic limb actuation device based on a spiking neural network, characterized in that, The prosthetic actuation device based on a spiking neural network includes: The signal processing module is used to acquire multimodal signals and preprocess the multimodal signals; The signal conversion module is used to convert the preprocessed multimodal signal into a pulse signal; The control strategy module is used to input the pulse signal into the pulse neural network for perception fusion processing and output a control strategy; The learning outcome module is used to dynamically adjust the control strategy based on behavioral feedback information using a reward function and impulse reinforcement learning to generate learning outcomes. The motion execution module is used to generate bionic motion commands based on the adjusted control strategy and learning results, and drive the prosthetic actuator to move according to the bionic motion commands.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a spiking neural network-based prosthesis driver stored in the memory and executable on the processor. When executed by the processor, the spiking neural network-based prosthesis driver implements the steps of the spiking neural network-based prosthesis driving method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a prosthesis driver program based on a spiking neural network, which, when executed by a processor, implements the steps of the prosthesis driving method based on a spiking neural network as described in any one of claims 1-7.

Images