Bionic intelligent artificial limb system with real-time visual feedback and control method thereof

Abstract

The invention discloses a bionic intelligent artificial limb system with real-time visual feedback and a control method thereof. The control method comprises the following steps: S1, carrying out preprocessing to generate a multi-dimensional feature vector group; s2, outputting a Vision Mama perceptual prediction tensor (Vision Mama); s3, obtaining a primary micro-action candidate sequence; s4, based on the primary micro-action candidate sequence and the perception confidence value, executing energy consumption-smoothness joint search in a spiral search stage, generating an intermediate micro-action candidate sequence, and obtaining an optimized micro-action sequence; and S5, issuing the optimized micro-action sequence to an embedded cooperative control layer, mapping the optimized micro-action sequence into a joint motor torque instruction set and an impedance parameter adjustment strategy to form a motor control instruction stream, and driving a bionic artificial limb joint to execute according to the motor control instruction stream on a microcontroller unit. According to the invention, the peak energy consumption of the servo system is reduced, and the action continuity and the tail end grasping success rate are guaranteed in the key micro-action stage.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent prosthetic systems, and in particular to a bionic intelligent prosthetic system with real-time visual feedback and a control method thereof. Background Art

[0002] With the development of wearable intelligent prostheses, bionic prostheses have become an important direction for rehabilitation assistance and enhanced human-computer interaction. In the field of real-time control of high-degree-of-freedom upper limb prostheses, intelligent prostheses not only require accurate recognition of the user's electromyographic signals and movement intentions, but also the ability to quickly and stably complete fine-grained action decomposition, such as object grasping and obstacle avoidance, in complex dynamic environments. However, existing technologies generally face two core challenges: First, traditional prostheses often use trajectory planning or model predictive control methods based on fixed step sizes, which makes it difficult to reorder and fine-grainedly segment action sequences in real time in dynamic environments. This leads to joint stalling, large movement shocks, or grasping failures in occlusion, interference, or unexpected scenarios. Second, existing deep learning-based vision-action recognition methods and swarm intelligence optimization algorithms typically require significant computing power and storage resources, failing to meet the low power consumption and high responsiveness requirements of embedded platforms. This leads to bottlenecks such as overheating, increased latency, and limited battery life during continuous use of prosthetic devices.

[0003] In recent years, some scholars have attempted to integrate methods such as deep visual networks and reinforcement learning with prosthetic control, but most of these solutions remain in the offline stage or rely on high-performance cloud computing power, lack end-side resource adaptation mechanisms, and find it difficult to achieve high-precision, low-latency control on the system-on-chip. In addition, existing multi-objective task segmentation optimization methods are mostly static or semi-real-time, lacking mechanisms for dynamic task switching and security redundancy protection in complex environments, and cannot meet the fine-grained, human-driven and high-robustness requirements in daily actual scenarios.

[0004] Therefore, how to achieve high degrees of freedom, multi-modal real-time perception fusion, efficient task segmentation and energy-adaptive bionic prosthetic control under limited resources remains a technical challenge that urgently needs to be overcome in this field. Summary of the Invention

[0005] One purpose of the present invention is to propose a bionic intelligent prosthesis system with real-time visual feedback and a control method thereof. The present invention not only reduces the peak energy consumption of the servo system, but also ensures the continuity of movement and the success rate of end grasping in the critical micro-movement stage.

[0006] A bionic intelligent prosthesis control method with real-time visual feedback according to an embodiment of the present invention includes the following steps:

[0007] S1. Acquire biomimetic signals through a multimodal sensor array and perform preprocessing to generate a multidimensional feature vector group;

[0008] S2. Input the multidimensional feature vector group into the Vision Mamba perception-prediction module in the system-on-chip, and output the Vision Mamba perception prediction tensor;

[0009] S3. Jointly map the Vision Mamba perception prediction tensor and the multidimensional feature vector group into a dynamic optimization parameter set. This dynamic optimization parameter set is input into the real-time task segmentation optimization module of the manta ray foraging optimization algorithm. In the chain search phase, a global energy-oriented search is performed to obtain a primary micro-action candidate sequence.

[0010] S4. Based on the primary micro-motion candidate sequence and the perception confidence value, a combined energy consumption and smoothness search is performed in the spiral search phase to generate an intermediate micro-motion candidate sequence. Based on the intermediate micro-motion candidate sequence and the scene semantic mask, a terminal precision constrained search is performed in the tumbling search phase to obtain the optimized micro-motion sequence.

[0011] S5. Send the optimized micro-action sequence to the embedded collaborative control layer, map it into a joint motor torque instruction set and impedance adjustment strategy, form a motor control instruction stream, and drive the bionic prosthetic joint to execute according to the motor control instruction stream on the microcontroller unit.

[0012] Optionally, S1 includes the following contents:

[0013] S11. Obtain the original Industrial Internet data stream to construct an Industrial Internet original data set. Each piece of Industrial Internet data includes a timestamp, numerical information, device identification, and data category label.

[0014] S12. Perform data cleaning on the original industrial Internet data set to obtain a cleaned industrial Internet data set;

[0015] S13. Perform standardization processing on the cleaned industrial Internet data set to form a standardized industrial Internet data set;

[0016] S14. Eliminate redundant information from the standardized industrial Internet data set to form a final standardized industrial Internet data set;

[0017] S15. Perform time series feature extraction on the final standardized Industrial Internet data set. The extracted time series features include the overall change trend of each dimension data at different times, the fluctuation amplitude in a short period of time, and the stability index of the long-term series. All time series features constitute a time series feature matrix.

[0018] S16. All time series features in the time series feature matrix are structured and organized according to the preset window to construct a multidimensional feature vector group of industrial Internet data. Each group of multidimensional feature vectors corresponds to the data statistical results within a time window.

[0019] Optionally, S2 includes the following contents:

[0020] S21. Input the synchronized multimodal input data stream as an input sequence to the VisionMamba perception-prediction module in the system-on-chip. The VisionMamba perception-prediction module uses a bidirectional state space structure to process image sequence and time series data.

[0021] S22. In the Vision Mamba perception-prediction module, the image frame sequence is encoded to generate a visual feature vector representing the semantic features of the image. Simultaneously, the time series containing all channel electromyographic and inertial signals is encoded to generate a temporal feature vector representing the user's motion trends.

[0022] S23. Align the visual feature vector and the temporal feature vector at the time step level and merge them into a unified fused feature tensor. The fused feature tensor structurally contains information from both the image channel and the sensor channel.

[0023] S24. Input the fused feature tensor into the decoder structure in the Vision Mamba perception-prediction module, and output four types of perception prediction information at each time point:

[0024] A set of hand-object keypoint trajectories, each of which represents a specific position coordinate in three-dimensional space and is used to characterize the spatial relationship between the user's hand or prosthetic terminal and the object;

[0025] Scene semantic mask tensor, used to represent the semantic category label corresponding to each pixel in each frame of the image;

[0026] Target motion trend vector field, used to predict the overall direction change trend of user actions;

[0027] The perception confidence value vector includes the key point confidence value, the semantic mask confidence value, and the motion trend confidence value, which respectively represent the credibility of the above three types of information at the current time step. The confidence value range is a real number between 0 and 1;

[0028] S25. Combine the hand-object keypoint trajectory set, the scene semantic mask tensor, the target motion trend vector field and the perception confidence value vector into the Vision Mamba perception prediction tensor.

[0029] Optionally, the S3 includes the following content:

[0030] S31. Jointly embed the Vision Mamba perception prediction tensor with a multidimensional feature vector group extracted from the synchronized multimodal input data stream to construct a fused mapping tensor in the system-on-chip. The fused mapping tensor is used to generate a dynamic optimization parameter set. The dynamic optimization parameter set consists of an optimization weight set and an initial micro-motion candidate set. The optimization weight set includes an energy consumption weight α, a smoothness weight β, a target error weight γ, and an end stability weight δ. The initial micro-motion candidate set represents the executable motion state space of each joint of the bionic prosthesis within the physical constraints in the current control cycle. Each micro-motion candidate state contains a triplet of joint displacement, desired torque, and target impedance characteristics.

[0031] S32. Constructing real-time task segmentation optimization objective function F based on dynamic optimization parameter set MRFO ,The real-time task segmentation optimization objective function combines the energy consumption control, trajectory smoothness, and terminal execution accuracy of the bionic prosthesis, and is defined in the following form in the chain search phase:

[0032] F MRFO =α·E joint +β·T smooth +γ·E pose +δ·D stab ;

[0033] Among them, E joint It represents the energy consumption per unit cycle of all servo motors by the current micro-action, which is obtained by sampling the output current and duration required by each joint. smooth It represents the speed first-order derivative change rate of the micro-motion sequence in time, E pose represents the Euclidean error between the current action execution state and the target end pose, D stab Represents the confidence level of target terminal stability under current environmental conditions, estimated based on the occlusion map and object motion trend field output by Vision Mamba;

[0034] S33. During the chain search phase, based on the swarm intelligence chain propagation mechanism of the manta ray foraging optimization algorithm, a virtual swarm of individuals is set up. Each optimized individual represents a complete sequence of micro-movements. Individuals in the swarm propagate their behaviors based on the position of the current optimal individual and the dynamic weight change trend within the control cycle.

[0035] S34. Introducing the energy consumption adjustment factor η based on the current myoelectric activation state of the bionic prosthesis emg , the energy consumption adjustment factor participates in the dynamic weighting of energy consumption index items, forming the energy consumption control mechanism of myoelectric drive perception:

[0036]

[0037] Among them, N e is the number of myoelectric channels, represents the normalized EMG signal of the i-th channel at time τ, where T is the sliding window length;

[0038] S35. All micro-action sequences after iterative optimization in the chain search phase are divided into optimization objective functions according to the modified real-time task Sort by, where the corrected energy consumption item is E′ joint =η emg ·E joint , the final real-time task segmentation optimization objective function is:

[0039]

[0040] S36. Write the primary micro-action candidate sequence into the SRAM area of ​​the system on chip through the shared cache interface.

[0041] Optionally, S4 includes the following contents:

[0042] S41. The primary micro-motion candidate sequence and the perception confidence value vector in the Vision Mamba perception prediction tensor are jointly input into the spiral search phase of the manta ray foraging optimization algorithm, and the confidence adjustment factor ω is introduced in the spiral search phase. conf ;

[0043] S42. Define the confidence-aware optimization objective function F in the spiral search phase spiral :

[0044] F spiral =ω conf ·(α·E′ joint +β·T smooth )+(1-ω conf )·(γ·E pose +δ·D stab );

[0045] Among them, ω conf is the perception confidence value output by Vision Mamba at the current moment;

[0046] S43. Optimize the objective function F based on confidence perception spiral In an embedded platform, local perturbations are performed on primary micro-motion candidate sequences. A set of intermediate perturbation action sequences is generated along the spiral trajectory rule of the manta ray foraging algorithm. The objective function values ​​corresponding to all perturbation action sequences are calculated, and the set with the smallest confidence-aware optimization objective function value is selected as the intermediate micro-motion candidate sequence.

[0047] S44. The intermediate micro-motion candidate sequence and the scene semantic mask tensor in the Vision Mamba perception prediction tensor are jointly input into the tumbling search stage, and the terminal occlusion probability P is introduced in the tumbling search stage. occl and trajectory disturbance change ΔT smooth , construct the terminal refinement optimization objective function F in the tumbling search phase roll :

[0048] F roll =γ·E pose +δ·(1-P occl )+∈·ΔT smooth ;

[0049] Among them, P occl represents the occlusion probability of the target terminal execution area in the semantic mask, provided by Vision Mamba, ΔT smooth represents the trajectory perturbation stability index between the spiral stage and the current micro-action, ∈ is the perturbation stability weight;

[0050] S45. During the roll search phase, a small roll perturbation is applied to the candidate intermediate micro-motion sequences according to the roll optimization objective function. Multiple rounds of iterative updates are performed based on terminal accuracy, occlusion perception, and perturbation smoothness. Ultimately, a set of micro-motion sequences with the minimum roll optimization objective function value is selected as the optimized micro-motion sequence.

[0051] S46. Map the optimized micro-motion sequence into execution commands of the bionic prosthesis, including the motor torque instructions and impedance adjustment parameter set of the target joint, and write them into the shared cache area of ​​the embedded control system; at the same time, write the optimized micro-motion sequence back to the system circular buffer.

[0052] Optionally, S5 includes the following contents:

[0053] S51. Input the optimized micro-motion sequence into the embedded collaborative control layer, construct a motor control mapping relationship based on the joint motion model in the embedded collaborative control layer, optimize each micro-motion sequence to include a joint displacement target, a target torque reference value, and a desired impedance parameter triple, and generate a motor control instruction stream for execution through the triple;

[0054] S52. The motor control instruction flow sets the execution parameter range according to different control modes;

[0055] S53 writes the formatted motor control instruction stream into the instruction cache queue in the microcontroller unit, which is scheduled and executed in real time by the microcontroller unit, and drives the joints of the bionic prosthesis to complete high-degree-of-freedom movements in accordance with the time sequence of the optimized micro-action sequence;

[0056] S54. During the execution of the motor control command, real-time collection of execution feedback information is performed and the difference between it and the target state in the optimized micro-action sequence is calculated to form an error signal;

[0057] S55. If the error signal exceeds the preset threshold range, the interrupt feedback update mechanism is immediately triggered, the most recently optimized micro-action sequence cache is activated, and the local disturbance re-optimization process is entered to form a new round of motor control instruction flow.

[0058] Optionally, the output rule of the motor control instruction stream is defined as follows:

[0059] If the target movement corresponds to the micro-movement sequence, the target joint displacement target change rate is greater than the set acceleration threshold, and the target end stability confidence D stab ≥0.75, the output type is a high-response compliant instruction flow, the motor torque is set to 95% of the target torque reference value, and the impedance parameter is set to medium stiffness;

[0060] If the target action corresponds to the micro-action sequence of the joint displacement target change rate is less than or equal to the acceleration threshold, and the target end occlusion probability P occl ≥0.5, the output type is a low-speed obstacle avoidance instruction flow, the motor torque is set to 70% of the target torque reference value, and the impedance parameter is lowered to enhance compliance;

[0061] If the trajectory disturbance stability index ΔT of the micro-motion sequence corresponding to the target action is smooth ≥0.3, and the energy consumption adjustment factor η emg ≥1.2, the output type is energy-suppression instruction flow, the motor torque is set to 60% of the target torque reference value, and the impedance parameter is set to low stiffness to relieve the load.

[0062] A bionic intelligent prosthesis control system with real-time visual feedback is used to execute a bionic intelligent prosthesis control method with real-time visual feedback, including the following subsystems:

[0063] The multimodal sensor acquisition subsystem is used to acquire raw electromyographic signals, inertial signals, and external visual information, and performs unified timestamp alignment and filtering preprocessing on various signals to output synchronized multimodal input data streams;

[0064] The perception-prediction processing subsystem, which is equipped with the Vision Mamba perception-prediction module in the system-on-chip, receives synchronized multimodal input data streams, performs fusion feature encoding and spatiotemporal alignment on the image frame sequence and time series signals, and generates the Vision Mamba perception prediction tensor;

[0065] The dynamic optimization scheduling subsystem includes an optimization parameter fusion unit and a manta ray foraging optimization algorithm processing module. The optimization parameter fusion unit is used to jointly embed the Vision Mamba perception prediction tensor and the multi-dimensional feature vector group to generate a dynamic optimization parameter set. The manta ray foraging optimization algorithm processing module follows a three-stage chain-spiral-tumble structure, optimizes the objective function based on real-time task segmentation, and iteratively outputs the optimized micro-action sequence.

[0066] The embedded collaborative control subsystem includes a microcontroller unit and a cache module. The microcontroller unit is used to map the optimized micro-action sequence into a joint motor torque instruction set and an impedance adjustment strategy to form a motor control instruction stream, and drive the joints of the bionic prosthesis to execute at a set servo frequency. At the same time, it collects execution feedback signals in real time to complete closed-loop adaptive control.

[0067] The beneficial effects of the present invention are:

[0068] (1) This paper deeply couples the Vision Mamba perception prediction module based on a bidirectional state-space network with the manta ray foraging optimization algorithm based on swarm intelligence, achieving fine-grained real-time task segmentation under multimodal input. By directly mapping the perception prediction tensor into multi-objective dynamic optimization parameters, the timeliness of action sequence reordering and micro-motion switching of bionic prostheses in complex dynamic scenes is significantly improved. In the case of occluded grasping tasks, the action segmentation accuracy and response speed are improved compared with traditional MPC or fixed-step trajectory planning.

[0069] (2) The present invention proposes an energy consumption adjustment factor based on real-time electromyographic activation state regulation, and introduces perception confidence, terminal occlusion probability and disturbance stability indicators in the chain-spiral-roll three-stage optimization process to achieve a dynamic trade-off between energy consumption, trajectory smoothness, execution accuracy and grasping safety. In the spiral and rolling stages, the optimization objective function is adaptively adjusted according to the current perception state and terminal safety, which not only reduces the peak energy consumption of the servo system, but also ensures the continuity of the movement and the success rate of terminal grasping in the critical micro-movement stage, which is significantly better than the existing single-target or static weighted optimization scheme. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0071] Figure 1 This is a flow chart of a bionic intelligent prosthesis system with real-time visual feedback and its control method proposed by the present invention. DETAILED DESCRIPTION

[0072] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0073] refer to Figure 1 A bionic intelligent prosthesis control method with real-time visual feedback comprises the following steps:

[0074] S1. Acquire raw electromyographic signals, inertial signals, and external visual information through a multimodal sensor array, and generate a multidimensional feature vector group after unified timestamp alignment and filtering preprocessing;

[0075] S2. Input the multidimensional feature vector group into the Vision Mamba perception-prediction module in the system-on-chip, which outputs a Vision Mamba perception prediction tensor containing hand-object keypoint trajectories, scene semantic masks, target motion trend vector fields, and perception confidence values;

[0076] S3. Jointly map the Vision Mamba perception prediction tensor and the multidimensional feature vector group into a dynamic optimization parameter set, which includes an optimization weight set and an initial micro-motion candidate set. This dynamic optimization parameter set is input into the real-time task segmentation optimization module of the manta ray foraging optimization algorithm. A global energy-oriented search is performed in the chain search phase to obtain a primary micro-motion candidate sequence.

[0077] S4. Based on the primary micro-motion candidate sequence and the perception confidence value, a combined energy consumption and smoothness search is performed in the spiral search phase to generate an intermediate micro-motion candidate sequence. Based on the intermediate micro-motion candidate sequence and the scene semantic mask, a terminal precision constrained search is performed in the tumbling search phase to obtain the optimized micro-motion sequence.

[0078] S5. Send the optimized micro-action sequence to the embedded collaborative control layer, map it into a joint motor torque instruction set and impedance adjustment strategy, form a motor control instruction stream, and drive the bionic prosthetic joint to execute according to the motor control instruction stream on the microcontroller unit.

[0079] In this embodiment, S1 includes the following contents:

[0080] S11. Obtain the original Industrial Internet data stream to construct an Industrial Internet original data set. Each piece of Industrial Internet data includes a timestamp, numerical information, device identification, and data category label. Each piece of data in the Industrial Internet original data set is recorded as a four-tuple, which is used to calibrate its collection time, numerical content, device to which it belongs, and category.

[0081] S12. Perform data cleansing on the original Industrial Internet data set to remove invalid data, duplicate data, and defective abnormal data to obtain a cleaned Industrial Internet data set;

[0082] S13. Perform standardization operations on the cleaned Industrial Internet data set, mapping all numerical fields to a unified numerical range to form a standardized Industrial Internet data set;

[0083] S14. Eliminate redundant information from the standardized Industrial Internet data set. During this elimination process, construct a feature correlation matrix to measure the degree of linear correlation between any two data dimensions. Each element in the feature correlation matrix represents the correlation coefficient between the corresponding two data dimensions. The correlation coefficient is calculated by calculating the ratio of the covariance of the corresponding elements between the two data dimensions to their respective standard deviations. When the correlation coefficient between any two data dimensions exceeds a set redundancy threshold, delete one of the data dimensions to form the final standardized Industrial Internet data set.

[0084] S15. Perform time series feature extraction on the final standardized Industrial Internet data set. The extracted time series features include the overall change trend of each dimension data at different times, the fluctuation amplitude in a short period of time, and the stability index of the long-term series. All time series features constitute a time series feature matrix.

[0085] S16. All time series features in the time series feature matrix are structured and organized according to the preset window to construct a multidimensional feature vector group of industrial Internet data. Each group of multidimensional feature vectors corresponds to the data statistical results within a time window.

[0086] In this embodiment, S2 includes the following contents:

[0087] S21. Input the synchronized multimodal input data stream as an input sequence to the VisionMamba perception-prediction module in the system-on-chip. The VisionMamba perception-prediction module uses a bidirectional state space structure to process image sequence and time series data.

[0088] S22. In the Vision Mamba perception-prediction module, the image frame sequence is encoded to generate a visual feature vector representing the semantic features of the image. The visual feature vector is used to extract the spatial feature information of each frame. Simultaneously, the time series containing all channels of electromyographic and inertial signals is encoded to generate a temporal feature vector representing the user's motion trends. The visual feature vector and the temporal feature vector are maintained as independent encoding spaces in terms of dimension.

[0089] S23. Align the visual feature vector and the temporal feature vector at the time step level and merge them into a unified fused feature tensor. The fused feature tensor structurally contains information from both the image channel and the sensor channel for unified perceptual expression.

[0090] S24. Input the fused feature tensor into the decoder structure in the Vision Mamba perception-prediction module, and output four types of perception prediction information at each time point:

[0091] A set of hand-object keypoint trajectories, each of which represents a specific position coordinate in three-dimensional space and is used to characterize the spatial relationship between the user's hand or prosthetic terminal and the object;

[0092] Scene semantic mask tensor, used to represent the semantic category label corresponding to each pixel in each frame of the image;

[0093] Target motion trend vector field, used to predict the overall direction change trend of user movements;

[0094] The perception confidence value vector includes the key point confidence value, the semantic mask confidence value, and the motion trend confidence value, which respectively represent the credibility of the above three types of information at the current time step. The confidence value range is a real number between 0 and 1;

[0095] In this system, a multimodal sensor acquisition subsystem obtains synchronized multimodal input data streams, including raw EMG signals, inertial signals, and external visual information that have been preprocessed and timestamp-aligned. These synchronized multimodal input data streams are input into the Vision Mamba perception-prediction module within the on-chip system. The Vision Mamba perception-prediction module uses a bidirectional state-space structure to perform feature encoding on the input image frame sequence and timing signals. Specifically, the image frame sequence is encoded into a visual feature vector that can represent the spatial structure and object shape through a neural network. Time series signals such as EMG and inertia are encoded into time series feature vectors that can represent movement trends and dynamic changes. The visual feature vector and the time series feature vector are then aligned and fused in time and space to form a unified fused feature tensor.

[0096] After further processing by the decoder, the fused feature tensor outputs four core perception results at each time step. Based on the spatial distribution information of the fused feature tensor, the keypoint detection head in the neural network extracts the coordinates of the hand and object's feature points in three-dimensional space. The sequence of coordinates at consecutive time steps forms a set of hand-object keypoint trajectories, which are used to describe the fine-grained movement trajectory of the hand when interacting with the object. Secondly, the semantic segmentation structure in the decoder classifies each pixel in each frame based on the spatial context, generating a scene semantic mask tensor for each pixel's category, reflecting the distribution of different regions of the hand, object, and background in the environment. The motion trend prediction branch within the network infers the hand and object's velocity vectors and motion trends based on current and historical feature vectors. The continuous output constitutes the target motion trend vector field. The system calculates the confidence level of the corresponding perception information at each output head. Based on the network's confidence scores for each feature extraction, environmental complexity, and data noise, the system outputs a perception confidence value vector, which quantifies the reliability of keypoint trajectories, scene segmentation, and motion trend prediction.

[0097] The hand-object key point trajectory set, scene semantic mask tensor, target motion trend vector field and perception confidence value vector are realized through the unified fusion and multi-head output of the Vision Mamba perception-prediction module.

[0098] S25. Combine the hand-object keypoint trajectory set, the scene semantic mask tensor, the target motion trend vector field and the perception confidence value vector into the Vision Mamba perception prediction tensor.

[0099] In this embodiment, S3 includes the following contents:

[0100] S31. Jointly embed the Vision Mamba perception prediction tensor with a multi-dimensional feature vector set extracted from the synchronized multimodal input data stream to construct a fused mapping tensor in the system-on-chip. The fused mapping tensor is used to generate a set of dynamic optimization parameters.

[0101] The dynamic optimization parameter set consists of an optimization weight set and an initial micro-action candidate set. The optimization weight set includes energy consumption weight α, smoothness weight β, target error weight γ, and terminal stability weight δ.

[0102] The initial micro-motion candidate set represents the executable motion state space of each joint of the bionic prosthesis within the physical constraints in the current control cycle. Each micro-motion candidate state contains a triplet of joint displacement, expected torque, and target impedance characteristics.

[0103] S32. Constructing real-time task segmentation optimization objective function F based on dynamic optimization parameter set MRFO,The real-time task segmentation optimization objective function combines the energy consumption control, trajectory smoothness, and terminal execution accuracy of the bionic prosthesis, and is defined in the following form in the chain search phase:

[0104] F MRFO =α·E joint +β·T smooth +γ·E pose +δ·D stab ;

[0105] Among them, E joint It represents the energy consumption per unit cycle of all servo motors by the current micro-action, which is obtained by sampling the output current and duration required by each joint. smooth It represents the rate of change of the first-order derivative of the velocity of the micro-motion sequence in time, reflecting the continuity of the trajectory in the space-time domain, E pose It represents the Euclidean error between the current action execution state and the target end pose, which is used to measure the spatial accuracy requirement. stab It represents the stability confidence of the target end under the current environmental conditions and the stability confidence of the end execution point within the grasping area, which is estimated based on the occlusion map and object motion trend field output by Vision Mamba;

[0106] S33. During the chain search phase, based on the swarm intelligence chain propagation mechanism of the manta ray foraging optimization algorithm, a virtual swarm of individuals is set up. Each optimized individual represents a complete sequence of micro-movements. Individuals in the swarm propagate their behaviors based on the position of the current optimal individual and the dynamic weight change trend within the control cycle.

[0107] S34. Introducing the energy consumption adjustment factor η based on the current myoelectric activation state of the bionic prosthesis emg , the energy consumption adjustment factor participates in the dynamic weighting of energy consumption index items to form the energy consumption control mechanism of myoelectric drive perception. The energy consumption adjustment factor is extracted from the RMS value of the standardized myoelectric tensor and is used to dynamically correct the energy consumption component in the optimization function:

[0108]

[0109] Among them, N e is the number of myoelectric channels, represents the normalized EMG signal of the i-th channel at time τ, where T is the sliding window length;

[0110] Energy consumption adjustment factor η proposed by S34 emgThe core construction principle is to directly introduce the muscle activation intensity reflected by the user's electromyographic signals in real time during the actual operation of the bionic prosthesis system into the energy consumption calculation module of the optimization target. Specifically, by performing root mean square calculation on all the normalized electromyographic signals collected in each optimization cycle, the statistics of the overall level of electromyographic activation are obtained, and the statistics are continuously updated in a sliding time window manner. The energy consumption adjustment factor is used to dynamically weight the unit cycle energy consumption index of each micro-action candidate in the chain search stage, and the actual physiological load of the bionic prosthesis execution end is tightly coupled with the energy consumption optimization model to realize an information closed loop of integrated perception-optimization. It breaks the bottleneck of traditional optimization that relies only on model parameters or external sensor data, so that each step of optimization output not only considers the ideal energy consumption, but also actively responds to changes in the user's current physiological state, thereby significantly improving the individual adaptability and dynamic real-time performance of energy consumption regulation.

[0111] The energy consumption adjustment factor has achieved a significant breakthrough based on existing technologies. First, the RMS characteristics of the physiological electromyographic signal are quantified into real-time differentiable weights, which serve as direct correction coefficients for the energy consumption term. The motor energy consumption assessment is dynamically adjusted in the core objective function of the optimization algorithm, achieving an immediate response to the user's intentions and physiological feedback. This makes the optimization algorithm no longer a static module isolated from the user, but a dynamic scheduler that is continuously coupled to the physiological state. Secondly, the introduction of the energy consumption adjustment factor effectively avoids the drawbacks of excessively pursuing the theoretical optimal energy consumption path while ignoring the user's load, and promotes the balance between high responsiveness and low energy consumption of bionic prostheses. It not only improves the comfort and durability of prosthetic wear, but also provides a paradigm demonstration for the adaptive optimization of bionic systems that introduce multimodal physiological feedback.

[0112] S35. All micro-action sequences after iterative optimization in the chain search phase are divided into optimization objective functions according to the modified real-time task Sort by, where the corrected energy consumption item is E′ joint =η emg ·E joint , the final real-time task segmentation optimization objective function is:

[0113]

[0114] The micro-motion sequence with the minimum value of the objective function of the replacement valence real-time task segmentation optimization is selected as the primary micro-motion candidate sequence. The primary micro-motion candidate sequence has energy consumption balance, trajectory continuity and end-execution reliability, and is temporally and spatially consistent with the current joint state of the bionic prosthesis and the user's movement intention.

[0115] S36. Write the primary micro-action candidate sequence into the SRAM area of ​​the system on chip through the shared cache interface.

[0116] In this embodiment, S4 includes the following contents:

[0117] S41. The primary micro-motion candidate sequence and the perception confidence value vector in the Vision Mamba perception prediction tensor are jointly input into the spiral search phase of the manta ray foraging optimization algorithm, and the confidence adjustment factor ω is introduced in the spiral search phase. conf , used to dynamically adjust the weight relationship between energy consumption, smoothness and spatial error;

[0118] S42. Define the confidence-aware optimization objective function F in the spiral search phase spiral :

[0119] F spiral =ω conf ·(α·E′ joint +β·T smooth )+(1-ω conf )·(γ·E pose +δ·D stab );

[0120] Among them, ω conf The perception confidence value output by Vision Mamba at the current moment, ranging from 0 to 1, is used to weight the action selection guided by perception.

[0121] S43. Optimize the objective function F based on confidence perception spiral In an embedded platform, local perturbations are performed on primary micro-motion candidate sequences. A set of intermediate perturbation action sequences is generated along the spiral trajectory rule of the manta ray foraging algorithm. The objective function values ​​corresponding to all perturbation action sequences are calculated, and the set with the smallest confidence-aware optimization objective function value is selected as the intermediate micro-motion candidate sequence.

[0122] S44. The intermediate micro-motion candidate sequence and the scene semantic mask tensor in the Vision Mamba perception prediction tensor are jointly input into the tumbling search stage, and the terminal occlusion probability P is introduced in the tumbling search stage. occl and trajectory disturbance change ΔT smooth , construct the terminal refinement optimization objective function F in the tumbling search phase roll :

[0123] F roll =γ·E pose +δ·(1-P occl )+∈·ΔT smooth ;

[0124] Among them, P occl represents the occlusion probability of the target terminal execution area in the semantic mask, provided by Vision Mamba, ΔT smoothIt represents the trajectory perturbation stability index between the spiral stage and the current micro-motion, which is used to reflect the trajectory perturbation stability. ∈ is the perturbation stability weight, which measures the trade-off between fast convergence and trajectory continuity.

[0125] S45. During the roll search phase, a small roll perturbation is applied to the candidate intermediate micro-motion sequences according to the roll optimization objective function. Multiple rounds of iterative updates are performed based on terminal accuracy, occlusion perception, and perturbation smoothness. Ultimately, a set of micro-motion sequences with the minimum roll optimization objective function value is selected as the optimized micro-motion sequence.

[0126] S46. The optimized micro-motion sequence is mapped into the execution command of the bionic prosthesis, including the motor torque instruction and impedance adjustment parameter set of the target joint, and written into the shared cache area of ​​the embedded control system; at the same time, the optimized micro-motion sequence is written back to the system circular buffer to support the iterative update and state tracking of the dynamic optimization parameter set in the next control cycle, completing the chain-spiral-tumbling three-stage task segmentation closed-loop structure.

[0127] In this embodiment, S5 includes the following contents:

[0128] S51. Input the optimized micro-motion sequence into the embedded collaborative control layer, construct a motor control mapping relationship based on the joint motion model in the embedded collaborative control layer, optimize each micro-motion sequence to include a joint displacement target, a target torque reference value, and a desired impedance parameter triple, and generate a motor control instruction stream for execution through the triple;

[0129] S52. Formatting the motor control command stream into a motor torque command set and an impedance parameter adjustment strategy set. The motor torque command set is used to drive the servo motors of each degree-of-freedom joint of the bionic prosthesis to perform rotational movements. The impedance parameter adjustment strategy set is used to dynamically adjust the force-displacement response relationship of the joint to match the compliance requirements of the current task and the execution stability target.

[0130] S53. The motor control instruction flow sets the execution parameter range according to different control modes;

[0131] S54. The formatted motor control instruction stream is written into the instruction cache queue in the microcontroller unit, and is scheduled and executed in real time by the microcontroller unit at a servo control frequency of 1kHz, and the joints of the bionic prosthesis are driven in sequence to complete high-degree-of-freedom movements in accordance with the time sequence of the optimized micro-action sequence;

[0132] S55. During the execution of the motor control command, real-time execution feedback information is collected, including motor current, joint angle and external force sensor data, and the difference between it and the target state in the optimized micro-action sequence is calculated to form an error signal;

[0133] S56. If the error signal exceeds the preset threshold range, the interrupt feedback update mechanism is immediately triggered, the most recently optimized micro-action sequence cache is activated and the local disturbance re-optimization process is entered to form a new round of motor control instruction flow.

[0134] In this embodiment, the output rules of the motor control instruction stream are defined as follows:

[0135] If the target movement corresponds to the micro-movement sequence, the target joint displacement target change rate is greater than the set acceleration threshold, and the target end stability confidence D stab ≥0.75, the output type is a high-response compliant instruction flow, the motor torque is set to 95% of the target torque reference value, and the impedance parameter is set to medium stiffness;

[0136] If the target action corresponds to the micro-action sequence of the joint displacement target change rate is less than or equal to the acceleration threshold, and the target end occlusion probability P occl ≥0.5, the output type is a low-speed obstacle avoidance instruction flow, the motor torque is set to 70% of the target torque reference value, and the impedance parameter is lowered to enhance compliance;

[0137] If the trajectory disturbance stability index ΔT of the micro-motion sequence corresponding to the target action is smooth ≥0.3, and the energy consumption adjustment factor η emg ≥1.2, the output type is energy-suppression instruction flow, the motor torque is set to 60% of the target torque reference value, and the impedance parameter is set to low stiffness to relieve the load.

[0138] A bionic intelligent prosthesis control system with real-time visual feedback is used to execute a bionic intelligent prosthesis control method with real-time visual feedback, including the following subsystems:

[0139] The multimodal sensor acquisition subsystem is used to acquire raw electromyographic signals, inertial signals, and external visual information, and performs unified timestamp alignment and filtering preprocessing on various signals to output synchronized multimodal input data streams;

[0140] The perception-prediction processing subsystem, which is equipped with the Vision Mamba perception-prediction module in the system-on-chip, receives synchronized multimodal input data streams, performs fusion feature encoding and spatiotemporal alignment on the image frame sequence and time series signals, and generates the Vision Mamba perception prediction tensor;

[0141] The dynamic optimization scheduling subsystem includes an optimization parameter fusion unit and a manta ray foraging optimization algorithm processing module. The optimization parameter fusion unit is used to jointly embed the Vision Mamba perception prediction tensor and the multi-dimensional feature vector group to generate a dynamic optimization parameter set. The manta ray foraging optimization algorithm processing module follows a three-stage chain-spiral-tumble structure, optimizes the objective function based on real-time task segmentation, and iteratively outputs the optimized micro-action sequence.

[0142] The embedded collaborative control subsystem includes a microcontroller unit and a cache module. The microcontroller unit is used to map the optimized micro-action sequence into a joint motor torque instruction set and an impedance adjustment strategy to form a motor control instruction stream, and drive the joints of the bionic prosthesis to execute at a set servo frequency. At the same time, it collects execution feedback signals in real time to complete closed-loop adaptive control.

[0143] Example 1: In a home experiment, participants wearing a bionic prosthesis with an integrated multimodal sensing system were tasked with completing a continuous grasping task: avoiding obstacles, picking up an object, and placing it down. The surface of the table was randomly strewn with stainless steel soup bowls, ceramic coffee cups, plastic bottles, and glass water cups. Simultaneously, two electric vehicles periodically moved in front of the table, simulating dynamic occlusion and sudden interference.

[0144] At the beginning of the task, the bionic prosthesis system automatically collected 8 channels of surface electromyographic signals from the subject's stump. The peak-to-peak amplitude of the signals ranged from 0.21 to 0.34 mV. The IMU detected the change in arm inertia, with a maximum X-axis acceleration of 2.4 m / s. 2 The maximum Z-axis angular velocity is 57 degrees per second. The binocular vision captures the image stream and enters the Vision Mamba module at a 72Hz frequency. When the system detects a 21cm tall, 7cm diameter glass on the counter, it generates a semantic mask in real time and automatically identifies the adjacent obstacle as a ceramic coffee cup, with a distance of 4.1cm between the obstacle and the target object.

[0145] The Vision Mamba module outputs a prediction of the hand-object keypoints with a confidence level of 0.92. The current action intention is "reach forward and grasp," and the scene occlusion probability is 0.27. At this point, the manta ray foraging optimization algorithm uses the above multimodal tensor to calculate in real time the energy consumption weight of 0.35, the smoothness weight of 0.22, the terminal error weight of 0.18, and the terminal stability weight of 0.25, and selects 12 groups of primary micro-motion candidate sequences in the optimization parameter space. Each group of micro-motion candidates includes: joint displacement 2.8deg, target motor torque command 0.87Nm, impedance parameter 11.2N·mm -1 .

[0146] After the first round of chain search screening, the average single-step energy consumption was 9.1mJ, the smoothness index was 0.78, and the motion response delay was 13ms. At this point, the dynamically occluded car's speed was 0.15m / s. After entering the occluded area, the Vision Mamba confidence score dropped to 0.65. The system automatically increased the smoothness weight and reduced the energy weight. The algorithm then generated local perturbations through a spiral search, selecting those micro-motion sequences with a speed change rate of no more than 1.9deg / ms and an end-point error of less than 1.6mm as intermediate micro-motion sequences. After screening, the remaining motion response delay was 11ms.

[0147] Entering the tumbling search phase, the scene semantic mask detected that the occlusion probability of the target area increased to 0.43, and the end-of-action accuracy index was 2.0mm. The system introduced an occlusion probability factor to weight all candidate micro-action sequences and selected the action sequence with an occlusion probability <0.45 and a disturbance stability <0.25 as the final optimized action sequence. The final output execution command was a target joint motor torque of 0.83Nm and an impedance parameter of 10.5N·mm -1 The deviation of the end point of the action is 1.4mm.

[0148] Under the same conditions, the traditional fixed-step trajectory planning method was used, with the initial joint displacement set to 3.1 degrees, the motor torque to 0.93 Nm, and the response delay to 34 ms. When encountering occlusion during the grasping process, the action sequence was not adjusted in time. The final end deviation was 4.2 mm, the action failure rate increased to 8.2%, the energy consumption was 12.6 mJ / step, and the action segmentation granularity was only 6 steps.

[0149] In actual operation, the method of the present invention completed the entire process of grasping, avoiding obstacles, and placing the utility table in a total of 7.2 seconds, with a success rate of 98.4%. The average latency of the optimized response under each occlusion was 12.9ms, reducing overall system energy consumption by 18%, and extending the continuous mission endurance to 5.1 hours. The traditional method took a total of 9.0 seconds in the same scenario, with a success rate of 85.7%, an occlusion response latency of 38ms, no significant reduction in energy consumption, and a maximum flight time of only 4.0 hours.

[0150] In further data comparison, in 120 continuous grasping tests, the average differential segmentation number of the method of the present invention was 13.8, the maximum amplitude of the end jitter was 0.17mm, the highest temperature rise was 37.2℃, and no joint jam occurred; the average differential segmentation number of the traditional method was 7.1, the maximum amplitude of the end jitter was 0.42mm, the temperature rise reached 53.7℃, and joint jam occurred 6 times.

[0151] During the sample training process, the method of the present invention selected 85,000 groups of training samples, each group containing electromyographic patterns, inertial states, image key points, terminal errors and action labels, etc. The prediction accuracy under occlusion during training reached 94.3%. The traditional method training sample structure remains unchanged, and the prediction accuracy under occlusion is 80.9%.

[0152] In summary, in daily dynamic grasping application scenarios, the method of the present invention deeply couples Vision Mamba with the manta ray foraging optimization algorithm, and utilizes myoelectric drive energy consumption adaptation, confidence dynamic weighting, terminal occlusion perception and multi-stage micro-motion segmentation optimization mechanism to achieve a bionic prosthetic control effect with high motion segmentation granularity, fast real-time response, low energy consumption and strong robustness, effectively solving the practical problems of traditional methods such as high grasping failure rate, large response delay, excessive energy consumption and limited embedded deployment in complex dynamic environments.

[0153] This paper deeply couples the Vision Mamba perception prediction module, based on a bidirectional state-space network, with a swarm intelligence-based manta ray foraging optimization algorithm, achieving fine-grained, real-time task segmentation under multimodal input. By directly mapping the perception prediction tensor into multi-objective dynamic optimization parameters, the proposed method significantly improves the timeliness of action sequence reordering and micro-motion switching in bionic prosthetics in complex dynamic scenes. In occluded grasping tasks, the accuracy and response speed of action segmentation are improved compared to traditional MPC or fixed-step trajectory planning.

[0154] The present invention proposes an energy consumption adjustment factor based on real-time electromyographic activation state regulation, and introduces perception confidence, terminal occlusion probability and disturbance stability indicators in the three-stage chain-spiral-roll optimization process to achieve a dynamic trade-off between energy consumption, trajectory smoothness, execution accuracy and grasping safety. In the spiral and rolling stages, the optimization objective function is adaptively adjusted according to the current perception state and terminal safety, which not only reduces the peak energy consumption of the servo system, but also ensures the continuity of the movement and the success rate of terminal grasping in the critical micro-movement stage, which is significantly better than the existing single-objective or static weighted optimization schemes.

[0155] To address the problem of limited computing resources in prosthetic embedded platforms, this invention sinks the spatiotemporal linear reasoning of Vision Mamba and the efficient low-parameter search of the manta ray foraging optimizer into the FPGA-SoC+MCU collaborative system, forming a data stream-level high-speed cache and low-latency control link. Through the closed-loop execution mechanism and the dynamic typing output of the motor control instruction stream, the prosthesis can complete the perception-decision-execution full-link closed loop within a period of <15ms under both continuous movement and sudden state switching.

[0156] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A bionic intelligent prosthesis control method with real-time visual feedback, characterized in that: The steps include: S1. Acquire biomimetic signals through a multimodal sensor array and perform preprocessing to generate a multidimensional feature vector group; S2. Input the multidimensional feature vector group into the Vision Mamba perception-prediction module in the system-on-chip, and output the Vision Mamba perception prediction tensor; S3. Jointly map the Vision Mamba perception prediction tensor and the multidimensional feature vector group into a dynamic optimization parameter set. This dynamic optimization parameter set is input into the real-time task segmentation optimization module of the manta ray foraging optimization algorithm. In the chain search phase, a global energy-oriented search is performed to obtain a primary micro-action candidate sequence. S4. Based on the primary micro-motion candidate sequence and the perception confidence value, a combined energy consumption and smoothness search is performed in the spiral search phase to generate an intermediate micro-motion candidate sequence. Based on the intermediate micro-motion candidate sequence and the scene semantic mask, a terminal precision constrained search is performed in the tumbling search phase to obtain the optimized micro-motion sequence. S5. Send the optimized micro-action sequence to the embedded collaborative control layer, map it into a joint motor torque instruction set and impedance adjustment strategy, form a motor control instruction stream, and drive the bionic prosthetic joint to execute according to the motor control instruction stream on the microcontroller unit.

2. The bionic intelligent prosthesis control method with real-time visual feedback according to claim 1, characterized in that: The S1 includes the following contents: S11. Obtain the original Industrial Internet data stream to construct an Industrial Internet original data set. Each piece of Industrial Internet data includes a timestamp, numerical information, device identification, and data category label. S12. Perform data cleaning on the original industrial Internet data set to obtain a cleaned industrial Internet data set; S13. Perform standardization processing on the cleaned industrial Internet data set to form a standardized industrial Internet data set; S14. Eliminate redundant information from the standardized industrial Internet data set to form a final standardized industrial Internet data set; S15. Perform time series feature extraction on the final standardized Industrial Internet data set. The extracted time series features include the overall change trend of each dimension data at different times, the fluctuation amplitude in a short period of time, and the stability index of the long-term series. All time series features constitute a time series feature matrix. S16. All time series features in the time series feature matrix are structured and organized according to the preset window to construct a multidimensional feature vector group of industrial Internet data. Each group of multidimensional feature vectors corresponds to the data statistical results within a time window.

3. The bionic intelligent prosthesis control method with real-time visual feedback according to claim 2, characterized in that: The S2 includes the following contents: S21. Input the synchronized multimodal input data stream as an input sequence to the Vision Mamba perception-prediction module in the system-on-chip. The Vision Mamba perception-prediction module uses a bidirectional state space structure to process image sequence and time series data. S22. In the Vision Mamba perception-prediction module, the image frame sequence is encoded to generate a visual feature vector representing the semantic features of the image. Simultaneously, the time series containing all channel electromyographic and inertial signals is encoded to generate a temporal feature vector representing the user's motion trends. S23. Align the visual feature vector and the temporal feature vector at the time step level and merge them into a unified fused feature tensor. The fused feature tensor structurally contains information from both the image channel and the sensor channel. S24. Input the fused feature tensor into the decoder structure in the Vision Mamba perception-prediction module, and output four types of perception prediction information at each time point: A set of hand-object keypoint trajectories, each of which represents a specific position coordinate in three-dimensional space and is used to characterize the spatial relationship between the user's hand or prosthetic terminal and the object; Scene semantic mask tensor, used to represent the semantic category label corresponding to each pixel in each frame of the image; Target motion trend vector field, used to predict the overall direction change trend of user actions; The perception confidence value vector includes the key point confidence value, the semantic mask confidence value, and the motion trend confidence value, which respectively represent the credibility of the above three types of information at the current time step. The confidence value range is a real number between 0 and 1; S25. Combine the hand-object keypoint trajectory set, the scene semantic mask tensor, the target motion trend vector field and the perception confidence value vector into the Vision Mamba perception prediction tensor.

4. The bionic intelligent prosthesis control method with real-time visual feedback according to claim 3, characterized in that: The S3 includes the following: S31. Jointly embed the Vision Mamba perception prediction tensor with a multidimensional feature vector group extracted from the synchronized multimodal input data stream to construct a fused mapping tensor in the system-on-chip. The fused mapping tensor is used to generate a dynamic optimization parameter set. The dynamic optimization parameter set consists of an optimization weight set and an initial micro-motion candidate set. The optimization weight set includes an energy consumption weight α, a smoothness weight β, a target error weight γ, and an end stability weight δ. The initial micro-motion candidate set represents the executable motion state space of each joint of the bionic prosthesis within the physical constraints in the current control cycle. Each micro-motion candidate state contains a triplet of joint displacement, desired torque, and target impedance characteristics. S32. Constructing real-time task segmentation optimization objective function F based on dynamic optimization parameter set MRFO ,The real-time task segmentation optimization objective function combines the energy consumption control, trajectory smoothness, and terminal execution accuracy of the bionic prosthesis, and is defined in the following form in the chain search phase: F MRFO =α·E joint +β·T smooth +γ·E pose +δ·D stab ; Among them, E joint It represents the energy consumption per unit cycle of all servo motors by the current micro-action, which is obtained by sampling the output current and duration required by each joint. smooth It represents the speed first-order derivative change rate of the micro-motion sequence in time, E pose represents the Euclidean error between the current action execution state and the target end pose, D stab Represents the confidence level of target terminal stability under current environmental conditions, estimated based on the occlusion map and object motion trend field output by Vision Mamba; S33. During the chain search phase, based on the swarm intelligence chain propagation mechanism of the manta ray foraging optimization algorithm, a virtual swarm of individuals is set up. Each optimized individual represents a complete sequence of micro-movements. Individuals in the swarm propagate their behaviors based on the position of the current optimal individual and the dynamic weight change trend within the control cycle. S34. Introducing the energy consumption adjustment factor η based on the current myoelectric activation state of the bionic prosthesis emg , the energy consumption adjustment factor participates in the dynamic weighting of energy consumption index items, forming the energy consumption control mechanism of myoelectric drive perception: Among them, N e is the number of myoelectric channels, represents the normalized EMG signal of the i-th channel at time τ, where T is the sliding window length; S35. All micro-action sequences after iterative optimization in the chain search phase are divided into optimization objective functions according to the modified real-time task Sort by, where the corrected energy consumption item is E′ joint =η emg ·E joint , the final real-time task segmentation optimization objective function is: S36. Write the primary micro-action candidate sequence into the SRAM area of ​​the system on chip through the shared cache interface.

5. The bionic intelligent prosthesis control method with real-time visual feedback according to claim 4, characterized in that: The S4 includes the following: S41. The primary micro-motion candidate sequence and the perception confidence value vector in the Vision Mamba perception prediction tensor are jointly input into the spiral search phase of the manta ray foraging optimization algorithm, and the confidence adjustment factor ω is introduced in the spiral search phase. conf ; S42. Define the confidence-aware optimization objective function F in the spiral search phase spiral : F spiral =ω conf ·(α·E′ joint +β·T smooth )+(1-ω conf )·(γ·E pose +δ·D stab ); Among them, ω conf is the perception confidence value output by Vision Mamba at the current moment; S43. Optimize the objective function F based on confidence perception spiral In an embedded platform, local perturbations are performed on primary micro-motion candidate sequences. A set of intermediate perturbation action sequences is generated along the spiral trajectory rule of the manta ray foraging algorithm. The objective function values ​​corresponding to all perturbation action sequences are calculated, and the set with the smallest confidence-aware optimization objective function value is selected as the intermediate micro-motion candidate sequence. S44. The intermediate micro-motion candidate sequence and the scene semantic mask tensor in the Vision Mamba perception prediction tensor are jointly input into the tumbling search stage, and the terminal occlusion probability P is introduced in the tumbling search stage. occl and trajectory disturbance variation ΔT smooth , construct the terminal refinement optimization objective function F in the tumbling search phase roll : F roll =γ·E pose +δ·(1-P occl )+∈·ΔT smooth ; Among them, P occl represents the occlusion probability of the target terminal execution area in the semantic mask, provided by Vision Mamba, ΔT smooth represents the trajectory perturbation stability index between the spiral stage and the current micro-action, ∈ is the perturbation stability weight; S45. During the roll search phase, a small roll perturbation is applied to the candidate intermediate micro-motion sequences according to the roll optimization objective function. Multiple rounds of iterative updates are performed based on terminal accuracy, occlusion perception, and perturbation smoothness. Ultimately, a set of micro-motion sequences with the minimum roll optimization objective function value is selected as the optimized micro-motion sequence. S46. Map the optimized micro-motion sequence into execution commands of the bionic prosthesis, including the motor torque instructions and impedance adjustment parameter set of the target joint, and write them into the shared cache area of ​​the embedded control system; at the same time, write the optimized micro-motion sequence back to the system circular buffer.

6. The bionic intelligent prosthesis control method with real-time visual feedback according to claim 5, characterized in that: The S5 includes the following: S51. Input the optimized micro-motion sequence into the embedded collaborative control layer, construct a motor control mapping relationship based on the joint motion model in the embedded collaborative control layer, optimize each micro-motion sequence to include a joint displacement target, a target torque reference value, and a desired impedance parameter triple, and generate a motor control instruction stream for execution through the triple; S52. The motor control instruction flow sets the execution parameter range according to different control modes; S53 writes the formatted motor control instruction stream into the instruction cache queue in the microcontroller unit, which is scheduled and executed in real time by the microcontroller unit, and drives the joints of the bionic prosthesis to complete high-degree-of-freedom movements in accordance with the time sequence of the optimized micro-action sequence; S54. During the execution of the motor control command, real-time execution feedback information is collected and calculated with the target state in the optimized micro-action sequence to form an error signal; S55. If the error signal exceeds the preset threshold range, the interrupt feedback update mechanism is immediately triggered, the most recently optimized micro-action sequence cache is activated, and the local disturbance re-optimization process is entered to form a new round of motor control instruction flow.

7. The bionic intelligent prosthesis control method with real-time visual feedback according to claim 6, characterized in that: The output rules of the motor control instruction stream are defined as follows: If the target movement corresponds to the micro-movement sequence, the target joint displacement target change rate is greater than the set acceleration threshold, and the target end stability confidence D stab ≥0.75, the output type is a high-response compliant instruction flow, the motor torque is set to 95% of the target torque reference value, and the impedance parameter is set to medium stiffness; If the target action corresponds to the micro-action sequence of the joint displacement target change rate is less than or equal to the acceleration threshold, and the target end occlusion probability P occl ≥0.5, the output type is a low-speed obstacle avoidance instruction flow, the motor torque is set to 70% of the target torque reference value, and the impedance parameter is lowered to enhance compliance; If the trajectory perturbation stability index ΔT of the micro-motion sequence corresponding to the target action is smooth ≥0.3, and the energy consumption adjustment factor η emg ≥1.2, the output type is energy consumption suppression instruction flow, the motor torque is set to 60% of the target torque reference value, and the impedance parameter is set to low stiffness to relieve the load.

8. A bionic intelligent prosthesis control system with real-time visual feedback, used to execute the bionic intelligent prosthesis control method with real-time visual feedback according to any one of claims 1 to 7, characterized in that: Includes the following subsystems: The multimodal sensor acquisition subsystem is used to acquire raw electromyographic signals, inertial signals, and external visual information, and performs unified timestamp alignment and filtering preprocessing on various signals to output synchronized multimodal input data streams; The perception-prediction processing subsystem, which is equipped with the Vision Mamba perception-prediction module in the system-on-chip, receives synchronized multimodal input data streams, performs fusion feature encoding and spatiotemporal alignment on the image frame sequence and time series signals, and generates the Vision Mamba perception prediction tensor; The dynamic optimization scheduling subsystem includes an optimization parameter fusion unit and a manta ray foraging optimization algorithm processing module. The optimization parameter fusion unit is used to jointly embed the Vision Mamba perception prediction tensor and the multi-dimensional feature vector group to generate a dynamic optimization parameter set. The manta ray foraging optimization algorithm processing module follows a three-stage chain-spiral-tumble structure, optimizes the objective function based on real-time task segmentation, and iteratively outputs the optimized micro-action sequence. The embedded collaborative control subsystem includes a microcontroller unit and a cache module. The microcontroller unit is used to map the optimized micro-action sequence into a joint motor torque instruction set and an impedance adjustment strategy to form a motor control instruction stream, and drive the joints of the bionic prosthesis to execute at a set servo frequency. At the same time, it collects execution feedback signals in real time to complete closed-loop adaptive control.

Images