Software mechanical arm sensing and control method and device, equipment, storage medium

Abstract

Embodiments of the present application disclose a kind of soft body mechanical arm perception and control method and device, equipment, storage medium, the neural network observer obtained by training is forward reasoning to the corresponding perception tensor of soft body mechanical arm, the real-time state vector of soft body mechanical arm is obtained, perception tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of soft body mechanical arm, real-time state vector is used to characterize three-dimensional interactive force and three-dimensional pose of soft body mechanical arm.According to three-dimensional pose and expected pose, determine feedforward control amount, according to three-dimensional interactive force, three-dimensional pose, expected pose and expected interactive force, calculate feedback control amount.According to digital twin intelligent agent, feedforward control amount and feedback control amount, determine pressure command, pressure command is used to control soft body mechanical arm to execute action.The present application fuses shape change and strain change to carry out real-time state prediction together, and combines two control amounts of feedforward and feedback control amount to carry out high-precision mechanical arm control.

Description

Technical Field

[0001] This application relates to the field of robotic arm control, and includes, but is not limited to, a soft robotic arm sensing and control method, device, equipment, and storage medium. Background Technology

[0002] Soft robotic arms, due to their compliance and safety, exhibit unique advantages in fields such as medical surgery, rehabilitation assistance, and flexible assembly. However, the complex deformation and extremely high degrees of freedom of soft robotic arms make it difficult to directly apply traditional rigid robot modeling and control methods. Current solutions for controlling soft robotic arms suffer from inaccuracies in perception and control due to factors such as the uncertainty of soft materials, insufficient utilization of sensing data, lack of full-state feedback capabilities, and lack of effective composite control architectures. Summary of the Invention

[0003] In view of this, the soft robotic arm sensing and control method, device, equipment, and storage medium provided in the embodiments of this application can achieve high-precision control of the soft robotic arm.

[0004] The soft robotic arm sensing and control method, device, equipment, and storage medium provided in this application embodiment are implemented as follows: One aspect of this application provides a soft robotic arm sensing and control method, the method comprising: The trained neural network observer performs forward reasoning on the perception tensor corresponding to the soft robotic arm to obtain the real-time state vector of the soft robotic arm. The perception tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm. The real-time state vector includes the three-dimensional interactive force and three-dimensional pose of the soft robotic arm. The feedforward control quantity is determined based on the three-dimensional pose and the desired pose of the soft robotic arm. The feedforward control quantity is used to characterize the air pressure value that the feedforward loop expects to apply to the soft robotic arm. The feedback control quantity is calculated based on the three-dimensional interaction force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interaction force. The feedback control quantity is used to characterize the feedback loop's suggestion to supplement or reduce the air pressure value applied to the soft robotic arm. The pressure command is determined based on the digital twin agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity. The pressure command is used to control the soft robotic arm to perform actions.

[0005] In one possible implementation, the method further includes: Acquire sensory data including the optical power and tactile sensing data of the macro-bent optical fiber corresponding to the soft robotic arm; The sensed data is stored in a circular buffer of a preset length; All sensing data within the annular buffer are subjected to baseline removal, bandpass filtering, and normalization to obtain a sensing tensor including bending morphology change tensor and surface strain change tensor.

[0006] In one possible implementation, the trained neural network observer performs forward reasoning on the perceptual tensor corresponding to the soft robotic arm to obtain the real-time state vector of the soft robotic arm, including: The trained neural network observer performs bi-branch feature extraction on the perception tensor corresponding to the soft robotic arm to obtain posture features and tactile features. Cross-attention fusion is performed on the posture features and the tactile features to obtain fused features; The fused features are decoded to obtain the real-time state vector of the soft robotic arm.

[0007] In one possible implementation, determining the feedforward control quantity based on the three-dimensional pose and the desired pose of the soft robotic arm includes: Determine the inverse kinematics mapping table and feedforward gain matrix of the digital twin agent of the soft robotic arm calibrated offline; Based on the three-dimensional pose, the corresponding expected air pressure is found in the inverse kinematics mapping table; The position error is determined based on the three-dimensional pose and the desired pose of the soft robotic arm; The feedforward control quantity is obtained by linearly mapping the position error, the desired air pressure value, and the feedforward gain matrix.

[0008] In one possible implementation, calculating the feedback control quantity based on the three-dimensional interaction force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interaction force includes: The position error and force error are calculated based on the three-dimensional interactive force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interactive force. Gain correction information is obtained by performing fuzzy inference based on the position error and force error using a fuzzy rule table; The calculation feedback control quantity is determined based on the gain correction information and the preset reference control information.

[0009] In one possible implementation, determining the pressure command based on the digital twin agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity includes: The digital twin intelligent agent corresponding to the soft robotic arm aims to minimize the absolute error of the integration time, and performs simulation hyperparameter optimization based on the position error and force error to obtain the fused parameters. The feedforward control quantity and the feedback control quantity are fused based on the fusion parameters, and the pressure command is determined according to the fused parameters.

[0010] In one possible implementation, the method further includes: Establish a digital twin intelligent agent corresponding to the soft robotic arm and obtain the corresponding Jacobian matrix; The neural network observer is trained based on a training set obtained from data collection under multiple operating conditions. The training set includes multiple sample tensors and corresponding calibration state vectors. The network loss is determined based on each sample tensor, the corresponding calibration state vector, the predicted state vector output by the neural network observer, and the Jacobian matrix, and the neural network observer is adjusted until a preset condition is met. The inverse kinematics mapping table and feedforward gain matrix are obtained based on the mathematical twin agent reasoning.

[0011] Another aspect of this application provides a soft robotic arm sensing and control device, the device comprising: The forward inference module is used to perform forward inference on the perceptual tensor corresponding to the soft robotic arm through the trained neural network observer to obtain the real-time state vector of the soft robotic arm. The perceptual tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm. The real-time state vector includes the three-dimensional interactive force and three-dimensional pose of the soft robotic arm. A feedforward control module is used to determine a feedforward control quantity based on the three-dimensional pose and the desired pose of the soft robotic arm. The feedforward control quantity is used to characterize the air pressure value that the feedforward loop expects to apply to the soft robotic arm. The feedback control module is used to calculate the feedback control quantity based on the three-dimensional interaction force, the three-dimensional pose, the desired pose and desired interaction force of the soft robotic arm. The feedback control quantity is used to characterize the air pressure value that the feedback loop suggests supplementing or reducing applied to the soft robotic arm. The instruction control module is used to determine a pressure instruction based on the digital twin intelligent agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity. The pressure instruction is used to control the soft robotic arm to perform actions.

[0012] In one possible implementation, the device further includes: The data acquisition module is used to acquire perception data including the optical power and tactile perception data of the macro-bent optical fiber corresponding to the soft robotic arm; The data storage module is used to store the sensed data into a circular buffer of a preset length; The tensor determination module is used to perform baseline removal, bandpass filtering, and normalization on all sensing data within the annular buffer to obtain sensing tensors including bending morphology change tensors and surface strain change tensors.

[0013] In one possible implementation, the forward inference module is further configured to: The trained neural network observer performs bi-branch feature extraction on the perception tensor corresponding to the soft robotic arm to obtain posture features and tactile features. Cross-attention fusion is performed on the posture features and the tactile features to obtain fused features; The fused features are decoded to obtain the real-time state vector of the soft robotic arm.

[0014] In one possible implementation, the feedforward control module is further configured to: Determine the inverse kinematics mapping table and feedforward gain matrix of the digital twin agent of the soft robotic arm calibrated offline; Based on the three-dimensional pose, the corresponding expected air pressure is found in the inverse kinematics mapping table; The position error is determined based on the three-dimensional pose and the desired pose of the soft robotic arm; The feedforward control quantity is obtained by linearly mapping the position error, the desired air pressure value, and the feedforward gain matrix.

[0015] In one possible implementation, the feedback control module is further configured to: The position error and force error are calculated based on the three-dimensional interactive force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interactive force. Gain correction information is obtained by performing fuzzy inference based on the position error and force error using a fuzzy rule table; The calculation feedback control quantity is determined based on the gain correction information and the preset reference control information.

[0016] In one possible implementation, the instruction control module is further configured to: The digital twin intelligent agent corresponding to the soft robotic arm aims to minimize the absolute error of the integration time, and performs simulation hyperparameter optimization based on the position error and force error to obtain the fused parameters. The feedforward control quantity and the feedback control quantity are fused based on the fusion parameters, and the pressure command is determined according to the fused parameters.

[0017] In one possible implementation, the device further includes: The agent establishment module is used to establish the digital twin agent corresponding to the soft robotic arm and obtain the corresponding Jacobian matrix; The network training module is used to train the neural network observer based on the training set obtained from data collection under multiple working conditions. The training set includes multiple sample tensors and corresponding calibration state vectors. The loss adjustment module is used to determine the network loss based on each sample tensor, the corresponding calibration state vector, the predicted state vector output by the neural network observer, and the Jacobian matrix, and adjust the neural network observer until a preset condition is met. The parameter reasoning module is used to derive the inverse kinematics mapping table and the feedforward gain matrix based on the mathematical twin agent.

[0018] The electronic device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the method described in this application.

[0019] The computer-readable storage medium provided in this application embodiment stores a computer program thereon, which, when executed by a processor, implements the method provided in this application embodiment.

[0020] The soft robotic arm sensing and control method, apparatus, device, and storage medium provided in this application embodiment perform forward inference on the sensing tensor corresponding to the soft robotic arm through a trained neural network observer to obtain the real-time state vector of the soft robotic arm. The sensing tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm, and the real-time state vector is used to characterize the three-dimensional interactive force and three-dimensional pose of the soft robotic arm. Feedforward control quantities are determined based on the three-dimensional pose and desired pose, and feedback control quantities are calculated based on the three-dimensional interactive force, three-dimensional pose, desired pose, and desired interactive force. Pressure commands are determined based on the digital twin intelligent agent, feedforward control quantities, and feedback control quantities, and the pressure commands are used to control the soft robotic arm to perform actions. This application integrates shape change and strain change for real-time state prediction and combines both feedforward and feedback control quantities for high-precision robotic arm control. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a soft robotic arm sensing and control method according to an embodiment of this application is shown; Figure 2 A schematic diagram of a soft robotic arm sensing and control device according to an embodiment of this application is shown; Figure 3 A schematic diagram of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0025] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0026] It should be noted that the terms "first, second, third" used in the embodiments of this application are used to distinguish similar or different objects and do not represent a specific order of objects. It can be understood that "first, second, third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0027] The soft robotic arm sensing and control method of this application embodiment can be executed by any electronic device, including but not limited to mobile phones, wearable devices (such as smartwatches, smart bracelets, smart glasses, etc.), tablet computers, laptops, vehicle terminals, PCs (Personal Computers), robots, etc. The functions implemented by this method can be achieved by a processor in the electronic device calling program code. Of course, the program code can be stored in a computer storage medium. Therefore, the electronic device includes at least a processor and a storage medium.

[0028] The soft robotic arm sensing and control method of this application can be used in any application scenario that requires soft robotic arm control. For example, in the medical and rehabilitation field, it can be used to assist in minimally invasive surgery; and in the industrial and precision operation field, it can be used to grasp fragile or irregularly shaped items.

[0029] Related technologies for soft robotic arm perception and control methods can include pure kinematics-based morphological perception schemes, physical model-based control methods, and force-position separation control schemes. Pure kinematics-based morphological perception schemes can reconstruct the soft body's shape using external vision systems (depth cameras, structured light, etc.) or deploy multiple IMUs on the soft body surface and estimate the posture through inertial measurement fusion. The former is affected by field-of-view occlusion and has significant visual data processing latency, while the latter has large cumulative errors and is difficult to accurately reconstruct the 3D shape. Physical model-based control methods can establish force-deformation relationships using constitutive models of hyperelastic materials (such as Neo-Hookean and Mooney-Rivlin models) or accurately simulate soft body deformation using the finite element method. The former is sensitive to material parameters and struggles to handle large deformations and contact problems; the latter has high computational complexity and cannot meet real-time control requirements. Force-position separation control schemes can perform position control as in traditional industrial robots, with force sensors added to the end effector to provide force feedback, or achieve compliant control by establishing a mapping relationship between the end effector position and contact force. The former, soft robotic arms, are difficult to model accurately, and the position of force sensors is limited; the latter relies on accurate dynamic models, which have limited effectiveness in soft robots.

[0030] Therefore, the relevant technologies have the following technical problems: Separation of form perception and force perception: Existing solutions usually treat form perception (position / attitude) and force perception as two independent modules, failing to fully utilize the physical relationship between the two.

[0031] Lack of full-state feedback capability: The state of a soft robotic arm includes three-dimensional shape (position + attitude) and interactive forces. Existing methods have difficulty acquiring all state variables at the same time and achieving real-time feedback.

[0032] Insufficient handling of model uncertainties: Factors such as the nonlinearity, viscoelasticity, and environmental stiffness variations of soft materials lead to significant uncertainties in the model, which existing methods struggle to effectively compensate for.

[0033] Lack of feedforward-feedback coordination mechanism: Existing control methods usually rely only on feedback control (such as PID) or only have open-loop feedforward control, lacking an effective composite control architecture.

[0034] Insufficient depth of multimodal data fusion: The signals from macro-bent fiber (attitude measurement) and fiber optic grating (tactile measurement) are not fully utilized, and the depth of multimodal data fusion needs to be improved.

[0035] Therefore, there is an urgent need for a solution that can fully utilize multimodal sensing data and full-state feedback capabilities to eliminate the uncertainty of soft materials, and achieve high-precision sensing and control of soft robotic arms through an effective composite control architecture.

[0036] The following describes in detail the soft robotic arm sensing and control scheme of the present application embodiment with reference to the accompanying drawings.

[0037] Figure 1 A flowchart illustrating a soft robotic arm sensing and control method according to an embodiment of this application is shown. Figure 1 As shown, the soft robotic arm sensing and control method of this application embodiment may include the following steps S10-S40.

[0038] For ease of description, the soft robotic arm sensing and control method of this application embodiment is described using an electronic device as the execution subject. It should be understood that the execution subject of this application embodiment can also be a processor or chip in an electronic device, and this application embodiment does not impose any limitations.

[0039] Step S10: Perform forward reasoning on the perception tensor corresponding to the soft robotic arm using the trained neural network observer to obtain the real-time state vector of the soft robotic arm.

[0040] In one possible implementation, when control of the soft robotic arm is required, the electronic device can first acquire the sensing tensor corresponding to the soft robotic arm. This sensing tensor can be acquired periodically, once per cycle, and the force and position of the soft robotic arm for that cycle can be controlled based on the acquired sensing tensor. The sensing tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm.

[0041] Optionally, the electronic device can first acquire sensing data including the optical power of the macro-bending fiber corresponding to the soft robotic arm and tactile sensing data. Then, the sensing data is stored in a pre-defined circular buffer. Next, baseline removal, bandpass filtering, and standardization are performed on all sensing data within the circular buffer to obtain a sensing tensor including a bending shape change tensor and a surface strain change tensor. The optical power of the macro-bending fiber is used to characterize the bending shape of the soft robotic arm, and the tactile sensing data is used to characterize the surface strain of the soft robotic arm. The electronic device can maintain a fixed-length circular buffer, removing the oldest data each time new sensing data is stored. Further, all sensing data within the circular buffer is acquired, and each sensing data point is subtracted from its initial static mean to perform baseline removal. Bandpass filtering is used to eliminate power frequency and high-frequency noise. Finally, the standardized value is obtained by calculating the ratio of the difference between the original value of each processed sensing data point and the mean of a set of preset sensing data values ​​to the standard deviation of the set of preset sensing data values. This standardized value, due to the introduction of sensing data from multiple periods, can characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm.

[0042] In some embodiments, the optical power of the macro-bending fiber in this application embodiment can be obtained through a macro-bending fiber (MB-FBG) deployed inside the soft robotic arm. This fiber is arranged in a serpentine pattern along the length of the soft robotic arm to monitor changes in micro-bending loss, reflecting local curvature. Tactile sensing data can be obtained through a fiber grating array (FBG) deployed inside the soft robotic arm. This fiber can be distributed on the soft surface in a grid layout of preset size to monitor surface strain, reflecting contact force and bending moment. The curvature sensitivity, spatial resolution, and other attributes of the MB-FBG, as well as the strain sensitivity and spatial resolution of the FBG, can all be preset.

[0043] Furthermore, after acquiring the perception tensor in each cycle, the electronic device uses a trained neural network observer to perform forward inference on the perception tensor corresponding to the soft robotic arm, obtaining the real-time state vector of the soft robotic arm. This process can begin by extracting bi-branch features from the perception tensor corresponding to the soft robotic arm using the trained neural network observer, obtaining posture and tactile features. Then, the posture and tactile features are fused through cross-attention to obtain fused features, and the real-time state vector of the soft robotic arm is obtained by decoding the fused features. The real-time state vector can include the three-dimensional interactive forces and three-dimensional pose of the soft robotic arm.

[0044] Optionally, the neural network observer includes a macro-bending branch and a strain branch. The macro-bending branch extracts the components representing the bending morphology change trajectory in the input tensor through three convolutional layers of different scales and a bidirectional long short-term memory network to obtain a 256-dimensional pose feature F_pose. The strain branch processes the components of the surface strain change trajectory in the input tensor through convolutional layers and attention pooling layers to obtain a 256-dimensional tactile feature. Then, weights are calculated based on the cross-attention projection matrix, pose features, and tactile features determined during model training. The fused features obtained by fusing the pose features and tactile features based on the weights are then used for state decoding to obtain a real-time state vector. The real-time state vector may include the three-dimensional interactive forces (Fx, Fy, Fz) and three-dimensional poses (x, y, z, Roll, Pitch, Yaw) of the soft robotic arm. Alternatively, depending on actual needs, the real-time state vector may also include three-dimensional torques (Mx, My, Mz).

[0045] In some embodiments, the electronic device can pre-train a neural network observer. To improve the accuracy of the training results, the training of the neural network observer can be assisted by establishing a digital twin agent of the soft robotic arm. This process can begin by establishing a digital twin agent corresponding to the soft robotic arm and obtaining the corresponding Jacobian matrix. Then, the neural network observer is trained based on a training set obtained from data acquisition under multiple operating conditions. The training set includes multiple sample tensors and corresponding calibration state vectors. The network loss is determined based on each sample tensor, the corresponding calibration state vector, the predicted state vector output by the neural network observer, and the Jacobian matrix. The neural network observer is then adjusted until preset conditions are met.

[0046] The electronic device first constructs a high-fidelity digital twin, importing the CAD geometric model of the soft robotic arm into finite element software and meshing it. Then, it assigns experimentally calibrated hyperelastic material parameters (Mooney-Rivlin coefficients C10, C01) and viscoelastic parameters. Finally, the end effector of the soft arm is set to be fully constrained and fixed, simulating the condition where the end effector is blocked by a hard object and cannot move. After completing the construction of the digital twin, the electronic device applies unit drive perturbations to each drive channel through digital twin simulation and extracts the change in end effector constraint reaction force to calculate the j-th column element of the Jacobian matrix. Thus, the Jacobian matrix is ​​obtained after traversing all drive channels.

[0047] Furthermore, the electronic device can acquire multiple sample tensors by collecting data under various operating conditions. Then, using a motion capture system and a six-dimensional force sensor, the calibration state vector corresponding to each sample tensor is determined, thus obtaining a training set. The tensors of each sample are then input into a neural network observer for forward prediction to obtain a predicted state vector. Finally, based on each sample tensor, the corresponding calibration state vector, the predicted state vector output by the neural network observer, and the Jacobian matrix, the network loss is determined, and the neural network observer is adjusted until the loss meets a preset condition.

[0048] Step S20: Determine the feedforward control quantity based on the three-dimensional pose and the desired pose of the soft robotic arm.

[0049] In one possible implementation, after determining the current real-time state vector of the soft robotic arm, the electronic device can determine a feedforward control quantity based on the 3D pose and the desired pose of the soft robotic arm. The feedforward control quantity characterizes the air pressure value that the feedforward loop expects to apply to the soft robotic arm. Optionally, this process can first determine the inverse kinematics mapping table and feedforward gain matrix calibrated offline by the digital twin agent of the soft robotic arm. Then, based on the 3D pose, the corresponding desired air pressure value is looked up in the inverse kinematics mapping table, and the position error is determined based on the 3D pose and the desired pose of the soft robotic arm. Finally, a linear mapping is performed based on the position error, the desired air pressure value, and the feedforward gain matrix to obtain the feedforward control quantity.

[0050] In some embodiments, the electronic device can pre-determine the inverse kinematics mapping table and the feedforward gain matrix through mathematical twin intelligence. This process is based on mathematical calculations performed by the digital twin intelligence. The calculation of the inverse kinematics mapping table can include generating 1000 dense desired pose points (grid sampling) covering the workspace in the digital twin space. For each point, the Levenberg-Marquardt (LM) numerical iterative algorithm is invoked to solve the inverse kinematics, obtaining the pressure combination required to achieve that pose. A discrete lookup table consisting of multiple desired poses and corresponding base pressures is determined as the inverse kinematics mapping table, which can be queried during runtime using multidimensional linear interpolation. The feedforward gain matrix can be obtained by least squares fitting, using the aforementioned 1000 sets of data to construct the matrix equation ΔU = K_ff·ΔPose. The feedforward gain matrix K_ff is obtained by solving the formula K_ff = (X^T·X)^{-1}·X^T·U using the least squares method. This matrix is ​​used to describe the linear optimal mapping of how much additional air pressure needs to be added for every 1 mm deviation at the end. The dimension of this matrix can be 3×6.

[0051] Based on a pre-calibrated inverse kinematics mapping table and feedforward gain matrix, after acquiring the three-dimensional pose in each cycle, the electronic device can look up the inverse kinematics mapping table to roughly calculate the desired air pressure q_d. Then, it calculates the small deviation ΔPose = Pose_des - Pose_pred between the desired pose and the three-dimensional pose, and performs a linear mapping to obtain the feedforward control quantity u_ff = q_d + K_ff·Δpose. Here, Pose_pred is the three-dimensional pose, Pose_des is the desired pose, and u_ff is the feedforward control quantity, which can include the desired pressure increment of the three air chambers, in kPa.

[0052] Step S30: Calculate the feedback control quantity based on the three-dimensional interactive force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interactive force.

[0053] In one possible implementation, after determining the real-time state vector in the current cycle, the electronic device can also calculate a feedback control quantity based on the included three-dimensional interaction force and three-dimensional pose, as well as the desired pose and desired interaction force of the soft robotic arm. This feedback control quantity is used to characterize the air pressure value that the feedback loop suggests supplementing or reducing applied to the soft robotic arm. Specifically, the electronic device can calculate position and force errors based on the three-dimensional interaction force, three-dimensional pose, and the desired pose and desired interaction force of the soft robotic arm. Then, fuzzy inference is performed based on the position and force errors using a fuzzy rule table to obtain gain correction information. Finally, the feedback control quantity is determined based on the gain correction information and preset reference control information.

[0054] Optionally, the fuzzy rule table and reference control information used in determining the feedback control quantity in this embodiment can be predetermined. The determination process for these parameters may include first obtaining reference control information (PID, Proportional-Integral-Derivative Controller) through a critical oscillation experiment. This process involves placing the soft robotic arm at a typical operating point, keeping it horizontally suspended, disabling the integral (I) and derivative (D) components, and enabling only proportional (P) control. Starting from 0, the proportional gain Kp is slowly increased, and a small step signal (e.g., a desired displacement of 5 mm) is applied to the system. The system response curve is observed until the output waveform exhibits a continuous oscillation with constant amplitude (neither decaying nor diverging). Record the critical gain Ku (e.g., 12.5 N / m) and critical oscillation period Tu (e.g., 0.8 s) at this time, and calculate the reference control information based on the Ziegler-Nichols (ZN) empirical formula. This reference control information may include the initial proportional gain reference value, the initial integral gain reference value, and the initial derivative gain reference value, which are used to determine the instantaneous response strength of the system to the current deviation, the elimination speed of the system to the historical accumulated deviation, and the damping suppression strength of the system to the deviation change trend (speed).

[0055] Furthermore, the process of determining the fuzzy rule table can begin by defining the error domain (position error [-5mm, 5mm], force error [-0.5N, 0.5N]). Pre-defined Mamdani-type fuzzy rules can then be formulated (e.g., "If e_pos is positive and e_force is zero, then ΔKp is positive and ΔKi is negative").

[0056] Based on a pre-determined fuzzy rule table and baseline control information, the electronic device can first calculate the position error e_pos(k) = Pose_des - Pose_pred and the force error e_force(k) = F_des - F_pred, where Pose_pred is the 3D pose, Pose_des is the desired pose, F_pred is the 3D interaction force, and F_des is the desired interaction force. The electronic device can then input e_pos and e_force into a fuzzy inference engine (Mamdani inference) to look up the fuzzy rule table and obtain the gain correction information ΔKp, ΔKi, and ΔKd. Further, based on the gain correction information, the baseline control information Kp0, Ki0, and Kd0 are corrected to obtain the corrected control information, including Kp = Kp0 + ΔKp, Ki = Ki0 + ΔKi, and Kd = Kd0 + ΔKd. Then, based on the corrected control information and the above position error and force error, the feedback control quantity u_fb(k)=Kp·e(k)+Ki·Σe(k)·Δt+Kd·(e(k)-e(k-1)) / Δt is calculated.

[0057] Step S40: Determine the pressure command based on the digital twin intelligent agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity.

[0058] In one possible implementation, after the electronic device determines the feedforward and feedback control quantities of the soft robotic arm, a pressure command can be determined based on the digital twin agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity. This pressure command is used to control the robotic arm to perform actions. This process can begin by optimizing the simulation hyperparameters based on position and force errors using the digital twin agent corresponding to the soft robotic arm, with the goal of minimizing the absolute error of the integral time. Then, fused parameters are applied to the feedforward and feedback control quantities based on these fused parameters, and the pressure command is determined according to the parameter fusion.

[0059] Optionally, this process can begin by acquiring the digital twin of the soft robotic arm and performing extensive offline simulations. Using automated hyperparameter optimization tools (such as Bayesian search), with the goal of minimizing the absolute error in integral time, simulation hyperparameters are optimized based on position and force errors to obtain fusion parameters w1, w2, and w3. After obtaining the fusion parameters, the fusion coefficient γ = sigmoid(w1·||e_pos|| + w2·||e_force|| + w3·||d(e_force) / dt||), where e_pos is the position error and e_force is the force error. Further, based on the fusion coefficient, the total control quantity is calculated as u_total = γ·u_ff + (1-γ)·u_fb, where u_ff is the feedforward control quantity and u_fb is the feedback control quantity. The final control data, u_final=clip(u_total,0,300kPa), is obtained by performing physical constraint projection based on the total control quantity, with the rate of change |Δu| < 50kPa / step. Furthermore, the electronic device can generate corresponding pressure commands based on this control data to control the soft robotic arm. After executing the pressure command, the electronic device re-acquires the corresponding sensing tensor of the soft robotic arm for the next cycle of soft robotic arm control until the preset stopping condition is met.

[0060] In some embodiments, the electronic device can also render and display the 3D soft arm shape and force cloud map of the soft robotic arm in real time through a digital twin intelligent agent after generating pressure commands, so that users can intuitively know the real-time status of the soft robotic arm.

[0061] Based on the above technical features, this application provides a force-position fusion sensing method for a soft robotic arm based on a neural network observer and feedforward-feedback composite control. By constructing an end-to-end deep neural network observer, it deeply fuses the multimodal sensing signals of macro-bending optical fibers and fiber Bragg gratings, achieving real-time synchronous estimation of three-dimensional shape and interactive force. This fundamentally solves the problems of perception separation and lack of full-state feedback in traditional solutions. At the control level, it innovatively combines the fast response and fine correction capabilities of the inverse kinematics of the feedforward controller with an adaptive fusion strategy, and introduces a digital twin advanced safety warning, making the position tracking accuracy more effective than that of pure feedback control force control. The improvement is that the data-driven nature of the neural network observer allows it to adapt to the nonlinear and viscoelastic changes of soft materials without relying on a precise physical model, and it still maintains good generalization under unseen conditions. The digital twin platform runs through the entire process of offline parameter calculation and online safety monitoring. It provides key prior knowledge such as physical constraint Jacobian matrix and inverse kinematics mapping table, and provides interpretability and active safety assurance for the system through real-time 3D visualization and advanced trajectory prediction. Finally, it forms a set of intelligent compliant control solutions that do not require external sensors and have both fast response and high steady-state accuracy. It can be widely used in complex human-computer interaction scenarios such as medical surgery, flexible assembly, and rehabilitation assistance.

[0062] It should be understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0063] Based on the foregoing embodiments, this application provides a soft robotic arm sensing and control device, which includes various modules and units included in each module, and can be implemented by a processor; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0064] Figure 2 A schematic diagram of a soft robotic arm sensing and control device according to an embodiment of this application is shown. Figure 2 As shown, the soft robotic arm sensing and control device in this embodiment includes: The forward inference module 20 is used to perform forward inference on the perceptual tensor corresponding to the soft robotic arm through the trained neural network observer to obtain the real-time state vector of the soft robotic arm. The perceptual tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm. The real-time state vector includes the three-dimensional interactive force and three-dimensional pose of the soft robotic arm. The feedforward control module 21 is used to determine the feedforward control quantity based on the three-dimensional pose and the desired pose of the soft robotic arm. The feedforward control quantity is used to characterize the air pressure value that the feedforward loop expects to apply to the soft robotic arm. Feedback control module 22 is used to calculate feedback control quantity based on the three-dimensional interaction force, the three-dimensional pose, the desired pose and desired interaction force of the soft robotic arm, and the feedback control quantity is used to characterize the air pressure value that the feedback loop suggests supplementing or reducing applied to the soft robotic arm. The instruction control module 23 is used to determine a pressure instruction based on the digital twin intelligent agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity. The pressure instruction is used to control the soft robotic arm to perform actions.

[0065] In one possible implementation, the device further includes: The data acquisition module is used to acquire perception data including the optical power and tactile perception data of the macro-bent optical fiber corresponding to the soft robotic arm; The data storage module is used to store the sensed data into a circular buffer of a preset length; The tensor determination module is used to perform baseline removal, bandpass filtering, and normalization on all sensing data within the annular buffer to obtain sensing tensors including bending morphology change tensors and surface strain change tensors.

[0066] In one possible implementation, the forward inference module 20 is further configured to: The trained neural network observer performs bi-branch feature extraction on the perception tensor corresponding to the soft robotic arm to obtain posture features and tactile features. Cross-attention fusion is performed on the posture features and the tactile features to obtain fused features; The fused features are decoded to obtain the real-time state vector of the soft robotic arm.

[0067] In one possible implementation, the feedforward control module 21 is further configured to: Determine the inverse kinematics mapping table and feedforward gain matrix of the digital twin agent of the soft robotic arm calibrated offline; Based on the three-dimensional pose, the corresponding expected air pressure is found in the inverse kinematics mapping table; The position error is determined based on the three-dimensional pose and the desired pose of the soft robotic arm; The feedforward control quantity is obtained by linearly mapping the position error, the desired air pressure value, and the feedforward gain matrix.

[0068] In one possible implementation, the feedback control module 22 is further configured to: The position error and force error are calculated based on the three-dimensional interactive force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interactive force. Gain correction information is obtained by performing fuzzy inference based on the position error and force error using a fuzzy rule table; The calculation feedback control quantity is determined based on the gain correction information and the preset reference control information.

[0069] In one possible implementation, the instruction control module 23 is further configured to: The digital twin intelligent agent corresponding to the soft robotic arm aims to minimize the absolute error of the integration time, and performs simulation hyperparameter optimization based on the position error and force error to obtain the fused parameters. The feedforward control quantity and the feedback control quantity are fused based on the fusion parameters, and the pressure command is determined according to the fused parameters.

[0070] In one possible implementation, the device further includes: The agent establishment module is used to establish the digital twin agent corresponding to the soft robotic arm and obtain the corresponding Jacobian matrix; The network training module is used to train the neural network observer based on the training set obtained from data collection under multiple working conditions. The training set includes multiple sample tensors and corresponding calibration state vectors. The loss adjustment module is used to determine the network loss based on each sample tensor, the corresponding calibration state vector, the predicted state vector output by the neural network observer, and the Jacobian matrix, and adjust the neural network observer until a preset condition is met. The parameter reasoning module is used to derive the inverse kinematics mapping table and the feedforward gain matrix based on the mathematical twin agent.

[0071] The descriptions of the above device embodiments are similar to those of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0072] It should be noted that, in the embodiments of this application... Figure 2The module division of the illustrated soft robotic arm sensing and control device is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, exist as separate physical units, or be integrated into one unit with two or more units. The integrated units can be implemented in hardware, as software functional units, or a combination of both.

[0073] It should be noted that, in the embodiments of this application, if the above-described methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0074] Figure 3 A schematic diagram of an electronic device according to an embodiment of this application is shown. For example... Figure 3 As shown in the figure, this application provides an electronic device, which can be a server, and its internal structure diagram can be as follows. Figure 3 As shown, the electronic device includes a processor 320, a memory, and a transceiver 340 connected via a system bus 310. The processor 320 provides computing and control capabilities. The memory includes a non-volatile storage medium 331 and internal memory 332. The non-volatile storage medium 331 stores an operating system, computer programs, and a database. The internal memory 332 provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium 331. The database stores data. The transceiver 340 communicates with external terminals via a network connection. The computer program is executed by the processor 320 to implement the aforementioned methods.

[0075] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor 320, implements the steps of the method provided in the above embodiments.

[0076] This application provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the steps in the method provided in the above-described method embodiments.

[0077] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0078] In one possible implementation, the apparatus provided in this application can be implemented as a computer program, which can be configured as follows: Figure 3 The device operates on the electronic device shown. The memory of the electronic device can store the various program modules that make up the above-described apparatus. The computer program composed of the various program modules causes the processor 320 to execute the steps of the methods in the various embodiments of this application described in this specification.

[0079] It should be noted that the descriptions of the storage medium and device embodiments above are similar to the descriptions of the method embodiments above, and have similar beneficial effects. For technical details not disclosed in the storage medium, storage medium, and device embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0080] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, phrases such as "in one possible implementation," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, they will not be repeated here.

[0081] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that there can be three kinds of relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist simultaneously, and object B exists alone.

[0082] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0083] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple modules or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or modules can be electrical, mechanical, or other forms.

[0084] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0085] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0086] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0087] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0088] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0089] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0090] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0091] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A sensing and control method for a soft robotic arm, characterized in that, The method includes: The trained neural network observer performs forward reasoning on the perception tensor corresponding to the soft robotic arm to obtain the real-time state vector of the soft robotic arm. The perception tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm. The real-time state vector includes the three-dimensional interactive force and three-dimensional pose of the soft robotic arm. The feedforward control quantity is determined based on the three-dimensional pose and the desired pose of the soft robotic arm. The feedforward control quantity is used to characterize the air pressure value that the feedforward loop expects to apply to the soft robotic arm. The feedback control quantity is calculated based on the three-dimensional interaction force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interaction force. The feedback control quantity is used to characterize the feedback loop's suggestion to supplement or reduce the air pressure value applied to the soft robotic arm. The pressure command is determined based on the digital twin agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity. The pressure command is used to control the soft robotic arm to perform actions.

2. The method according to claim 1, characterized in that, The method further includes: Acquire sensory data including the optical power and tactile sensing data of the macro-bent optical fiber corresponding to the soft robotic arm; The sensed data is stored in a circular buffer of a preset length; All sensing data within the annular buffer are subjected to baseline removal, bandpass filtering, and normalization to obtain a sensing tensor including bending morphology change tensor and surface strain change tensor.

3. The method according to claim 1, characterized in that, The trained neural network observer performs forward inference on the perception tensor corresponding to the soft robotic arm to obtain the real-time state vector of the soft robotic arm, including: The trained neural network observer performs bi-branch feature extraction on the perception tensor corresponding to the soft robotic arm to obtain posture features and tactile features. Cross-attention fusion is performed on the posture features and the tactile features to obtain fused features; The fused features are decoded to obtain the real-time state vector of the soft robotic arm.

4. The method according to claim 1, characterized in that, The step of determining the feedforward control quantity based on the three-dimensional pose and the desired pose of the soft robotic arm includes: Determine the inverse kinematics mapping table and feedforward gain matrix of the digital twin agent of the soft robotic arm calibrated offline; Based on the three-dimensional pose, the corresponding expected air pressure is found in the inverse kinematics mapping table; The position error is determined based on the three-dimensional pose and the desired pose of the soft robotic arm; The feedforward control quantity is obtained by linearly mapping the position error, the desired air pressure value, and the feedforward gain matrix.

5. The method according to claim 1, characterized in that, The step of calculating the feedback control quantity based on the three-dimensional interaction force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interaction force includes: The position error and force error are calculated based on the three-dimensional interactive force, the three-dimensional pose, the desired pose of the soft robotic arm, and the desired interactive force. Gain correction information is obtained by performing fuzzy inference based on the position error and force error using a fuzzy rule table; The calculation feedback control quantity is determined based on the gain correction information and the preset reference control information.

6. The method according to claim 5, characterized in that, The step of determining the pressure command based on the digital twin intelligent agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity includes: The digital twin intelligent agent corresponding to the soft robotic arm aims to minimize the absolute error of the integration time, and performs simulation hyperparameter optimization based on the position error and force error to obtain the fused parameters. The feedforward control quantity and the feedback control quantity are fused based on the fusion parameters, and the pressure command is determined according to the fused parameters.

7. The method according to claim 1, characterized in that, The method further includes: Establish a digital twin intelligent agent corresponding to the soft robotic arm and obtain the corresponding Jacobian matrix; The neural network observer is trained based on a training set obtained from data collection under multiple operating conditions. The training set includes multiple sample tensors and corresponding calibration state vectors. The network loss is determined based on each sample tensor, the corresponding calibration state vector, the predicted state vector output by the neural network observer, and the Jacobian matrix, and the neural network observer is adjusted until a preset condition is met. The inverse kinematics mapping table and feedforward gain matrix are obtained based on the mathematical twin agent reasoning.

8. A soft robotic arm sensing and control device, characterized in that, The device includes: The forward inference module is used to perform forward inference on the perceptual tensor corresponding to the soft robotic arm through the trained neural network observer to obtain the real-time state vector of the soft robotic arm. The perceptual tensor is used to characterize the bending shape change trajectory and surface strain change trajectory of the soft robotic arm. The real-time state vector includes the three-dimensional interactive force and three-dimensional pose of the soft robotic arm. A feedforward control module is used to determine a feedforward control quantity based on the three-dimensional pose and the desired pose of the soft robotic arm. The feedforward control quantity is used to characterize the air pressure value that the feedforward loop expects to apply to the soft robotic arm. The feedback control module is used to calculate the feedback control quantity based on the three-dimensional interaction force, the three-dimensional pose, the desired pose and desired interaction force of the soft robotic arm. The feedback control quantity is used to characterize the air pressure value that the feedback loop suggests supplementing or reducing applied to the soft robotic arm. The instruction control module is used to determine a pressure instruction based on the digital twin intelligent agent corresponding to the soft robotic arm, the feedforward control quantity, and the feedback control quantity. The pressure instruction is used to control the soft robotic arm to perform actions.

9. An electronic device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images