A Slip Sensing Method for End-Factory Actuators in Minimally Invasive Surgery Based on Temporal Convolutional Networks

Abstract

This invention discloses a slip sensing method for a minimally invasive surgical end effector based on a temporal convolutional network. It is used in minimally invasive surgical end effectors with slip sensing capabilities. When the upper and lower clamps of the end effector clamp an object, causing vibration in the flexible vibration transmission layer due to tissue slippage, this vibration is initially amplified by a vibration signal amplification unit and transmitted to a piezoresistive cantilever beam. This allows the change in resistance of the piezoresistive cantilever beam to be converted into a voltage signal. Based on the voltage signal, a slip detection model based on a temporal convolutional network is used to detect the slippage, obtaining a K-dimensional probability vector. A two-level judgment mechanism is introduced: the first level uses a dual-threshold hysteresis judgment based on the comprehensive slip probability to determine the "slippage / non-slippage" state of the end effector; the second level identifies the slippage direction based on the direction sub-probability when the first level judgment result is "slippage". Based on this invention, the "slippage / non-slippage" state of the end effector can be determined, and the slippage direction can be identified based on the relative probabilities of each direction subclass within the same probability vector.

Description

Technical Field

[0001] This invention relates to a method for sensing the sliding motion of an end effector in minimally invasive surgery based on a temporal convolutional network, belonging to the field of medical device control technology. Background Technology

[0002] Minimally invasive surgery (MIS) has gradually replaced some traditional open surgeries and become the preferred choice for many surgical procedures due to its advantages such as less trauma, faster postoperative recovery, and fewer complications. In minimally invasive surgery, the surgeon uses a minimally invasive surgical robot (typically represented by the da Vinci Surgical System) to remotely manipulate tissues through end effectors to perform a series of delicate actions such as clamping, traction, separation, and suturing. Among these, stable clamping of soft tissues is a fundamental element for the successful execution of the surgery.

[0003] However, human soft tissue is typically moist, highly compliant, and easily damaged. When the clamping force of the end effector is insufficient or the clamping posture is incorrect, relative slippage between the tissue and the clamping surface is highly likely to occur. Studies have shown that in minimally invasive procedures such as laparoscopic colectomy, the incidence of grasping failure due to tissue slippage can reach approximately 7%, and may further induce serious complications such as tissue tearing, bleeding, and loss of critical field of vision. Since the end effectors of existing minimally invasive surgical robots generally lack tactile sensing capabilities comparable to human fingertips, surgeons rely primarily on visual feedback and personal experience to judge the clamping status. This significantly increases the reliance on surgeon experience during surgery and limits the development of surgical robots towards autonomy and intelligence. Therefore, providing a reliable tissue slippage sensing method for end effectors in minimally invasive surgery is of great significance for improving surgical safety, reducing the risk of tissue damage, and promoting the intelligent operation of surgical robots.

[0004] Existing research on tissue slip sensing in end effectors for minimally invasive surgery mainly follows the following technical routes, but all have significant shortcomings:

[0005] 1. Slip sensing methods based on the principle of heat flow. This type of method integrates micro-heating elements and several thermistors on a sensor chip, and determines slip events by measuring changes in the thermal gradient distribution in the area where the tissue contacts the sensor. Although the feasibility of this method has been verified in isolated animal tissues, it has the following shortcomings: First, the sensor needs to continuously apply a heat source to the contacting tissue, and the heating power and temperature rise range are strictly constrained by the tissue thermal damage safety threshold; Second, the smallest tissue displacement that can be detected is usually on the order of millimeters or larger (about 2 mm), making it difficult to effectively identify sub-millimeter-level micro-displacements in the initial stage of slip; Third, the single-point sensing structure lacks spatial distribution information, making it difficult to simultaneously reflect the directional and local features of slip; Fourth, its response is easily affected by environmental factors such as tissue humidity, thickness, and initial temperature, and its robustness is limited in the complex and variable in vivo thermal environment.

[0006] 2. Slippage sensing methods based on force / torque feedback. These methods integrate force / torque sensors within clamps or surgical instruments, detecting abrupt changes in clamping force or friction coefficient to determine if slippage has occurred. Since the clamping force often changes significantly by the time slippage occurs, this type of method is essentially a reactive sensing mechanism, unable to predict early signs of slippage, leaving a very limited window for the control system to intervene and prevent slippage. Furthermore, active manipulations during surgery, such as traction and dissection, can themselves cause fluctuations in clamping force, easily confused with actual slippage events.

[0007] 3. Initial Slip Sensing Method Based on Displacement Measurement of Curved Clamping Surface and Movable Island Array. This method designs the clamping surface as a curved surface and divides it into several independently movable "islands," using the difference in the relative displacement of each island to determine the initial slip. This approach achieves a high detection success rate on silicone tissue phantoms, but the success rate drops significantly on ex vivo biological tissues (such as pig liver). The main reasons are: uneven thickness and abnormal tissue morphology (such as pores, tears, and fat / solid mixtures) of biological tissues can lead to an imbalance in the mechanical engagement of the islands, resulting in false triggering or missed detection; in addition, this approach requires significant modification to the mechanical structure of the end effector, resulting in a complex structure that is difficult to miniaturize, making it unsuitable for integration and deployment in existing standard minimally invasive surgical instruments.

[0008] 4. Slip recognition methods based on traditional vibration signal analysis or shallow convolutional neural networks. Since slip events are often accompanied by micro-frictional vibrations between the clamping surface and the tissue, one type of research attempts to collect vibration signals from the contact surface and use power spectral density (PSD) thresholding or frequency domain features derived from short-time Fourier transforms to determine slip. While these methods are simple to implement, in real surgical environments, background noise such as mechanical vibrations transmitted from the robotic arm chassis and high-frequency electromagnetic interference from the electrosurgical unit easily triggers misjudgments, resulting in limited accuracy under real-world noise conditions. Subsequent research introduced convolutional neural networks (CNNs) to classify the frequency domain feature maps, improving performance. However, standard CNNs are essentially modeling tools for local spatial features. For the dynamic process of slip events, which evolves gradually over time, their long-range temporal dependence modeling capability is insufficient, making it difficult to stably capture the subtle temporal precursor features at the moment of slip initiation. Furthermore, when relying solely on a single probability threshold for slip / non-slip judgment, high-frequency false triggers are easily generated when the model output fluctuates near the critical value, causing repeated jitter in the end effector control state and threatening surgical safety.

[0009] In summary, existing minimally invasive surgical end effector slip sensing technology still has significant shortcomings in the following aspects: (1) The sensing methods generally rely on active heating or major mechanical structure modifications, making it difficult to achieve thermal damage-free tissue and compact integration compatible with existing clamping surfaces; (2) Most solutions adopt single-point sensing or post-event reactive force feedback, lacking early, distributed sensing capabilities for millimeter and sub-millimeter level micro-slippage; (3) For background interference such as surgical arm vibration and electrosurgical noise, there is a lack of systematic suppression methods that combine hardware damping and software feature screening; (4) At the algorithm level, there is a lack of deep modeling capabilities for the long-term temporal evolution of slippage events, and there is a lack of robust judgment mechanisms against critical jitter, resulting in limited reliability in complex clinical scenarios.

[0010] Therefore, there is an urgent need to propose a method for slip sensing of the end effector in minimally invasive surgery that combines compact integration, multi-channel distributed micro-vibration sensing, background noise suppression, and temporal deep feature modeling capabilities, so as to achieve early, robust, and stable identification of tissue slip events during surgery. Summary of the Invention

[0011] To better address the technical problem of the lack of early, distributed, interference-resistant, and stable slip sensing methods for existing minimally invasive surgical end effectors during soft tissue clamping, this invention provides a slip sensing method for minimally invasive surgical end effectors based on temporal convolutional networks to determine the "slip / non-slip" state of the end effector; and further performs slip direction identification based on direction probability.

[0012] The technical solution of this invention is:

[0013] A method for slip sensing in a minimally invasive surgical end effector based on a temporal convolutional network is disclosed. This method is used in a minimally invasive surgical end effector with slip sensing functionality. The end effector includes an upper clamp 1, a lower clamp 2, a flexible vibration transmission layer 3, a piezoresistive slip sensor 4, and a protective shell 6. One end of the upper clamp 1 and the lower clamp 2 are hinged, while the other end is a free end. The side of the upper clamp 1 and the lower clamp 2 that is close together serves as a clamping surface. The clamping surface has a first groove for mounting the protective shell 6. The protective shell 6 has a second groove for mounting the piezoresistive slip sensor 4 and the flexible vibration transmission layer 3, which are arranged vertically and fitted together. The opening of the second groove is located on the side away from the first groove and within the second groove. The piezoresistive slip sensor 4 is arranged relative to the flexible vibration transmission layer 3 on the side closer to the first groove; the flexible vibration transmission layer 3 is provided with a contact point 9 on the side closer to the piezoresistive slip sensor 4, and a vibration signal amplification unit 11 is provided on the side of the flexible vibration transmission layer 3 away from the piezoresistive slip sensor 4; a piezoresistive cantilever beam 8 is provided on the side of the piezoresistive slip sensor 4 closer to the flexible vibration transmission layer 3. When the upper clamp 1 and lower clamp 2 clamp the object, causing the flexible vibration transmission layer 3 to be subjected to vibration generated by tissue slippage, the vibration is initially amplified by the vibration signal amplification unit 11 and transmitted to the piezoresistive cantilever beam 8, so as to realize the conversion of the resistance change corresponding to the piezoresistive cantilever beam 8 into a voltage signal; the slippage is detected by a slippage detection model based on a temporal convolutional network based on the voltage signal.

[0014] Furthermore, the slip detection model based on temporal convolutional networks includes:

[0015] Input layer: 8-channel voltage timing window data are acquired based on the piezoresistive sliding sensor 4. The input tensor is... Among them, the eight piezoresistive cantilever beams 8 on the piezoresistive sliding sensor 4 are arranged in a crisscross pattern;

[0016] Channel attention module: First, The input sequence is compressed into a global average pooling operation over time. The channel descriptors are then processed through two fully connected layers to learn the non-linear dependencies between channels, generating eight channel weight coefficients between 0 and 1. Finally, these eight weight coefficients are compared with the original... The input tensor is multiplied element-wise to obtain the output of the channel attention module;

[0017] Temporal Convolutional Residual Stack: The feature sequence output from the channel attention module is input to the temporal convolutional residual stack; the temporal convolutional residual stack consists of n concatenated residual blocks, and the inflation factor of each residual block is... It increases exponentially; each residual block contains: for fusing cross-channel features. Channel hybrid convolutional layers and convolutional kernels are The expansion factor is The dilated causal convolutional layer; the input and output branches of each residual block are fused element-wise through skip connections; finally, the output shape of the temporal convolutional residual stack is... Advanced temporal feature maps;

[0018] Output layer: First, for The advanced temporal feature map is subjected to global average pooling along the temporal dimension to obtain a global feature vector of length 64. This global feature vector is then fed into a two-stage fully connected network: the first fully connected layer maps the 64-dimensional global feature vector to an H-dimensional hidden representation, which is then activated by ReLU and supplemented with Dropout regularization; the second fully connected layer maps the H-dimensional hidden representation to a K-dimensional output, which is then activated by Softmax to output a K-dimensional probability vector. ,in Corresponding to the "no slip" category, Each corresponds to a different sliding direction subcategory.

[0019] Furthermore, a two-level judgment mechanism is introduced for the K-dimensional probability vector: the first level is used to determine the "slip / non-slip" state of the end effector by using a double-threshold hysteresis judgment based on the slip comprehensive probability; the second level is used to identify the slip direction based on the direction subprobability when the first level judgment result is "slip".

[0020] Furthermore, the dual-threshold hysteresis judgment based on the sliding synthesis probability is specifically as follows:

[0021] Define the slip probability Set a high threshold With low threshold ,in Both can take values ​​in the interval (0,1):

[0022] 1) In the minimally invasive surgical end effector with slip sensing function, which is currently in a "non-slip state", and At that time, the state of the end effector is switched to "slip state";

[0023] 2) The minimally invasive surgical end effector with sliding sensing function is currently in a "sliding state," and When this happens, the state of the end effector is switched to "non-slip state";

[0024] 3) When At this time, the end effector does not switch states and maintains the judgment result of the previous moment.

[0025] Furthermore, the slip direction recognition based on direction subprobabilities specifically involves: when the end effector is in a "slip state", identifying the direction subprobabilities in the same probability vector. Take argmax and output the category of the direction sub-probability corresponding to the maximum probability as the sliding direction within the current time window.

[0026] Furthermore, the slip detection model based on temporal convolutional networks employs multi-class cross-entropy with class weights and label smoothing as the loss function. :

[0027] ;

[0028] in: The first output of the model Class probability; For the first Class weight coefficient; This represents the target distribution after label smoothing.

[0029] The beneficial effects of this invention are:

[0030] Firstly, in terms of sensing method, this invention avoids active heating of the tissue and provides directional redundancy information. Because the present invention embeds a sensing array composed of multiple piezoresistive cantilever beams spatially distributed on the clamping surface, the micro-vibrations caused by the slippage event are converted into multiple independent voltage timing signals by the piezoresistive effect of each cantilever beam. This allows the present invention to break free from the active heating link relied upon by existing heat flow type slip sensors in terms of physical sensing principle, thereby avoiding the contradiction between heating power and the safety threshold of tissue thermal damage. Simultaneously, compared to heat flow type or force feedback type schemes using single-point sensing, multi-point distributed sampling provides spatial redundancy features for subsequent models. Therefore, even when the sensor is partially obstructed or the signal-to-noise ratio of a certain channel is low, the detection capability can still be maintained by relying on other channels.

[0031] Secondly, the invention overcomes the severe attenuation of micro-vibrations at the contact interface during signal acquisition. Considering the extremely weak amplitude of mechanical vibrations generated on the clamping surface during the initial slippage stage, direct acquisition by the sensor substrate would result in significant attenuation due to impedance mismatch at the contact interface. Therefore, this invention incorporates a flexible vibration transmission layer with a vibration amplification unit above the piezoresistive slippage sensor. This vibration amplification unit can be fabricated into microstructures at the micrometer scale to form localized micro-contacts with the tissue surface, geometrically guiding and amplifying the relative motion components of the contact area and directionally transmitting them to the piezoresistive cantilever beam below. Therefore, compared to solutions that rely solely on passive vibration pickup from a flat clamping surface, this invention can acquire slippage precursor signals before a significant change in clamping force occurs, essentially transforming reactive detection into precursor detection.

[0032] Thirdly, a hardware-software collaborative dual-layer suppression mechanism is formed to enhance anti-interference capabilities. Considering that the vibration transmitted from the surgical arm chassis and the high-frequency electromagnetic interference from the electrosurgical unit are the main causes of misjudgments by traditional power spectral density thresholding methods and shallow frequency-domain convolutional neural networks, this invention utilizes the inherent viscoelastic damping properties of PDMS polymer material to attenuate high-frequency background noise at the physical source of the vibration transmission path. Furthermore, a channel attention module is introduced at the front end of the temporal convolutional network to generate adaptive weights for the eight voltage signals based on their signal-to-noise ratio and feature saliency, enhancing effective channels and suppressing noise-dominant channels. This hardware-software collaborative dual-layer mechanism acts on the "transmission link" and "discrimination link" of noise, respectively. Therefore, compared to existing schemes that rely solely on a single frequency-domain threshold or frequency-domain CNN, this invention improves the robustness of distinguishing between real slip signals and operational vibrations in strong noise backgrounds.

[0033] Fourthly, the slip detection model and two-level judgment structure of this invention balance long-range temporal modeling capability with the stability of critical state judgment. Considering that tissue slip events exhibit a gradual evolution from micro-vibration accumulation to macro-slip in time, the feature span of such processes is much larger than the local receptive field of standard convolutional neural networks in the frequency domain feature map. Therefore, this invention adopts a temporal convolutional network composed of multi-scale dilated causal convolution and residual connections. Its receptive field expands exponentially with the network depth without leaking future information, thus enabling the modeling of long-range temporal dependencies within a millisecond time window, overcoming the shortcomings of standard CNNs in characterizing the gradual evolution of slip. Furthermore, a dual-threshold hysteresis judgment based on slip synthesis probability (i.e., the sum of probabilities of non-"no slip") is introduced at the output end. It has Schmitt triggering characteristics, maintaining the judgment result of the previous moment unchanged when the slip synthesis probability is in the hysteresis interval composed of high and low thresholds; at the same time, the slip direction is identified based on the relative probability of each directional subclass in the same probability vector. As can be seen from the above, compared with the single threshold judgment scheme, the present invention not only avoids the frequent switching of the end effector state and mechanical vibration caused by the fluctuation of the critical probability, but also provides directional feedback for the adjustment of the anti-slip force of the end effector.

[0034] Fifthly, in terms of engineering integration, it is compatible with standard clamp structures and can be adapted to different surgical scenarios. The piezoresistive cantilever beam array, flexible vibration transmission layer, and integrated PDMS protective shell involved in this invention are geometrically matched with the grooves of standard clamps, thus achieving embedded integration without requiring significant modifications to the shape of existing end effectors. At the same time, the sensor size, piezoresistive structure, and the material and thickness of the flexible vibration transmission layer can all be adjusted according to specific surgical scenarios. Compared to curved island array solutions that require significant modifications to the clamp shape, this invention offers better adaptability for engineering deployment on different minimally invasive surgical instruments. Attached Figure Description

[0035] Figure 1 This is a structural diagram of the slip detection model based on temporal convolutional networks in this invention.

[0036] Figure 2 This is a schematic diagram of a minimally invasive surgical end effector with sliding sensing function (the vibration signal amplification unit is not shown).

[0037] Figure 3 This is a multi-view schematic diagram of a piezoresistive sliding sensor. Figure 3 (a) is a 3D diagram. Figure 3 (b) The right half is the main view.

[0038] Figure 4 These are schematic diagrams of various structures of flexible vibration transmission layers.

[0039] Figure 5 This is a multi-view structural diagram of the PDMS protective shell (from top to bottom and left to right: front view, left view, top view, and perspective view).

[0040] Figure 6 This is a Wheatstone bridge diagram of a single varistor.

[0041] Figure 7 This is a graph showing the loss function curves of the training set and validation set during the training process of the slip detection model based on a temporal convolutional network in this embodiment.

[0042] Figure 8 This is a confusion matrix diagram of the slip detection model based on temporal convolutional networks in this embodiment for four categories of slip events on the test set.

[0043] The labels in the diagram are as follows: 1-Upper clamp, 2-Lower clamp, 3-Flexible vibration transmission layer, 4-Pierre resistance sliding sensor, 5-Fixing pin, 6-PDMS protective shell, 7-Positioning hole, 8-Pierre resistance cantilever beam, 9-Contact point, 10-Positioning post, 11-Vibration signal amplification unit. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other.

[0045] Example 1: As Figures 1-8As shown, a method for slip sensing in a minimally invasive surgical end effector based on a temporal convolutional network is disclosed. This method is used in a minimally invasive surgical end effector with slip sensing functionality. The end effector includes an upper clamp 1, a lower clamp 2, a flexible vibration transmission layer 3, a piezoresistive slip sensor 4, and a PDMS protective shell 6. One end of the upper clamp 1 and lower clamp 2 is hinged, while the other end is a free end. The side of the upper clamp 1 and lower clamp 2 that is close together serves as a clamping surface. The clamping surface has a first groove for mounting the PDMS protective shell 6. The PDMS protective shell 6 has a second groove for mounting the piezoresistive slip sensor 4 and the flexible vibration transmission layer 3, which are arranged vertically and close together. The opening of the second groove is located away from the first groove, and the piezoresistive slip sensor 4 located in the second groove is positioned relative to the flexible vibration transmission layer 3. Layer 3 is arranged near the first groove; the flexible vibration transmission layer 3 is provided with a contact 9 near the piezoresistive sliding sensor 4, and a vibration signal amplification unit 11 is provided on the side of the flexible vibration transmission layer 3 away from the piezoresistive sliding sensor 4; the piezoresistive sliding sensor 4 is provided with a piezoresistive cantilever beam 8 near the flexible vibration transmission layer 3. When the upper clamp 1 and lower clamp 2 clamp the object and cause the flexible vibration transmission layer 3 to be subjected to vibration generated by tissue slippage, the vibration is initially amplified by the vibration signal amplification unit 11 and transmitted to the piezoresistive cantilever beam 8, so as to realize the conversion of the resistance change corresponding to the piezoresistive cantilever beam 8 into a voltage signal; wherein, the piezoresistive sliding sensor 4 includes a substrate, and piezoresistive cantilever beams 8 are arranged on the substrate in a one-to-one correspondence with the contact 9. Specifically, the piezoresistive cantilever beam 8 is formed by integrating a piezoresistive resistor on the elastic cantilever beam by ion implantation.

[0046] Slip is detected using a slip detection model based on a temporal convolutional network, based on the voltage signal.

[0047] Furthermore, the upper clamp 1 and lower clamp 2 are positioned by a fixing pin 5 and driven by a steel wire rope to open and close around the fixing pin to complete the clamping action. Square grooves are formed on the clamping surfaces of the upper and lower clamps as first grooves for embedding the protective shell 6. The protective shell 6 has a second groove for embedding the piezoresistive sliding sensor 4 and the flexible vibration transmission layer 3, which are arranged vertically. The flexible vibration transmission layer 3 completely covers the piezoresistive sliding sensor 4 using a spin-coating process and is flush with the upper end face of the clamping surface of the end effector, used to transmit the vibration generated on the clamping surface during tissue slippage. Figure 4As shown, the vibration signal amplification unit on the flexible vibration transmission layer 3 can include various structural designs and arrangements, such as the "ridge" structure, the conventional cylindrical contact structure, and the hexagonal micro-pad structure resembling the foot of a tree frog. The flexible vibration transmission layer 3 is assembled and fixed with the positioning holes 7 on the sensor via the positioning posts 10, ensuring good contact between each contact 9 and the piezoresistive cantilever beam 8 below. The piezoresistive slip sensor 4 is embedded in the second groove of the integrally formed PDMS protective shell 6, and the second groove fits snugly with the piezoresistive slip sensor 4, thereby avoiding loosening, detachment, or structural interference. The PDMS protective shell 6 can be firmly fixed to the upper and lower clamps by an adhesive process. The flexible vibration transmission layer 3 is provided on the surface of the piezoresistive slip sensor 4. When the clamps stably hold the tissue, the flexible vibration transmission layer 3 forms contact with the tissue surface and can uniformly respond to frictional shear forces; when a slip event occurs, the small relative motion will be transmitted to the piezoresistive cantilever beam 8 of the piezoresistive slip sensor 4 in the form of local vibration. The entire piezoresistive sliding sensor 4 is encapsulated and sealed by a PDMS protective shell 6. This solution offers the following advantages: strong sealing and environmental adaptability; PDMS is a high-molecular-weight elastic material with excellent resistance to moisture and biocompatibility. High flexibility and buffering performance; the encapsulated sliding sensor structure possesses certain buffering characteristics under stress, absorbing excess mechanical impact and protecting the sensor element from damage. Signal integrity is protected; the excellent damping performance of PDMS filters background noise and improves the transmission efficiency of vibration signals between the flexible vibration transmission layer and the piezoresistor, ensuring that weak sliding signals are effectively amplified and detected.

[0048] Furthermore, the flexible vibration transmission layer 3 is made of polydimethylsiloxane (PDMS), with a typical substrate-to-crosslinking agent ratio of 10:1. It is prepared using a mold replication method, and the contact structure of the prepared surface includes, but is not limited to, the aforementioned "ridge" structure, conventional cylindrical contact structures, and hexagonal micro-pad structures resembling the distribution of a tree frog's foot. The PDMS protective shell 6 is prepared using a mold replication method (with a typical substrate-to-crosslinking agent ratio of 5:1).

[0049] As can be seen from the above technical solution, the flexible vibration transmission layer 3 in this invention is disposed on the side of the piezoresistive sliding sensor 4 away from the first groove. It is made of low-modulus PDMS material, and its surface structure is a vibration amplification unit. This structure can be a micron-scale "ridge" structure, a regular cylindrical protrusion structure, or a hexagonal micro-pad structure simulating the foot of a tree frog. These protrusions are designed to enhance microscopic contact with the tissue surface, so that the tiny mechanical vibrations generated by tissue sliding can be effectively amplified and transmitted to the piezoresistive element below. By guiding and amplifying the local vibration amplitude through structural geometry, early identification and intensity enhancement of the sliding signal can be achieved, thereby improving the sensitivity and response accuracy of the sensor system. The above-mentioned flexible vibration transmission layer is made of low-hardness PDMS material, and the mass ratio of substrate to crosslinking agent is 10:1, which can obtain a low Young's modulus. This gives the structure good flexibility and deformation ability when under force contact, ensuring that the protrusion structure will not cause damage or irritation when in contact with tissue, while also possessing good mechanical adaptability. In contrast, the outer PDMS protective shell 6 of the overall device is made of a high-hardness PDMS material (typically, the ratio of substrate to crosslinking agent is 5:1). It primarily supports the sliding sensor body and protects the sensor elements and flexible vibration layer from external mechanical impacts and environmental interference. This shell structure has higher mechanical strength and structural stability, ensuring stable engagement and durable encapsulation of the entire sensing unit during clamping, while also providing a certain degree of sealing against dust and liquid penetration.

[0050] Furthermore, since the resistance change of the varistor output is very small, a Wheatstone bridge and an amplifier circuit are needed to convert the resistance change into a larger voltage change. In this scheme, the eight varistors are arranged in an eight-half-bridge array. The output of each bridge is sent to an amplifier circuit, which uses an instrumentation amplifier (INA332). Finally, the data is passed through a digital-to-analog converter (ADC) and connected to a host computer for data storage. The data acquisition frequency is 1kHz. Figure 6 This is a Wheatstone bridge diagram for a single varistor; the input voltage of the Wheatstone bridge is 5V, and it needs to be calibrated before use. The value is 0V, and the calibration conditions are as follows: ; The piezoresistive resistor on the piezoresistive cantilever beam has a resistance that varies with the strain on the cantilever beam and serves as the sensing arm in the bridge circuit. To fix the precision resistor, a bridge reference arm is formed.

[0051] Due to the moist and compliant nature of human tissues during minimally invasive surgery, traditional single-threshold or shallow feature extraction methods are insufficient to meet the complex and variable stress environment. Therefore, this invention constructs a slip detection model based on a temporal convolutional network. This model can perform deep feature extraction, feature recalibration, and temporal fusion classification on multi-channel voltage time-series signals acquired by a piezoresistive array, thereby determining whether slip risk exists and its direction during clamping. The voltage signals output from the sensors are first synchronously sampled at a sampling frequency of 1kHz. To suppress background noise and highlight the transient high-frequency characteristics accompanying slip events, this embodiment uses stationary wavelet transform (SWT) to perform time-frequency decomposition on each channel voltage signal, extracting its detail coefficients as feature representations for subsequent modeling. The feature sequence after the above processing is formatted with a length of... Sliding time window of sampling points ( Preferably 100~400, in this embodiment The data is segmented according to a step size S (S is preferably 10~100, S=25 in this embodiment). Each time-series window can be described as a two-dimensional tensor with dimension 1. (in For time points, (where is the number of channels), and this tensor is used as the input to a slip detection model based on a temporal convolutional network.

[0052] refer to Figure 1 The slip detection model based on temporal convolutional networks includes:

[0053] Input layer: 8-channel voltage timing window data are acquired based on the piezoresistive sliding sensor 4. The input tensor is... Among them, the piezoresistive cantilever beams 8 on the piezoresistive sliding sensor 4 are arranged in a crisscross pattern. In this embodiment, there are specifically 8 beams, thereby obtaining 8 channels of voltage timing window data. In this embodiment, the time point number is used. The preferred value is 100-400 sampling points, corresponding to a time span of 100ms-400ms at a sampling frequency of 1kHz. .

[0054] Channel Attention Module: Because the signal-to-noise ratio and feature importance of each channel signal differ when the eight piezoresistors are clamping tissues of different shapes and under different stress states, the channel attention module first... The input sequence is compressed into a global average pooling (GAP) over the time dimension. The channel descriptors are then processed through two fully connected layers (the first layer reduces the dimension from 8 to 4 and performs ReLU activation, the second layer increases the dimension from 4 back to 8 and performs Sigmoid activation) to learn the non-linear dependencies between channels, generating eight channel weight coefficients between 0 and 1; finally, these eight weight coefficients are compared with the original... The input tensor is multiplied element-wise to achieve adaptive enhancement of key slip feature channels and suppression of noise channels.

[0055] Temporal Convolutional Residual Stack: The feature sequence output from the channel attention module is input to the temporal convolutional residual stack. The temporal convolutional residual stack consists of n concatenated residual blocks, where n is preferably 4 to 8, and the inflation factor of each residual block is... It increases exponentially; in this embodiment, the number of residual blocks n=6, and the inflation factor sequence The number of output channels is Each residual block contains: features for fusing cross-channel characteristics. Channel hybrid convolutional layers and convolutional kernels are (This embodiment) The expansion factor is The model employs dilated causal convolutional layers; weight normalization and ReLU activation are applied after both convolutional layers. These dilated causal convolutional layers allow the model to significantly expand its receptive field without losing temporal resolution or leaking future information, accurately capturing the long-term dynamic evolution features of tissue slippage in its early stages. In this embodiment, the overall receptive field is 253, which can completely cover a length of... The time window is defined; to prevent the gradient vanishing problem in deep networks, the input and output branches of each residual block are fused element-wise through skip connections; finally, the output shape of the temporal convolution residual stack is... Advanced temporal feature maps.

[0056] Output layer: To eliminate the influence of relative positional shifts in the slip features within the time window, the output layer is first... The advanced temporal feature map is subjected to global average pooling (GAP) along the temporal dimension to obtain a global feature vector of length 64. This global feature vector is then fed into a two-stage fully connected network: the first fully connected layer maps the 64-dimensional global feature vector to an H-dimensional hidden representation (H is preferably 16~64, H=32 in this embodiment), is activated by ReLU and supplemented by Dropout regularization (preferably a dropout probability of 0.1~0.3, 0.15 in this embodiment); the second fully connected layer maps the H-dimensional hidden representation to a K-dimensional output, is activated by Softmax, and outputs a K-dimensional probability vector. ,in Corresponding to the "no slip" category, These correspond to several slip direction subcategories. In this embodiment, K=4, corresponding to "no slip", "slip along the +X direction", "slip along the +Y direction", and "slip along the -Y direction" (e.g., ...). Figure 3 (b) Taking the center point of the positioning hole as the origin, the vertical direction as the X direction, the top-down direction as the positive direction, the horizontal direction as the Y direction, and the left-to-right direction as the positive direction); It should be noted that, considering that the main operation in the operation is to lift and pull the tissue, the main component of the traction force points to the free end of the clamp, i.e., +X. The operation of applying traction force to the hinge end is very rare in clinical practice and does not conform to the logic of surgical operation. The positive and negative Y directions correspond to the direction of lateral displacement of the tissue when it is pulled laterally. The two directions need to be identified separately in order to provide the correct force adjustment direction feedback for the end effector.

[0057] In the precise operation of minimally invasive surgery, if hard judgment is performed frame-by-frame independently based solely on time windows, the slight fluctuations in the model's output probability near the class boundary can easily lead to high-frequency false triggering and mechanical oscillations (jitter) in the end effector, seriously threatening surgical safety. Therefore, a two-level judgment mechanism is introduced for the K-dimensional probability vector: the first level uses a dual-threshold hysteresis judgment based on the slip probability to determine the "slip / non-slip" state of the end effector; the second level is used to identify the slip direction based on the direction probability when the first level's judgment result is "slip" (no execution occurs when the first level's judgment result is "non-slip"). Specifically:

[0058] Level 1: Dual-threshold hysteresis judgment based on sliding synthesis probability. Define the sliding synthesis probability. Set a high threshold With low threshold ,in Both can take values ​​in the interval (0,1), with the latter being preferred. Take 0.6~0.9, The value is 0.2~0.5, and is preferred in this embodiment. :

[0059] 1) In the minimally invasive surgical end effector with slip sensing function, which is currently in a "non-slip state", and When this occurs, it is determined that slippage has occurred, and the state of the end effector is switched to "slippage state";

[0060] 2) The minimally invasive surgical end effector with sliding sensing function is currently in a "sliding state," and When the slippage risk is determined to be eliminated, the state of the end effector is switched to "non-slippage state";

[0061] 3) When When the end effector does not switch states, it maintains the judgment result of the previous moment, thereby introducing the Schmitt trigger characteristic to suppress frequent oscillations near the threshold caused by small signal fluctuations.

[0062] Level 2: Slip direction recognition based on direction factor probabilities. When the end effector is in a "slip state", the direction factors in the same probability vector are identified. Take argmax and output the category of the direction sub-probability corresponding to the maximum probability as the slip direction within the current time window. This direction information can be used as a feedback signal to guide the end effector to adjust the clamping force direction or attitude, thus forming a comprehensive perception of tissue slip together with the first-level "slip presence or absence" judgment.

[0063] The slip detection model based on the temporal convolutional network is trained using supervised learning. The specific data construction process is as follows: In the experiment, the end effector clamp is controlled to clamp the tissue and apply traction force to the clamped tissue in different directions to induce physical displacement. Throughout this process, the system continuously samples and records the multi-channel voltage time sequence output by the slip sensor at a sampling frequency of 1kHz, which serves as the input feature X of the slip detection model based on the temporal convolutional network. The real labels required for supervised learning are obtained as follows: A stationary wavelet transform (SWT) is applied to the original voltage signals of each channel to extract their detail coefficients (the detail coefficients are used to characterize the high-frequency transient components caused by frictional vibration and abrupt changes in contact state during the slip process). These coefficients are then sequentially rectified and processed by root mean square (RMS) envelopes (where rectification converts the positive and negative fluctuations in the detail coefficients into unipolar amplitude signals, and the RMS envelope reflects the vibration intensity of the signal within a local time range, thus obtaining a slip intensity envelope curve that smoothly changes over time). Subsequently, a dual-threshold determination mechanism is used to identify the slip state of this envelope curve: when the envelope value rises from a steady state below the lower threshold and exceeds the upper threshold, the corresponding moment is determined to be the start of slip; when the envelope value falls from a slip state above the upper threshold and falls below the lower threshold, the corresponding moment is determined to be the end of slip. This transforms each channel signal into a binary time series of "slip / non-slip". After a logical OR operation, each channel's binary sequence can automatically determine whether it is in a slip state and its start and end times. The slip direction is labeled according to the known direction of the applied traction force in the experiment: the samples are labeled according to the following categories (taking K=4 as an example): category 0 corresponds to no slip, category 1 corresponds to slip along the +X direction, category 2 corresponds to slip along the +Y direction, and category 3 corresponds to slip along the -Y direction.

[0064] After time alignment is completed, the data is divided into sliding windows with a step size S. S is preferably 10 to 100 sampling points, and in this embodiment, S=25. There is an overlap of L-S sampling points between adjacent time windows to ensure that the moment of sliding can be completely covered by at least one time window. When the same window contains multiple category labels, the one with the larger sliding degree is taken as the final label of that window.

[0065] Considering that the steady state occupies the majority of the time in actual clamping operations, there is a class imbalance problem between the no-slip class samples and the samples in each slip direction class. To avoid the network tending to predict "no-slip", this invention uses multi-class cross-entropy with class weights and label smoothing as the loss function:

[0066] ;

[0067] in: The first output of the model Class probability; For the first The class weight coefficient is inversely proportional to the number of samples of that class in the training set. For the target distribution after label smoothing, the smoothing coefficient used in the smoothing process is... The preferred value is 0 to 0.1. In this embodiment... =0.05. During the training phase, the AdamW optimizer, combined with a cosine annealing learning rate scheduling strategy, is used for end-to-end training.

[0068] Through the optimized network topology and training strategy described above, this invention can effectively decode the multi-dimensional micro-vibration signals transmitted by the sensor within a millisecond-level time window, thereby achieving highly robust online monitoring of tissue clamping status and identification of slip direction.

[0069] The working principle of this invention is:

[0070] S1. Clamping contact and micro-vibration amplification:

[0071] When the clamp contacts and stably holds the tissue, the flexible vibration transmission layer 3 adheres to the tissue surface. If relative slippage occurs, the vibration signal amplification unit on the surface of the flexible vibration transmission layer amplifies the slippage-induced micro-vibrations and directionally transmits them to the contact point 9 below, causing the piezoresistive cantilever beam 8 at the corresponding position to enter a dynamic strain state. Simultaneously, the material and geometry of this transmission layer, combined with the overall PDMS protective shell 6, utilizes the unique viscoelastic damping properties of PDMS polymer material to effectively filter and isolate high-frequency mechanical background noise transmitted from the surgical robotic arm chassis, thereby suppressing environmental interference at its physical source and significantly improving the detectability of weak slippage signals.

[0072] S2. Piezoresistive effect and signal amplification:

[0073] Eight distributed varistors (cantilever beam type) are positioned at different locations. Upon excitation by micro-vibration, each varistor generates a small strain, resulting in subtle changes in its resistance. This multi-point distribution provides orientation information and redundancy, enhancing the robustness of slip detection. Each varistor is connected to a corresponding Wheatstone bridge measurement branch; the minute changes in resistance cause bridge imbalance, generating millivolt-level differential voltage signals, which are amplified by an instrumentation amplifier to obtain eight amplified voltages. .

[0074] S3, Signal Sampling and Preprocessing

[0075] Eight voltages are synchronously sampled at a frequency of 1kHz. The time window with a length of L sampling points (L=225 in this embodiment) is divided by a step size S (S=25 in this embodiment) to form a time tensor of shape L×8, which is then fed as a single sample into the slip detection model based on a time-series convolutional network.

[0076] S4, will The temporal tensor is fed into a slip detection model based on a temporal convolutional network. The model first adaptively weights each sensing channel using a channel attention module, then sequentially passes through a temporal convolutional residual stack composed of multi-scale dilated causal convolutions and residual connections, a global average pooling layer, and a two-stage fully connected network. Finally, it is activated by Softmax to output a K-dimensional probability vector p (in this embodiment, K=4, corresponding to four categories: "no slip," "+X direction slip," "+Y direction slip," and "-Y direction slip"). Subsequently, a two-stage judgment is performed on this probability vector: the first stage uses the slip comprehensive probability... As a judgment variable, the "slip / non-slip" state of the end effector is determined by a double-threshold hysteresis mechanism with Schmitt triggering characteristics; in the second stage, in the "slip" state, the opposing direction probability is... The argmax parameter is used to obtain the slip direction within the current time window. The first-level judgment result is used to trigger the anti-slip intervention of the end effector, and the second-level judgment result is used to guide the end effector to adjust the clamping posture.

[0077] S5. Preliminary simulation verification of this embodiment:

[0078] To verify the feasibility of the slip detection model based on temporal convolutional networks described in this invention under 8-channel input, before the completion of the 8-channel hardware prototype of the end effector, an 8-channel simulation dataset was constructed based on the measured voltage timing data of a four-channel piezoresistive slip sensor, combined with the sensor's spatial layout. Specifically, the first four channels were directly retained from the four-channel measured data; based on the geometric positional relationship between each newly added corner point in the 8-cantilever beam array and the adjacent known channels, interpolation was performed using the channel mean and spatial gradient, and a small random perturbation was superimposed to generate the last four channels, to approximately simulate the response characteristics of the 8-cantilever beam array in different spatial orientations. The generated 8-channel samples were divided into a training set, a validation set, and a test set in a 7:1.5:1.5 ratio for end-to-end training of the slip detection model based on temporal convolutional networks. Figure 7 The curves showing the change of the loss function between the training set and the validation set during the training process are shown. Figure 8 The confusion matrices for the four categories (no slip, +X slip, +Y slip, and -Y slip) on the test set are shown. The simulation results above are used to illustrate the feasibility of the technical solution of this invention. Specific performance indicators can be further optimized as the 8-channel hardware prototype is deployed and the scale of real samples increases.

[0079] S6, adaptable to different surgical scenarios

[0080] This invention allows for the adjustment of the size of the end effector and sensor as needed, optimization of the structure of the piezoresistor (e.g., using a thin film type), and the material and thickness of the flexible vibration transmission layer to achieve optimal results, thus making it suitable for end effectors in different surgical scenarios (such as laparoscopic surgery).

[0081] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for sensing the sliding motion of an end effector in minimally invasive surgery based on temporal convolutional networks, characterized in that, This is a minimally invasive surgical end effector with a slip sensing function. The minimally invasive surgical end effector with a slip sensing function includes an upper clamp (1), a lower clamp (2), a flexible vibration transmission layer (3), a piezoresistive slip sensor (4), and a protective shell (6). One end of the upper clamp (1) and the lower clamp (2) are hinged, and the other end of the upper clamp (1) and the lower clamp (2) is a free end. The side of the upper clamp (1) and the lower clamp (2) that are close to each other serves as a clamping surface. The clamping surface has a first groove for installing the protective shell (6). The protective shell (6) has a second groove for installing the piezoresistive slip sensor (4) and the flexible vibration transmission layer (3) that are arranged in an upper and lower fit. The opening of the second groove is opened on the side away from the first groove, and the piezoresistive slip sensor (4) located in the second groove is relative to the first groove. The flexible vibration transmission layer (3) is arranged near the first groove; the flexible vibration transmission layer (3) is provided with a contact (9) near the piezoresistive slip sensor (4), and the flexible vibration transmission layer (3) is provided with a vibration signal amplification unit (11) away from the piezoresistive slip sensor (4); the piezoresistive slip sensor (4) is provided with a piezoresistive cantilever beam (8) near the flexible vibration transmission layer (3). When the upper clamp (1) and lower clamp (2) clamp the object and cause the flexible vibration transmission layer (3) to be subjected to vibration caused by tissue slippage, the vibration is initially amplified by the vibration signal amplification unit (11) and transmitted to the piezoresistive cantilever beam (8) so as to realize the conversion of the resistance change of the piezoresistive cantilever beam (8) into a voltage signal; the slippage is detected by a slippage detection model based on a time-series convolutional network based on the voltage signal.

2. The method for sliding sensing of the end effector in minimally invasive surgery based on temporal convolutional networks according to claim 1, characterized in that, The slip detection model based on temporal convolutional networks includes: Input layer: Based on the piezoresistive sliding sensor (4), 8-channel voltage timing window data are obtained, and the input tensor is... Among them, the eight piezoresistive cantilever beams (8) on the piezoresistive sliding sensor (4) are arranged in a crisscross pattern; Channel attention module: First, The input sequence is compressed into a global average pooling operation over time. The channel descriptors are then processed through two fully connected layers to learn the non-linear dependencies between channels, generating eight channel weight coefficients between 0 and 1. Finally, these eight weight coefficients are compared with the original... The input tensor is multiplied element-wise to obtain the output of the channel attention module; Temporal Convolutional Residual Stack: The feature sequence output from the channel attention module is input to the temporal convolutional residual stack; the temporal convolutional residual stack consists of n concatenated residual blocks, and the inflation factor of each residual block is... It increases exponentially; each residual block contains: for fusing cross-channel features. Channel hybrid convolutional layers and convolutional kernels are The expansion factor is The dilated causal convolutional layer; the input and output branches of each residual block are fused element-wise through skip connections; finally, the output shape of the temporal convolutional residual stack is... Advanced temporal feature maps; Output layer: First, for The advanced temporal feature map is subjected to global average pooling along the temporal dimension to obtain a global feature vector of length 64. This global feature vector is then fed into a two-stage fully connected network: the first fully connected layer maps the 64-dimensional global feature vector to an H-dimensional hidden representation, which is then activated by ReLU and supplemented with Dropout regularization; the second fully connected layer maps the H-dimensional hidden representation to a K-dimensional output, which is then activated by Softmax to output a K-dimensional probability vector. ,in Corresponding to the "no slip" category, Each corresponds to a different sliding direction subcategory.

3. The method for slip sensing of a minimally invasive surgical end effector based on a temporal convolutional network according to claim 2, characterized in that, A two-level judgment mechanism is introduced for the K-dimensional probability vector: the first level is used to determine the "slip / non-slip" state of the end effector by using a double threshold hysteresis judgment based on the slip comprehensive probability; The second level is used to identify the slip direction based on the direction probability when the first level judgment result is "slip".

4. The method for sliding sensing of the end effector in minimally invasive surgery based on temporal convolutional networks according to claim 3, characterized in that, The dual-threshold hysteresis judgment based on the sliding synthesis probability is specifically as follows: Define the slip probability Set a high threshold With low threshold ,in Both can take values ​​in the interval (0,1): 1) In the minimally invasive surgical end effector with slip sensing function, it is currently in a "non-slip state," and At this time, the state of the end effector is switched to "slip state"; 2) The minimally invasive surgical end effector with sliding sensing function is currently in a "sliding state," and When this happens, the state of the end effector is switched to "non-slip state"; 3) When At this time, the end effector does not switch states and maintains the judgment result from the previous moment.

5. The method for sliding sensing of the end effector in minimally invasive surgery based on temporal convolutional networks according to claim 3, characterized in that, The slip direction recognition based on direction sub-probability specifically involves: when the end effector is in a "slip state", identifying the direction sub-probabilities in the same probability vector. Take argmax and output the category of the direction sub-probability corresponding to the maximum probability as the sliding direction within the current time window.

6. The method for sliding sensing of the end effector in minimally invasive surgery based on temporal convolutional networks according to claim 1, characterized in that, The slip detection model based on temporal convolutional networks uses multi-class cross-entropy with class weights and label smoothing as the loss function. : ； in: The first output of the model Class probability; For the first Class weight coefficient; This represents the target distribution after label smoothing.

Images