Flexible machine finger instrument control method and system based on visual guidance

Abstract

The invention relates to a flexible machine finger instrument control method and system based on visual guidance, and relates to the technical field of robot control. Performing catheter tail end feature point segmentation of the lightweight convolutional neural network, and calling a PnP algorithm to solve a rotation variable and a translation variable under a camera coordinate system; three-dimensional coordinate system positioning information is called to serve as target track input, and contact type pressure sensing waveform data of the flexible machine finger and the catheter wall are output; synchronously loading original waveform data of a pressure sensor in the conduit, executing wavelet packet decomposition to extract frequency band energy distribution of dual-channel signals, and inputting detail coefficient components into a support vector regression machine for dimensionality reduction so as to obtain viscosity change rate and flow resistance gradient data of fluid in the conduit; and outputting a dredging control sequence of the vibration frequency, the amplitude and the action duration of the flexible machine finger. According to the invention, cooperative control of intelligent pre-judgment of the blockage risk and active dredging of the flexible machine finger is realized.

Description

Technical Field

[0001] This invention relates to the field of robot control technology, and more specifically, to a vision-guided flexible robot finger instrument control method and system. Background Technology

[0002] Current intelligent conduit condition monitoring technology mainly relies on rigid sensor arrays and threshold triggering mechanisms. It collects fluid dynamics signals within the conduit using fixed pressure probes, activating a passive alarm when monitored parameters exceed preset safety ranges. However, it lacks active physical intervention capabilities and visual detection methods. This technology system generally lacks a visual intelligent perception layer, resulting in the inability to acquire spatial geometric information of the target area. This technical architecture has three inherent defects: First, rigid sensing elements are prone to mechanical stress concentration when in contact with flexible conduit walls, leading to structural damage to the conduit wall or distortion of the sensing signal. Second, existing systems can only achieve single-point signal monitoring and cannot obtain the precise spatial pose relationship between the end effector and the target area through visual servoing, resulting in a lack of spatial targeting in intervention actions. Third, signal processing mostly uses traditional frequency domain analysis or static threshold comparison, without integrating fluid constitutive equation inversion and multimodal sensing information, making early identification difficult in the initial stage of blockage.

[0003] To address these limitations, traditional solutions often add redundant sensors or expand the threshold tolerance range. However, such approaches only delay the alarm timing and fail to build a closed-loop control link from perception to execution. Especially in high-precision minimally invasive operation scenarios, existing technologies cannot achieve the coordinated control requirements of millimeter-level positioning accuracy and millisecond-level response speed, let alone meet the active intervention requirements of visual intelligent guidance. Summary of the Invention

[0004] The purpose of this invention is to provide a vision-guided flexible machine finger instrument control method and system to improve the aforementioned problems. To achieve the above objective, the technical solution adopted by this invention is as follows: In a first aspect, this application provides a vision-guided flexible machine finger instrument control method, comprising: Acquire real-time visual images of the indwelling catheter exit area, raw waveform data from the catheter internal pressure sensor, and initial signals from the multimodal sensor at the tip of the flexible robotic finger. Based on the acquired real-time visual images, a lightweight convolutional neural network is used to segment the feature points at the end of the catheter. The PnP algorithm is called to solve for the rotation and translation variables in the camera coordinate system. The rotation and translation variables are then transformed to the robot base coordinate system to generate the three-dimensional coordinate system positioning information of the end of the catheter. The three-dimensional coordinate system positioning information is called as the target trajectory input. The initial signal obtained is loaded and zero-point compensation and environmental interference correction are performed to obtain the calibrated sensor reference parameters. Based on the calibrated sensor reference parameters, the flexible robotic finger is driven to move and the end pressure response is collected. The contact pressure sensing waveform data between the flexible robotic finger and the duct wall is output. The system receives waveform data from a contact pressure sensor, synchronously loads the original waveform data from the pressure sensor inside the catheter, performs wavelet packet decomposition to extract the frequency band energy distribution of the dual-channel signal, obtains the dual-channel frequency band energy feature matrix, separates the approximate coefficient component and the detail coefficient component from the dual-channel frequency band energy feature matrix, substitutes the approximate coefficient component into the Navier-Stokes equation parameter identifier, and inputs the detail coefficient component into the support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter. The viscosity change rate and flow resistance gradient data of the fluid in the duct are used as inputs to the fuzzy inference system. The blockage level domain is set as mild, moderate and severe and a triangular membership function is configured. The defuzzified blockage level value is mapped to the vibration parameter library of the machine finger, and the unblocking control sequence of the vibration frequency, amplitude and duration of the flexible machine finger is output.

[0005] Preferably, based on the acquired real-time visual image, a lightweight convolutional neural network is used to segment the feature points at the end of the catheter, and the PnP algorithm is called to solve for the rotation and translation variables in the camera coordinate system. The rotation and translation variables are then transformed to the robot base coordinate system to generate the three-dimensional coordinate system positioning information of the catheter end, including: The grayscale image is input into the encoder of the MobileNetV3 architecture to extract the depth feature map. The decoder uses transposed convolution to upsample to the original image resolution and outputs a binary mask image of the duct end contour. Sixteen Harris corner points are extracted around the centroid of the mask image as a feature point set. Each feature point contains horizontal and vertical pixel coordinates and response intensity values. The homogeneous coordinates of 16 feature point pixels are paired with the spatial coordinates in the corresponding three-dimensional measurement coordinate system to construct a collinearity equation system. The Levenberg-Marquardt iteration is used to minimize the reprojection error, and the rotation matrix and translation vector of the camera relative to the duct coordinate system are solved. The 4*4 homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system is obtained through hand-eye calibration. The transformation matrix is ​​multiplied by the three-dimensional coordinates of the feature points to obtain the duct end positioning coordinate vector in the robot base coordinate system.

[0006] Preferably, the step of calling the three-dimensional coordinate system positioning information as the target trajectory input, loading the acquired initial signal for zero-point compensation and environmental interference correction, obtaining calibrated sensor reference parameters, driving the flexible robotic finger to move based on the calibrated sensor reference parameters and acquiring the end-effector pressure response, and outputting contact pressure sensing waveform data between the flexible robotic finger and the duct wall, including: The reference deviation statistics are extracted from the reference signal vector of the multimodal sensor, the drift compensation is calculated, and the compensation is subtracted from the readings of each axis of the triaxial micro-force sensor to obtain the compensated force vector. The target position is determined by using the positioning coordinate vector at the end of the catheter. The vector composed of the rotation angles of each joint is solved by the forward kinematics equation of the flexible robotic finger. A smooth trajectory from the current position to the target position is generated by fifth-order polynomial interpolation. The trajectory execution period is set to 500 milliseconds. The dimension of the joint rotation angle vector is equal to the number of degrees of freedom of the flexible robotic finger. During trajectory execution, triaxial micro-force sensor data is continuously acquired at a frequency of 2000 Hz. The triaxial data is then synthesized into a contact pressure scalar, forming a contact pressure sensing waveform data vector that shows pressure changing over time.

[0007] Preferably, the receiving contact pressure sensing waveform data is synchronously loaded with the original waveform data from the pressure sensor inside the catheter, and wavelet packet decomposition is performed to extract the frequency band energy distribution of the dual-channel signal, obtaining a dual-channel frequency band energy feature matrix. Approximate coefficient components and detail coefficient components are separated from the dual-channel frequency band energy feature matrix. The approximate coefficient components are substituted into the Navier-Stokes equation parameter identifier, and the detail coefficient components are input into a support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter, including: The contact pressure sensor waveform data vector is used as the first channel, and the seventh column of the pressure waveform matrix is ​​used as the second channel to form a dual-channel signal. The dual-channel signal is decomposed into three layers of wavelet packets. The db4 wavelet basis is selected, and the energy values ​​of 8 frequency bands are extracted to form a frequency band energy feature matrix with a dimension of 2*8. Normalize each row of the frequency band energy feature matrix, sum the data in the first row to obtain the approximate coefficient components, and calculate the standard deviation of the data in columns 2 to 8 to obtain the detail coefficient components. The approximate coefficient components are input into a parameter identifier based on minimizing the residuals of the Navier-Stokes equations to solve for the viscosity change rate. The detailed coefficient components are input into a support vector regression machine with a radial basis kernel function to map and obtain the flow resistance gradient data.

[0008] Secondly, this application also provides a vision-guided flexible machine finger instrument control system, comprising: Acquisition module: used to acquire real-time visual images of the indwelling catheter exit area, raw waveform data from the catheter internal pressure sensor, and initial signals from the flexible robotic fingertip multimodal sensor; Segmentation generation module: Based on the acquired real-time visual image, it performs feature point segmentation of the catheter tip using a lightweight convolutional neural network, calls the PnP algorithm to solve for rotation and translation variables in the camera coordinate system, transforms the rotation and translation variables to the robot base coordinate system, and generates the three-dimensional coordinate system positioning information of the catheter tip. The output module is used to call the three-dimensional coordinate system positioning information as the target trajectory input, load the acquired initial signal for zero-point compensation and environmental interference correction, obtain the calibrated sensor reference parameters, drive the flexible robotic finger to move based on the calibrated sensor reference parameters and collect the end pressure response, and output the contact pressure sensing waveform data between the flexible robotic finger and the duct wall. Separation and Substitution Module: This module receives waveform data from contact pressure sensors, synchronously loads the original waveform data from the pressure sensor inside the catheter, performs wavelet packet decomposition to extract the frequency band energy distribution of the dual-channel signal, obtains the dual-channel frequency band energy feature matrix, separates the approximate coefficient component and the detail coefficient component from the dual-channel frequency band energy feature matrix, substitutes the approximate coefficient component into the Navier-Stokes equation parameter identifier, and inputs the detail coefficient component into the support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter. The mapping module is used to input the fluid viscosity change rate and flow resistance gradient data in the duct as fuzzy inference system input, set the blockage level domain as mild, moderate and severe and configure the triangular membership function, map the defuzzified blockage level value to the machine finger vibration parameter library, and output the unblocking control sequence of flexible machine finger vibration frequency, amplitude and duration.

[0009] Thirdly, this application also provides a vision-guided flexible machine finger instrument control device, comprising: Memory, used to store computer programs; A processor is used to implement the steps of the vision-guided flexible machine finger instrument control method when executing the computer program.

[0010] Fourthly, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described vision-guided flexible machine finger instrument control method.

[0011] The beneficial effects of this invention are as follows: This invention acquires real-time visual images of the catheter tip region by deploying an industrial camera and combining it with visual inspection methods. It then calls the MobileNetV3 lightweight convolutional neural network to perform pixel-level contour segmentation and Harris corner extraction. Combined with the PnP algorithm and hand-eye calibration homogeneous transformation matrix, it calculates the catheter's three-dimensional pose, achieving sub-millimeter-level spatial positioning accuracy and a 30Hz real-time response speed. This avoids the stress concentration and structural damage to the tube wall caused by mechanical contact with traditional rigid probes. By integrating a flexible robotic finger with multimodal sensors, it performs contact pressure signal acquisition along a planned trajectory and uses a polynomial drift compensation model to calibrate the reference parameters, significantly improving the signal-to-noise ratio of multimodal signals and system stability, thus achieving non-invasive dynamic monitoring.

[0012] This invention constructs a dual-channel signal by combining the contact pressure sensing waveform and the pipe pressure waveform. It then performs three-layer wavelet packet decomposition to extract an 8-band energy feature matrix, separates the approximation coefficient components and detail coefficient components, and substitutes them into the Navier-Stokes equation residual minimization identifier to invert the fluid viscosity change rate. Finally, it inputs the radial basis function support vector regression machine to map the flow resistance gradient. This invention overcomes the limitation of traditional threshold triggering, which can only alarm in the later stages of blockage. It can identify blockage risk in the early stage when the viscosity change rate is on the order of 0.1 mPa·s per minute, and extends the early identification time window to more than 30 minutes, forming a complete visual intelligent perception and control closed loop.

[0013] This invention constructs a dual-input single-output fuzzy inference system, using viscosity change rate and resistance gradient as input variables. It configures a triangular membership function and centroid method to defuzzify the system, mapping the clear value of the blockage level to a vibration parameter library to generate frequency, amplitude, and duration control sequences. This forms a complete closed-loop control link from visual intelligent perception, multimodal fusion, fluid parameter inversion to active intervention, realizing the coordinated control of intelligent prediction of blockage risk and active unblocking by flexible machines.

[0014] Other features and advantages of the invention will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing embodiments of the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1This is a schematic diagram of the visual-guided flexible machine finger instrument control method described in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the vision-guided flexible machine finger instrument control system described in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the vision-guided flexible machine finger instrument control device described in an embodiment of the present invention.

[0017] In the diagram: 701, Acquisition module; 702, Segmentation generation module; 703, Call output module; 704, Separation and substitution module; 705, Setting mapping module; 800, Vision-guided flexible machine finger instrument control device; 801, Processor; 802, Memory; 803, Multimedia component; 804, I / O interface; 805, Communication component. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0019] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0020] Example 1: This embodiment provides a vision-guided flexible machine finger instrument control method.

[0021] See Figure 1 The figure shows that the method includes steps S100, S200, S300, S400 and S500.

[0022] S100: Acquire real-time visual images of the indwelling catheter outlet area, raw waveform data from the catheter internal pressure sensor, and initial signals from the flexible machine fingertip multimodal sensor.

[0023] It is understood that step S100 includes S101, S102, and S103, wherein: S101. A high-resolution industrial camera fixed to the side of the conduit mounting bracket acquires a 768*1024 pixel grayscale image, including the conduit's end outlet, at a rate of 30 frames per second. Simultaneously, the camera's intrinsic parameter calibration plate is triggered to triangulate the marker points on the outer wall of the conduit, establishing a three-dimensional measurement coordinate system with the conduit inlet as the origin. The projected coordinates of the marker points in the image coordinate system satisfy the formula: In the formula, This represents the two-dimensional pixel coordinate vector corresponding to the mark point. This represents a 3x3 camera intrinsic parameter matrix. Represents the rotation matrix. Represents the translation vector. This represents the homogeneous coordinate vector of the mark point in the three-dimensional measurement coordinate system; S102. Seven miniature fiber Bragg grating pressure sensors are embedded at 5 mm intervals along the length of the conduit to form a sensing array. The sampling frequency is set to 1000 Hz to continuously collect 24 hours of pressure pulsation data. The time series data of each sensor are aligned according to the timestamp to generate an N-row by 7-column pressure waveform matrix, where N is the number of sampling points and the j-th column of the pressure waveform matrix corresponds to the pressure time series of the j-th sensor. S103. A triaxial micro-force sensor is integrated into the silicone fingertip of the flexible machine finger. After power-on, a static signal of 100 milliseconds is continuously collected to calculate the zero-point offset. The zero-point offset is subtracted from the subsequent dynamic signal to obtain the corrected multimodal sensor reference signal vector. The reference signal vector contains the axial components of the triaxial force vector.

[0024] S200: Based on the acquired real-time visual image, perform feature point segmentation of the duct tip using a lightweight convolutional neural network, call the PnP algorithm to solve for the rotation and translation variables in the camera coordinate system, transform the rotation and translation variables to the robot base coordinate system, and generate the three-dimensional coordinate system positioning information of the duct tip.

[0025] It is understood that step S200 includes S201, S202, and S203, wherein: S201. Input the grayscale image into the encoder of the MobileNetV3 architecture to extract the depth feature map. The decoder uses transposed convolution to upsample to the original image resolution and outputs a binary mask image of the duct end contour. Extract 16 Harris corner points around the centroid of the mask image as a feature point set. Each feature point contains horizontal and vertical pixel coordinate values ​​and response intensity values. It should be noted that the MobileNetV3 encoder employs depthwise separable convolution and a linear bottleneck structure to compress the features of the duct mesh texture and the end opening region into a 128-dimensional depth feature vector. This feature map retains the original spatial dimensions. Figure 1 The resolution is reduced to 8 / 8 to decrease computational load. The decoder is not a simple stack of transposed convolutions; instead, skip connections are introduced to concatenate the multi-scale feature maps from layers 3, 7, and 11 of the encoder, then upsample to the original resolution. This design effectively alleviates the blurring problem at the duct edges. The output binary mask is not a single-channel image but a dual-channel probability map. The first channel identifies the main duct region, and the second channel specializes in identifying high-risk blockage areas at the terminal opening. Each pixel value represents the confidence probability that the region belongs to the target contour, and the confidence threshold is dynamically set to 0.73 to balance the detection rate and false alarm rate. Harris corner extraction faces unique challenges in this scenario: the reflection on the surface of the silicone conduit causes instability in the local gradient calculation. Therefore, in the actual implementation, anisotropic diffusion filtering is first applied to the mask image to enhance edge coherence before calculating the structure tensor matrix. Finally, the top 16 corner points are selected in descending order of response intensity within a 15-pixel neighborhood of the centroid. The data structure of each corner point contains 16-bit unsigned integer u and v coordinate values ​​and a 32-bit floating-point response intensity value. This intensity value is subsequently used for weighted least squares optimization in step S202.

[0026] In other words, the homogeneous coordinates of 16 feature point pixels are paired with their corresponding spatial coordinates in the 3D measurement coordinate system to construct a collinearity equation system. The Levenberg-Marquardt iteration is used to minimize the reprojection error, solving for the rotation matrix and translation vector of the camera relative to the duct coordinate system. A 4×4 homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system is obtained through hand-eye calibration. The transformation matrix is ​​then multiplied on the left by the 3D coordinates of the feature points to obtain the duct end-effector positioning coordinate vector in the robot base coordinate system. It should be noted that the hardware functions of the neural network need to be transformed into a visual perception processing device. This device should be further described in the specification as follows: Image acquisition unit: a physical entity containing an industrial camera, lens, light source, and fixed support; Lightweight convolution operation unit: a fixed-point quantitative inference module deployed on a GPU or NPU hardware acceleration chip; Feature point extraction unit: a Harris corner parallel computing kernel implemented based on a hardware description language.

[0027] S202. Pair the homogeneous coordinates of the 16 feature point pixels with their corresponding spatial coordinates in the 3D measurement coordinate system to construct a collinearity equation system. Use the Levenberg-Marquardt iteration to minimize the reprojection error and solve for the rotation matrix and translation vector of the camera relative to the duct coordinate system. The formula for calculating the reprojection error function is as follows: In the formula, This represents the sum of squared reprojection errors. This represents the pixel coordinates of the k-th feature point. Represents the projection function. Represents the rotation matrix. Represents the translation vector. Represents the three-dimensional coordinates of the k-th feature point; It should be noted that the construction of homogeneous coordinates for the 16 feature points is not a simple dimensionality increase, but rather a mapping of pixel coordinates (u,v) to a normalized image plane. The corresponding spatial coordinates in the 3D measurement coordinate system are derived from the Mark point triangulation network established in sub-step 1.1. Nearest neighbor search is used to match three Mark points within a 5mm radius for each feature point, and sub-millimeter spatial coordinates are calculated using barycentric coordinate interpolation. The initial values ​​for the Levenberg-Marquardt iteration are provided by a fast estimation using the EPnP algorithm. During the iteration, the Jacobian matrix is ​​used in analytical form rather than numerical difference. The condition number is calculated synchronously with each LM step size adjustment to avoid numerical instability caused by singular duct posture. A triple criterion is set for the iteration termination condition. The final output rotation matrix is ​​stored in compact form as an Euler angle vector, and the translation vector retains three decimal places of precision. Together, they constitute the 6-DOF pose parameters.

[0028] S203. Obtain the 4*4 homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system through hand-eye calibration. Multiply the transformation matrix by the three-dimensional coordinates of the feature points to obtain the positioning coordinate vector of the duct end in the robot base coordinate system. The formula for calculating the positioning coordinate vector is as follows: In the formula, Let be the homogeneous coordinates of the catheter tip in the robot's base coordinate system. This represents the transformation matrix from the camera to the robot's base coordinate system. This represents the pose matrix of the camera relative to the duct coordinate system. The homogeneous coordinate vector representing the feature point.

[0029] It should be noted that the accuracy of this step depends on the reliability of the hand-eye calibration. In actual clinical deployment, an "eye outside the hand" configuration is adopted. The calibration process uses a 3D printed ArUco calibration board. Twenty different poses are randomly generated within the robot's workspace, and camera images and robot end-effector poses are simultaneously acquired in each pose. After solving the rotation matrix using the Tsai-Lenz algorithm, the translation vector is obtained through least-squares fitting. This 4×4 matrix is ​​stored in column-major order. The left multiplication operation is implemented in the embedded GPU as a single-instruction multiple-data-stream parallel computation. The 3D coordinates of the feature points are regarded as homogeneous coordinates [x,y,z,1]^T. The matrix multiplication is split into four vector dot products and one addition. The batch conversion of 16 feature points is completed within 0.8 milliseconds using the CUDA core. The catheter end-effector positioning coordinate vector contains XYZ 3D coordinates and pose quaternions. The data structure uses 128-bit aligned contiguous memory blocks for easy direct transmission to the robot controller. In practical applications, it is necessary to dynamically compensate for the deformation error of the catheter itself. This model calculates the micron-level offset caused by gravity and blood flow impact in real time based on the catheter material characteristics parameters, ensuring that the positioning accuracy is maintained within ±0.15 mm, which meets the stringent requirements for the subsequent precise attachment of flexible machine fingers.

[0030] S300: Call the three-dimensional coordinate system positioning information as the target trajectory input, load the acquired initial signal for zero-point compensation and environmental interference correction, obtain the calibrated sensor reference parameters, drive the flexible robotic finger to move based on the calibrated sensor reference parameters and collect the end pressure response, and output the contact pressure sensing waveform data between the flexible robotic finger and the duct wall.

[0031] It is understood that step S300 includes S301, S302, and S303, wherein: S301. Extract the reference deviation statistics from the reference signal vector of the multimodal sensor, calculate the drift compensation amount, and subtract the compensation amount from the readings of each axis of the triaxial micro-force sensor to obtain the compensated force vector. The calculation formula for the compensation amount is as follows: In the formula, Indicates the amount of drift compensation. Represents the linear drift coefficient. This indicates the current benchmark deviation value. Indicates the standard reference deviation value. Indicates the second-order drift coefficient; S302. Using the positioning coordinate vector of the catheter end as the target position, solve the vector composed of the rotation angles of each joint through the forward kinematics equation of the flexible robotic finger, and use fifth-order polynomial interpolation to generate a smooth trajectory from the current position to the target position. The trajectory execution period is set to 500 milliseconds, and the dimension of the joint rotation angle vector is equal to the number of degrees of freedom of the flexible robotic finger. It should be noted that the forward kinematic equations of the flexible robotic finger are not based on traditional DH parameter modeling, but rather on an equivalent joint mapping model based on the piecewise constant curvature assumption: the pneumatically driven silicone finger is discretized into 5 virtual rigid segments, each 15 mm long, with the segments parameterized by the radius of curvature and bending angle, constructing a transition from the driving space (air pressure P1~P3) to the configuration space (curvature vector κ∈ℝ). 5 The mapping relationship between the target contact force (preset 0.8N safety threshold) and the operation space (end pose T∈SE(3)) is then established. When solving the joint rotation angle vector, the target contact force (preset 0.8N safety threshold) is first solved into the required air pressure through the constitutive equation of the airbag, and then the Jacobi pseudo-inverse iteration algorithm is used to solve the increment of each air pressure chamber. This process takes about 12ms in the embedded ARM Cortex-M7 processor to ensure real-time performance. The trajectory execution cycle is set to 500ms, which is the result of balancing the elastic response time constant of the duct (about 200ms) and the visual servo delay (about 100ms). During this cycle, the robot controller sends interpolation points at a frequency of 1kHz to ensure motion stability. The dimension of the joint angle vector is strictly equal to the number of degrees of freedom of the flexible finger, but in actual research, it is extended to 4 degrees of freedom to enhance redundancy, and the zero-space projection algorithm is introduced to optimize the joint force distribution and avoid airbag overload.

[0032] This step achieves a smooth transition from visual positioning to flexible execution. Through air pressure-curvature decoupling modeling and zero-space optimization, the impact acceleration of the robot finger at the moment of contact with the duct is less than 0.5g, the contact force overshoot is reduced, and the peak stress on the duct wall is reduced compared to the point-to-point motion of traditional rigid robotic arms.

[0033] S303. During trajectory execution, triaxial micro-force sensor data is continuously acquired at a frequency of 2000 Hz. The triaxial data is synthesized into a contact pressure scalar, forming a contact pressure sensing waveform data vector that changes pressure over time. The formula for calculating the contact pressure scalar is as follows: in, This represents a scalar value indicating contact pressure. , , These represent the axial components of the triaxial force vector. It should be noted that the contact pressure sensing waveform data vector is stored using a timestamp-pressure value binary structure. Each millisecond data point is appended with a 32-bit timestamp and a 32-bit pressure value, forming a compact 8-byte data packet, which is transmitted to the edge computing unit in real time via the UDP protocol. The study found that the radius of curvature of the conduit surface (typically 5-15 mm) significantly affects the calculation of the normal force. Therefore, curvature parameters need to be collected synchronously in the trajectory planning step S302 and corrected in real time using a Laplace mesh deformation algorithm.

[0034] This step constructs a high-fidelity, low-latency pressure sensing data stream. Through multi-axis fusion and dynamic compensation, the contact force measurement error is controlled within ±0.02N, providing a reliable data foundation for subsequent fluid parameter inversion. Compared with traditional single-axis pressure monitoring, the sensitivity of early blockage feature identification is significantly improved. The S400 receives waveform data from a contact pressure sensor, synchronously loads the original waveform data from the pressure sensor inside the catheter, performs wavelet packet decomposition to extract the frequency band energy distribution of the dual-channel signal, obtains the dual-channel frequency band energy feature matrix, separates the approximate coefficient component and the detail coefficient component from the dual-channel frequency band energy feature matrix, substitutes the approximate coefficient component into the Navier-Stokes equation parameter identifier, and inputs the detail coefficient component into a support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter.

[0035] It is understood that in this step, S400 includes S401, S402, and S403, wherein: S401. The contact pressure sensor waveform data vector is used as the first channel, and the seventh column of the pressure waveform matrix is ​​used as the second channel to form a dual-channel signal. The dual-channel signal is decomposed into three-level wavelet packets. The db4 wavelet basis is selected, and the energy values ​​of 8 frequency bands are extracted to form a frequency band energy feature matrix. The feature matrix has a dimension of 2*8. Understandably, the innovation of this step lies in unifying heterogeneous physical signals (pipe wall mechanical response and in-pipe fluid dynamics signal) into a single time-frequency analysis framework through wavelet packet decomposition, constructing a feature tensor that reflects the multi-scale characteristics of blockage evolution. The first channel selects a contact pressure sensing waveform data vector. This signal is essentially the micro-strain response of the duct wall under a contact force of 0.5~2N from a flexible mechanical finger, with a spectral distribution in the range of 10~500Hz, containing viscoelastic hysteresis information of the pipe wall material. The second channel uses the near-end data from the seventh column of the pressure waveform matrix, originating from a fiber optic grating sensor 5mm from the duct end. This location has the highest sensitivity to fluid viscosity changes, requiring a measured signal-to-noise ratio greater than 35dB. Its dominant frequency is distributed in the 0.1~50Hz range, reflecting macroscopic pressure pulsations in the fluid. The dual-channel synchronization time window is 2 seconds, with a sliding step of 0.5 seconds, ensuring the capture of quasi-steady-state flow characteristics.

[0036] Furthermore, the three-layer wavelet packet decomposition employs an asymmetric tree structure: the low-frequency subband is further bisected to the third layer, forming eight terminal nodes, while the high-frequency subband is only decomposed to the second layer. This design is based on the physical mechanism of catheter occlusion—thrombus accumulation dominates low-frequency energy migration (0~20Hz), while transient events such as microbubble rupture and platelet adhesion excite mid-to-high-frequency components (50~400Hz). The choice of the db4 wavelet basis has been experimentally verified: its support length of 7 sampling points precisely matches the effective signal length within the 0.5-second machine finger contact duration, and the vanishing moment order of 4 is sufficient to suppress the polynomial trend term in the pressure signal. Moreover, the generated 2×8 feature matrix E has its row vectors normalized to unit power, while the column vectors retain relative energy ratios, ensuring that the matrix elements possess both cross-channel comparability and intra-channel time-frequency localization characteristics, laying a structured data foundation for subsequent physical parameter inversion.

[0037] S402. Normalize each row of the frequency band energy feature matrix, sum the data in the first row to obtain the approximate coefficient components, and calculate the standard deviation of the data in columns 2 to 8 to obtain the detail coefficient components. It should be noted that before calculating the standard deviation in columns 2 to 8, median filtering must be performed to remove outliers: for each column of data, outlier points that deviate from the median by 3 times the absolute median difference must be removed using a 3-point window.

[0038] S403. Input the approximation coefficient components into the parameter identifier based on minimizing the residuals of the Navier-Stokes equations. The calculation formula for the optimized objective function is as follows: in, Describe the objective function of the residual integral. Represents the velocity field vector. Represents a time variable. Indicates fluid viscosity. Represents a pressure field. Indicates fluid density, The volume force vector is represented; the viscosity change rate is solved, and the detailed coefficient components are input into a support vector regression machine with a radial basis kernel function to obtain the flow resistance gradient data.

[0039] It should be noted that this step combines the approximation coefficient components with the residual minimization framework of the Navier-Stokes equation to construct a viscosity inversion device constrained by partial differential equations. The approximation coefficient components are embedded as prior information into the parameterized expression of the viscosity term in the Navier-Stokes equation, and the Navier-Stokes equation residuals are incorporated into the network training as a loss function. The network architecture consists of a 3-layer, 30-node fully connected layer with the activation function tanh. During training, the approximation coefficient components and detail coefficient components are input together, simultaneously predicting the viscosity change rate and drag gradient, achieving end-to-end physical constraint learning. During deployment, quantization-aware training is used to compress the network weights to INT8, reducing inference time to 5ms.

[0040] S500: The viscosity change rate and flow resistance gradient data of the fluid in the duct are used as inputs to the fuzzy inference system. The blockage level domain is set as mild, moderate and severe and a triangular membership function is configured. The defuzzified blockage level value is mapped to the vibration parameter library of the machine finger and the unblocking control sequence of the vibration frequency, amplitude and duration of the flexible machine finger is output.

[0041] It should be noted that the fuzzy inference system with dual inputs and single outputs of viscosity change rate and drag gradient is defined as having a viscosity change rate domain of 0 to 5 mPa·s per minute and a drag gradient domain of 0 to 10 kPa per meter, each divided into three triangular membership functions. The fuzzy rule base contains 9 IF-THEN rules. Understandably, in this step, the centroid method is used to defuzzify the fuzzy inference output and calculate the clear value of the congestion level. The calculation formula is as follows: Where L represents the congestion level clarity value, This represents the activation level of the i-th rule. This represents the center point of the output universe of discourse corresponding to the i-th rule, and the sharpness value is normalized to the interval between 0 and 1. In this step, a lookup table is established to correlate blockage level clarity values ​​with vibration parameters. When the clarity value is in the range of 0 to 0.3, it is mapped to a frequency of 20 Hz, an amplitude of 0.5 mm, and an action duration of 3 seconds; when the clarity value is in the range of 0.3 to 0.7, it is mapped to a frequency of 50 Hz, an amplitude of 1.2 mm, and an action duration of 5 seconds; when the clarity value is in the range of 0.7 to 1.0, it is mapped to a frequency of 80 Hz, an amplitude of 2.0 mm, and an action duration of 8 seconds. The unblocking control sequence vector is output, thereby generating an unblocking control sequence for the vibration frequency, amplitude, and action duration of the flexible machine finger.

[0042] In summary, this invention proposes an active pre-intervention control architecture that integrates lightweight visual servoing and flexible multimodal execution. It acquires real-time image data of the catheter tip region using an industrial camera, employs a lightweight convolutional neural network to perform pixel-level contour segmentation and extract Harris corner features, and uses the PnP algorithm to solve the homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system, achieving sub-millimeter-level positioning of the catheter tip's three-dimensional pose. The multimodal sensing reference is calibrated based on a zero-point drift compensation model, and contact pressure signal acquisition is performed along the planned trajectory. The contact pressure waveform and the intra-tube pressure waveform are combined to form a dual-channel signal. The system extracts the frequency band energy feature matrix by performing wavelet packet decomposition, separates the approximation coefficient component and the detail coefficient component, and inputs them into the Navier-Stokes equation residual minimization identifier and support vector regression machine, respectively, to invert the fluid viscosity change rate and flow resistance gradient. A dual-input fuzzy inference system is constructed, with the viscosity change rate and resistance gradient as input variables, to defuzzify the clear value of the blockage level and map it to the vibration parameter library, generating a dredging control sequence of flexible machine finger vibration frequency, amplitude and duration, forming a complete closed-loop control link from visual perception, force-thermal multimodal fusion, fluid parameter inversion to active intervention.

[0043] Example 2: like Figure 2 As shown, this embodiment provides a vision-guided flexible machine finger instrument control system. See [link to relevant documentation]. Figure 2 The system includes: Acquisition module 701: used to acquire real-time visual images of the indwelling catheter outlet area, raw waveform data of the catheter internal pressure sensor, and initial signals of the flexible machine fingertip multimodal sensor; Segmentation generation module 702: Based on the acquired real-time visual image, it performs feature point segmentation of the duct tip using a lightweight convolutional neural network, calls the PnP algorithm to solve for rotation and translation variables in the camera coordinate system, transforms the rotation and translation variables to the robot base coordinate system, and generates three-dimensional coordinate system positioning information of the duct tip. Call output module 703: Used to call the three-dimensional coordinate system positioning information as the target trajectory input, load the acquired initial signal for zero-point compensation and environmental interference correction, obtain the calibrated sensor reference parameters, drive the flexible robotic finger to move based on the calibrated sensor reference parameters and collect the end pressure response, and output the contact pressure sensing waveform data between the flexible robotic finger and the duct wall. Separation and Substitution Module 704: This module receives waveform data from a contact pressure sensor, synchronously loads the original waveform data from the pressure sensor inside the catheter, performs wavelet packet decomposition to extract the frequency band energy distribution of the dual-channel signal, obtains the dual-channel frequency band energy feature matrix, separates the approximate coefficient component and the detail coefficient component from the dual-channel frequency band energy feature matrix, substitutes the approximate coefficient component into the Navier-Stokes equation parameter identifier, and inputs the detail coefficient component into the support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter. The mapping module 705 is used to input the fluid viscosity change rate and flow resistance gradient data in the duct as fuzzy inference system input, set the blockage level domain as mild, moderate and severe and configure the triangular membership function, map the defuzzified blockage level value to the machine finger vibration parameter library, and output the unblocking control sequence of flexible machine finger vibration frequency, amplitude and duration.

[0044] The segmentation generation module 702 includes: The first extraction unit is used to extract depth feature maps from the encoder of the MobileNetV3 architecture, which inputs grayscale images. The decoder uses transposed convolution to upsample to the original image resolution and outputs a binary mask image of the duct end contour. Sixteen Harris corner points are extracted around the centroid of the mask image as a feature point set. Each feature point contains horizontal and vertical pixel coordinates and response intensity values. The construction unit is used to pair the homogeneous coordinates of 16 feature point pixels with their corresponding spatial coordinates in the 3D measurement coordinate system, construct a collinearity equation system, and use the Levenberg-Marquardt iteration to minimize the reprojection error. The rotation matrix and translation vector of the camera relative to the duct coordinate system are then solved. The formula for calculating the reprojection error function is as follows: In the formula, This represents the sum of squared reprojection errors. This represents the pixel coordinates of the k-th feature point. Represents the projection function. Represents the rotation matrix. Represents the translation vector. Represents the three-dimensional coordinates of the k-th feature point; The acquisition unit is used to obtain the 4*4 homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system through hand-eye calibration. Multiplying the transformation matrix by the 3D coordinates of the feature points on the left yields the positioning coordinate vector of the duct end in the robot base coordinate system. The formula for calculating the positioning coordinate vector is as follows: In the formula, Let be the homogeneous coordinates of the catheter tip in the robot's base coordinate system. This represents the transformation matrix from the camera to the robot's base coordinate system. This represents the pose matrix of the camera relative to the duct coordinate system. The homogeneous coordinate vector representing the feature point.

[0045] Specifically, the call output module 703 includes: The second extraction unit is used to extract the reference deviation statistics from the reference signal vector of the multimodal sensor, calculate the drift compensation amount, and subtract the compensation amount from the readings of each axis of the triaxial micro-force sensor to obtain the compensated force vector. The calculation formula for the compensation amount is as follows: In the formula, Indicates the amount of drift compensation. Represents the linear drift coefficient. This indicates the current benchmark deviation value. Indicates the standard reference deviation value. Indicates the second-order drift coefficient; The first solving unit is used to take the positioning coordinate vector of the catheter end as the target position, solve the vector composed of the rotation angles of each joint through the forward kinematic equation of the flexible robotic finger, and generate a smooth trajectory from the current position to the target position using fifth-order polynomial interpolation. The trajectory execution period is set to 500 milliseconds, and the dimension of the joint rotation angle vector is equal to the number of degrees of freedom of the flexible robotic finger. The data acquisition unit continuously acquires triaxial micro-force sensor data at a frequency of 2000 Hz during trajectory execution, synthesizes the triaxial data into a contact pressure scalar, and forms a contact pressure sensing waveform data vector that changes pressure over time. The calculation formula for the contact pressure scalar is as follows: in, This represents a scalar value indicating contact pressure. , , These represent the axial components of the three-axis force vector.

[0046] Specifically, the separation and substitution module 704 includes: The decomposition unit is used to take the contact pressure sensor waveform data vector as the first channel and the seventh column of the pressure waveform matrix as the second channel to form a dual-channel signal. The dual-channel signal is decomposed into three-level wavelet packets, the db4 wavelet basis is selected, and the energy values ​​of 8 frequency bands are extracted to form a frequency band energy feature matrix with a dimension of 2*8. Processing unit: Normalizes each row of the frequency band energy feature matrix, sums the data in the first row to obtain the approximate coefficient components, and calculates the standard deviation of the data in columns 2 to 8 to obtain the detail coefficient components; Optimization unit: Used to input the approximation coefficient components into the parameter identifier based on minimizing the residuals of the Navier-Stokes equations. The formula for calculating the objective function is as follows: in, Describe the objective function of the residual integral. Represents the velocity field vector. Represents a time variable. Indicates fluid viscosity. Represents a pressure field. Indicates fluid density, Represents the volume force vector; The second solution unit is used to solve for the viscosity change rate. It inputs the detailed coefficient components into a support vector regression machine with a radial basis kernel function and maps them to obtain the flow resistance gradient data.

[0047] It should be noted that the specific methods by which each module performs operations in the system described in the above embodiments have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0048] Example 3: Corresponding to the above method embodiments, this embodiment also provides a vision-guided flexible machine finger instrument control device. The vision-guided flexible machine finger instrument control device described below and the vision-guided flexible machine finger instrument control method described above can be referred to in correspondence.

[0049] Figure 3 This is a block diagram illustrating a vision-guided flexible machine finger instrument control device 800 according to an exemplary embodiment. Figure 3 As shown, the vision-guided flexible machine finger instrument control device 800 includes a processor 801 and a memory 802. The vision-guided flexible machine finger instrument control device 800 also includes one or more of a multimedia component 803, an I / O interface 804, and a communication component 805.

[0050] The processor 801 controls the overall operation of the vision-guided flexible machine finger instrument control device 800 to complete all or part of the steps in the aforementioned vision-guided flexible machine finger instrument control method. The memory 802 stores various types of data to support the operation of the vision-guided flexible machine finger instrument control device 800. This data may include, for example, instructions for any application or method operating on the vision-guided flexible machine finger instrument control device 800, as well as application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 802 can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The multimedia component 803 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in the memory 802 or transmitted via the communication component 805. The audio component also includes at least one speaker for outputting audio signals. I / O interface 804 provides an interface between processor 801 and other interface modules, such as keyboards, mice, or buttons. These buttons can be virtual or physical. Communication component 805 is used for wired or wireless communication between the vision-guided flexible machine instrument control device 800 and other devices. Wireless communication includes, for example, Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination thereof. Therefore, the corresponding communication component 805 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0051] In an exemplary embodiment, the vision-guided flexible machine finger instrument control device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to execute the vision-guided flexible machine finger instrument control method described above.

[0052] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the vision-guided flexible machine finger instrument control method described above. For example, the computer-readable storage medium may be the memory 802 including the program instructions described above, which may be executed by the processor 801 of the vision-guided flexible machine finger instrument control device 800 to complete the vision-guided flexible machine finger instrument control method described above.

[0053] Example 4: Corresponding to the above method embodiments, this embodiment also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the vision-guided flexible machine finger instrument control method described above.

[0054] A computer program is stored on a readable storage medium, and when executed by a processor, the computer program implements the steps of the vision-guided flexible machine finger instrument control method described in the above method embodiments.

[0055] Specifically, the readable storage medium can be a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other readable storage medium capable of storing program code.

[0056] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A vision-guided flexible machine finger instrument control method, characterized in that, include: Acquire real-time visual images of the indwelling catheter exit area, raw waveform data from the catheter internal pressure sensor, and initial signals from the multimodal sensor at the tip of the flexible robotic finger. Based on the acquired real-time visual images, a lightweight convolutional neural network is used to segment the feature points at the end of the catheter. The PnP algorithm is called to solve for the rotation and translation variables in the camera coordinate system. The rotation and translation variables are then transformed to the robot base coordinate system to generate the three-dimensional coordinate system positioning information of the end of the catheter. The three-dimensional coordinate system positioning information is called as the target trajectory input. The initial signal obtained is loaded and zero-point compensation and environmental interference correction are performed to obtain the calibrated sensor reference parameters. Based on the calibrated sensor reference parameters, the flexible robotic finger is driven to move and the end pressure response is collected. The contact pressure sensing waveform data between the flexible robotic finger and the duct wall is output. The system receives waveform data from a contact pressure sensor, synchronously loads the original waveform data from the pressure sensor inside the catheter, performs wavelet packet decomposition to extract the frequency band energy distribution of the dual-channel signal, obtains the dual-channel frequency band energy feature matrix, separates the approximate coefficient component and the detail coefficient component from the dual-channel frequency band energy feature matrix, substitutes the approximate coefficient component into the Navier-Stokes equation parameter identifier, and inputs the detail coefficient component into the support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter. The viscosity change rate and flow resistance gradient data of the fluid in the duct are used as inputs to the fuzzy inference system. The blockage level domain is set as mild, moderate and severe and a triangular membership function is configured. The defuzzified blockage level value is mapped to the vibration parameter library of the machine finger, and the unblocking control sequence of the vibration frequency, amplitude and duration of the flexible machine finger is output.

2. The vision-guided flexible machine finger instrument control method according to claim 1, characterized in that, Based on the acquired real-time visual images, a lightweight convolutional neural network is used for catheter tip feature point segmentation. The PnP algorithm is then called to solve for rotation and translation variables in the camera coordinate system. These rotation and translation variables are then transformed to the robot base coordinate system to generate three-dimensional coordinate system positioning information for the catheter tip, including: The grayscale image is input into the encoder of the MobileNetV3 architecture to extract the depth feature map. The decoder uses transposed convolution to upsample to the original image resolution and outputs a binary mask image of the duct end contour. Sixteen Harris corner points are extracted around the centroid of the mask image as a feature point set. Each feature point contains horizontal and vertical pixel coordinates and response intensity values. The homogeneous coordinates of the 16 feature point pixels are paired with their corresponding spatial coordinates in the 3D measurement coordinate system to construct a collinearity equation system. The Levenberg-Marquardt iteration is used to minimize the reprojection error, and the rotation matrix and translation vector of the camera relative to the duct coordinate system are solved. The formula for calculating the reprojection error function is as follows: In the formula, This represents the sum of squared reprojection errors. This represents the pixel coordinates of the k-th feature point. Represents the projection function. Represents the rotation matrix. Represents the translation vector. Represents the three-dimensional coordinates of the k-th feature point; The 4x4 homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system is obtained through hand-eye calibration. The transformation matrix is ​​then multiplied by the three-dimensional coordinates of the feature points to obtain the positioning coordinate vector of the duct tip in the robot base coordinate system. The formula for calculating the positioning coordinate vector is as follows: In the formula, Let be the homogeneous coordinates of the catheter tip in the robot's base coordinate system. This represents the transformation matrix from the camera to the robot's base coordinate system. This represents the pose matrix of the camera relative to the duct coordinate system. The homogeneous coordinate vector representing the feature point.

3. The vision-guided flexible machine finger instrument control method according to claim 1, characterized in that, The process involves calling the three-dimensional coordinate system positioning information as the target trajectory input, loading the acquired initial signal for zero-point compensation and environmental interference correction, obtaining calibrated sensor reference parameters, driving the flexible robotic finger to move based on these parameters, acquiring the end-effector pressure response, and outputting contact pressure sensing waveform data between the flexible robotic finger and the duct wall, including: The reference deviation statistics are extracted from the reference signal vector of the multimodal sensor, and the drift compensation is calculated. The compensation is then subtracted from the readings of each axis of the triaxial micro-force sensor to obtain the compensated force vector. The formula for calculating the compensation is as follows: In the formula, Indicates the amount of drift compensation. Represents the linear drift coefficient. This indicates the current benchmark deviation value. Indicates the standard reference deviation value. Indicates the second-order drift coefficient; The target position is determined by using the positioning coordinate vector at the end of the catheter. The vector composed of the rotation angles of each joint is solved by the forward kinematics equation of the flexible robotic finger. A smooth trajectory from the current position to the target position is generated by fifth-order polynomial interpolation. The trajectory execution period is set to 500 milliseconds. The dimension of the joint rotation angle vector is equal to the number of degrees of freedom of the flexible robotic finger. During trajectory execution, triaxial micro-force sensor data is continuously acquired at a frequency of 2000 Hz. The triaxial data is then synthesized into a contact pressure scalar, forming a contact pressure sensing waveform data vector that varies with time. The formula for calculating the contact pressure scalar is as follows: in, This represents a scalar value indicating contact pressure. , , These represent the axial components of the three-axis force vector.

4. The vision-guided flexible machine finger instrument control method according to claim 1, characterized in that, The system receives contact pressure sensor waveform data, synchronously loads the original waveform data from the pressure sensor inside the catheter, performs wavelet packet decomposition to extract the frequency band energy distribution of the dual-channel signal, obtains the dual-channel frequency band energy feature matrix, separates the approximate coefficient component and the detail coefficient component from the dual-channel frequency band energy feature matrix, substitutes the approximate coefficient component into the Navier-Stokes equation parameter identifier, and inputs the detail coefficient component into a support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter, including: The contact pressure sensor waveform data vector is used as the first channel, and the seventh column of the pressure waveform matrix is ​​used as the second channel to form a dual-channel signal. The dual-channel signal is decomposed into three layers of wavelet packets. The db4 wavelet basis is selected, and the energy values ​​of 8 frequency bands are extracted to form a frequency band energy feature matrix with a dimension of 2*8. Normalize each row of the frequency band energy feature matrix, sum the data in the first row to obtain the approximate coefficient components, and calculate the standard deviation of the data in columns 2 to 8 to obtain the detail coefficient components. The approximation coefficient components are input into a parameter identifier based on minimizing the residuals of the Navier-Stokes equations, and the formula for calculating the objective function is as follows: in, Describe the objective function of the residual integral. Represents the velocity field vector. Represents a time variable. Indicates fluid viscosity. Represents a pressure field. Indicates fluid density, Represents the volume force vector; The viscosity change rate is solved, and the detailed coefficient components are input into a support vector regression machine with a radial basis kernel function to obtain the flow resistance gradient data.

5. A vision-guided flexible robotic finger instrument control system, based on the vision-guided flexible robotic finger instrument control method of claim 1, characterized in that, include: Acquisition module: used to acquire real-time visual images of the indwelling catheter exit area, raw waveform data from the catheter internal pressure sensor, and initial signals from the flexible robotic fingertip multimodal sensor; Segmentation generation module: Based on the acquired real-time visual image, it performs feature point segmentation of the catheter tip using a lightweight convolutional neural network, calls the PnP algorithm to solve for rotation and translation variables in the camera coordinate system, transforms the rotation and translation variables to the robot base coordinate system, and generates the three-dimensional coordinate system positioning information of the catheter tip. The output module is used to call the three-dimensional coordinate system positioning information as the target trajectory input, load the acquired initial signal for zero-point compensation and environmental interference correction, obtain the calibrated sensor reference parameters, drive the flexible robotic finger to move based on the calibrated sensor reference parameters and collect the end pressure response, and output the contact pressure sensing waveform data between the flexible robotic finger and the duct wall. Separation and Substitution Module: This module receives waveform data from contact pressure sensors, synchronously loads the original waveform data from the pressure sensor inside the catheter, performs wavelet packet decomposition to extract the frequency band energy distribution of the dual-channel signal, obtains the dual-channel frequency band energy feature matrix, separates the approximate coefficient component and the detail coefficient component from the dual-channel frequency band energy feature matrix, substitutes the approximate coefficient component into the Navier-Stokes equation parameter identifier, and inputs the detail coefficient component into the support vector regression machine for dimensionality reduction, thereby obtaining the fluid viscosity change rate and flow resistance gradient data inside the catheter. The mapping module is used to input the fluid viscosity change rate and flow resistance gradient data in the duct as fuzzy inference system input, set the blockage level domain as mild, moderate and severe and configure the triangular membership function, map the defuzzified blockage level value to the machine finger vibration parameter library, and output the unblocking control sequence of flexible machine finger vibration frequency, amplitude and duration.

6. The vision-guided flexible robotic finger instrument control system according to claim 5, characterized in that, The segmentation generation module includes: The first extraction unit is used to extract depth feature maps from the encoder of the MobileNetV3 architecture, which inputs grayscale images. The decoder uses transposed convolution to upsample to the original image resolution and outputs a binary mask image of the duct end contour. Sixteen Harris corner points are extracted around the centroid of the mask image as a feature point set. Each feature point contains horizontal and vertical pixel coordinates and response intensity values. The construction unit is used to pair the homogeneous coordinates of 16 feature point pixels with their corresponding spatial coordinates in the 3D measurement coordinate system, construct a collinearity equation system, and use the Levenberg-Marquardt iteration to minimize the reprojection error. The rotation matrix and translation vector of the camera relative to the duct coordinate system are then solved. The formula for calculating the reprojection error function is as follows: In the formula, This represents the sum of squared reprojection errors. This represents the pixel coordinates of the k-th feature point. Represents the projection function. Represents the rotation matrix. Represents the translation vector. Represents the three-dimensional coordinates of the k-th feature point; The acquisition unit is used to obtain the 4*4 homogeneous transformation matrix from the camera coordinate system to the robot base coordinate system through hand-eye calibration. Multiplying the transformation matrix by the 3D coordinates of the feature points on the left yields the positioning coordinate vector of the duct end in the robot base coordinate system. The formula for calculating the positioning coordinate vector is as follows: In the formula, Let be the homogeneous coordinates of the catheter tip in the robot's base coordinate system. This represents the transformation matrix from the camera to the robot's base coordinate system. This represents the pose matrix of the camera relative to the duct coordinate system. The homogeneous coordinate vector representing the feature point.

7. The vision-guided flexible robotic finger instrument control system according to claim 5, characterized in that, The call output module includes: The second extraction unit is used to extract the reference deviation statistics from the reference signal vector of the multimodal sensor, calculate the drift compensation amount, and subtract the compensation amount from the readings of each axis of the triaxial micro-force sensor to obtain the compensated force vector. The calculation formula for the compensation amount is as follows: In the formula, Indicates the amount of drift compensation. Represents the linear drift coefficient. This indicates the current benchmark deviation value. Indicates the standard reference deviation value. Indicates the second-order drift coefficient; The first solving unit is used to take the positioning coordinate vector of the catheter end as the target position, solve the vector composed of the rotation angles of each joint through the forward kinematic equation of the flexible robotic finger, and generate a smooth trajectory from the current position to the target position using fifth-order polynomial interpolation. The trajectory execution period is set to 500 milliseconds, and the dimension of the joint rotation angle vector is equal to the number of degrees of freedom of the flexible robotic finger. The data acquisition unit continuously acquires triaxial micro-force sensor data at a frequency of 2000 Hz during trajectory execution, synthesizes the triaxial data into a contact pressure scalar, and forms a contact pressure sensing waveform data vector that changes pressure over time. The calculation formula for the contact pressure scalar is as follows: in, This represents a scalar value indicating contact pressure. , , These represent the axial components of the three-axis force vector.

8. The vision-guided flexible machine finger instrument control system according to claim 5, characterized in that, The separation and substitution module includes: The decomposition unit is used to take the contact pressure sensor waveform data vector as the first channel and the seventh column of the pressure waveform matrix as the second channel to form a dual-channel signal. The dual-channel signal is decomposed into three-level wavelet packets, the db4 wavelet basis is selected, and the energy values ​​of 8 frequency bands are extracted to form a frequency band energy feature matrix with a dimension of 2*8. Processing unit: Normalizes each row of the frequency band energy feature matrix, sums the data in the first row to obtain the approximate coefficient components, and calculates the standard deviation of the data in columns 2 to 8 to obtain the detail coefficient components; Optimization unit: Used to input the approximation coefficient components into the parameter identifier based on minimizing the residuals of the Navier-Stokes equations. The formula for calculating the objective function is as follows: in, Describe the objective function of the residual integral. Represents the velocity field vector. Represents a time variable. Indicates fluid viscosity. Represents a pressure field. Indicates fluid density, Represents the volume force vector; The second solution unit is used to solve for the viscosity change rate. It inputs the detailed coefficient components into a support vector regression machine with a radial basis kernel function and maps them to obtain the flow resistance gradient data.

9. A vision-guided flexible machine finger instrument control device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the vision-guided flexible machine finger instrument control method as described in any one of claims 1 to 4 when executing the computer program.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program that, when executed by a processor, implements the vision-guided flexible machine finger instrument control method as described in any one of claims 1 to 4.

Images