Fiber Optic Force-Form Sensing Device and Method via Natural Cavity

Abstract

This invention provides a fiber optic force-shape sensing device and method via a natural cavity, relating to the field of fiber optic sensing technology. The device includes a manipulator arm mechanism, specifically comprising a front-end connecting joint, several movable joints, and an end-end connecting joint, sequentially rotatably connected by a ball head and a ball socket. At least one limiting protrusion is distributed on the circumferential surface of the ball head, and a limiting groove corresponding to each limiting protrusion is formed in the ball socket. The limiting protrusion and the limiting groove slide together to restrict in-situ torsion between adjacent joints. The interaction between the limiting groove and the limiting protrusion reduces in-situ torsion and improves torsional strength. A conformal embedding fiber optic sensor integration method is proposed to reduce shape reconstruction errors caused by motion misalignment and protect the optical fiber. Embedded in the manipulator arm mechanism are a first drive wire, four second drive wires, two fiber optic shape sensing modules, and four fiber optic force sensing modules, enabling real-time and accurate measurement of interactive forces and the bending shape of the manipulator arm mechanism.

Description

Technical Field

[0001] This invention relates to the field of fiber optic sensing technology, and in particular to a fiber optic force-shape sensing device and method via a natural cavity. Background Technology

[0002] Because the nasal cavity and the base of the nasal skull have complex and narrow structures, surgery can easily damage surrounding blood vessels and nerves, resulting in a high risk of endoscopic nasal surgery.

[0003] However, due to the narrow and tortuous natural cavities exposed to the human body, existing nasal surgical manipulators are usually only small in size due to limited operating space. If multiple fiber optic sensing mechanisms are to be embedded simultaneously when embedding force-driven wires, the integration is difficult and it is hard to achieve the optimal balance between measurement accuracy and protection of optical fibers. Summary of the Invention

[0004] The purpose of this invention is to provide a fiber optic force-shape sensing device through a natural cavity, in order to solve the problem mentioned in the background art that traditional nasal surgical manipulators are difficult to achieve an optimal balance between measurement accuracy and protection of the optical fiber.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a fiber optic force-shape sensing device for natural cavities, comprising: a clamp for clamping cavity tissue; a scraper for scraping cavity tissue; and two parallel operating arm mechanisms, wherein the end of one operating arm mechanism is fixedly connected to the clamp, and the end of the other operating arm mechanism is fixedly connected to the scraper. Each operating arm mechanism includes a front connecting joint, several movable joints, and an end connecting joint sequentially rotatably connected. Adjacent joints rotate through a clearance fit between a ball head and a ball socket arranged coaxially. At least one limiting protrusion is distributed on the circumferential surface of the ball head, and the ball socket has a corresponding limiting protrusion. A one-to-one corresponding limiting groove, with the limiting protrusion slidingly engaging with the limiting groove to restrict in-situ torsion between adjacent joints; a first drive wire, coaxially passing through the operating arm mechanism and connected to the clamp, the first drive wire driving the clamp to open and close under axial force; four second drive wires, two fiber shape sensing modules, and four fiber force sensing modules, circumferentially distributed inside the operating arm mechanism in the same direction as the first drive wires, the second drive wires driving the operating arm mechanism to bend or straighten under force, the fiber shape sensing modules being used to sense the shape of the operating arm mechanism, and the fiber force sensing modules being used to measure the three-dimensional force on the clamp.

[0006] Optionally, it also includes: an endoscope for providing a view inside the cavity; two drive mechanisms arranged in parallel, each drive mechanism being fixedly connected to the front end of the two operating arm mechanisms via a connecting base, one of the drive mechanisms being fixedly connected to the endoscope via an endoscope support; the drive mechanism includes an angle adjustment mechanism and a translation mechanism, the angle adjustment mechanism being used to drive the second drive wire to cause the operating arm mechanism to bend or straighten, and the translation mechanism being used to drive the operating arm mechanism and / or the endoscope to move back and forth.

[0007] Optionally, the four second drive wires are distributed circumferentially at 90° intervals, the four optical fiber force sensing modules are distributed circumferentially at 90° intervals, and the two optical fiber shape sensing modules are distributed circumferentially at 90° intervals.

[0008] Optionally, the fiber shape sensing module includes a fiber grating string and two support wires. The two support wires are arranged in a triangular shape with the cross section of the fiber grating string to provide support for the fiber grating string. The fiber grating string includes four spaced-apart first fiber gratings, and the center wavelengths of the four first fiber gratings are distributed in a stepped manner.

[0009] Optionally, the clamp includes a fixed clamping part, a movable clamping part, and an elastic support mechanism; the movable clamping part is rotatably connected to the fixed clamping part via a pivot pin, and the movable clamping part is connected to the first drive wire. Under the drive of the first drive wire, the movable clamping part rotates around the pivot pin to move closer to or away from the fixed clamping part; one end of the elastic support mechanism is fixedly connected to the fixed clamping part, and the other end is fixedly connected to the operating arm mechanism. The elastic support mechanism includes four hollow elastic support beams; the fiber optic force sensing module includes a second fiber grating with an attached metal layer, and the four second fiber gratings pass through the four corresponding hollow elastic support beams and are fixedly connected to the fixed clamping part.

[0010] On the other hand, the present invention also provides a fiber optic force-shape sensing method based on the above-mentioned fiber optic force-shape sensing device with natural cavity, comprising the following steps: acquiring a binocular image of an operating arm mechanism; extracting pixel coordinate information of each feature point of the operating arm mechanism in the binocular image; and calculating the three-dimensional coordinate information of the feature points based on parallax and camera calibration parameters using the principle of triangulation; acquiring center wavelength drift data of two fiber optic shape sensing modules in the operating arm mechanism; using the three-dimensional coordinate information obtained by triangulation as a reference value, training and establishing a deep learning model with the center wavelength drift as an input value; inputting the center wavelength drift data into the deep learning model to predict the feature point position information of the operating arm mechanism; and reconstructing the shape of the operating arm mechanism based on the three-dimensional coordinate information and the feature point position information using the constant curvature deformation assumption.

[0011] Optionally, the step of acquiring a binocular image of the manipulator mechanism and extracting the pixel coordinate information of each feature point of the manipulator mechanism in the binocular image specifically includes: calibrating the binocular camera and determining the intrinsic and extrinsic parameters of the binocular camera; marking the feature points of the manipulator mechanism and acquiring a binocular image of the manipulator mechanism, and extracting the pixel coordinate information of each feature point of the manipulator mechanism in the binocular image.

[0012] Optionally, the step of calculating the three-dimensional coordinate information of the feature point based on the principle of triangulation, parallax, and camera calibration parameters specifically includes: The imaging planes of the two cameras are parallel to each other. Given the known distance between them and their equal focal lengths, the triangles formed by the projections of the point to be measured onto the imaging projection positions of both cameras, and the triangles formed by the edges of the projections to the measurement point, are similar. Therefore, the depth calculation formula is: In the formula, Z represents depth, and T represents the distance between the two cameras. and Let f be the distance from the left and right edges of the imaging plane to the measurement point, f be the focal length, and d be the parallax. Establish a three-dimensional coordinate system. The formulas for calculating the X and Y coordinates of the measurement point in the three-dimensional coordinate system are as follows: , , bring in We can obtain: , Let d / T = W, then the process of converting pixel coordinates to camera coordinates can be represented in matrix form: By converting the camera coordinate system to the world coordinate system, the three-dimensional coordinates of the measurement point in the world coordinate system can be obtained. In the formula, x and y are the normalized coordinates of the measurement point on the camera's imaging plane, f is the focal length, X and Y are the coordinate components of the measurement point in the three-dimensional coordinate system, Z is the depth coordinate of the measurement point in the three-dimensional coordinate system, d is the parallax, and u and v are the pixel coordinates of the measurement point in the left camera image. and The point where the principal ray intersects the imaging plane, and W is the normalization factor. Let R be the three-dimensional coordinates of the measurement point in the world coordinate system, R be the third-order orthogonal rotation matrix, and To be the translation matrix.

[0013] Optionally, the step of obtaining the center wavelength drift data of the two fiber shape sensing modules in the operating arm mechanism specifically includes: when the operating arm mechanism bends, according to the coupled-mode theory, the axial strain generated by the fiber shape sensing module will cause the center wavelength to drift, and the amount of drift can be expressed as: In the formula, and They are respectively The center wavelength drift and the initial center wavelength, where index i = 1, 2; index j = 1, 2, 3, 4; For the effective elastic coefficient, For what happened The response on the surface.

[0014] Optionally, the step of using the three-dimensional coordinate information obtained by triangulation as a reference value, using the center wavelength drift as an input value to train and establish a deep learning model, and inputting the center wavelength drift data into the deep learning model to predict the feature point position information of the manipulator mechanism specifically includes: the center wavelength offset corresponding to the input layer of the deep learning model. The output layer corresponds to the three-dimensional coordinates of each feature point. Then the model output formula is: In the formula, R is the predicted value output by the model, and L is the number of hidden layer nodes. The input weights for the j-th hidden layer node, Let be the bias of the j-th hidden layer node. Let the output weights be those of the j-th hidden layer node. For activation function, This represents the wavelength shift.

[0015] Compared with the prior art, the beneficial effects of the present invention are:

[0016] This application configures the manipulator mechanism as a front-end connecting joint, several movable joints, and an end-end connecting joint, which are sequentially rotatably connected by a ball head and a ball socket. This allows the manipulator mechanism to have a larger bending angle and a smaller bending radius, and also provides a larger axial space. The circumferential surface of the ball head has at least one limiting protrusion, and the ball socket has a limiting groove corresponding to the limiting protrusion. The limiting protrusion and the limiting groove slide together to limit the in-situ torsion between adjacent joints. Through the cooperation of the limiting groove and the limiting protrusion, the in-situ torsion is reduced and the torsional strength is improved. A conformal embedding fiber optic sensor integration method is proposed to reduce the shape reconstruction error caused by motion misalignment and to protect the optical fiber. A first drive wire is embedded in the manipulator mechanism, coaxially passing through the manipulator mechanism and connected to the clamp. The first drive wire drives the clamp to open and close under the drive of axial force. Four second drive wires, two fiber optic shape sensing modules, and four fiber optic force sensing modules are circumferentially distributed inside the manipulator mechanism in the same direction as the first drive wires. The second drive wires drive the manipulator mechanism to bend or straighten under the drive of force. The fiber optic shape sensing modules are used to sense the shape of the manipulator mechanism, and the fiber optic force sensing modules are used to measure the three-dimensional force on the clamp. This enables real-time and accurate measurement of interactive forces and the bending shape of the manipulator mechanism, greatly improving the sense of presence in surgery and the safety and accuracy of surgical operations. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the overall structure of the device of the present invention.

[0018] Figure 2 This is a schematic diagram of the operation execution structure of the present invention.

[0019] Figure 3 This is a schematic diagram of the operating arm mechanism of the present invention.

[0020] Figure 4 This is a three-dimensional structural diagram of the end-connecting joint of the present invention.

[0021] Figure 5 This is a schematic diagram of the top structure of the end-connecting joint of the present invention.

[0022] Figure 6 This is a three-dimensional structural diagram of the movable joint of the present invention.

[0023] Figure 7 This is a schematic diagram of the front-end connection joint structure of the present invention.

[0024] Figure 8 This is a schematic diagram of the clamp structure of the present invention.

[0025] Figure 9 This is a schematic diagram of the scraper of the present invention.

[0026] Figure 10 This is a schematic diagram of the drive mechanism structure of the present invention.

[0027] Figure 11 This is a cross-sectional view of the drive mechanism of the present invention.

[0028] Figure 12 This is a schematic diagram of the joint space mapping of the manipulator mechanism of the present invention.

[0029] Figure 13 This is a schematic diagram of the bending state of the operating arm mechanism of the present invention.

[0030] Figure 14 This is a schematic diagram of the theoretical model of the operating arm mechanism of the present invention.

[0031] Figure 15 This is a schematic diagram of the theoretically reachable space of this invention.

[0032] Figure 16 This is a schematic diagram of the combined workspace of the present invention.

[0033] Figure 17 This is a schematic diagram of the fiber optic shape sensing module structure of the present invention.

[0034] Figure 18 This is a diagram showing the cable arrangement in a single movable joint of the present invention.

[0035] Figure 19 This is a schematic diagram of the in-situ calibration method for binocular vision according to the present invention.

[0036] Figure 20 This is a schematic diagram of the shape reconstruction process of the Extreme Learning Machine of the present invention. Detailed Implementation

[0037] The present invention will now be clearly and completely described in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0038] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0039] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0040] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0041] It should be understood that the sequence number and size of each step in this embodiment do not imply the order of execution. The execution order of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of this application embodiment.

[0042] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0043] Please refer to Figures 1-11 as well as Figure 17 and Figure 18The present invention discloses a fiber optic force-shape sensing device for natural cavities, comprising: a clamp 1 for clamping cavity tissue; a scraper 2 for scraping cavity tissue; and two parallel operating arm mechanisms, one of which is fixedly connected at the end to the clamp 1, and the other is fixedly connected at the end to the scraper 2. Each operating arm mechanism includes a front connecting joint 23, several movable joints 22, and an end connecting joint 21 sequentially rotatably connected. Adjacent joints rotate through a clearance fit between a ball head 221 and a ball socket 222 arranged coaxially. The circumferential surface of the ball head 221 has at least one limiting protrusion 223, and the ball socket 222 has a limiting groove 224 corresponding to each limiting protrusion 223. The limiting protrusion 223 slides with the limiting groove 224 to restrict in-situ torsion between adjacent joints; the first drive wire 115 coaxially passes through the operating arm mechanism and is connected to the clamp 1, and the first drive wire 115 drives the clamp 1 to open and close under the drive of axial force; four second drive wires 24, two fiber shape sensing modules 25 and four fiber force sensing modules 116 are circumferentially distributed inside the operating arm mechanism in the same direction as the first drive wire 115; the second drive wires 24 drive the operating arm mechanism to bend or straighten under the drive of force; the fiber shape sensing module 25 is used to sense the shape of the operating arm mechanism; and the fiber force sensing module 116 is used to measure the three-dimensional force on the clamp 1.

[0044] Specifically, this application includes a first operating arm mechanism and a second operating arm mechanism, a clamp 1, a scraper 2, an optical fiber force sensing module 116, an optical fiber shape sensing module 25, an integrated channel, a first drive wire 115 for driving the clamp 1 to open and close, and a second drive wire 24 for driving the operating arm mechanism to bend. The clamp 1 and the scraper 2 are made of titanium alloy metal additive manufacturing, and the drive wire is made of nickel-titanium alloy material. The first operating arm mechanism and the second operating arm mechanism are arranged in parallel. The clamp 1 is fixedly connected to the end of the first operating arm mechanism, and the scraper 2 is fixedly connected to the end of the second operating arm mechanism. The clamp 1 and the scraper 2 are driven by the operating arm mechanism to perform clamping and scraping operations. The operating arm mechanism is composed of eighteen coaxially arranged hinge ball units connected in series. The eighteen hinge ball units include a front end connecting joint 23, sixteen movable joints 22, and an end connecting joint 21 rotatably connected by a ball head 221 and a ball socket 222. The clamp 1 and the scraper 2 are respectively fixedly connected to the end connecting joints 21 of the two operating arm mechanisms.

[0045] Specifically, the hinge ball unit is machined from aluminum alloy. The hinge ball unit has an axial cavity to accommodate the first drive wire 115, and ten through holes circumferentially to accommodate four second drive wires 24, two fiber optic shape sensing modules 25, and four fiber optic force sensing modules 116. This integration method provides more circumferential space for the operating arm mechanism to embed more sensing mechanisms. Furthermore, by rotating the ball head 221 and the ball socket 222, each fiber optic sensor is arranged on the outside of the ball head 221 and the ball socket 222, ensuring that the fiber optic sensing mechanisms do not deform with the ball head during deformation. Interference occurs between the ball joint 221 and the socket 222, and four centrally symmetrically arranged limiting protrusions 223 are distributed on the circumferential surface of the ball joint 221. The socket 222 has limiting grooves 224 that correspond one-to-one with the limiting protrusions 223. The limiting protrusions 223 and the limiting grooves 224 are arranged at 90-degree intervals along the circumference. The limiting protrusions 223 and the limiting grooves 224 slide in fit to limit the in-situ torsion between adjacent joints, thereby improving the torsional strength. The modular design allows for timely replacement of the ball joint unit when it is damaged without affecting the overall function. The end of the operating arm mechanism adopts a universal connector design, which can be connected to a variety of surgical actuators.

[0046] It is understood that this application configures the operating arm mechanism as a front connecting joint 23, several movable joints 22, and an end connecting joint 21 that are sequentially rotatably connected by a ball head 221 and a ball socket 222. This allows the operating arm mechanism to have a larger bending angle and a smaller bending radius, and also provides a larger accommodating space in the axial direction of the operating arm mechanism. At least one limiting protrusion 223 is distributed on the circumferential surface of the ball head 221, and a limiting groove 224 corresponding to the limiting protrusion 223 is provided in the ball socket 222. The limiting protrusion 223 and the limiting groove 224 slide in cooperation to limit the in-situ torsion between adjacent joints. Through the cooperation of the limiting groove 224 and the limiting pin, the in-situ torsion is reduced and the torsional strength is improved. A conformal embedded fiber optic sensor integration method is proposed to reduce the shape reconstruction error caused by motion misalignment and to protect the optical fiber. A first drive wire 115 is embedded in the operating arm mechanism, coaxially passing through the operating arm mechanism and connected to the clamp 1. The first drive wire 115 drives the clamp 1 to open and close under the drive of axial force. Four second drive wires 24, two fiber shape sensing modules 25, and four fiber force sensing modules 116 are circumferentially distributed inside the operating arm mechanism in the same direction as the first drive wires 115. The second drive wires 24 drive the operating arm mechanism to bend or straighten under the drive of force. The fiber shape sensing modules 25 are used to sense the shape of the operating arm mechanism, and the fiber force sensing modules 116 are used to measure the three-dimensional force on the clamp 1. This enables real-time and accurate measurement of interactive forces and the bending shape of the operating arm mechanism, greatly improving the sense of presence in surgery and the safety and accuracy of surgical operations.

[0047] In some embodiments, the device further includes: an endoscope for providing intracavitary visualization; two parallel drive mechanisms, each drive mechanism being fixedly connected to the front end of one of the two manipulator arm mechanisms via a connecting base, and one of the drive mechanisms being fixedly connected to the endoscope via an endoscope support 3; the drive mechanism includes an angle adjustment mechanism and a translation mechanism, the angle adjustment mechanism being used to drive the second drive wire 24 to bend or straighten the manipulator arm mechanism, and the translation mechanism being used to drive the manipulator arm mechanism and / or the endoscope to move back and forth.

[0048] Specifically, the device includes two drive mechanisms, namely a first operating arm mechanism and a second operating arm mechanism arranged in parallel, such that the two operating arm mechanisms connected to it are also arranged in parallel. The drive mechanism includes an angle adjustment mechanism and a translation mechanism. There are four angle adjustment mechanisms, each corresponding to one of the first drive wires 115. The angle adjustment mechanism drives the first drive wires 115 through axial reciprocating movement at its output end, thereby causing the operating arm mechanism to bend or straighten. The translation mechanism is located at the bottom of the drive mechanism. By translating along a fixed track or fixed slide, the drive mechanism is translated, thereby driving the operating arm mechanism and / or the endoscope to move back and forth.

[0049] More specifically, the integrated drive unit, comprising two drive mechanisms, includes eight brushed DC motors, a fixed base, a coupling, a rear support base, a trapezoidal lead screw, a nut, a front support base, a guide shaft, two push rod motors, a guide wire guide tube, an inner tube fixing clamp, an inner tube support frame, a housing, a fixed plate, a rear support plate, a front support plate, and two guide rail motors. One end of the drive guide wire is fixedly connected to the nut, and the other end is fixedly connected to the end connection unit via metal adhesive, passing through the circumferential through hole of the ball joint unit. The eight brushed DC motors are divided into two groups: the first group of four brushed DC motors controls two bending degrees of freedom of the first operating arm mechanism, and the second group of four brushed DC motors controls two bending degrees of freedom of the second operating arm mechanism. The two push rod motors control the opening and closing of the clamp 1 and the forward and backward movement of the nasal endoscope, respectively. The two guide rail motors control the forward and backward movement of the two drive motor housings, respectively.

[0050] More specifically, the drive unit 4 includes two drive mechanisms, specifically a first drive housing 41, a second drive housing 42, a linear guide motor 43, and a frame 44. The first drive housing 41 and the second drive housing 42 are respectively responsible for driving the first and second manipulator mechanisms to bend. The first drive housing 41 and the second drive housing 42 are respectively bolted to their corresponding linear guide motors 43. The linear guide motors 43 control the forward and backward movement of their respective drive housings, thereby realizing the relative forward and backward movement of the manipulator mechanism 2. The linear guide motors 43 are fixed to the frame 44, and the frame 44 is connected to a commercial collaborative robotic arm to realize preoperative overall posture adjustment.

[0051] More specifically, the first drive housing 41 includes a DC brushed motor 411, a fixed base 412, a coupling 413, a rear support base 414, a trapezoidal lead screw 415, a nut 416, a front support base 417, a guide shaft 418, a push rod motor 419, a guide wire guide tube 4110, an inner tube fixing clamp 4111, an inner tube support frame 4112, an outer shell 4113, a fixing plate 4114, a rear support plate 4115, a front support plate 4116, and an angle iron 4117. The fixed plate 4114, rear support plate 4115, and front support plate 4116 are respectively fixed to the outer shell 4113 by angle iron 4117. The fixed seat 412 is fixed on the fixed plate 4114, the rear support seat 414 is fixed on the rear support plate 4115, and the front support seat 417 is fixed on the front support plate 411. The DC brushed motor 411 is fixed by the fixed seat 412. The coupling 413 is used to connect the output shaft of the DC brushed motor 411 to the trapezoidal lead screw 415. The nut 416 moves along the trapezoidal lead screw 415 and passes through the guide shaft 418 to limit the rotation of the nut 416 and ensure that the nut 416 can only move along the guide shaft 418, thus converting the rotational motion of the output shaft of the DC brushed motor 411 into the translational motion of the nut 416. The two ends of the trapezoidal lead screw 415 are respectively fixed in the rear support seat 414 and the front support seat 417. The inner tube fixing clamp 4111 and the inner tube support frame 4112 are respectively fixedly connected to the outer shell 4113. The inner tube 31 is fixed in the inner tube fixing clamp 4111, and the inner tube support frame 4112 is used to provide support for the inner tube 31 and improve stability. The push rod motor 419 is used to push and pull the opening and closing drive guide wire 115 to open and close the clamp 1. The opening and closing drive guide wire 115 is led out from the inner tube 31 and then fixedly connected to the push rod motor 419 after passing through the guide wire guide tube 4110. The guide wire guide tube 4110 is used to limit the additional bending of the opening and closing drive guide wire 115 and improve transmission efficiency. The second drive housing 42 adopts a similar structure. The end of the push tube motor in the second drive housing 42 is connected to the endoscope support 3, thereby controlling the back and forth movement of the nasal endoscope 32; the two operating arm mechanisms are in the straight state, and according to the needs of the operation, the two arms are controlled to bend to the designated position respectively. The host computer sends a command to the DC brushed motor 411, which drives the coupling 413 and the trapezoidal lead screw 415 to rotate, thereby driving the nut 416 to move along the guide shaft 418, driving the first drive screw to move, thereby causing the operating arm mechanism 2 to bend in the specified direction.

[0052] In some embodiments, the four second drive wires 24 are distributed circumferentially at 90° intervals, the four fiber force sensing modules 116 are distributed circumferentially at 90° intervals, and the two fiber shape sensing modules are distributed circumferentially at 90° intervals.

[0053] Specifically, each manipulator arm mechanism uses four second drive wires 24 to control its bending. That is, the arrangement direction of the four second drive wires 24 is parallel to the axis of the manipulator arm mechanism, and the four second drive wires 24 are centrally symmetrically distributed with the axis of the manipulator arm mechanism as the center. The four second drive wires 24 are arranged at 90° intervals. Every two drive wires are arranged at 180° intervals as a group to control the bending degree of freedom of the manipulator arm mechanism. The two fiber optic shape sensing modules are distributed at 90° intervals around the circumference to sense the shape of the manipulator arm mechanism.

[0054] In some embodiments, the fiber shape sensing module 25 includes a fiber grating string 2-5-1 and two support wires 2-5-2. The two support wires 2-5-2 are arranged in a triangular shape with the cross section of the fiber grating string 2-5-1 to provide support for the fiber grating string 2-5-1. The fiber grating string 2-5-1 includes four spaced-apart first fiber gratings, and the center wavelengths of the four first fiber gratings are distributed in a stepped manner.

[0055] Specifically, the fiber optic shape sensing module 25 includes a fiber grating string 2-5-1 and two supporting nickel-titanium alloy wires. The cross-section of the fiber grating string 2-5-1 and the two supporting nickel-titanium alloy wires is arranged in a triangular structure. Each fiber is etched with four fiber gratings (FBGs), with a grating length of 3 mm and an axial spacing of 10 mm between adjacent FBGs. The center wavelengths of the four FBGs are 1535 nm, 1540 nm, 1545 nm, and 1550 nm, respectively. Each of the manipulator mechanisms accommodates two fiber optic shape sensing modules 25, which are arranged through pre-reserved through holes in the ball joint unit. When the manipulator mechanism bends, the center wavelengths of the eight FBGs in the embedded fiber optic shape sensing module 25 drift accordingly. By decoupling them, the real-time three-dimensional shape of the manipulator mechanism can be reconstructed.

[0056] In some embodiments, the clamp 1 includes a fixed clamping part 111, a movable clamping part 112, and an elastic support mechanism 117; the movable clamping part 112 is rotatably connected to the fixed clamping part 111 via a pivot pin 113, and the movable clamping part 112 is connected to the first drive wire 115. Under the drive of the first drive wire 115, the movable clamping part 112 rotates about the pivot pin 113 to move closer to or away from the fixed clamping part 111; one end of the elastic support mechanism 117 is fixedly connected to the fixed clamping part 111, and the other end is fixedly connected to the operating arm mechanism. The elastic support mechanism 117 includes four hollow elastic support beams; the fiber optic force sensing module 116 includes a second fiber grating with an attached metal layer, the four second fiber gratings passing through the four corresponding hollow elastic support beams, and integrally formed with the fixed clamping part 111 by metal additive manufacturing.

[0057] Specifically, the clamp 1 includes a fixed clamping part 111, a movable clamping part 112, a pivot pin 113, a connecting rod 114, and an elastic support mechanism 117. The fixed clamping part 111 is fixed to the end connecting unit 21 by metal adhesive or laser welding. The pivot pin 113 passes through the through holes at corresponding positions on the fixed clamping part 111, the movable clamping part 112, and the connecting rod 114, and is fixed to the fixed clamping part 111 by metal adhesive. The movable clamping part 112 rotates around the pivot pin 113. The first drive wire 115 is bonded to the bottom end of the connecting rod 114 by metal adhesive to form a crank-slider mechanism. The other end of the first drive wire 115 is fixed to the drive mechanism. The clamp 1 is opened and closed by pushing and pulling the first drive wire 115 through the drive mechanism. The elastic support mechanism 117 consists of four hollow elastic support beams. The fiber optic force sensing module 116 consists of four surface-metallized fiber optic gratings that pass through the hollow part of the elastic support beams 117. Both ends are fixed to the fixed clamping part 111 by laser welding. The elastic support mechanism 117 amplifies the strain generated by the received three-dimensional force and transmits it to the fiber optic force sensing mechanism 116. The three-dimensional force on the clamp 1 is obtained by decoupling the wavelength change of the fiber optic force sensing mechanism 116. The connection method between the second operating arm mechanism and the scraper 2 is the same as the connection method between the first operating arm and the clamp 1. However, the first drive wire 115 in the second operating arm mechanism is connected to the endoscope support 3 and is used to drive the endoscope 32 to move axially.

[0058] Please refer to Figures 12-16 as well as Figure 19 and Figure 20On the other hand, the present invention also provides a fiber optic force-form sensing method based on the above-mentioned fiber optic force-form sensing device with natural cavity, the steps of which include: establishing a coordinate system with the axis of the operating arm mechanism as the Z-axis, and the radial directions of the connecting base, the front connecting joint 23, and the end connecting joint 21 as the X-axis and Y-axis, respectively; establishing a homogeneous transformation matrix between each coordinate system according to the bending angle of the operating arm mechanism and the changing length of the second drive wire 24, and obtaining the mapping relationship between each coordinate system; and using simulation analysis to analyze the reachable workspace of the operating arm mechanism to obtain the reachable area information of the two arms.

[0059] In some embodiments, the step of establishing a coordinate system with the axial direction of the manipulator mechanism as the Z-axis and the radial directions of the base, the front connecting joint 23, and the end connecting joint 21 as the X-axis and Y-axis, respectively, specifically includes: establishing a world coordinate system on the base fixedly connected to the manipulator mechanism. ,in, The plane is parallel to the upper and lower surfaces of the base. Parallel to the central axis of the base; establish a base coordinate system on the front connecting joint 23. ,in, The plane is parallel to the lower surface of the front connecting joint 23. Parallel to the central axis of the bottom hinge unit shaft and The included angle of the axis is Establish an end-effector coordinate system on the end-effector joint 21. ,in, The plane is parallel to the lower surface of the end-joint 21. Parallel to the central axis of the bottom hinge unit The axis points to the second drive wire 24.

[0060] In some embodiments, the step of establishing a homogeneous transformation matrix between each coordinate system based on the bending angle of the operating arm mechanism and the changing length of the second drive wire 24, and obtaining the mapping relationship between each coordinate system, includes: Base coordinate system To the world coordinate system The translation length, For curved plane and The included angle between the axes determines the position of the manipulator mechanism from the world coordinate system. To the end coordinate system homogeneous transformation matrix It can be represented as ; In the formula, From the world coordinate system To the end coordinate system The homogeneous transformation matrix, From the world coordinate system To the base coordinate system The homogeneous transformation matrix, From the base coordinate system To the end coordinate system The homogeneous transformation matrix, From the base coordinate system To the end coordinate system The rotation matrix, From the base coordinate system To the end coordinate system The translation matrix, From the world coordinate system To the end coordinate system The rotation matrix, From the world coordinate system To the end coordinate system The translation vector, Let Z be the transformation matrix for rotation about the Z-axis. Let be the transformation matrix for translation along the Z-axis.

[0061] In some embodiments, geometric analysis reveals that when adjacent joints deform, the distance from the center of the ball joint 222 to the center of the ball head 221 is... Since the bending of the manipulator mechanism is approximately constant curvature, the homogeneous transformation matrix can be expressed as: ; ; ; In the formula, The total bending angle of the manipulator mechanism. This represents the total number of joints. From the base coordinate system To the end coordinate system The homogeneous transformation matrix, From the base coordinate system To the end coordinate system The rotation matrix, From the base coordinate system To the end coordinate system The translation matrix, For the base system Z-axis rotation angle , For the surrounding base system Y-axis rotation angle , For the base system Z-axis reverse rotation angle , The bending angle of a single joint. , , The origin of the final coordinate system is in the base coordinate system. The coordinate components in the figure are: h is the length of a single joint, and i is the i-th joint.

[0062] In some embodiments, the mapping from the operating space to the joint space has been established so far. Next, the mapping from the joint space to the drive space is established. Pulling any adjacent second drive wire 24 can cause the operating arm mechanism to bend at any angle in space. After bending, the change in the length of the second drive wire 24 is proportional to the distance between the second drive wire 24 and the neutral plane. The lengths of the four drive guide wires after bending are as follows: ; ; ; In the formula, The original length of the second drive wire 24, , , , For the length of the four second drive wires 24 after bending, This represents the total number of joints. The bending angle of a single joint. for and Distance to the neutral axis for and The distance to the neutral axis, d is the center distance between the two second drive wire holes 225 with an included angle of 180°. Four second drive wires 24 respectively Projection in a plane. Four second drive wires 24 respectively Projection in a plane.

[0063] In some embodiments, the bending angle and deflection angle of the manipulator mechanism can be obtained based on the length of the drive guide wire, and the calculation formula is as follows: ; In the formula, For curved plane and The angle between the axes, , , , For the length of the four second drive wires 24 after bending, The total bending angle of the manipulator mechanism. d represents the total number of joints, and d is the center distance between the two second drive wire holes with an included angle of 180°.

[0064] In some embodiments, the mapping from joint space to operation space is obtained iteratively by a homogeneous transformation matrix, and the homogeneous transformation matrix of the local coordinate system of the i-th and i+1-th joints is: In the formula, Let be the homogeneous transformation matrix from the i-th joint to the (i+1)-th joint. The translation transformation matrix is... Let be the rotation transformation matrix about the Y-axis.

[0065] In some embodiments, the homogeneous transformation matrix from the world coordinate system to the lowest joint is: In the formula, Let be the homogeneous transformation matrix of the lowest joint. Let be the rotation transformation matrix about the X-axis.

[0066] In some embodiments, the total homogeneous transformation matrix from the world coordinate system to the end joint can be expressed as: In the formula, This is the homogeneous transformation matrix from the world coordinate system to the end joint. This is the initial transformation from the base to the front joint. It transmits local bending between joints. This represents the global pose of the distal joint.

[0067] In some embodiments, simulation analysis is used to analyze the reachable workspace of the manipulator mechanism to obtain information on the reachable area of ​​both arms.

[0068] In some embodiments, the steps further include: acquiring a binocular image of the manipulator mechanism, extracting pixel coordinate information of each feature point of the manipulator mechanism in the binocular image, and calculating the three-dimensional coordinate information of the feature points based on parallax and camera calibration parameters using the principle of triangulation; acquiring center wavelength drift data of the two fiber shape sensing modules 25 in the manipulator mechanism; establishing a deep learning model, inputting the center wavelength drift data into the deep learning model, predicting the feature point position information of the manipulator mechanism, and reconstructing the shape of the manipulator mechanism based on the three-dimensional coordinate information and the feature point position information using the constant curvature deformation assumption.

[0069] In some embodiments, the step of acquiring a binocular image of the manipulator mechanism and extracting the pixel coordinate information of each feature point of the manipulator mechanism in the binocular image specifically includes: calibrating the binocular camera and determining the intrinsic and extrinsic parameters of the binocular camera; marking the feature points of the manipulator mechanism and acquiring a binocular image of the manipulator mechanism, and extracting the pixel coordinate information of each feature point of the manipulator mechanism in the binocular image.

[0070] In some embodiments, the step of calculating the three-dimensional coordinate information of the feature point based on the principle of triangulation, parallax, and camera calibration parameters specifically includes: the imaging planes of the two cameras are parallel to each other, and given the known distance between them and their equal focal lengths, the triangles formed by the projection positions of the point to be measured onto the imaging projection positions of the two cameras and the projection edges to the measurement point positions are similar. Therefore, the depth calculation formula is: In the formula, Z represents depth, and T represents the distance between the two cameras. and Let f be the distance from the left and right edges of the imaging plane to the measurement point, f be the focal length, and d be the parallax. Establish a three-dimensional coordinate system. The formulas for calculating the X and Y coordinates of the measurement point in the three-dimensional coordinate system are as follows: , , bring in We can obtain: , Let d / T = W, then the process of converting pixel coordinates to camera coordinates can be represented in matrix form: By converting the camera coordinate system to the world coordinate system, the three-dimensional coordinates of the measurement point in the world coordinate system can be obtained. In the formula, x and y are the normalized coordinates of the measurement point on the camera's imaging plane, f is the focal length, X and Y are the coordinate components of the measurement point in the three-dimensional coordinate system, Z is the depth coordinate of the measurement point in the three-dimensional coordinate system, d is the parallax, and u and v are the pixel coordinates of the measurement point in the left camera image. and The point where the principal ray intersects the imaging plane, and W is the normalization factor. Let R be the three-dimensional coordinates of the measurement point in the world coordinate system, R be the third-order orthogonal rotation matrix, and To be the translation matrix.

[0071] In some embodiments, the step of obtaining the center wavelength drift data of the two fiber shape sensing modules 25 in the manipulator mechanism specifically includes: when the manipulator mechanism bends, according to the coupled-mode theory, the axial strain generated by the fiber shape sensing module 25 will cause the center wavelength to drift, and the amount of drift can be expressed as: In the formula, and They are respectively The center wavelength drift and the initial center wavelength, where index i = 1, 2; index j = 1, 2, 3, 4; For the effective elastic coefficient, For what happened The response on the surface.

[0072] In some embodiments, the step of establishing a deep learning model, inputting the center wavelength drift data into the deep learning model, and predicting the feature point position information of the manipulator mechanism specifically includes: the center wavelength offset corresponding to the input layer of the deep learning model. The output layer corresponds to the three-dimensional coordinates of each feature point. Then the model output formula is: In the formula, R is the predicted value output by the model, and L is the number of hidden layer nodes. The input weights for the j-th hidden layer node, Let be the bias of the j-th hidden layer node. Let the output weights be those of the j-th hidden layer node. For activation function, This represents the wavelength shift.

[0073] In some embodiments, when the clamp 1 is subjected to an external three-dimensional force, the four hollow support beams in the elastic support mechanism 117 are stretched and compressed, thereby causing strain in the fiber gratings in its cavity, resulting in a drift of the center wavelength of the fiber gratings. By decoupling the drift of the center wavelengths of the four fiber gratings, the three-dimensional interactive force value of the actuator can be obtained.

[0074] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0075] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0076] The above are merely embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention's specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A fiber optic force-shape sensing device via a natural cavity, characterized in that, include: Clamps are used to hold tissues in cavities. A curette is used to scrape away tissue from cavities. Two parallel operating arm mechanisms are provided. The end of one operating arm mechanism is fixedly connected to the clamp, and the end of the other operating arm mechanism is fixedly connected to the scraper. Each operating arm mechanism includes a front connecting joint, several movable joints, and an end connecting joint that are rotatably connected in sequence. Adjacent joints rotate through a clearance fit between a ball head and a ball socket arranged coaxially. At least one limiting protrusion is distributed on the circumferential surface of the ball head, and a limiting groove corresponding to the limiting protrusion is formed in the ball socket. The limiting protrusion and the limiting groove slide to restrict the in-situ torsion between adjacent joints. A first drive wire coaxially passes through the operating arm mechanism and is connected to the clamp. The first drive wire drives the clamp to open and close under the drive of axial force. Four second drive wires, two fiber shape sensing modules, and four fiber force sensing modules are circumferentially distributed inside the operating arm mechanism in the same direction as the first drive wires. The second drive wires drive the operating arm mechanism to bend or straighten under the drive of force. The fiber shape sensing modules are used to sense the shape of the operating arm mechanism, and the fiber force sensing modules are used to measure the three-dimensional force on the clamp.

2. The fiber optic force-shape sensing device via a natural cavity according to claim 1, characterized in that, Also includes: Endoscopes are used to provide a view inside cavities; Two drive mechanisms are arranged in parallel. The two drive mechanisms are respectively fixedly connected to the front ends of the two operating arm mechanisms through a connecting base. One of the drive mechanisms is also fixedly connected to the endoscope through an endoscope support. The driving mechanism includes an angle adjustment mechanism and a translation mechanism. The angle adjustment mechanism is used to drive the second drive wire to bend or straighten the operating arm mechanism. The translation mechanism is used to drive the operating arm mechanism and / or the endoscope to move back and forth.

3. The fiber optic force-shape sensing device via a natural cavity according to claim 1, characterized in that, The four second drive wires are circumferentially spaced at 90° intervals, the four optical fiber force sensing modules are circumferentially spaced at 90° intervals, and the two optical fiber shape sensing modules are circumferentially spaced at 90° intervals.

4. The fiber optic force-shape sensing device via a natural cavity according to claim 1, characterized in that, The fiber shape sensing module includes a fiber grating string and two support wires. The two support wires are arranged in a triangular shape with the cross section of the fiber grating string to provide support for the fiber grating string. The fiber grating string includes four spaced-apart first fiber gratings, and the center wavelengths of the four first fiber gratings are distributed in a stepped manner.

5. The fiber optic force-shape sensing device via a natural cavity according to claim 1, characterized in that, The clamp includes a fixed clamping part, a movable clamping part, and an elastic support mechanism; The movable clamping part is rotatably connected to the fixed clamping part via a pivot pin. The movable clamping part is connected to the first drive wire. Under the drive of the first drive wire, the movable clamping part rotates around the pivot pin to move closer to or away from the fixed clamping part. One end of the elastic support mechanism is fixedly connected to the fixed clamping part, and the other end is fixedly connected to the operating arm mechanism. The elastic support mechanism includes four hollow elastic support beams. The fiber optic force sensing module includes a second fiber grating with an attached metal layer, four of which pass through four corresponding hollow elastic support beams and are fixedly connected to the fixed clamping part.

6. A fiber optic force-form sensing method based on the fiber optic force-form sensing device via a natural cavity as described in any one of claims 1-5, characterized in that the steps include... include: A binocular image of the manipulator mechanism is acquired, the pixel coordinate information of each feature point of the manipulator mechanism in the binocular image is extracted, and the three-dimensional coordinate information of the feature point is calculated based on the parallax and camera calibration parameters using the principle of triangulation. Acquire the center wavelength drift data of the two fiber shape sensing modules in the manipulator mechanism; Using the three-dimensional coordinate information obtained by triangulation as a reference value, and the center wavelength drift as an input value, a deep learning model is trained and established. The center wavelength drift data is input into the deep learning model to predict the feature point position information of the manipulator mechanism. Based on the three-dimensional coordinate information and the feature point position information, the shape of the manipulator mechanism is reconstructed using the constant curvature deformation assumption.

7. The fiber optic force-shape sensing method via natural cavity as described in claim 6, characterized in that, The step of acquiring a binocular image of the manipulator mechanism and extracting the pixel coordinate information of each feature point of the manipulator mechanism in the binocular image specifically includes: The binocular camera is calibrated to determine its intrinsic and extrinsic parameters; The feature points of the manipulator mechanism are marked, and a binocular image of the manipulator mechanism is acquired. The pixel coordinate information of each feature point of the manipulator mechanism in the binocular image is extracted.

8. The fiber optic force-shape sensing method via natural cavity as described in claim 6, characterized in that, The step of calculating the three-dimensional coordinate information of the feature point based on the principle of triangulation, parallax, and camera calibration parameters specifically includes: Given that the imaging planes of the two cameras are parallel to each other and their focal lengths are equal, the triangles formed by the projections of the point to be measured onto the images of both cameras and the points on either side of the projections to the measurement point are similar. Therefore, the depth calculation formula is: ； In the formula, Z represents depth, and T represents the distance between the two cameras. and Let f be the distance from the left and right edges of the imaging plane to the measurement point, f be the focal length, and d be the parallax. ; Establish a three-dimensional coordinate system. The formulas for calculating the X and Y coordinates of the measurement point in the three-dimensional coordinate system are as follows: , , bring in We can obtain: , ; Let d / T = W, then the process of converting pixel coordinates to camera coordinates can be represented in matrix form: ； By converting the camera coordinate system to the world coordinate system, the three-dimensional coordinates of the measurement point in the world coordinate system can be obtained: ； In the formula, x and y are the normalized coordinates of the measurement point on the camera's imaging plane, f is the focal length, X and Y are the coordinate components of the measurement point in the three-dimensional coordinate system, Z is the depth coordinate of the measurement point in the three-dimensional coordinate system, d is the parallax, and u and v are the pixel coordinates of the measurement point in the left camera image. and The point where the principal ray intersects the imaging plane, and W is the normalization factor. Let R be the three-dimensional coordinates of the measurement point in the world coordinate system, R be the third-order orthogonal rotation matrix, and To be the translation matrix.

9. The fiber optic force-shape sensing method via natural cavity as described in claim 6, characterized in that, The step of obtaining the center wavelength drift data of the two fiber optic shape sensing modules in the manipulator mechanism specifically includes: When the manipulator mechanism bends, according to coupled-mode theory, the axial strain generated by the fiber optic shape sensing module will cause a shift in the center wavelength, and the amount of shift can be expressed as: ； In the formula, and They are respectively The center wavelength drift and the initial center wavelength, where index i = 1, 2; index j = 1, 2, 3, 4; For the effective elastic coefficient, For what happened The response on the surface.

10. The fiber optic force-shape sensing method via natural cavity as described in claim 6, characterized in that, The step of using the three-dimensional coordinate information obtained by triangulation as a reference value, using the center wavelength drift as an input value to train and establish a deep learning model, and inputting the center wavelength drift data into the deep learning model to predict the feature point position information of the manipulator mechanism specifically includes: The center wavelength offset corresponding to the input layer of the deep learning model The output layer corresponds to the three-dimensional coordinates of each feature point. Then the model output formula is: ； In the formula, R is the predicted value output by the model, and L is the number of hidden layer nodes. The input weights for the j-th hidden layer node, Let be the bias of the j-th hidden layer node. Let the output weights be those of the j-th hidden layer node. For activation function, This represents the wavelength shift.

Images