Probe having an image-based force-recording assembly

Abstract

The invention relates to a probe (01) having a probe body (01a), a probe head (01b), a camera (74), and a protective grating (73) attached to the probe head. The probe additionally has a force-recording assembly which is formed by the camera and the protective grating (73). The protective grating is arranged at least partially in the field of view of the camera (74) and is elastically deformable under the action of a force. The resulting deformation of the protective grating (73) can be imaged by the camera (74).

Description

[0001] Probe with an image-based force detection arrangement

[0002] The invention relates to a probe that is particularly, but not exclusively, suitable for use in the medical field. In addition to its conventional functionalities, the probe is intended to enable force measurement. Here, a probe is understood in a broader sense as a unit that can be inserted into a medium or cavity to perform investigations, collect data, or carry out manipulations. Such probes have a probe body in which, for example, a cavity is provided for transporting sensors or microtools. A probe head is attached to the probe body, which, for example, has a probe opening for the exit of a microtool and / or for receiving samples. The probe opening can be partially covered or protected by a protective grid, particularly to protect sensors from mechanical damage.

[0003] Probes are used, among other things, in minimally invasive surgery (MIS). In this procedure, probes with treatment instruments or sensors are inserted into the body via rigid rod-shaped mechanisms or flexible catheter systems to perform medical examinations and / or treatment manipulations at the target site. The probes are inserted manually by the treating physician or assisted by external robotic manipulation systems.

[0004] German patent application DE 20 2021 003 909 Ul describes a device for inspecting and / or cleaning pipes and / or channels. The device comprises a probe and a measuring device for measuring the geometry of a pipe or channel and for measuring the distance between the probe and an object located in the pipe or channel. The probe is equipped with a camera. The measuring device is coupled to a control unit. The measuring device includes a time-of-flight (TOF) sensor. The control unit is configured to adjust a coupled focusing unit of the camera and / or an illumination unit based on measurement data from the TOF sensor.

[0005] From WO 2024 / 013035 Al, a probe is known that is designed as a microrobotic unit for locomotion and positioning in organic cavities. The microrobotic unit comprises a front and a rear expansion module, a thrust module arranged between the two expansion modules, and a supply connection which is axially connected to the rear expansion module and serves to supply at least one supply medium. Each expansion module has at least one outer balloon shell, the expansion of which increases upon supply of the supply medium. In a particular embodiment, this microrobotic unit has a micro camera on its probe head, which is preferably shaped as a frustoconical mandrel, for observing the surroundings. The camera can be protected by a protective grille.

[0006] In many cases, it is desirable to detect, and preferably qualitatively measure, the force exerted on the probe, particularly on the probe head and any instruments positioned there. This force measurement could improve the monitoring of movement patterns and provide the treating physician with important information for current navigation or for the qualitative assessment of surrounding organ structures, especially for the detection of diseases, anomalies, or other medical conditions. Due to the technical complexity and the challenges of signal differentiation (ratio of measured signal to noise), especially with small forces, force measurement has largely been avoided in medical practice to date.In W02007 / 120329 A2, a modular force sensor device and a method for improving force and torque measurement and feedback to the surgeon performing telerobotic surgery are described. In one embodiment, a modular force sensor comprises a tube section containing multiple strain gauges, a proximal tube section for connection to the shaft of a surgical instrument, which may be connected to a manipulator arm of a surgical robotic system, and a distal tube section for proximal connection to a wrist, which is connected to an end section.

[0007] DE 10 2005 029 002 A1 discloses a method and a device for the contact measurement of a force acting on a rigid body. The device has an elastic deformation element designed as a spring element, against whose restoring force a force transducer with its probe tip is movable in a receptacle. In order to avoid undesirable influences from an electrical power supply as well as obstructive data lines during measurement, the force transducer and the receptacle are each equipped with a reference mark, the spatial positions of which can be automatically detected by a camera of an image processing system. In this way, the relative change in distance can be determined by a control unit, and the magnitude of the acting force can be determined from this based on the known spring constant.

[0008] One object of the invention, based on WO 2024 / 013035, is to provide an improved probe or microrobotic unit which, by utilizing the sensors already present in the probe, additionally allows the detection of the effect of a force, preferably enabling the directionally resolved determination of the forces and moments acting on the probe head. The probe is to be equipped with a force detection arrangement without increasing the design complexity or the size of the probe.

[0009] This problem is solved by a probe according to the attached claim 1.

[0010] The probe according to the invention is characterized in that the protective grid, which conventionally serves only to mechanically protect the probe head and any sensors optionally arranged in the probe body, is at least partially positioned within the camera's field of view. This is preferably achieved by suitable positioning of the camera, by appropriate design of the camera's imaging optics, and / or by suitable positioning of the protective grid. The protective grid is also designed to be elastically deformable, so that an elastic deformation of at least a section of the protective grid occurs when a force is applied. The resulting deformation of the protective grid is imaged by the camera and evaluated by a corresponding image processing routine, so that a signal can be generated which represents the occurrence of a force or moment and preferably also its magnitude.

[0011] Crucial to the invention is the design of the protective grid such that, when subjected to one- or multi-dimensional forces or moments, it undergoes defined deformations that are reversible due to the grid's elasticity. These deformations cause a displacement and / or a change in size of sections of the protective grid within the camera's field of view. Compared to a reference image captured by the camera in the undeformed state of the protective grid (for example, during calibration), a modified measurement image is obtained when a force is applied. Using appropriately configured image processing software, the acting forces can be deduced from this image change. Thus, the protective grid serves not only to protect the camera, for example, when penetrating flexible tissue structures, but also as a deformation system for force measurement.

[0012] One advantage of the invention is that it provides a simple and cost-effective solution for multidimensional force measurement at the probe head, particularly at the head of a microrobotic unit. This allows for the control of reaction forces between the probe and its environment, and enables better automation of the probe's movement control. In particular, the reaction forces between the microrobotic unit and surrounding tissue can be minimized. Simultaneously, in combination with the mobility of the microrobotic unit, tissue properties can be "detected," as feedback of the measured force action is possible, for example, to a control unit.

[0013] The invention can generally be used with different probes. It is irrelevant whether the probe serves merely for observation or whether it provides access for microtools for manipulation in cavities or on organs. According to a particularly preferred embodiment, the probe is configured as a microrobotic unit. In particular, the microrobotic unit is realized according to one of the embodiments described in WO 2024 / 013035 A1. The disclosure content of WO 2024 / 013035 A1 regarding the structure and function of the microrobotic unit is expressly incorporated here, so that a description of these details can be omitted.

[0014] According to a preferred embodiment, the probe further comprises at least one sensor for detecting its movement. Preferably, the probe incorporates multiple accelerometers and / or gyroscopes and / or magnetic field sensors for motion detection. By combining the camera-based force measurement signals with the signals from the integrated motion sensor (IMU sensor) and, optionally, the camera-based environmental images, further qualitative properties of the surrounding structures, such as tissue elasticity upon palpation, can be determined and documented. The probe can also be used for in vivo palpation, potentially revealing organ hardening or similar findings.

[0015] The protective grid on the probe head initially serves, in a known manner, to keep, for example, tissue segments away from the camera or from penetrating the probe body. According to the invention, the structure of the protective grid is adapted to fulfill the function as a three-dimensional force transducer in combination with the camera as an image acquisition unit.

[0016] According to an advantageous embodiment, the protective grille consists of a central protective ring and four support arms extending from it. The position, or force-based displacement, of the central protective ring within the camera's field of view is a measure of the forces acting from the X and Y directions. A frontal force (Z direction) acting on the protective grille causes an increase in the ring diameter (or similar structures) within the camera's field of view. A particularly preferred embodiment has two solid-state hinges on each of the preferably four support arms that carry the protective ring. The solid-state hinges cause a defined deformation of the protective grille when a force is applied (double parallelogram).

[0017] The invention thus enables the use of an elastically deformable protective grid on a probe for detecting and preferably also measuring a force by capturing an image of the protective grid using a camera mounted in the probe and performing numerical image analysis. The image analysis is preferably carried out by pattern recognition and analysis, edge detection, and similar image processing methods. As a result of real-time calculations, three force values ​​are preferably continuously transmitted digitally to a subsequent signal processing unit.

[0018] It is advantageous to calibrate the system with respect to applied reference forces before the actual force measurement. For a force measurement specific to a particular application, determining three to five reference measurement points is beneficial. Real-time measurement is then performed, for example, by determining the measured values ​​using polynomial calibration functions. In practical applications, the relative changes in magnitude with respect to different force measurement points are of less interest than the absolute force values. The determined force values ​​can be combined with the dynamic parameters for the probe's movement. From this, for example, elastic properties of the contacted material can be derived.

[0019] Preferably the measurement results are presented in a

[0020] Signal processing unit prepared and for

[0021] The data is permanently stored for documentation purposes. Depending on the possible formats, the measurement results can be displayed as numerical values ​​or as a graphical signal progression in conjunction with color coding. Further output options include the use of optical or acoustic signals when limit values ​​are exceeded, the generation of haptic signals (vibrations, force feedback) at control units, or the consideration of the measured values ​​for the (partially) autonomous motion control of the microrobotic unit.

[0022] According to a preferred embodiment, the probe body has a probe cavity with a longitudinally extending central axis. The camera is positioned in the probe cavity such that its line of sight is perpendicular (non-parallel) to the central axis. The camera's viewing direction towards the protective grid is (slightly) inclined relative to the Z-axis, so that the observation field (field of view) and the central protective ring of the grid are clearly depicted with sharp contours in the captured camera image.

[0023] In general, image analysis software can employ a variety of methods to deduce the force acting on the protective grille from its deformation. Using the aforementioned design of the protective grille with four support arms and two rigid joints each, a deflection of the protective ring in the X-direction indicates a force acting laterally in the X-direction; a deflection in the Y-direction indicates a force acting laterally in the Y-direction; and an enlarged view of the protective ring in the camera's field of view indicates a force acting in the Z-direction.

[0024] One main area of ​​application of the invention is the described

[0025] Force measurement on microrobotic units. This leads to significant advantages, for example, in the field of intracorporeal diagnostics and navigation of robotic systems, instruments, and probes. Corresponding probes can also be advantageously used in technical, geological, and biological structures. Furthermore, the invention can also be used in other fields of application, some of which are listed here:

[0026] - Extracorporeal diagnostics and medical robotics;

[0027] - Collaborative robotics, sensor-equipped gripping systems, prosthetics, rehabilitation systems;

[0028] - Industrial manufacturing, especially assembly and quality control;

[0029] - Transport systems, aviation, aerospace, security technology;

[0030] - Material and surface characterization;

[0031] - Sensor-guided additive manufacturing processes.

[0032] Further advantages and details will become apparent from the following description of preferred embodiments of the invention with reference to the drawing. The drawing shows:

[0033] Fig. 1 shows a perspective view of a probe in the form of a microrobotic unit with an elastically deformable protective grid;

[0034] Fig. 2 shows an enlarged perspective detail view of a probe head of the probe according to Fig. 1;

[0035] Fig. 3 shows a frontal view and a side view of the protective grille in detail.

[0036] Fig. 1 shows a preferred embodiment of a probe 01 according to the invention in a perspective view, comprising a probe body O la and a probe head 01b. The probe head 01b is shown again in an enlarged perspective view in Fig. 2.

[0037] In the embodiment shown in Figures 1 and 2, the probe is designed as a microrobotic unit 01, as described in detail in WO 2024 / 013035 A1, to which reference can be made for further details. The microrobotic unit 01 has a front spreading module 02 and a rear spreading module 03, which are spaced apart from each other and (in the rest state) axially aligned with each other. A thrust module 04 extends between the two spreading modules 02 and 03. A supply connection 05 is located on the rear spreading module 03, to which an axially extending hose system 06 is connected. At the front end of the unit 01, in the forward direction, is the probe head 01b, which here is frustoconical in shape and has a front opening 08.

[0038] Miniaturized tools, for example, can be guided through the hose system 06 and through the probe body 01a to the front opening 08 and exit from it.

[0039] The two spreading modules 02, 03 each have a balloon sheath 20 that can be expanded or contracted with the aid of the supply medium. The balloon sheaths 20 are each provided with a surface structuring 21, for example in the form of bristles, nubs, or cascaded structures integrated into the balloon sheath. In the expanded state, the surface structuring 21 causes anchorage in the surrounding tissue without damaging the tissue.

[0040] A protective grille 73 is attached to the probe head 01b in front of the opening 08, protecting the units behind it. This includes, in particular, a camera 74, which is positioned in the probe head 01b. Fig. 3 shows, in particular, the protective grille 73 in a front view and a side view. In addition to its conventional protective function, the protective grille 73 performs a further function according to the invention and is specifically configured for this purpose. The protective grille allows elastic deformation when forces / moments are applied and, in the illustrated embodiment, comprises four support arms 80 that carry a central protective ring 81. The protective ring can alternatively have a modified shape, for example, in the form of a polygon.The support arms 80 are preferably attached uniformly around the circumference of the probe head and extend to the protective ring 81, the main extension plane of which, in the unloaded state, is preferably perpendicular to a central axis of the probe 01.

[0041] For easier understanding, a coordinate system is shown in Fig. 2. The Z-axis lies on the central axis of the probe, which usually coincides with the main feed direction of the probe. The protective ring 81 then lies in the XY plane.

[0042] The camera 74 is positioned in the probe 01 such that the protective grid 73 lies at least partially within the camera's field of view, allowing deformations or displacements of the protective grid to be captured by the camera. For image acquisition, it is advantageous in many environments for the probe to have an illumination unit (not shown) that illuminates the camera's field of view. In particular, the protective ring 81 lies within the field of view of the camera 74.

[0043] Preferably, the support arms 80 j each have two

[0044] Solid body joints 82, so that when a force is applied to the

[0045] The protective ring 81 is capable of a double parallelogram displacement. The camera 74 detects the displacement of the protective ring 81. An image processing unit coupled to the camera determines the direction and magnitude of the acting force from the captured images. The protective grid undergoes defined deformations when subjected to multidimensional reaction forces. These deformations cause a positional displacement of the protective ring 81 and thus a change in size within the image field of the camera 74.

[0046] The method used when applying the probe enables force measurement and, in a first step, captures an image of at least part of the protective grid, in particular the protective ring. In a second step, the image is analyzed (e.g., by pattern recognition and evaluation, edge detection, etc.). As a result of real-time calculations, three force values, for example, are transmitted to a subsequent signal processing unit. The real-time measurement is then performed by determining the measured values ​​using polynomial calibration functions. In a further step, the determined force values ​​can be combined with dynamic parameters for the movement of the probe head, which can be determined by additional sensors. From this, for example, elastic properties of the contacted material can be derived.The measurement results are then processed in the signal processing unit and can be saved for documentation purposes. In a further step, the measured values ​​can be displayed as numerical values ​​or as a graphical signal waveform.

[0047] Before the actual measurement in the probe's operating state, it is advantageous to calibrate the system with respect to applied reference forces. For application-specific force measurements, determining three to five reference measurement points is advisable. Reference symbol

[0048] 01 Probe / microrobotic unit

[0049] O la probe body

[0050] 01b Probe head

[0051] 02 front spreader module

[0052] 03 rear spreader module

[0053] 04 Thrust module

[0054] 05 Supply connection

[0055] 06 Hose system

[0056] 07

[0057] 08 front opening

[0058] 20 balloon envelopes

[0059] 21 Surface texturing on the balloon envelope

[0060] 73 protective grilles

[0061] 74 Micro camera

[0062] 80 support arms

[0063] 81 Protective ring

[0064] 82 solid body joints

Claims

Patent claims 1. Probe ( 01 ) with : - a probe body (Ola) ; - a probe head (01b) ; - a camera (74) ; - a protective grid (73) attached to the probe head; characterized in that the probe comprises a force detection arrangement formed by the protective grid (73), which is at least partially arranged in the field of view of the camera (74) and is elastically deformable when a force is applied, wherein the resulting deformation of the protective grid (73) can be detected image-wise by the camera (74).

2. Probe (01) according to claim 1, characterized in that the probe body (Ola) has a probe cavity with a longitudinally extending central axis, and that the camera (74) is positioned in the probe cavity such that its viewing axis is not parallel to the central axis.

3. Probe (01) according to claim 1 or 2, characterized in that the protective grid (73) has several support arms (80) distributed around the circumference of the probe head (01b), and that the support arms carry a protective ring (81) which lies in the field of view of the camera (74) and whose center point is preferably coaxial with the central axis of the probe body (01a).

4. Probe (01) according to claim 3, characterized in that the protective ring (81) is supported by at least four support arms (80), and that two solid body joints (82) are arranged on each of the support arms to allow the protective ring (81) to be displaced according to the principle of the double parallelogram when force is applied.

5. Probe (01) according to one of claims 1 to 4, characterized in that it further comprises at least one sensor for detecting the movement of the probe, preferably selected from the list: accelerometers, Gyro sensors, magnetic field sensors.

6. Probe (01) according to one of claims 1 to 5, characterized in that it is designed as a microrobotic unit for locomotion and positioning in organic cavities.

7. Probe (01) according to one of claims 1 to 6, characterized in that it comprises a lighting means which illuminates the field of view of the camera (74).

8. Use of an elastically deformable protective grid (73) attached to a microrobotic unit (01) and a camera (74) arranged in the microrobotic unit for detecting a force acting on the protective grid (73) by optically detecting the deformation of the protective grid with the camera and supplying the detected image to image processing software which derives a Force signal determined.

9. Method for determining a force acting on an elastically deformable protective grid (73) on a microrobotic unit (01), comprising the following steps: - Capturing a reference image of the protective grid (73) in its undeformed state with a camera (74) located in the microrobotic unit; - Capturing a measurement image of the protective grid (73) deformed by force with the camera (74) ; - Determining the acting force by comparing the measured image with the reference image using a Image processing software.

Images