A method for instrument tip detection

Abstract

This invention relates to a method for detecting the end effector of an instrument. The method involves receiving images captured during the instrument's operation, the images being sorted according to the instrument's operational process; extracting the instrument from the images, including the tube body and the actuator; identifying the position of the actuator's tip in each image; obtaining the position information of the actuator's tip and end effector in a preset image; obtaining the tilt angle θ0 of the instrument in the preset image; calculating the tilt angle θ of each image; and obtaining the end effector position P of the actuator in each image based on the position of the actuator's tip in the image. end It enables the estimation of the position of an invisible needle tip, which can be used to detect data such as needle tip vibration and depth.

Description

Technical Field

[0001] This invention relates to the field of image recognition, and more specifically, to a method for detecting the end effector of an instrument. Background Technology

[0002] For retinal surgery robots, data such as instrument tremors and depth are crucial safety indicators, and obtaining these indicators helps compare the performance of the robot with that of the human hand. To obtain these indicators for testing, the position of the instrument's actuator tip is needed. However, because the actuator tip is located deep beneath the retina, it is invisible. Existing needle tip detection methods primarily focus on detecting the target, mainly using convolutional neural networks and spatiotemporal frames to detect the needle tip. Related technologies include: using convolutional neural networks to identify surgical needles in the presence of blur or occlusion; or using spatiotemporal frames to detect needle tip displacement and thus obtain needle positioning. These methods all require detecting the visible position of the actuator tip to achieve detection, therefore they cannot be used when the position information of the actuator tip cannot be directly obtained, and thus cannot detect instrument tremors and depth. Summary of the Invention

[0003] To overcome the problem in the prior art that the position of the actuator end cannot be detected when the actuator end is invisible, the present invention provides a method for instrument end detection, which can estimate the position of the invisible actuator end, and then use it to obtain the actuator end vibration and depth.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: a method for detecting the end effector of an instrument, comprising the following steps:

[0005] Step 1: Receive several images captured during the working process of the instrument, and arrange the images according to the working process of the instrument;

[0006] Step 2: Use a neural network to extract the instruments from the image. The extraction method is to use the neural network to perform semantic segmentation on the instruments and extract them from the image. The instruments include tubes and actuators installed on the tubes.

[0007] Step 3: Identify the position A(x) of the beginning of the actuator in each image. a ,y a ); Obtain the position information of the beginning and end of the actuator in the preset image, and then obtain the pixel length l of the actuator. pThe preset image is any one of the first N images, along with the slope k0. The tilt angle θ0 of the device in the preset image is obtained. During operation, the movement of the actuator tip gradually deepens. After reaching a certain depth, the actuator tip cannot be accurately identified. However, before reaching a certain depth, the position information of the actuator tip can be identified. Therefore, any one of the first N images where the actuator tip can still be identified is selected to obtain the position information of the actuator tip and the actuator end. The smaller the value of N, the better. The position information is specifically coordinate points. After obtaining two coordinate points, the distance between the two coordinate points can be calculated, which is the pixel length l of the actuator. p The tilt angle θ0 of the instrument is essentially the angle between the instrument and the camera's shooting plane when the instrument is working. This angle will change as the instrument moves. It can be obtained through external hardware that controls the movement of the instrument, or it can be obtained through image recognition.

[0008] Step 4: For each image, perform the following processing: fit a straight line to the outer contour points on both sides of the pipe axis, which are respectively the first pipe contour line and the second pipe contour line; obtain the distance dis from the intersection of the first pipe contour line and the second pipe contour line to the beginning of the pipe in each image other than the preset image. cp The distance between the intersection of the first and second pipe contour lines in the preset image and the beginning of the pipe is compared. cp0 The tilt angle θ in each image is estimated by combining the cotangent and arctangent functions, specifically as follows:

[0009] θ = arctan(1 / (cotan(θ0) / dis) cp0 ·dis cp ))

[0010] During the instrument's movement, vibrations or other factors can cause changes in its tilt angle. These changes affect the estimation of the actuator's end effector position. Therefore, by first calculating the instrument's tilt angle in each image, and then calculating the actuator's end effector position P in each image, a more precise calculation is made. end The obtained position information of the end position of the actuator is more accurate.

[0011] Step 5: Based on the information obtained in Step 4, obtain the end position P of the actuator in each image. end Specifically:

[0012]

[0013] Preferably, in step three, the process of obtaining the instrument tilt angle in the preset image is as follows:

[0014] S3.1: Obtain the actual length l of the actuator mapped onto the plane in the preset image. rm0 ;

[0015] S3.2: Obtain the actual length l of the actuator. r ;

[0016] S3.3: Calculate the tilt angle θ0, specifically:

[0017]

[0018] Based on the projection relationship, the actual length of the actuator corresponds to the hypotenuse of a triangle, and the length of the actuator mapped onto the plane corresponds to the adjacent side of the triangle. The tilt angle is obtained through this trigonometric relationship. This method is achieved through image recognition, which avoids the need to add hardware to the instrument compared to obtaining it through external hardware. Since surgical instruments are delicate and precise instruments, adding external hardware will affect the movement accuracy of the instrument, requiring additional calculation compensation, and the adverse effects on the instrument are significant.

[0019] Preferably, the specific process of step S3.1 is as follows:

[0020] Obtain any two non-coincident midpoints of the tube body and the midpoint of the actuator in the preset image;

[0021] Calculate the pixel distance between the diameters of the tube at the two midpoints; obtain the actual size of the instrument diameter at the two midpoints;

[0022] The actual pixel ratio at the two midpoints of the tube is calculated based on the pixel distance of the tube diameter and the actual size of the instrument diameter; the actual pixel ratio refers to the ratio of the actual distance to the pixel distance.

[0023] Based on the actual pixel ratio at the two midpoints and the pixel distances from the midpoint of the actuator to the two midpoints, the actual pixel ratio r at the midpoint of the actuator is obtained. m ;

[0024] In practice, the actuating element has a length l mapped to the plane. rm0 The actual pixel ratio r at the midpoint of the actuator. m The pixel length l of the actuator o The product of.

[0025] Preferably, the positions of the first and last ends of the actuator, the midpoint of the actuator, and the two non-coincident midpoints of the tube body are obtained by marking the outer contour points of the device. By extracting the device through semantic segmentation, the outer contour points of the device can be obtained, and the required key points can be marked or calculated based on these outer contour points.

[0026] Preferably, the method for obtaining the two midpoints of the tube body is as follows:

[0027] The skeleton of the axis is extracted from the tube body using the skeleton extraction method, and then the skeleton is fitted into a straight line as the axis of the tube body;

[0028] Select any two auxiliary points on the tube frame. Based on the positions of the two auxiliary points and the vertical slope perpendicular to the tube axis, obtain the first intersection line and the second intersection line that intersect the first tube outline and the second tube outline, respectively. The farther apart the auxiliary points are, the more accurate the result will be. If the two auxiliary points are too close, the error will be too large. For example, a distance of 5 pixels will result in an error of 1 pixel, while a distance of 50 pixels will also result in an error of only 1 pixel.

[0029] The first intersecting line intersects with the first pipe body outline and the second pipe body outline respectively, and obtains the first intersection point and the second intersection point;

[0030] The second intersecting line intersects with the first pipe body outline and the second pipe body outline respectively, and obtains the third intersection point and the fourth intersection point;

[0031] The midpoint between the first and second intersection points is taken as one of the midpoints of the tube, and the third and fourth intersection points are taken as the other midpoint of the tube.

[0032] Preferably, the position of the start end of the actuator is obtained as follows:

[0033] Calculate the projection points of the contour points on the axis of the tube body on both sides of the connection with the needle tip to obtain a number of contour point projection points;

[0034] Calculate the mean of all the projection points of the contour points to obtain the root point of the needle tip, which is the position of the head end of the actuator.

[0035] Preferably, the preset image is the first image.

[0036] Preferably, in step one, the device's working process video is received and each frame in the video is broken down to obtain several images.

[0037] An instrument end detection system for implementing the above-described method for instrument end detection includes an image processing module for receiving and processing instrument images and an estimation module for calculating the needle tip position.

[0038] Preferably, the image processing module includes a semantic segmentation and extraction module and a key point annotation module; the semantic segmentation and extraction module is used to decompose the video into image frames and extract the instrument contours from the image frames; the key point annotation module is used to extract key points based on the instrument contours and annotate them.

[0039] A computer-readable storage medium for storing a computer program that, when executed by a processor, implements the above-described method for detecting instrument tremors.

[0040] Compared with existing technologies, the beneficial effects of this invention are: by combining images of the instrument's usage process with the instrument's actual parameters, the position of the invisible needle tip under the retina can be inferred, thereby accurately detecting data such as needle tip tremors and depth. This allows the stability and safety of the instrument during retinal surgery to be verified, contributing to improved stability and safety of the surgical robot. Furthermore, compared to other existing needle tip positioning methods, this method allows for the selection of microneedles with smaller diameters, sometimes even one-tenth or less than the diameter of other needles. Attached Figure Description

[0041] Figure 1 This is a flowchart of a device tremor detection method according to the present invention;

[0042] Figure 2 This is a schematic diagram of the tilt angle θ0 of the present invention. Detailed Implementation

[0043] The accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. To better illustrate this embodiment, some components in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings. The positional relationships described in the drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0044] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "long," and "short" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0045] The technical solution of the present invention will be further described in detail below through specific embodiments and in conjunction with the accompanying drawings:

[0046] Example 1

[0047] like Figure 1 The following is an embodiment of a method for detecting instrument tremor, comprising the following steps:

[0048] Step 1: Receive several images captured during the working process of the instrument, and arrange the images according to the working process of the instrument;

[0049] Step 2: Use a neural network to extract the instruments from the image. The extraction method is to use the neural network to perform semantic segmentation on the instruments and extract them from the image. The instruments include tubes and actuators installed on the tubes.

[0050] Step 3: Identify the position A(x) of the beginning of the actuator in each image. a ,y a ); Obtain the position information of the beginning and end of the actuator in the preset image, and then obtain the pixel length l of the actuator. p The preset image is any one of the first N images, along with the slope k0. The tilt angle θ0 of the device in the preset image is obtained. During operation, the movement of the actuator tip gradually deepens. After reaching a certain depth, the actuator tip cannot be accurately identified. However, before reaching a certain depth, the position information of the actuator tip can be identified. Therefore, any one of the first N images where the actuator tip can still be identified is selected to obtain the position information of the actuator tip and the actuator end. The smaller the value of N, the better. In this embodiment, the preset image is the first image. In this embodiment, the actuator tip needs to be obtained by projecting the point where the tube and actuator connect onto the axis of the tube and taking the average point. The position information is specifically coordinate points. After obtaining two coordinate points, the distance between the two coordinate points can be calculated, which is the pixel length l of the actuator. p And the slope k0. In this embodiment, the tilt angle θ0 of the instrument in the preset image is calculated by combining the ratio of the actual length to the pixel length and the actual length of the actuator to calculate the pixel length of the actuator, and then using the inverse cosine function to derive the tilt angle θ0 of the instrument in the first frame. The specific process is as follows:

[0051] The process for obtaining the instrument tilt angle in a preset image is as follows:

[0052] S3.1: Obtain the length k of the actuator mapped to the plane in practice. rm0 The specific process is as follows:

[0053] Obtain any two non-coincident midpoints of the tube in the preset image, namely P. ef and P gh and the midpoint P of the actuator. ab ;

[0054] Calculate the pixel distance between the tube diameters at the two midpoints. and Obtain the actual diameter φ of the instrument body at the two midpoints. r ;

[0055] Calculate the actual pixel ratio at the two midpoints of the tube based on the pixel distance of the tube diameter and the actual size of the instrument diameter;

[0056] Based on the actual pixel ratio at the two midpoints and the pixel distances from the midpoint of the actuator to the two midpoints, the actual pixel ratio r at the midpoint of the actuator is obtained. m ;

[0057] In practice, the actuating element has a length l mapped to the plane. rm0 The actual pixel ratio r at the midpoint of the actuator. m The pixel length l of the actuator o The product of l rm0 =l p ·r m ;

[0058] S3.2: Obtain the actual length l of the actuator. r ;

[0059] S3.3: Calculate the tilt angle θ0, specifically:

[0060]

[0061] According to the projection relationship, such as Figure 2 As shown, the actual length of the actuator corresponds to the hypotenuse of a triangle, and the length of the actuator mapped onto the plane corresponds to the adjacent side of the triangle. The tilt angle is obtained through trigonometric relationships. This method is achieved through image recognition, which avoids the need to add hardware to the instrument compared to obtaining it through external hardware. Since surgical instruments are delicate and precise instruments, adding external hardware will affect the movement accuracy of the instrument, requiring additional calculation compensation, and the adverse effects on the instrument are significant.

[0062] Step 4: For each image, perform the following processing: fit a straight line to the outer contour points on both sides of the pipe axis, which are respectively the first pipe contour line and the second pipe contour line; obtain the distance dis from the intersection of the first pipe contour line and the second pipe contour line to the beginning of the pipe in each image other than the preset image. cp The distance between the intersection of the first and second pipe contour lines in the preset image and the beginning of the pipe is compared. cp0 The tilt angle θ in each image is estimated by combining the cotangent and arctangent functions, specifically as follows:

[0063] θ = arctan(1 / (cotan(θ0) / dis) cp0 ·dis_cp))

[0064] During the instrument's movement, vibrations or other factors can cause changes in its tilt angle. These changes affect the estimation of the actuator's end effector position. Therefore, by first calculating the instrument's tilt angle in each image, and then calculating the actuator's end effector position P in each image, a more precise calculation is made. end The obtained position information of the end position of the actuator is more accurate.

[0065] Step 5: Based on the information obtained in Step 4, obtain the end position P of the actuator in each image. end Specifically:

[0066]

[0067] In this embodiment, the positions of the first and last ends of the actuator, the midpoint of the actuator, and the two non-coincident midpoints of the tube body are obtained by annotating the outer contour points of the device. The device is extracted through semantic segmentation, yielding its outer contour points. Based on these outer contour points, the required key points are annotated or calculated. The position of the last end of the actuator is clear in the first N images, especially the first image, and can therefore be directly annotated using contour points. The position B(x) of the last end of the actuator is obtained by annotating the outer contour points. b ,y b The remaining locations were obtained through calculation. The calculation is as follows:

[0068] The skeleton of the axis is extracted from the tube body using the skeleton extraction method, and then the skeleton is fitted into a straight line as the axis of the tube body;

[0069] Select any two auxiliary points on the tube frame, and obtain the first intersection line and the second intersection line that intersect the first tube outline and the second tube outline respectively based on the position of the two auxiliary points and the vertical slope perpendicular to the tube axis.

[0070] The first intersecting line intersects with the first pipe body outline and the second pipe body outline respectively, and obtains the first intersection point E(x). e ,y e ) and the second intersection point F(x) f ,y f );

[0071] The second intersection line intersects the first pipe body outline and the second pipe body outline respectively, and obtains a third intersection point G(x). g ,y g ) and the fourth intersection point H(x) h ,y h );

[0072] The midpoint between the first and second intersection points is taken as one of the midpoints P of the tube. ef The third and fourth intersection points are taken as the other midpoint P of the tube body. gh .

[0073]

[0074]

[0075] The above two needle pixel distances Specifically:

[0076]

[0077]

[0078] The actual pixel ratio at the two midpoints is as follows:

[0079]

[0080]

[0081] The specific method for obtaining the position of the start end of the actuator is as follows:

[0082] Calculate the projection points of the contour points on the axis of the tube body on both sides of the connection with the needle tip to obtain a number of contour point projection points;

[0083] Calculate the mean of all the projection points of the contour points to obtain the root point of the needle tip, which is the position A(x) of the head end of the actuator. a ,y a ).

[0084] The midpoint P of the aforementioned actuator ab for The midpoint of the actuator is used to calculate the pixel ratio, so the point used is taken from the preset image.

[0085] The actual pixel ratio r at the midpoint of the actuator m for:

[0086]

[0087] The beneficial effects of this embodiment are as follows: By combining images of the instrument's usage process with the instrument's actual parameters, the position of the invisible needle tip under the retina can be inferred. This allows for the detection of data such as needle tip tremors and depth, enabling the verification of the instrument's stability and safety during retinal surgery and contributing to improved stability and safety of the surgical robot. Furthermore, compared to other existing needle tip positioning methods, this method allows for the selection of microneedles with smaller diameters, sometimes as small as one-tenth or less than the diameter of other needles.

[0088] Example 2

[0089] An instrument end-of-care device (IAV) detection system for implementing the aforementioned method for IAV end-of-care device detection, comprising an image processing module for receiving and processing IAV images and an estimation module for calculating the needle tip position, wherein the image processing module includes a semantic segmentation and extraction module and a key point annotation module; the semantic segmentation and extraction module is used to decompose a video into individual images and extract the IAV contour from the image frames; the key point annotation module is used to extract and annotate key points based on the IAV contour.

[0090] In this embodiment, a video is input to the image processing module, and the semantic segmentation and extraction module decomposes the video into individual images, extracting the instrument contour in each image frame. The key point annotation module annotates the following information: the position A(x) of the actuator's head in each image. a ,y a The preset image contains the position information of the beginning and end of the actuator, the first tube outline, the second tube outline, any two non-overlapping midpoints of the tube in the preset image, the tube axis, and the outline points on both sides of the connection between the tube and the needle tip.

[0091] The estimation module is used to calculate the pixel length l of the actuator. p and slope k0, tilt angle θ0 of the instrument in the preset image, intersection of the first tube contour line and the second tube contour line in each image other than the preset image, intersection of the first tube contour line and the second tube contour line in the preset image, and distance dis cp Distance comparison cp0 Tilt angle θ, actuator end position P end The actual length l of the actuator mapped onto the plane in the preset image rm0 The actual pixel ratio r at the midpoint of the row element m Contour point projection point and needle tip root point.

[0092] Example 3

[0093] A computer-readable storage medium for storing a computer program that, when executed by a processor, implements the instrument tremor detection method of Embodiment 1 or Embodiment 2.

[0094] The remaining features and working principles of this embodiment are the same as those of Embodiment 1.

[0095] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for detecting the end effector of an instrument, characterized in that, Includes the following steps: Step 1: Receive several images captured during the working process of the instrument, and arrange the images according to the working process of the instrument; Step 2: Use a neural network to extract the device from the image, the device including a tube and an actuator mounted on the tube; Step 3: Identify the position of the start of the actuator in each image. ; Obtain the position information of the beginning and end of the actuator in the preset image, and then obtain the pixel length of the actuator in the preset image. and slope The preset image is any one of the first N images that can be recognized at the end of the actuator; the tilt angle of the instrument in the preset image is obtained. ; Step 4: For each image, perform the following processing: fit a straight line to the outer contour points on both sides of the pipe axis, which are respectively the first pipe contour line and the second pipe contour line; obtain the distance from the intersection of the first pipe contour line and the second pipe contour line to the beginning of the pipe in each image except the preset image. The distance between the intersection of the first tube outline and the second tube outline in the preset image and the beginning of the tube is compared. The tilt angle in each image is estimated by combining the cotangent and arctangent functions. Specifically: Step 5: Based on the information obtained in Step 4, determine the end position of the actuator in each image. Specifically: 。 2. The method for detecting the end of an instrument according to claim 1, characterized in that, In step three, the process for obtaining the instrument tilt angle in the preset image is as follows: S3.1: Obtain the actual length of the actuator mapped to the plane in the preset image. ; S3.2: Obtain the actual length of the actuator. ; S3.3: Calculate the tilt angle Specifically: 。 3. The method for detecting the end of an instrument according to claim 2, characterized in that, The specific process of step S3.1 is as follows: Obtain any two non-coincident midpoints of the tube body and the midpoint of the actuator in the preset image; Calculate the pixel distance between the diameters of the tube at the two midpoints; obtain the actual size of the instrument diameter at the two midpoints; Calculate the actual pixel ratio at the two midpoints of the tube based on the pixel distance of the tube diameter and the actual size of the instrument diameter; The actual pixel ratio at the midpoint of the actuator is obtained based on the actual pixel ratio at the two midpoints and the pixel distances from the midpoint of the actuator to the two midpoints. ; In practice, the length of the actuator mapped to the plane It is the actual pixel ratio at the midpoint of the actuator. Pixel length of the actuator The product of.

4. The method for instrument tip detection according to claim 3, characterized in that, The positions of the first and last ends of the actuator, the midpoint of the actuator, and the two non-coincident midpoints of the tube body are obtained by marking the outer contour points of the instrument.

5. The method for detecting the end effector of an instrument according to claim 3, characterized in that, The specific method for obtaining the two non-coincident midpoints of the tube body is as follows: The skeleton of the axis is extracted from the tube body using the skeleton extraction method, and then the skeleton is fitted into a straight line as the axis of the tube body; Select any two auxiliary points on the tube frame, and obtain the first intersection line and the second intersection line that intersect the first tube outline and the second tube outline respectively based on the position of the two auxiliary points and the vertical slope perpendicular to the tube axis. The first intersecting line intersects with the first pipe body outline and the second pipe body outline respectively, and obtains the first intersection point and the second intersection point; The second intersecting line intersects with the first pipe body outline and the second pipe body outline respectively, and obtains the third intersection point and the fourth intersection point; The midpoint between the first and second intersection points is taken as one of the midpoints of the tube, and the third and fourth intersection points are taken as the other midpoint of the tube.

6. The method for detecting the end of an instrument according to claim 3, characterized in that, The specific method for obtaining the position of the start end of the actuator is as follows: Calculate the projection points of the contour points on the axis on both sides of the connection between the tube body and the needle tip to obtain a number of contour point projection points; Calculate the mean of all the projection points of the contour points to obtain the root point of the needle tip, which is the position of the head end of the actuator.

7. The method for detecting the end of an instrument according to claim 3, characterized in that, The preset image is the first image.

8. An instrument end detection system for implementing the method for instrument end detection according to any one of claims 1-7, characterized in that, It includes an image processing module for receiving and processing instrument images and an estimation module for calculating the needle tip position.

9. The instrument end detection according to claim 8, characterized in that, The image processing module includes a semantic segmentation and extraction module and a key point annotation module; the semantic segmentation and extraction module is used to decompose the video into image frames and extract the instrument outlines from the image frames; the key point annotation module is used to extract key points based on the instrument outlines and annotate them.

10. A computer-readable storage medium for storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for instrument end detection as described in any one of claims 1-7.

Images