Attitude estimation method, system and equipment of fundus surgical needle and navigation positioning method

Abstract

The invention relates to the technical field of computer medical treatment, in particular to a posture estimation method, system and device of a fundus surgical needle and a navigation positioning method.The posture estimation method comprises the steps that at the current moment, after a two-dimensional RGB image, obtained through camera shooting, of the fundus surgical needle is processed, and a posture of the fundus surgical needle is obtained; obtaining a foreground mask by creating a two-dimensional mask, extracting a plurality of connected component sequences in the foreground mask, and matching the plurality of connected component sequences with the plurality of three-dimensional reference point sequences to obtain a plurality of matching pairs; and for each matching pair, performing iteration by adopting an RANSAC algorithm and a PnP algorithm to obtain optimal attitude estimation information, and obtaining attitude estimation information of the fundus surgical needle at the current moment according to the optimal attitude estimation information. The posture of the fundus operation needle can be accurately estimated in real time, an operator is helped to make accurate judgment and adjustment, and therefore the safety and success rate of the fundus operation are improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer medical technology, and in particular to a method, system and device for estimating the posture of a fundus surgical needle, and a navigation and positioning method. Background Art

[0002] In fundus surgery, accurate positioning and navigation of the needle tip are crucial to successful surgery. During the operation, changes in the posture of the surgical needle may affect the safety and effectiveness of the operation. By continuously monitoring the posture of the surgical needle, the position and angle of the surgical needle relative to the target area (such as the fundus lesion site) can be calculated more accurately, and possible wrong operations can be discovered and corrected in time, thereby performing more precise surgical operations and reducing surgical risks.

[0003] Due to the complex structure of the fundus and the narrow surgical space, traditional positioning methods usually rely on manual operation or simple mechanical devices, which are difficult to meet the needs of high precision and high efficiency. Therefore, there is an urgent need for automated and intelligent positioning of fundus surgical needles. Summary of the invention

[0004] In order to solve the problems in the related art, the embodiments of the present disclosure provide a method, system and device for estimating the posture of a fundus surgical needle, and a navigation and positioning method.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for estimating the posture of a fundus surgical needle, wherein a positioning area is provided on the fundus surgical needle, wherein the positioning area includes a plurality of positioning patterns, and the positions of the plurality of positioning patterns are used to determine a plurality of three-dimensional reference points of the fundus surgical needle, wherein the plurality of three-dimensional reference points are divided into a plurality of three-dimensional reference point sequences, and an area other than the plurality of positioning patterns in the positioning area has a specified color, wherein the method includes:

[0006] At the current moment, a two-dimensional RGB image of the fundus surgical needle captured by a camera is acquired;

[0007] Convert the two-dimensional RGB image into a two-dimensional HSV image;

[0008] Creating a two-dimensional mask using a set color threshold, and segmenting the two-dimensional HSV image using the two-dimensional mask to obtain a foreground mask, wherein the color threshold is related to the specified color;

[0009] Extracting multiple connected components in the foreground mask, dividing the multiple connected components, and obtaining multiple connected component sequences corresponding to the multiple three-dimensional reference point sequences respectively;

[0010] For each connected component sequence in the plurality of connected component sequences, calculate the center coordinates of each connected component in the camera coordinate system and set label information, and create a feature list of each connected component in the connected component sequence, wherein the feature list includes the center coordinates and label information of the corresponding connected component;

[0011] According to the feature list of each connected component in the multiple connected component sequences, one-to-one matching is performed between each three-dimensional reference point in the three-dimensional reference point sequence and the central coordinate of each connected component in the corresponding connected component sequence to obtain multiple matching pairs, each matching pair including a three-dimensional reference point and a central coordinate;

[0012] For the multiple matching pairs, a random sampling consistency RANSAC algorithm is used to perform multiple RANSAC iterations. In each RANSAC iteration process, a preset number of matching pairs are randomly selected from the multiple matching pairs, and a perspective n-point PnP algorithm is used to perform camera pose estimation on the preset number of matching pairs and calculate the number of inliers, and the camera pose estimation result obtained in the iteration process with the largest number of inliers is used as the pose information of the camera relative to the fundus surgical needle;

[0013] By inverse transformation of the camera attitude, the attitude information of the camera relative to the fundus surgical needle is converted into attitude estimation information of the fundus surgical needle in the world coordinate system at the current moment.

[0014] According to an embodiment of the present disclosure, each of the connected component sequences includes some of the connected components among the plurality of connected components;

[0015] Each 3D reference point sequence includes some 3D reference points in the plurality of 3D reference points.

[0016] According to an embodiment of the present disclosure, converting the two-dimensional RGB image into a two-dimensional HSV image includes implementing image conversion using the following formula:

[0017]

[0018] Among them, H represents hue, R represents red brightness value, G represents green brightness value, B represents blue brightness value, max(R,G,B) is the maximum value among red brightness value, green brightness value and blue brightness value, min(R,G,B) is the minimum value among red brightness value, green brightness value and blue brightness value.

[0019] According to an embodiment of the present disclosure, each connected component sequence is set with a preset range, and the method further includes:

[0020] Checking whether the connected components in each connected component sequence are within a corresponding preset range;

[0021] If so, the connected component sequence is judged to be a valid connected component sequence;

[0022] If not, the connected component sequence is determined to be an abnormal connected component sequence, and the abnormal connected component sequence is not used for camera pose estimation.

[0023] According to an embodiment of the present disclosure, each connected component sequence is provided with a preset threshold of the number of inliers, and the method further comprises:

[0024] Count the number of inliers in each connected component sequence;

[0025] Checking whether the number of inliers in each connected component sequence is within a corresponding preset inlier number threshold;

[0026] If so, the connected component sequence is judged to be a valid connected component sequence;

[0027] If not, the connected component sequence is determined to be an abnormal connected component sequence, and the abnormal connected component sequence is not used for camera pose estimation.

[0028] According to an embodiment of the present disclosure, the RANSAC algorithm has a set number of iterations;

[0029] The using the perspective n-point PnP algorithm to perform camera pose estimation on the preset number of matching pairs and calculate the number of interior points includes:

[0030] Use the PnP algorithm to perform camera pose estimation on the preset number of matching pairs to obtain corresponding camera pose estimation results;

[0031] Projecting the three-dimensional reference points in the remaining matching pairs of the multiple matching pairs except the preset number of matching pairs into the camera coordinate system using the corresponding camera pose estimation results to obtain a plurality of three-dimensional reprojection results;

[0032] Comparing the multiple three-dimensional reprojection results with the center coordinates of the remaining matching pairs to obtain a reprojection error of each matching pair in the remaining matching pairs;

[0033] The matching pairs whose reprojection error is less than the reprojection error threshold are regarded as inliers.

[0034] According to an embodiment of the present disclosure, the method further includes:

[0035] Acquire positional relationship information between the positioning area and the needle tip of the fundus surgery needle in a world coordinate system;

[0036] The estimated posture information of the needle tip in the world coordinate system is located according to the positional relationship information and the estimated posture information of the fundus surgery needle in the world coordinate system at the current moment.

[0037] In a second aspect, the present disclosure provides a navigation and positioning method for a fundus surgical needle, the method comprising:

[0038] At the current moment, a two-dimensional RGB image of the fundus surgical needle captured by a camera is acquired;

[0039] Convert the two-dimensional RGB image into a two-dimensional HSV image;

[0040] Creating a two-dimensional mask using a set color threshold, and segmenting the two-dimensional HSV image using the two-dimensional mask to obtain a foreground mask;

[0041] Extracting multiple connected components in the foreground mask, dividing the multiple connected components, and obtaining multiple connected component sequences corresponding to multiple three-dimensional reference point sequences respectively;

[0042] For each connected component sequence in the plurality of connected component sequences, calculate the center coordinates of each connected component in the camera coordinate system and set label information, and create a feature list of each connected component in the connected component sequence, wherein the feature list includes the center coordinates and label information of the corresponding connected component;

[0043] According to the feature list of each connected component in the plurality of connected component sequences, one-to-one matching is performed between each three-dimensional reference point in the three-dimensional reference point sequence and the central coordinate of each connected component in the corresponding connected component sequence to obtain a plurality of matching pairs, each matching pair including a three-dimensional reference point and a central coordinate;

[0044] For the multiple matching pairs, a random sampling consistency RANSAC algorithm is used to perform multiple RANSAC iterations. In each RANSAC iteration process, a preset number of matching pairs are randomly selected from the multiple matching pairs, and a PnP algorithm is used to perform camera pose estimation on the preset number of matching pairs and calculate the number of inliers, and the camera pose estimation result obtained in the iteration process with the largest number of inliers is used as the pose information of the camera relative to the fundus surgical needle;

[0045] By inverse transformation of the camera attitude, converting the attitude information of the camera relative to the fundus surgical needle into the attitude estimation information of the fundus surgical needle in the world coordinate system at the current moment;

[0046] Repeat the above steps for any moment after the current moment to obtain real-time posture estimation information of the fundus surgery needle, thereby achieving navigation positioning of the fundus surgery needle;

[0047] The real-time posture estimation information of the fundus surgical needle is sent to a display screen for display.

[0048] In a third aspect, the present disclosure provides a system for estimating the posture of a fundus surgical needle, comprising:

[0049] An image acquisition module is configured to acquire, at a current moment, a two-dimensional RGB image of the fundus surgical needle captured by a camera;

[0050] The image preprocessing module is configured to convert the two-dimensional RGB image into a two-dimensional HSV image, create a two-dimensional mask using a set color threshold, and use the two-dimensional mask to segment the two-dimensional HSV image to obtain a foreground mask; extract multiple connected components in the foreground mask, divide the multiple connected components, obtain multiple connected component sequences corresponding to multiple three-dimensional reference point sequences respectively, calculate the center coordinates of each connected component in the camera coordinate system for each connected component sequence in the multiple connected component sequences and set label information, and create a feature list of each connected component in the connected component sequence, wherein the feature list includes the center coordinates and label information of the corresponding connected component;

[0051] a feature matching module configured to perform one-to-one matching between each three-dimensional reference point in the three-dimensional reference point sequence and the center coordinate of each connected component in the corresponding connected component sequence according to the feature list of each connected component in the plurality of connected component sequences, so as to obtain a plurality of matching pairs, each matching pair including a three-dimensional reference point and a center coordinate;

[0052] a posture calculation module, configured to perform multiple RANSAC iterations for the multiple matching pairs using a random sampling consistency RANSAC algorithm, randomly select a preset number of matching pairs from the multiple matching pairs in each RANSAC iteration process, use a PnP algorithm to perform camera posture estimation on the preset number of matching pairs and calculate the number of inliers, and use the camera posture estimation result obtained in the iteration process with the largest number of inliers as the posture information of the camera relative to the fundus surgical needle;

[0053] The posture estimation module is configured to convert the posture information of the camera relative to the fundus surgical needle into posture estimation information of the fundus surgical needle in the world coordinate system at the current moment through inverse transformation of the camera posture.

[0054] In a fourth aspect, an embodiment of the present disclosure provides a device for estimating the posture of a fundus surgical needle, comprising: a camera, a computing device, and a microscope, wherein the camera is communicatively connected to the computing device, and the computing device is communicatively connected to the microscope; wherein:

[0055] The microscope is used to display the surgical area of ​​the fundus surgical needle;

[0056] The camera is used to obtain a two-dimensional RGB image of the fundus surgical needle and send the two-dimensional RGB image to the computing device;

[0057] The computing device is used to obtain posture estimation information of the fundus surgical needle based on the two-dimensional RGB image, wherein the computing device includes a memory and a processor, the memory is used to store computer instructions, and the computer instructions are executed by the processor to implement the method as described in any one of the first aspect or the second aspect.

[0058] According to the technical solution provided by the embodiment of the present disclosure, at the current moment, after preprocessing the two-dimensional RGB image of the fundus surgical needle obtained by camera shooting, a foreground mask is obtained by creating a two-dimensional mask, and multiple connected components in the obtained foreground mask are extracted, and the multiple connected components are divided into connected component sequences corresponding to multiple three-dimensional reference point sequences, and then each three-dimensional reference point in each three-dimensional reference point sequence is matched with the center coordinates of each connected component in the corresponding connected component sequence to obtain multiple matching pairs; for each matching pair, the RANSAC algorithm and the PnP algorithm are used for iteration to obtain the best posture estimation information, and the posture estimation information of the fundus surgical needle at the current moment is obtained according to the best posture estimation information. The present disclosure can automatically and accurately locate the posture of the fundus surgical needle in real time, and the entire processing flow is real-time, efficient, and highly automated, thereby helping operators make accurate judgments and adjustments, thereby improving the safety and success rate of the operation.

[0059] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Other features, objectives and advantages of the present disclosure will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0061] Figure 1 A flowchart showing a method for estimating the posture of a fundus surgical needle according to an embodiment of the present disclosure is shown;

[0062] Figure 2 A cross-sectional view of a fundus surgery needle according to an embodiment of the present disclosure is shown;

[0063] Figure 3 A stereoscopic diagram showing a fundus surgery needle according to an embodiment of the present disclosure;

[0064] Figure 4A schematic diagram showing division of multiple connected components in a foreground mask according to an embodiment of the present disclosure is shown;

[0065] Figure 5 A logic flow chart showing a method for estimating the posture of a fundus surgical needle according to an embodiment of the present disclosure is shown;

[0066] Figure 6 A flowchart showing a navigation and positioning method for a fundus surgical needle according to an embodiment of the present disclosure is shown;

[0067] Figure 7 A structural block diagram of a fundus surgery needle posture estimation system according to an embodiment of the present disclosure is shown;

[0068] Figure 8 A structural block diagram of a fundus surgery needle posture estimation device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0069] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement them. In addition, for the sake of clarity, parts not related to the description of the exemplary embodiments are omitted in the accompanying drawings.

[0070] In the present disclosure, it should be understood that terms such as "include" or "have" are intended to indicate the presence of features, numbers, steps, behaviors, components, parts, or a combination thereof disclosed in the present specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, behaviors, components, parts, or a combination thereof exist or are added.

[0071] It should also be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0072] In the present disclosure, if it involves operations of obtaining user information or user data or displaying user information or user data to others, the operations are all authorized and confirmed by the user, or actively selected by the user.

[0073] As mentioned above, due to the complex structure of the fundus and the narrow surgical space, traditional positioning methods usually rely on manual operation or simple mechanical devices, which are difficult to meet the needs of high precision and high efficiency. The inventors found in their research that since the position and posture of the fundus surgical needle will constantly change due to the surgical operation, and are subject to the limitations of the operating accuracy when manually operating or using a robotic arm to operate the fundus surgical needle, the actual posture of the fundus surgical needle is different from the posture required by the operator. This difference is exactly the factor that is expected to be minimized as much as possible during fundus surgery. By estimating the posture of the fundus surgical needle in real time, its position and orientation in three-dimensional space can be dynamically tracked, which is crucial for precise control and guidance of surgery.

[0074] The present disclosure provides a method for estimating the posture of a fundus surgical needle, wherein a positioning area is provided on the fundus surgical needle, the positioning area includes a plurality of positioning patterns, the positions of the plurality of positioning patterns are used to determine a plurality of three-dimensional reference points of the fundus surgical needle, the plurality of three-dimensional reference points are divided into a plurality of three-dimensional reference point sequences, and the area in the positioning area other than the plurality of positioning patterns has a specified color, the method comprising:

[0075] At the current moment, a two-dimensional RGB image of the fundus surgical needle taken by a camera is obtained; the two-dimensional RGB image is converted into a two-dimensional HSV image; a two-dimensional mask is created using a set color threshold, and the two-dimensional HSV image is segmented using the two-dimensional mask to obtain a foreground mask, wherein the color threshold is related to the specified color; multiple connected components in the foreground mask are extracted, and the multiple connected components are divided to obtain multiple connected component sequences corresponding to the multiple three-dimensional reference point sequences respectively; for each connected component sequence in the multiple connected component sequences, the center coordinates of each connected component in the camera coordinate system are calculated and the marking information is set, and a feature list of each connected component in the connected component sequence is created, wherein the feature list includes the center coordinates and marking information of the corresponding connected component; according to the feature list of each connected component in the multiple connected component sequences, the three Each three-dimensional reference point in a three-dimensional reference point sequence is matched one-to-one with the central coordinates of each connected component in a corresponding connected component sequence to obtain multiple matching pairs, each matching pair including a three-dimensional reference point and a central coordinate; for the multiple matching pairs, a random sampling consistency RANSAC algorithm is used to perform multiple RANSAC iterations, and in each RANSAC iteration process, a preset number of matching pairs are randomly selected from the multiple matching pairs, and a perspective n-point PnP algorithm is used to perform camera posture estimation on the preset number of matching pairs and calculate the number of inliers, and the camera posture estimation result obtained in the iteration process with the largest number of inliers is used as the posture information of the camera relative to the fundus surgical needle; through a camera posture inverse transformation, the posture information of the camera relative to the fundus surgical needle is converted into the posture estimation information of the fundus surgical needle in the world coordinate system at the current moment.

[0076] The present invention can automatically and accurately position the posture of the fundus surgery needle in real time. The entire processing flow is real-time, efficient, and highly automated, thereby helping operators make accurate judgments and adjustments, thereby improving the safety and success rate of fundus surgery.

[0077] Figure 1 A flow chart of a method for estimating the posture of a fundus surgical needle according to an embodiment of the present disclosure is shown. Figure 1 As shown, the posture estimation method includes the following steps S101-S108:

[0078] In the present disclosure, a positioning area is provided on the fundus surgical needle, and the positioning area includes a plurality of positioning patterns, the positions of the plurality of positioning patterns are used to determine a plurality of three-dimensional reference points of the fundus surgical needle, the plurality of three-dimensional reference points are divided into a plurality of three-dimensional reference point sequences, and the area other than the plurality of positioning patterns in the positioning area has a specified color. In a specific embodiment, the positioning area is realized by pasting a positioning sticker on the needle handle of the fundus surgical needle.

[0079] The positions of the multiple positioning patterns in the positioning area can be the coordinate positions of the multiple positioning patterns measured in the three-dimensional space, and the coordinate positions of the multiple positioning patterns in the world coordinate system are used as the multiple three-dimensional reference points of the fundus surgical needle. The three-dimensional reference points can be divided according to the pattern features of each three-dimensional reference point or the distance between the three-dimensional reference points, and each three-dimensional reference point sequence includes some of the three-dimensional reference points in the multiple three-dimensional reference points.

[0080] The inventors note that although the coordinates of multiple positioning patterns on the fundus surgical needle in the world coordinate system can be directly obtained by measurement and other technical means, in actual applications, due to various factors (such as camera calibration errors, image distortion, small movements during surgery, etc.), directly using these points for surgical positioning may introduce errors. Therefore, the present disclosure uses the coordinates of the positioning pattern obtained by measurement as a three-dimensional reference point, and uses the three-dimensional reference point for posture estimation, so that more information (such as image features, camera internal parameters, etc.) can be used to reduce these errors, so as to improve the accuracy of fundus surgical needle positioning.

[0081] Combine the following Figure 2 and Figure 3 The positioning area in the fundus surgery needle of the present disclosure is specifically described as follows:

[0082] Figure 2 A cross-sectional view of a fundus surgery needle according to an embodiment of the present disclosure is shown, Figure 3 A stereoscopic diagram of a fundus surgical needle according to an embodiment of the present disclosure is shown. Figure 2 and Figure 3 The positioning area in the example shown in the figure is only a specific implementation of the positioning area of ​​the present disclosure. Those skilled in the art will appreciate that the position of the positioning area and the arrangement of the positioning pattern in the specific implementation are not technical means to limit the protection scope of the present disclosure. Figure 2 and Figure 3 The positioning pattern is not only an ellipse as shown in FIG. , but the positioning pattern can also be a rectangle, a polygon, etc.

[0083] like Figure 2 and Figure 3 As shown, a positioning area is provided at the needle handle position of the fundus surgical needle, and inside the positioning area, there are various types of elliptical positioning patterns. In the positioning area, except for the area with the elliptical positioning pattern, the remaining area is green, so that in the positioning area, the positioning pattern and the area outside the positioning pattern have a clear dividing line.

[0084] In step S101, at the current moment, a two-dimensional RGB image of the fundus surgical needle photographed by a camera is acquired.

[0085] The two-dimensional RGB image is a digital image synthesized based on three basic color channels of red (Red), green (Green), and blue (Blue), and can be directly obtained by shooting with a camera. Those skilled in the art should understand that the camera is only a way to obtain the two-dimensional RGB image. For example, the two-dimensional RGB image of the fundus surgical needle can also be obtained by a camera, a mobile phone camera, etc.

[0086] In step S102, the two-dimensional RGB image is converted into a two-dimensional HSV image.

[0087] According to an embodiment of the present disclosure, the image conversion is achieved using the following formula:

[0088]

[0089] Among them, H represents hue, R represents red brightness value, G represents green brightness value, B represents blue brightness value, max(R,G,B) is the maximum value among red brightness value, green brightness value and blue brightness value, min(R,G,B) is the minimum value among red brightness value, green brightness value and blue brightness value.

[0090] In a two-dimensional HSV image, H stands for hue, S stands for saturation, and V stands for value. Therefore, converting a two-dimensional RGB image into a two-dimensional HSV image can make it easier to select a specific color by setting the hue value.

[0091] According to an embodiment of the present disclosure, before converting the two-dimensional RGB image into a two-dimensional HSV image, the method further includes: loading the internal parameter matrix and distortion coefficient of the camera, and performing dedistortion processing on the RGB image. Dedistortion processing can correct the distorted part in the image, improve the detail performance of the image edge and corner, and lay the foundation for further processing of the subsequent image.

[0092] In step S103, a two-dimensional mask is created using a set color threshold, and the two-dimensional HSV image is segmented using the two-dimensional mask to obtain a foreground mask, wherein the color threshold is related to the designated color.

[0093] Furthermore, pixels in the two-dimensional HSV image that meet the set color threshold are set to white, and the remaining pixels are set to black, so as to obtain the foreground mask.

[0094] For example, in Figure 2 and Figure 3In the example shown, since the designated color is green, the color threshold is set to [35°, 85°], so that the color threshold can be used to create a two-bit mask to segment the positioning area and the positioning pattern in the two-dimensional RGB image. For another example, if the designated color is set to yellow, the color threshold can be set to [15°, 35°].

[0095] In step S104, a plurality of connected components in the foreground mask are extracted, and the plurality of connected components are divided to obtain a plurality of connected component sequences respectively corresponding to the plurality of three-dimensional reference point sequences.

[0096] The connected component is a set of interconnected pixel points in the image. In the present disclosure, each connected component is the outline of each positioning pattern in the positioning area.

[0097] The principle of dividing the multiple three-dimensional reference points is the same as that of dividing the multiple connected components. The division of the multiple connected components can be matched according to the distance or pattern features between the connected components. Figure 4 A schematic diagram showing division of multiple connected components in a foreground mask according to an embodiment of the present disclosure is shown.

[0098] exist Figure 4 In the example shown, based on the pattern features of each connected component, multiple connected components in the positioning area can be divided into 5 groups: connected component sequence 1 to connected component sequence 5.

[0099] In step S105, for each connected component sequence in the multiple connected component sequences, the center coordinates of each connected component in the camera coordinate system are calculated and the marking information is set, and a feature list of each connected component in the connected component sequence is created, and the feature list includes the center coordinates and marking information of the corresponding connected component.

[0100] The feature list of each connected component includes the center coordinates of the connected component (the geometric center of the positioning pattern) and marking information. Each center coordinate is provided with marking information, which can facilitate the subsequent marking and identification of each center coordinate when selecting it.

[0101] Further, each of the connected component sequences includes some of the connected components among the multiple connected components.

[0102] In step S106, based on the feature list of each connected component in the multiple connected component sequences, a one-to-one match is performed between each three-dimensional reference point in the three-dimensional reference point sequence and the central coordinates of each connected component in the corresponding connected component sequence to obtain multiple matching pairs, each matching pair including a three-dimensional reference point and a central coordinate.

[0103] Furthermore, since multiple 3D reference point sequences and corresponding multiple connected component sequences have been obtained, the 3D reference point sequences and connected component sequences can be matched. For example, in a specific embodiment, multiple 3D reference points are divided into 3 3D reference point sequences, and multiple connected components are divided into 3 connected component sequences, then the connected component sequence matching them can be found based on the positional relationship features between each 3D reference point in the first 3D reference point sequence.

[0104] The inventors have noticed that after dividing multiple connected components to obtain multiple connected component sequences, there may be some abnormal sequences that do not meet the requirements, so these abnormal sequences need to be screened and eliminated.

[0105] In a specific embodiment, each connected component sequence is set with a preset range, and the method further includes: checking whether the connected components in each connected component sequence are within the corresponding preset range; if so, judging that the connected component sequence is a valid connected component sequence; if not, judging that the connected component sequence is an abnormal connected component sequence, and not using the abnormal connected component sequence for camera posture estimation.

[0106] In another specific embodiment, each connected component sequence is set with a preset inlier number threshold, and the method further includes: calculating the number of inliers in each connected component sequence; checking whether the number of inliers in each connected component sequence is within the corresponding preset inlier number threshold; if so, judging that the connected component sequence is a valid connected component sequence; if not, judging that the connected component sequence is an abnormal connected component sequence, and not using the abnormal connected component sequence for camera pose estimation.

[0107] The inner points in each connected component sequence represent connected components that meet the requirements of the connected component sequence (which can be set according to requirements).

[0108] The RANSAC algorithm is an uncertainty algorithm that assumes that data consists of "inliers" and "outliers", where "inliers" are data that conform to the model parameters and "outliers" are data that do not conform to the model parameters. For example, in a two-dimensional plane, the straight line model assumes that data points should be distributed along a straight line. The RANSAC algorithm randomly selects two points as a sample set and uses these two points to calculate the equation of the line. The RANSAC algorithm then evaluates the distance of each data point to the line and considers data points whose distance is less than a certain threshold as inliers.

[0109] In the present disclosure, the preset range can be determined based on the selected samples, and the preset internal point quantity threshold can be customized according to user needs.

[0110] In step S107, for the multiple matching pairs, the RANSAC algorithm is used for multiple RANSAC iterations. During each RANSAC iteration, a preset number of matching pairs are randomly selected from the multiple matching pairs, and the PnP algorithm is used to estimate the camera pose for the preset number of matching pairs and calculate the number of inliers. The camera pose estimation result obtained during the iteration with the largest number of inliers is used as the pose information of the camera relative to the fundus surgical needle.

[0111] According to an embodiment of the present disclosure, the camera pose estimation result includes a rotation angle vector and a translation vector.

[0112] Wherein, the RANSAC algorithm has a set number of iterations, and this number of iterations defines the number of iterations for the multiple matching pairs. The setting of this number of iterations can also be set according to user requirements.

[0113] The following will detail the use of the RANSAC algorithm for multiple RANSAC iterations on the multiple matching pairs (3D reference points - central coordinates of connected components):

[0114] Use the PnP algorithm to estimate the camera pose for the preset number of matching pairs to obtain the corresponding camera pose estimation result; project the 3D reference points in the remaining matching pairs among the multiple matching pairs except the preset number of matching pairs into the camera coordinate system using the corresponding camera pose estimation result to obtain multiple 3D reprojection results; compare the multiple 3D reprojection results with the central coordinates in the remaining matching pairs to obtain the reprojection error for each matching pair in the remaining matching pairs; and use the matching pairs with a reprojection error less than the reprojection error threshold as inliers.

[0115] For example, assume that the number of iterations of the RANSAC algorithm is M times and there are N matching pairs. Then, during the first RANSAC iteration, n matching pairs are randomly selected, and the PnP algorithm is used for camera pose estimation to obtain the corresponding camera pose estimation result, where the preset number n is the number of the minimum samples required for camera pose estimation, and n < N. Then, project the 3D reference points in the remaining N - n matching pairs into the camera coordinate system using the obtained camera pose estimation result to obtain multiple 3D reprojection results; compare these multiple 3D reprojection results with the corresponding central coordinates in the remaining N - n matching pairs, and then obtain the reprojection error for each matching pair in the remaining N - n matching pairs. Use the matching pairs with a reprojection error less than the reprojection error threshold in the remaining N - n matching pairs as inliers.

[0116] Specifically, if for any matching pair (X i , x i ) among the multiple matching pairs, the 3D reference point Xi =[X i ,Y i ,Z i ] T , the center coordinates x of the connected components i =[u i ,v i ] T , when the PnP algorithm is used to estimate the camera pose for each of the n randomly selected matching pairs, the following projection model formula is used for implementation:

[0117]

[0118] in, is the projection point (the center coordinate of the matching pair), K is the camera internal parameter matrix, R is the rotation matrix of the camera to be obtained, and t is the translation vector of the camera to be obtained.

[0119] Furthermore, the rotation matrix R can be calculated based on the camera internal parameter matrix K, as shown in the following formula:

[0120]

[0121] Where I is the identity matrix.

[0122] Combined with the rotation matrix R, first use the formula Calculate the camera's rotation angle θ, and then use the formula Calculate the rotation axis u of the camera, and finally calculate the rotation angle vector r = θu of the camera, where trace(R) is the trace of R. ij is the element in the i-th row and j-th column of the rotation matrix R.

[0123] When calculating the reprojection error, assume that any of the remaining matching pairs is (X j ,x j ), then according to x j =K(RX j +t), calculate the reprojection error Among them, X j is the three-dimensional reference point coordinate of any matching pair, x j is the center coordinate of any matching pair, x j [2] is x j The scale factor of .

[0124] The reprojection errors of all the remaining matching pairs are calculated, and the number of inliers is counted using the following formula:

[0125]

[0126] Among them, ∈ is the reprojection error threshold, δ is the indicator function, and the indicator function is usually defined as: when the condition is met, the function value is 1; when the condition is not met, the function value is 0. Specifically in the present disclosure, when the reprojection error is less than the reprojection error threshold ∈, 1 is returned, otherwise 0 is returned.

[0127] In step S108, the posture information of the camera relative to the fundus surgical needle is converted into posture estimation information of the fundus surgical needle in the world coordinate system at the current moment through inverse transformation of the camera posture.

[0128] Those skilled in the art can understand that the posture of the camera relative to the fundus surgical needle (three-dimensional reference point) is the posture of the fundus surgical needle in the camera coordinate system.

[0129] Among them, the position relationship information between the positioning area and the needle tip of the fundus surgical needle in the world coordinate system is obtained; based on the position relationship information and the posture estimation information of the fundus surgical needle in the world coordinate system at the current moment, the posture estimation information of the needle tip in the world coordinate system is located.

[0130] The present invention sets a positioning area on the fundus surgical needle, and combines the positioning pattern set on the positioning area, and then utilizes the PnP algorithm and the RANSAC algorithm, wherein the PnP algorithm is used to derive the position and posture in the three-dimensional space from the two-dimensional image points, and the RANSAC algorithm improves the robustness of the estimation result through iterative optimization, thereby effectively reducing the influence of noise and outliers, and realizing high-precision needle tip posture estimation, helping the operator to make accurate judgments, so that the operator can control the surgical operation more accurately, and promptly discover and correct possible operational errors, thereby improving the safety and success rate of fundus surgery.

[0131] For ease of understanding, the process of the method for estimating the posture of a fundus surgical needle disclosed in the present invention is described in detail below using a specific embodiment.

[0132] Figure 5 A logic flow chart of a method for estimating the posture of a fundus surgery needle according to an embodiment of the present disclosure is shown.

[0133] like Figure 5As shown, the fundus surgical needle is first calibrated to determine multiple three-dimensional reference points according to multiple positioning patterns in the positioning area on the fundus surgical needle, and then the fundus surgical needle is photographed by a camera to obtain a two-dimensional RGB image, the two-dimensional RGB image is converted into a two-dimensional HSV image, and then a two-dimensional mask is created according to the specified color of the positioning area, so as to segment the two-dimensional HSV image to obtain a foreground mask; after obtaining the foreground mask, multiple connected components in the foreground mask are extracted, and the multiple connected components are divided to obtain multiple connected component sequences corresponding to multiple three-dimensional reference point sequences respectively, and then the center coordinates of each connected component in the connected component sequence are calculated, and the multiple three-dimensional reference points are converted into a three-dimensional HSV image. Each three-dimensional reference point in the point sequence is matched one-to-one with the central coordinates of each connected group in the corresponding connected component sequence to obtain multiple matching pairs; for multiple matching pairs, the RANSAC algorithm is used to perform multiple RANSAC iterations, and in each RANSAC iteration process, a preset number of matching pairs are randomly selected from the multiple matching pairs, and the PnP algorithm is used to perform camera posture estimation and calculate the number of inliers, and the camera posture estimation result obtained in the iteration process with the largest number of inliers is used as the posture information of the camera relative to the fundus surgical needle; finally, through the camera posture inverse transformation, the posture information of the camera relative to the fundus surgical needle is converted into the posture estimation information of the fundus surgical needle in the world coordinate system.

[0134] The present invention not only has a high degree of automation and strong robustness, but also ensures robustness and stability in different environments by using a variety of image processing technologies, such as connected component analysis, sequence grouping, and abnormal sequence removal. It also has a wide range of applications. This method is not only suitable for the positioning of fundus surgical needles, but can also be promoted to other medical devices that require high-precision positioning and posture estimation, industrial robots, augmented reality systems and other fields, and has broad application prospects.

[0135] Figure 6 A flow chart of a navigation and positioning method for a fundus surgical needle according to an embodiment of the present disclosure is shown. Figure 6 The method shown is Figure 1 Based on the method shown, steps S601-S602 are added:

[0136] In step S101, at the current moment, a two-dimensional RGB image of the fundus surgical needle photographed by a camera is acquired.

[0137] In step S102, the two-dimensional RGB image is converted into a two-dimensional HSV image.

[0138] In step S103, a two-dimensional mask is created using a set color threshold, and the two-dimensional HSV image is segmented using the two-dimensional mask to obtain a foreground mask, wherein the color threshold is related to the designated color.

[0139] In step S104, a plurality of connected components in the foreground mask are extracted, and the plurality of connected components are divided to obtain a plurality of connected component sequences respectively corresponding to a plurality of three-dimensional reference point sequences.

[0140] In step S105, for each connected component sequence in the multiple connected component sequences, the center coordinates of each connected component in the camera coordinate system are calculated and the marking information is set, and a feature list of each connected component in the connected component sequence is created, and the feature list includes the center coordinates and marking information of the corresponding connected component.

[0141] In step S106, based on the feature list of each connected component in the multiple connected component sequences, a one-to-one match is performed between each three-dimensional reference point in the three-dimensional reference point sequence and the central coordinates of each connected component in the corresponding connected component sequence to obtain multiple matching pairs, each matching pair including a three-dimensional reference point and a central coordinate.

[0142] In step S107, a random sampling consistency RANSAC algorithm is used to perform multiple RANSAC iterations for the multiple matching pairs. In each RANSAC iteration, a preset number of matching pairs are randomly selected from the multiple matching pairs, and a perspective n-point PnP algorithm is used to perform camera pose estimation on the preset number of matching pairs and calculate the number of inliers. The camera pose estimation result obtained in the iteration with the largest number of inliers is used as the pose information of the camera relative to the fundus surgical needle.

[0143] In step S108, the posture information of the camera relative to the fundus surgical needle is converted into posture estimation information of the fundus surgical needle in the world coordinate system at the current moment through inverse transformation of the camera posture.

[0144] like Figure 6 As shown, in step S601, for any moment after the current moment, the above steps are repeated to obtain real-time posture estimation information of the fundus surgery needle, thereby realizing navigation positioning of the fundus surgery needle;

[0145] In step S602, the real-time posture estimation information of the fundus surgical needle is sent to a display screen in real time for display.

[0146] The present invention adopts an efficient algorithm and an optimized image processing process, which can run in real time during the operation, provide timely positioning and posture estimation results, and meet the real-time requirements of the operation. It also provides an intuitive result visualization function, which helps operators make accurate judgments and adjustments by displaying the positioning information and posture changes of the needle tip, thereby improving the safety and success rate of the operation. It also reduces the dependence on professional equipment and can be implemented in combination with conventional surgical microscopes and ordinary cameras, reducing costs.

[0147] Figure 7 A structural block diagram of a fundus surgery needle posture estimation system according to an embodiment of the present disclosure is shown.

[0148] like Figure 7 As shown, the system 700 includes: an image acquisition module 710 , an image preprocessing module 720 , a feature matching module 730 , a posture calculation module 740 and a posture estimation module 750 .

[0149] The image acquisition module 710 is configured to acquire, at a current moment, a two-dimensional RGB image of the fundus surgical needle captured by a camera;

[0150] The image preprocessing module 720 is configured to convert the two-dimensional RGB image into a two-dimensional HSV image, create a two-dimensional mask using a set color threshold, and use the two-dimensional mask to segment the two-dimensional HSV image to obtain a foreground mask; extract multiple connected components in the foreground mask, divide the multiple connected components, obtain multiple connected component sequences corresponding to multiple three-dimensional reference point sequences respectively, for each connected component sequence in the multiple connected component sequences, calculate the center coordinates of each connected component in the camera coordinate system and set label information, and create a feature list of each connected component in the connected component sequence, wherein the feature list includes the center coordinates and label information of the corresponding connected component;

[0151] The feature matching module 730 is configured to perform one-to-one matching between each 3D reference point in the 3D reference point sequence and the center coordinates of each connected component in the corresponding connected component sequence according to the feature list of each connected component in the multiple connected component sequences, to obtain multiple matching pairs, each matching pair including a 3D reference point and a center coordinate;

[0152] The posture calculation module 740 is configured to perform multiple RANSAC iterations for the multiple matching pairs using the random sampling consistency RANSAC algorithm, randomly select a preset number of matching pairs from the multiple matching pairs in each RANSAC iteration process, use the PnP algorithm to perform camera posture estimation on the preset number of matching pairs and calculate the number of inliers, and use the camera posture estimation result obtained in the iteration process with the largest number of inliers as the posture information of the camera relative to the fundus surgical needle;

[0153] The posture estimation module 750 is configured to convert the posture information of the camera relative to the fundus surgical needle into posture estimation information of the fundus surgical needle in the world coordinate system at the current moment through inverse transformation of the camera posture.

[0154] Figure 8 A structural block diagram of a fundus surgery needle posture estimation device according to an embodiment of the present disclosure is shown.

[0155] like Figure 8 As shown, the device includes: a camera, a computing device and a microscope, wherein the camera is communicatively connected to the computing device, and the computing device is communicatively connected to the microscope;

[0156] Wherein, the microscope is used to display the surgical area of ​​the fundus surgical needle;

[0157] The camera is used to obtain a two-dimensional RGB image of the fundus surgical needle and send the two-dimensional RGB image to the computing device;

[0158] The computing device is used to obtain the posture estimation information of the fundus surgical needle based on the two-dimensional RGB image, wherein the computing device includes a memory and a processor, the memory is used to store computer instructions, and the computer instructions are executed by the processor to implement the method as described above.

[0159] The present disclosure not only provides timely positioning and posture estimation results to meet the real-time requirements of surgery, but also reduces the dependence on professional equipment. It can be implemented in combination with conventional surgical microscopes and ordinary cameras, thus reducing costs.

[0160] In particular, according to an embodiment of the present disclosure, the method described above can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program tangibly contained on a machine-readable medium, and the computer program includes a program code for executing the above method. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium.

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or programmable hardware. The units or modules described may also be set in a processor, and the names of these units or modules do not constitute limitations on the units or modules themselves in some cases.

[0163] As another aspect, the present disclosure further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the electronic device or computer system in the above embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device. The computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present disclosure.

[0164] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the present disclosure is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the inventive concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other.

Claims

1. A method for estimating the posture of a fundus surgical needle, characterized in that: The fundus surgery needle is provided with a positioning area, the positioning area includes a plurality of positioning patterns, the positions of the plurality of positioning patterns are used to determine a plurality of three-dimensional reference points of the fundus surgery needle, the plurality of three-dimensional reference points are divided into a plurality of three-dimensional reference point sequences, and the area in the positioning area other than the plurality of positioning patterns has a specified color, the method comprising: At the current moment, a two-dimensional RGB image of the fundus surgical needle captured by a camera is acquired; Convert the two-dimensional RGB image into a two-dimensional HSV image; Creating a two-dimensional mask using a set color threshold, and segmenting the two-dimensional HSV image using the two-dimensional mask to obtain a foreground mask, wherein the color threshold is related to the specified color; Extracting multiple connected components in the foreground mask, dividing the multiple connected components, and obtaining multiple connected component sequences corresponding to the multiple three-dimensional reference point sequences respectively; For each connected component sequence in the plurality of connected component sequences, calculate the center coordinates of each connected component in the camera coordinate system and set label information, and create a feature list of each connected component in the connected component sequence, wherein the feature list includes the center coordinates and label information of the corresponding connected component; According to the feature list of each connected component in the multiple connected component sequences, one-to-one matching is performed between each three-dimensional reference point in the three-dimensional reference point sequence and the central coordinate of each connected component in the corresponding connected component sequence to obtain multiple matching pairs, each matching pair including a three-dimensional reference point and a central coordinate; For the multiple matching pairs, a random sampling consistency RANSAC algorithm is used to perform multiple RANSAC iterations. In each RANSAC iteration process, a preset number of matching pairs are randomly selected from the multiple matching pairs, and a perspective n-point PnP algorithm is used to perform camera pose estimation on the preset number of matching pairs and calculate the number of inliers, and the camera pose estimation result obtained in the iteration process with the largest number of inliers is used as the pose information of the camera relative to the fundus surgical needle; By inverse transformation of the camera attitude, the attitude information of the camera relative to the fundus surgical needle is converted into attitude estimation information of the fundus surgical needle in the world coordinate system at the current moment.

2. The method according to claim 1, characterized in that: Each of the connected component sequences includes some of the connected components in the plurality of connected components; Each 3D reference point sequence includes some 3D reference points in the plurality of 3D reference points.

3. The method according to claim 1, characterized in that The converting the two-dimensional RGB image into a two-dimensional HSV image includes using the following formula to achieve image conversion: Among them, H represents hue, R represents red brightness value, G represents green brightness value, B represents blue brightness value, max(R,G,B) is the maximum value among red brightness value, green brightness value and blue brightness value, min(R,G,B) is the minimum value among red brightness value, green brightness value and blue brightness value.

4. The method according to claim 1, characterized in that: Each connected component sequence is set with a preset range, and the method further includes: Checking whether the connected components in each connected component sequence are within a corresponding preset range; If so, the connected component sequence is judged to be a valid connected component sequence; If not, the connected component sequence is determined to be an abnormal connected component sequence, and the abnormal connected component sequence is not used for camera pose estimation.

5. The method according to claim 1, characterized in that Each connected component sequence is provided with a preset threshold of the number of inliers, and the method further comprises: Count the number of inliers in each connected component sequence; Checking whether the number of inliers in each connected component sequence is within a corresponding preset inlier number threshold; If so, the connected component sequence is judged to be a valid connected component sequence; If not, the connected component sequence is determined to be an abnormal connected component sequence, and the abnormal connected component sequence is not used for camera pose estimation.

6. The method according to claim 1, characterized in that The RANSAC algorithm has a set number of iterations; The using the perspective n-point PnP algorithm to perform camera pose estimation on the preset number of matching pairs and calculate the number of interior points includes: Use the PnP algorithm to perform camera pose estimation on the preset number of matching pairs to obtain corresponding camera pose estimation results; Projecting the three-dimensional reference points in the remaining matching pairs of the multiple matching pairs except the preset number of matching pairs into the camera coordinate system using the corresponding camera pose estimation results to obtain a plurality of three-dimensional reprojection results; Comparing the multiple three-dimensional reprojection results with the center coordinates of the remaining matching pairs to obtain a reprojection error of each matching pair in the remaining matching pairs; The matching pairs whose reprojection error is less than the reprojection error threshold are regarded as inliers.

7. The method according to claim 1, characterized in that The method further comprises: Acquire positional relationship information between the positioning area and the needle tip of the fundus surgery needle in a world coordinate system; The estimated posture information of the needle tip in the world coordinate system is located according to the positional relationship information and the estimated posture information of the fundus surgery needle in the world coordinate system at the current moment.

8. A navigation and positioning method for fundus surgery needle, characterized in that: The method comprises: At the current moment, a two-dimensional RGB image of the fundus surgical needle captured by a camera is acquired; Convert the two-dimensional RGB image into a two-dimensional HSV image; Creating a two-dimensional mask using a set color threshold, and segmenting the two-dimensional HSV image using the two-dimensional mask to obtain a foreground mask; Extracting multiple connected components in the foreground mask, dividing the multiple connected components, and obtaining multiple connected component sequences corresponding to multiple three-dimensional reference point sequences respectively; For each connected component sequence in the plurality of connected component sequences, calculate the center coordinates of each connected component in the camera coordinate system and set label information, and create a feature list of each connected component in the connected component sequence, wherein the feature list includes the center coordinates and label information of the corresponding connected component; According to the feature list of each connected component in the plurality of connected component sequences, one-to-one matching is performed between each three-dimensional reference point in the three-dimensional reference point sequence and the central coordinate of each connected component in the corresponding connected component sequence to obtain a plurality of matching pairs, each matching pair including a three-dimensional reference point and a central coordinate; For the multiple matching pairs, a random sampling consistency RANSAC algorithm is used to perform multiple RANSAC iterations. In each RANSAC iteration process, a preset number of matching pairs are randomly selected from the multiple matching pairs, and a PnP algorithm is used to perform camera pose estimation on the preset number of matching pairs and calculate the number of inliers, and the camera pose estimation result obtained in the iteration process with the largest number of inliers is used as the pose information of the camera relative to the fundus surgical needle; By inverse transformation of the camera attitude, converting the attitude information of the camera relative to the fundus surgical needle into the attitude estimation information of the fundus surgical needle in the world coordinate system at the current moment; Repeat the above steps for any moment after the current moment to obtain real-time posture estimation information of the fundus surgery needle, thereby achieving navigation positioning of the fundus surgery needle; The real-time posture estimation information of the fundus surgical needle is sent to a display screen for display.

9. A fundus surgery needle posture estimation system, characterized in that: include: An image acquisition module is configured to acquire, at a current moment, a two-dimensional RGB image of the fundus surgical needle captured by a camera; The image preprocessing module is configured to convert the two-dimensional RGB image into a two-dimensional HSV image, create a two-dimensional mask using a set color threshold, and use the two-dimensional mask to segment the two-dimensional HSV image to obtain a foreground mask; extract multiple connected components in the foreground mask, divide the multiple connected components, obtain multiple connected component sequences corresponding to multiple three-dimensional reference point sequences respectively, calculate the center coordinates of each connected component in the camera coordinate system for each connected component sequence in the multiple connected component sequences and set label information, and create a feature list of each connected component in the connected component sequence, wherein the feature list includes the center coordinates and label information of the corresponding connected component; a feature matching module configured to perform one-to-one matching between each three-dimensional reference point in the three-dimensional reference point sequence and the center coordinate of each connected component in the corresponding connected component sequence according to the feature list of each connected component in the plurality of connected component sequences, so as to obtain a plurality of matching pairs, each matching pair including a three-dimensional reference point and a center coordinate; a posture calculation module, configured to perform multiple RANSAC iterations for the multiple matching pairs using a random sampling consistency RANSAC algorithm, randomly select a preset number of matching pairs from the multiple matching pairs in each RANSAC iteration process, use a PnP algorithm to perform camera posture estimation on the preset number of matching pairs and calculate the number of inliers, and use the camera posture estimation result obtained in the iteration process with the largest number of inliers as the posture information of the camera relative to the fundus surgical needle; The posture estimation module is configured to convert the posture information of the camera relative to the fundus surgical needle into posture estimation information of the fundus surgical needle in the world coordinate system at the current moment through inverse transformation of the camera posture.

10. A device for estimating the posture of a fundus surgical needle, characterized in that: include: A camera, a computing device and a microscope, wherein the camera is communicatively connected to the computing device, and the computing device is communicatively connected to the microscope; wherein, The microscope is used to display the surgical area of ​​the fundus surgical needle; The camera is used to obtain a two-dimensional RGB image of the fundus surgical needle and send the two-dimensional RGB image to the computing device; The computing device is used to obtain the posture estimation information of the fundus surgical needle based on the two-dimensional RGB image, wherein the computing device includes a memory and a processor, the memory is used to store computer instructions, and the computer instructions are executed by the processor to implement any one of claims 1 to 8.

Images