Method and device for determining needle insertion depth in fundus needle insertion image, equipment and medium

Abstract

This application provides a method, apparatus, device, and medium for determining the needle insertion depth in fundus needle insertion images. The method involves acquiring a target fundus needle insertion image and, based on the FAST corner detection algorithm, determining multiple first key points of the surgical needle in the target fundus needle insertion image. Based on these multiple first key points, multiple first feature points of a reference image, and a feature point matching algorithm, multiple second feature points of the target fundus needle insertion image are determined. Based on the multiple second feature points and the first feature points, the multiple second feature points are perspective-transformed to the plane of the reference image to obtain multiple third feature points of the perspective-transformed target fundus needle insertion image. Based on the multiple third feature points and the first feature points, the needle insertion depth of the surgical needle in the target fundus needle insertion image is determined. Transforming fundus needle insertion images from different shooting angles to the same shooting angle as the reference image unifies the ratio of image pixels to actual distances, thus improving the accuracy of needle insertion depth measurement in fundus needle insertion images.

Description

Technical Field

[0001] This application relates to the field of medical image processing technology, and in particular to methods, apparatus, equipment and media for determining the depth of needle insertion in fundus needle insertion images. Background Technology

[0002] Needle insertion depth is an important evaluation indicator for the clinical registration of ophthalmic surgical robots, used to assess whether ophthalmic surgical robots are more stable than human hands.

[0003] Existing methods for measuring needle insertion depth involve manually marking the characteristic points of the surgical needle in fundus needle insertion images and manually measuring the insertion depth of the surgical needle in the fundus needle insertion images. The measurement error is large in fundus needle insertion images under different shooting angles, and the accuracy of needle insertion depth measurement in fundus needle insertion images is low. Summary of the Invention

[0004] In view of this, embodiments of this application at least provide a method, apparatus, device and medium for determining the needle insertion depth in fundus needle insertion images, transforming fundus needle insertion images from different shooting angles to the same shooting angle as the reference image, unifying the ratio of image pixels to actual distance, and improving the accuracy of needle insertion depth measurement in fundus needle insertion images.

[0005] This application mainly includes the following aspects:

[0006] In a first aspect, embodiments of this application provide a method for determining the needle insertion depth in a fundus needle insertion image, the method comprising:

[0007] A target fundus needle insertion image is acquired, and multiple first key points of the surgical needle in the target fundus needle insertion image are determined based on the FAST corner detection algorithm; the target fundus needle insertion image is an image acquired during the fundus needle insertion surgery.

[0008] Based on the plurality of first key points, the plurality of first feature points of the reference image, and the feature point matching algorithm, a plurality of second feature points of the target fundus needle insertion image are determined; the plurality of second feature points are points that match the plurality of first feature points among the plurality of first key points; the reference image is a pre-acquired image before the surgical needle is inserted; the plurality of first feature points are used to describe the salient features of the surgical needle in the reference image;

[0009] Based on the plurality of second feature points and the plurality of first feature points, the plurality of second feature points are perspective-transformed to the plane where the reference image is located to obtain a plurality of third feature points of the target fundus needle insertion image after perspective transformation.

[0010] Based on the plurality of third feature points and the plurality of first feature points, the insertion depth of the surgical needle in the target fundus needle insertion image is determined.

[0011] Secondly, embodiments of this application also provide a device for determining the needle insertion depth in a fundus needle insertion image, the device comprising:

[0012] The acquisition module is used to acquire the target fundus needle insertion image and, based on the FAST corner detection algorithm, determine multiple first key points of the surgical needle in the target fundus needle insertion image; the target fundus needle insertion image is an image acquired during the fundus needle insertion surgery.

[0013] A matching module is used to determine multiple second feature points of the target fundus needle insertion image based on the multiple first key points, multiple first feature points of the reference image, and a feature point matching algorithm; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image before the surgical needle is inserted; the multiple first feature points are used to describe the salient features of the surgical needle in the reference image;

[0014] The conversion module is used to perform perspective transformation of the plurality of second feature points to the plane where the reference image is located, based on the plurality of second feature points and the plurality of first feature points, to obtain a plurality of third feature points of the perspective-transformed target fundus needle insertion image;

[0015] The determination module is used to determine the insertion depth of the surgical needle in the target fundus needle insertion image based on the plurality of third feature points and the plurality of first feature points.

[0016] Thirdly, embodiments of this application also provide an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the bus. The machine-readable instructions are executed by the processor to perform the steps of the method for determining the needle insertion depth in the fundus needle insertion image as described above.

[0017] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the method for determining the needle insertion depth in a fundus needle insertion image as described above.

[0018] The present application provides a method, apparatus, device, and medium for determining the needle insertion depth in a fundus needle insertion image. The method includes: acquiring a target fundus needle insertion image and determining multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner detection algorithm; the target fundus needle insertion image is an image acquired during fundus needle insertion surgery; determining multiple second feature points of the target fundus needle insertion image based on the multiple first key points, multiple first feature points of a reference image, and a feature point matching algorithm; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image before needle insertion; the multiple first feature points are used to describe the salient features of the surgical needle in the reference image; based on the multiple second feature points and the multiple first feature points, performing a perspective transformation on the plane of the reference image to obtain multiple third feature points of the perspective-transformed target fundus needle insertion image; and determining the needle insertion depth of the surgical needle in the target fundus needle insertion image based on the multiple third feature points and the multiple first feature points. In this way, the fundus needle insertion images from different shooting angles are transformed to the same shooting angle as the reference image, and the ratio of image pixels to actual distance is unified, which improves the accuracy of needle insertion depth measurement in fundus needle insertion images.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart illustrating a method for determining needle insertion depth in a fundus needle insertion image provided in an embodiment of this application is shown;

[0022] Figure 2 This illustration shows a schematic diagram of the surgical needle from a first shooting perspective in an embodiment of this application;

[0023] Figure 3 This illustration shows a schematic diagram of the surgical needle from a second shooting perspective in an embodiment of this application;

[0024] Figure 4 A schematic diagram of a reference image in an embodiment of this application is shown;

[0025] Figure 5 A schematic diagram of the target fundus needle insertion image in an embodiment of this application is shown;

[0026] Figure 6 A schematic diagram of the target fundus needle insertion image after perspective transformation is shown in an embodiment of this application;

[0027] Figure 7 This paper illustrates a functional block diagram of a device for determining needle insertion depth in a fundus needle insertion image provided in an embodiment of this application;

[0028] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of this application is shown. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0030] The following is a detailed description of a method for determining the needle insertion depth in a fundus needle insertion image provided by an embodiment of this application. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating a method for determining the needle insertion depth in a fundus needle insertion image, provided as an embodiment of this application. Figure 1 As shown, the method includes:

[0031] S101, acquire the target fundus needle insertion image, and based on the FAST corner detection algorithm, determine multiple first key points of the surgical needle in the target fundus needle insertion image; the target fundus needle insertion image is an image acquired during the fundus needle insertion surgery.

[0032] In this step, the first step is to acquire an image of the target fundus where needle insertion is performed. In this embodiment, the target fundus image is an image captured during the fundus needle insertion surgery. Specifically, images during the surgery are acquired using equipment such as a microscope or endoscope. To ensure image quality and measurement accuracy, high-resolution imaging equipment is required, and image distortion and noise must be reduced. Specifically, after acquiring the target fundus image, preprocessing can be performed on the target fundus image, including steps such as noise reduction, contrast enhancement, and distortion correction, to improve the clarity of the target fundus image.

[0033] After acquiring the target fundus needle insertion image, multiple first key points of the surgical needle in the target fundus needle insertion image are determined based on the Features From Accelerated Segment Test (FAST) corner detection algorithm. The FAST corner detection algorithm is a feature point detection method used in computer vision, which can effectively detect corner points in an image, serving as key points / feature points for image feature extraction. In this step, multiple first key points of the surgical needle in the target fundus needle insertion image are extracted using the FAST corner detection algorithm.

[0034] Specifically, the FAST corner detection algorithm works by selecting a pixel p from the image, whose grayscale value is I. p A preset threshold t is set, and a discretized Bresenham circle with a radius of 3 is drawn centered on this pixel using the Bresenham circle algorithm. The circle has 16 pixels on its boundary. If n consecutive pixels on the Bresenham circle have grayscale values ​​less than I... p -t or greater than I p If n is +t, then this point can be identified as a corner point (n can be 12 or 9).

[0035] The FAST corner detection algorithm first requires selecting multiple images of the scene to be detected, choosing an appropriate threshold n (n<12), and extracting multiple key points / feature points as training data.

[0036] For a pixel p in an image, the pixels at position x, x∈{1,…,16} on its neighborhood circle are represented as p→x. The following formula can be used to classify the point p→x into three categories:

[0037]

[0038] Let P be the set of all pixels in the training image set. Let x be one of the 16 possible positions on the neighborhood circle surrounding pixel p. The set P can be divided into three parts: Pi, Pp, and Pi. d P s and P b , where P d The definition is as follows, P s and P b Definition and P d similar:

[0039] P d ={p∈P∶S p→s =d};

[0040] In other words, for any given position x, it can classify all points in the image into three categories, the first category being P. d This includes all pixels at position x that are darker than the center pixel p at a threshold t, and the second type P. s This includes all pixels at position x that approximate the center pixel p under a threshold t. b This includes all pixels at position x that are brighter than the center pixel p under a threshold t.

[0041] Define a boolean variable K for each selected pixel p. p If p is a corner point, then K p If true, then false.

[0042] The extracted keypoint / feature point set is trained, and a decision tree is built using the ID3 algorithm, with the x-th pixel used for partitioning. Specifically, the ID3 algorithm is a greedy algorithm used to construct the decision tree. Originating from Concept Learning Systems (CLS), ID3 uses the rate of decrease in information entropy as the criterion for selecting test attributes. That is, at each node, the attribute with the highest information gain that has not yet been used for partitioning is selected as the partitioning criterion. This process continues until the generated decision tree can perfectly classify the training examples. For set P, the entropy value is:

[0043]

[0044] Where c is the number of corner points. This represents the number of non-corner points. The resulting information gain is:

[0045] ΔH=H(P)-H(P d )-H(P s )-H(P b );

[0046] The decision tree is obtained by splitting the data at the position with the maximum information gain.

[0047] For cases where multiple feature points exist in a neighboring location, further non-maximum suppression (NMPS) is required. Each detected corner point is assigned a quantized value V, and then the V values ​​of adjacent corner points are compared, retaining the point with the largest V value in its local neighborhood. The V value can be defined as the sum of the absolute differences in grayscale between the keypoint / feature point and its 16 neighboring pixels, i.e.:

[0048]

[0049] In the above formula, S bright It is the gray value greater than I among 16 neighboring pixels. pThe set of pixels with +t, and S dark This refers to those grayscale values ​​less than I. p -t is the set of pixels.

[0050] The FAST corner detection algorithm can identify multiple key points of the surgical needle in the target fundus needle insertion image.

[0051] S102, based on multiple first key points, multiple first feature points of the reference image, and a feature point matching algorithm, determine multiple second feature points of the target fundus needle insertion image; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image before the surgical needle is inserted; the multiple first feature points are used to describe the salient features of the surgical needle in the reference image.

[0052] In this step, a feature matching algorithm is used to match multiple first key points of the surgical needle in the target fundus needle insertion image with multiple first feature points in the reference image. The successfully matched first key points are then used as second feature points and marked in the target fundus needle insertion image, thus identifying them as multiple second feature points of the target fundus needle insertion image. The reference image is a pre-acquired image before needle insertion. Specifically, the fundus needle insertion image includes an image before needle insertion and an image after needle insertion. The image before needle insertion is further subdivided into images where the needle tip approaches but does not reach the retina, images where the needle tip just reaches the retina but does not puncture it, and images where the needle tip is withdrawn away from the retina. In this embodiment, the first clear image of the image before needle insertion where the needle tip approaches but does not reach the retina is selected as the reference image. The multiple first feature points are used to describe the salient features of the surgical needle in the reference image, such as the needle tip, the inflection point of the needle bend, the connection point between the needle tip and the robotic arm, and feature markings on the robotic arm.

[0053] S103, based on multiple second feature points and multiple first feature points, the multiple second feature points are perspective-transformed to the plane of the reference image to obtain multiple third feature points of the target fundus needle insertion image after perspective transformation.

[0054] In this step, due to the complexity of the human eye structure and the limitations of the surgical environment, the fundus images acquired are often taken from different angles and heights. Therefore, the resolution of the shooting angles varies, and the number of pixels for the surgical needle differs in different images. For details, please refer to [link to relevant documentation]. Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of the surgical needle from a first shooting perspective in an embodiment of this application. Figure 3 This is a schematic diagram of the surgical needle from a second shooting perspective in an embodiment of this application. Figure 2 and Figure 3 As shown, Figure 2 and Figure 3 The surgical needles used in the images are all the same, with a length of L. However, due to differences in the shooting angle and resolution, the number of pixels for the needle varies in different images. Therefore, we need to unify the fundus needle insertion images to the same viewpoint before performing pixel statistics to facilitate subsequent needle insertion depth calculations. Specifically, based on multiple second feature points in the target fundus needle insertion image and multiple first feature points in the reference image, a mapping relationship is established between the two image coordinate systems. The multiple second feature points in the target fundus needle insertion image are then unified to the plane of the reference image through perspective transformation, resulting in multiple third feature points in the perspective-transformed target fundus needle insertion image.

[0055] S104, based on multiple third feature points and multiple first feature points, determine the insertion depth of the surgical needle in the target fundus needle insertion image.

[0056] In this embodiment, since the portion of the surgical needle that pierces the retina is not shown in the fundus needle insertion image, the insertion depth of the surgical needle in the target fundus needle insertion image can be obtained by comparing the length of the surgical needle in the reference image and the perspective-transformed target fundus needle insertion image. The length of the surgical needle in the reference image and the perspective-transformed target fundus needle insertion image can be calculated based on multiple third feature points and multiple first feature points respectively.

[0057] Further, multiple first feature points of the reference image are determined according to the following steps:

[0058] Step a1: Based on the FAST corner detection algorithm, determine multiple second key points of the surgical needle in the reference image.

[0059] In this step, the FAST corner detection algorithm can identify multiple secondary key points of the surgical needle on the reference image. Specifically, the description of extracting key points using the FAST corner detection algorithm in step a1 can be found in S101, and the same technical effect can be achieved, so it will not be repeated here.

[0060] Step a2: Based on preset filtering rules, select multiple target points from multiple second key points, and determine multiple target points as multiple first feature points of the reference image; multiple target points are second key points of the significant feature position of the surgical needle in the reference image.

[0061] In this step, after identifying multiple second key points of the surgical needle in the reference image, second key points with significant features of the surgical needle in the reference image are selected from these second key points according to a preset selection rule. These second key points are then used as multiple first feature points of the reference image to generate a mask for the surgical needle. This allows the target fundus needle insertion image to extract structural features similar to the mask from the image, i.e., the surgical needle in the insertion image at the significant feature position of the surgical needle in the target fundus reference image. The preset selection rule can be implemented by manually annotating the selected second key points or by using a neural network algorithm to identify significant features of the surgical needle in the reference image for algorithmic annotation; no specific limitation is imposed here.

[0062] In one possible implementation, the feature point matching algorithm includes the ORB algorithm. In S102, based on multiple first key points, multiple first feature points of the reference image, and the feature point matching algorithm, multiple second feature points of the target fundus needle insertion image are determined, specifically including:

[0063] Step b1: For each first key point in the target fundus needle insertion image, determine the direction of the first key point based on the gray centroid of the first key point.

[0064] In this embodiment, the feature point matching algorithm employs the OrientedFAST and RotatedBRIEF (ORB) algorithm. Since the corner points detected by the FAST corner detection algorithm, i.e., each first key point in the target fundus needle insertion image, do not possess directionality, the ORB algorithm assigns an orientation to each first key point by calculating the gray-level centroid of the pixel.

[0065] Specifically, the direction vector is calculated as follows:

[0066] Calculate the image moments in the neighborhood of the keypoints:

[0067] m pq =∑ x,y x p y q I(x,y);

[0068] Where I(x,y) represents the gray value of a pixel, (x,y) is the coordinates in the neighborhood of the key point, and p and q are non-negative integers.

[0069] The position of the centroid is calculated using the following formula:

[0070]

[0071] Where, m 00 It is the zeroth moment, representing the sum of the gray levels of all pixels in the neighborhood, m 10 and m01 These are the first moments with respect to x and y, respectively.

[0072] The angle θ of the direction vector is:

[0073]

[0074] In this way, each first key point is assigned a direction.

[0075] Step b2: Determine the feature descriptor of the first keypoint based on the BRIEF algorithm and the orientation of the first keypoint.

[0076] In this step, the feature descriptor of each first key point is determined based on the Binary Robust Independent Elementary Features (BRIEF) algorithm and the orientation of each first key point calculated in step a1.

[0077] BRIEF is a binary feature descriptor generated by comparing the gray values ​​of randomly selected pixel pairs within the neighborhood of a keypoint / feature point.

[0078] Specifically, for each keypoint / feature point, a pair of pixels is randomly selected within its neighborhood; the grayscale values ​​of each pair are compared. If the grayscale value of the first pixel is less than that of the second pixel, the comparison result for that pixel pair is 1; otherwise, it is 0. The comparison results of all pixel pairs are concatenated to form a binary string, which is the feature descriptor for each keypoint / feature point. To enhance the rotation invariance of the BRIEF feature descriptor, ORB modifies BRIEF so that its feature descriptor rotates with the orientation of the feature point.

[0079] Step b3: Based on the feature descriptor of the first key point and the feature descriptors of each first feature point, determine whether there is a first feature point among the multiple first feature points that matches the first key point.

[0080] In this step, the method for determining the feature descriptors of each first feature point is the same as the method for determining the feature descriptors of each first key point. The specific implementation method can be referred to the description in step b2, and the same technical effect can be achieved. It will not be elaborated here.

[0081] The feature descriptor of each first key point is compared with the feature descriptor of each first feature point to determine the similarity between the two feature descriptors. If the similarity is high, the first feature point of the reference image can be matched with the key point of the target fundus needle insertion image.

[0082] Step b4: If so, determine the first key point as the second feature point of the target fundus needle insertion image.

[0083] In this step, if the feature descriptor of the first keypoint matches the feature descriptor of the first feature point, then the first keypoint is determined as the second feature point of the target fundus needle insertion image. Through the above steps, the ORB algorithm can effectively detect and describe feature points in an image and achieve feature point matching.

[0084] Furthermore, feature point matching algorithms may also include scale-invariant feature transformation algorithms or accelerated robust feature algorithms.

[0085] In this embodiment of the application, in addition to the ORB algorithm, the feature point matching algorithm can also use the Scale-invariant feature transform (SIFT) algorithm or the Speed ​​Up Robust Feature (SURF) algorithm to achieve feature point matching.

[0086] Furthermore, step b3 specifically includes:

[0087] Step c1: For each first key point in the target fundus needle insertion image, calculate the Hamming distance between the feature descriptor of the first key point and the feature descriptor of each first feature point.

[0088] In this step, since the BRIEF feature descriptors are binary strings, the similarity between feature descriptors can be determined by calculating the Hamming distance between the feature descriptor of the first keypoint and the feature descriptors of each of the first feature points. The Hamming distance is a measure of the difference between two strings, defined as the number of bits that differ at each position in the two strings. The larger the Hamming distance between two strings, the greater the difference; conversely, the smaller the Hamming distance, the smaller the difference, and therefore the more similar they are.

[0089] Step c2: If the minimum value in the Hamming distance is less than or equal to the preset minimum Hamming distance threshold, the first feature point corresponding to the minimum value in the Hamming distance is determined as the first feature point that matches the first key point.

[0090] In this step, if the minimum Hamming distance is less than or equal to the preset minimum Hamming distance threshold, it indicates that the similarity of the first feature point most similar to the first keypoint meets the feature point matching requirements, and the first feature point can be determined as the first feature point matching the first keypoint. The preset minimum Hamming distance threshold is the minimum Hamming distance that can achieve keypoint matching, determined based on the actual measurement accuracy.

[0091] Step c3: If the minimum value of the Hamming distance is greater than the preset minimum Hamming distance threshold, determine that there is no first feature point among the multiple first feature points that matches the first key point.

[0092] In this step, if the minimum value of the Hamming distance is greater than the preset minimum Hamming distance threshold, it means that there is no first feature point that matches the first key point in the first feature point, that is, the first key point is not a key point at the salient feature position of the surgical needle.

[0093] Furthermore, in S103, based on multiple second feature points and multiple first feature points, the multiple second feature points are perspective transformed to the plane where the reference image is located, specifically including:

[0094] Step d1: Select at least four sets of corresponding feature points from multiple second feature points and multiple first feature points. Based on the coordinates of the at least four sets of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system, calculate the perspective transformation matrix between the target fundus needle insertion image coordinate system and the reference image coordinate system, and determine the perspective transformation matrix as the coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located.

[0095] In this step, since multiple second feature points in the target fundus needle insertion image have corresponding first feature points in the reference image, and the two correspond to the same position of the surgical needle, a coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located can be established based on the correspondence between multiple second feature points and multiple first feature points.

[0096] In this embodiment, at least four sets of corresponding feature points are selected from multiple second feature points and multiple first feature points. Based on the coordinates of these four sets of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system, the perspective transformation matrix between the target fundus needle insertion image coordinate system and the reference image coordinate system is calculated. Specifically, since the perspective transformation matrix can be completely solved using only four sets of feature points, this application uses four sets of feature points as an example. The specific steps for solving the perspective transformation matrix are as follows:

[0097] (0,0)→(x0,y0),(1,0)→(x1,y1),(1,1)→(x2,y2),(0,1)→(x3,y3);

[0098] Where (0,0)(1,0)(1,1)(0,1) are the coordinates of the four second feature points selected in the target fundus needle insertion image coordinate system; (x0,y0)(x1,y1)(x2,y2)(x3,y3) are the coordinates of the four first feature points corresponding to the four second feature points in the reference image coordinate system;

[0099] The coordinate mapping relationship between the plane containing the target fundus needle insertion image and the plane containing the reference image can be expressed as:

[0100]

[0101] Where u, v, and ω are the coordinates of the target fundus needle insertion image, specifically, the perspective transformation matrix. It is a 3×3 matrix used to transform a point (u,v) in the target fundus needle insertion image to a point (x,y) in the reference image, where In this secondary coordinate system, these points are represented as (u,v,1) and (x,y,ω). The perspective transformation matrix H can be divided into four parts. Specifically, the perspective transformation matrix...

[0102] Part One Used to represent linear transformations of images, including image scaling and rotation operations;

[0103] Part Two Used to represent perspective transformation operations in images;

[0104] Part 3 T3 = [a 31 a 32 Used to represent image translation operations;

[0105] Part 4a 33 To achieve a full scaling transformation, 'a' is typically set to 'a'. 33 =1.

[0106] Because usually a is set 33 =1. The perspective transformation matrix has a total of 8 unknown parameters, so 4 sets of coordinate pairs, or 8 equations, are needed to solve it. Therefore, the expression for the transformed (x,y) is:

[0107]

[0108] Based on the above formula transformation, the following correspondence is obtained:

[0109] x0 = a 31 ;

[0110] x1=a 11 +a 31 -a 13 x1;

[0111] x2=a 11 +a 21 +a 31 -a 13 x2-a 23 x2;

[0112] x3=a 21 +a 31 -a 23 x3;

[0113] y0=a 32 ;

[0114] y1=a 12 +a 32 -a 13 y1;

[0115] y2=a 12 +a 22 +a 32 -a 13 y2-a 23 y2;

[0116] y3=a 22 +a 32 -a 23 y3;

[0117] Several auxiliary variables are defined here:

[0118] Δx1=x1-x2; Δx2=x3-x2; Δx3=x0-x1+x2-x3;

[0119] Δy1=y1-y2; Δy2=y3-y2; Δy3=y0-y1+y2-y3;

[0120] During perspective transformation, when both Δx3 and Δy3 are 0, it indicates that the plane containing the target fundus needle insertion image is parallel to the plane containing the reference image. This is an affine transformation, and the specific parameters of the perspective transformation matrix are:

[0121] a 11 =x1-x0;

[0122] a 21 =x2-x1;

[0123] a 31 =x0;

[0124] a 12 =y1-y0;

[0125] a 22 =y2-y1;

[0126] a 32 =y0;

[0127] a 13 =0;

[0128] a 23 =0.

[0129] During perspective transformation, when Δx3 and Δy3 are not both 0, it is a perspective transformation. The specific parameters of the perspective transformation matrix are:

[0130] a 11 =x1-x0+a 12 x1;

[0131] a 21 =x3-x0+x 12 x2;

[0132] a 31 =x0;

[0133] a 12 =y1-y y +a 13 y1;

[0134] a 22 =y3-y0+a 23 y3;

[0135] a 32 =y0;

[0136]

[0137] Step d2: Based on the coordinate mapping relationship and multiple second feature points, calculate the coordinates of the multiple second feature points in the coordinate system of the reference image, and perform perspective transformation on the plane of the reference image.

[0138] In this step, based on the coordinate mapping relationship obtained in step d1, the coordinates of each second feature point in the target fundus needle insertion image are transformed, and the multiple second feature points are transformed to the plane of the reference image.

[0139] Furthermore, S104 specifically includes:

[0140] Step e1: Select multiple fourth feature points from multiple first feature points to describe the length of the surgical needle from the tip to the end of the needle tip, and calculate the length of the surgical needle in the reference image based on the coordinates of the multiple fourth feature points in the reference image coordinate system.

[0141] In this step, please refer to Figure 4 , Figure 4 This is a schematic diagram of the reference image in an embodiment of this application. Figure 4 As shown, multiple fourth feature points are selected from multiple first feature points marked on the reference image to describe the length of the surgical needle from the tip to the end of the needle. Based on the coordinates of the multiple fourth feature points in the reference image coordinate system, the length of the surgical needle in the reference image is calculated using a multi-point fitting curve calculation method.

[0142] Since the surgical needle image in fundus needle insertion surgery is not a straight line, a multi-point fitting curve algorithm is needed. Based on multiple fourth feature points used to describe the length from the tip of the surgical needle to the tip, the algorithm uses tools such as least squares and nonlinear least squares to determine the best fitting curve. This ensures that the fitting curve is as close as possible to the actual curve of the surgical needle, thereby improving the accuracy of calculating the length of the surgical needle in the reference image.

[0143] Step e2: Calculate the length of the surgical needle in the target fundus needle insertion image after perspective transformation based on the coordinates of the third feature points corresponding to multiple fourth feature points in the target fundus needle insertion image after perspective transformation.

[0144] Please see Figure 5 and Figure 6 , Figure 5 This is a schematic diagram of the target fundus needle insertion image in an embodiment of this application; Figure 6 This is a schematic diagram of the target fundus needle insertion image after perspective transformation in an embodiment of this application. For example... Figure 5 and Figure 6 As shown, multiple second feature points in the target fundus needle insertion image are subjected to perspective transformation, so that the feature points at different angles and heights are transformed to the same height and angle as the reference image, thus obtaining multiple third feature points in the perspective-transformed target fundus needle insertion image.

[0145] In this step, since it is necessary to calculate the length of the surgical needle in the target fundus needle insertion image after perspective transformation, multiple third feature points that correspond to the fourth feature points are obtained. That is, multiple feature points in the transformed target fundus needle insertion image are used to describe the length from the end of the surgical needle to the tip of the needle. The coordinates of these third feature points in the target fundus needle insertion image after perspective transformation are used to calculate the length of the surgical needle in the target fundus needle insertion image after perspective transformation by using the multi-point fitting curve calculation method. The description of the multi-point fitting curve calculation method can refer to the description in step e2, and it can achieve the same technical effect, so it will not be elaborated here.

[0146] Step e3: The difference between the length of the surgical needle in the reference image and the length in the target fundus needle insertion image after perspective transformation is determined as the insertion depth of the surgical needle in the target fundus needle insertion image.

[0147] In this step, the difference between the length of the surgical needle in the reference image, calculated using a multi-point fitting curve, and the length in the target fundus insertion image after perspective transformation, is determined as the insertion depth of the surgical needle in the target fundus insertion image. Specifically, the calculated insertion depth can be accurate to within 3 pixels of the true depth. Since the calculated insertion depth is a pixel length, it needs to be divided by the resolution of the reference image to obtain the actual length of the surgical needle insertion in the target fundus insertion image.

[0148] This application provides a method for determining the needle insertion depth in a fundus needle insertion image, comprising: acquiring a target fundus needle insertion image, and determining multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner detection algorithm; the target fundus needle insertion image is an image acquired during fundus needle insertion surgery; determining multiple second feature points of the target fundus needle insertion image based on the multiple first key points, multiple first feature points of a reference image, and a feature point matching algorithm; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image before needle insertion; the multiple first feature points are used to describe the salient features of the surgical needle in the reference image; based on the multiple second feature points and the multiple first feature points, performing perspective transformation on the plane of the reference image to obtain multiple third feature points of the perspective-transformed target fundus needle insertion image; and determining the needle insertion depth of the surgical needle in the target fundus needle insertion image based on the multiple third feature points and the multiple first feature points. In this way, the fundus needle insertion images from different shooting angles are transformed to the same shooting angle as the reference image, and the ratio of image pixels to actual distance is unified, which improves the accuracy of needle insertion depth measurement in fundus needle insertion images.

[0149] Based on the same application concept, this application also provides a device for determining the needle depth in a fundus needle image, which corresponds to the method for determining the needle depth in a fundus needle image provided in the above embodiments. Since the principle of the device in this application is similar to the method for determining the needle depth in a fundus needle image in the above embodiments of this application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0150] Please see Figure 7 , Figure 7 This is a functional block diagram of a device for determining the needle insertion depth in a fundus needle insertion image, provided in an embodiment of this application. Figure 7 As shown, the device 700 for determining the needle insertion depth in a fundus needle insertion image includes:

[0151] The acquisition module 710 is used to acquire the target fundus needle insertion image and, based on the FAST corner detection algorithm, determine multiple first key points of the surgical needle in the target fundus needle insertion image; the target fundus needle insertion image is an image acquired during the fundus needle insertion surgery.

[0152] The matching module 720 is used to determine multiple second feature points of the target fundus needle insertion image based on multiple first key points, multiple first feature points of the reference image, and a feature point matching algorithm; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image before the surgical needle is inserted; the multiple first feature points are used to describe the salient features of the surgical needle in the reference image.

[0153] The conversion module 730 is used to perform perspective transformation on multiple second feature points and multiple first feature points to the plane of the reference image, thereby obtaining multiple third feature points of the target fundus needle insertion image after perspective transformation.

[0154] The determination module 740 is used to determine the insertion depth of the surgical needle in the target fundus needle insertion image based on multiple third feature points and multiple first feature points.

[0155] Furthermore, the matching module 720 is used to determine multiple first feature points of the reference image according to the following steps:

[0156] Based on the FAST corner detection algorithm, multiple secondary key points of the surgical needle in the reference image were determined.

[0157] Based on preset filtering rules, multiple target points are selected from multiple second key points, and these multiple target points are identified as multiple first feature points of the reference image; the multiple target points are second key points at the significant feature positions of the surgical needle in the reference image.

[0158] In one possible implementation, the feature point matching algorithm includes the ORB algorithm. When the matching module 720 determines multiple second feature points of the target fundus needle insertion image based on multiple first key points, multiple first feature points of the reference image, and the feature point matching algorithm, the matching module 720 is specifically used for:

[0159] For each first key point in the target fundus needle insertion image, the orientation of the first key point is determined based on its gray centroid.

[0160] Based on the BRIEF algorithm and the orientation of the first keypoint, determine the feature descriptor of the first keypoint;

[0161] Based on the feature descriptor of the first key point and the feature descriptor of each first feature point, determine whether there is a first feature point among multiple first feature points that matches the first key point;

[0162] If so, the first key point is determined as the second feature point of the target fundus needle insertion image.

[0163] Furthermore, when determining whether there is a first feature point matching the first key point among multiple first feature points based on the feature descriptor of the first key point and the feature descriptors of each first feature point, the matching module 720 is specifically used for:

[0164] For each first key point in the target fundus needle insertion image, calculate the Hamming distance between the feature descriptor of the first key point and the feature descriptor of each first feature point;

[0165] If the minimum value in the Hamming distance is less than or equal to the preset minimum Hamming distance threshold, the first feature point corresponding to the minimum value in the Hamming distance is determined as the first feature point that matches the first key point;

[0166] If the minimum value of the Hamming distance is greater than the preset minimum Hamming distance threshold, it is determined that there is no first feature point among the multiple first feature points that matches the first key point.

[0167] Furthermore, when the conversion module 730 is used to perform perspective transformation of multiple second feature points to the plane of the reference image based on multiple second feature points and multiple first feature points, the conversion module 730 is specifically used for:

[0168] Select at least four sets of corresponding feature points from multiple second feature points and multiple first feature points. Based on the coordinates of the at least four sets of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system, calculate the perspective transformation matrix between the target fundus needle insertion image coordinate system and the reference image coordinate system, and determine the perspective transformation matrix as the coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located.

[0169] Based on the coordinate mapping relationship and multiple second feature points, the coordinates of the multiple second feature points in the reference image coordinate system are calculated, and the multiple second feature points are perspective transformed to the plane where the reference image is located.

[0170] Furthermore, when determining the insertion depth of the surgical needle in the target fundus needle insertion image based on multiple third feature points and multiple first feature points, the determining module 740 is specifically used for:

[0171] Multiple fourth feature points are selected from multiple first feature points to describe the length of the surgical needle from the tip to the tip, and the length of the surgical needle in the reference image is calculated based on the coordinates of the multiple fourth feature points in the reference image coordinate system.

[0172] The length of the surgical needle in the target fundus needle insertion image after perspective transformation is calculated based on the coordinates of the third feature points corresponding to multiple fourth feature points in the target fundus needle insertion image after perspective transformation.

[0173] The difference between the length of the surgical needle in the reference image and the length in the target fundus needle insertion image after perspective transformation is determined as the insertion depth of the surgical needle in the target fundus needle insertion image.

[0174] Furthermore, feature point matching algorithms include scale-invariant feature transformation algorithms or accelerated robust feature algorithms.

[0175] This application provides a device for determining the needle insertion depth in a fundus needle insertion image, comprising: an acquisition module for acquiring a target fundus needle insertion image and determining multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner detection algorithm; the target fundus needle insertion image is an image acquired during fundus needle insertion surgery; a matching module for determining multiple second feature points of the target fundus needle insertion image based on the multiple first key points, multiple first feature points of a reference image, and a feature point matching algorithm; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image before needle insertion; the multiple first feature points are used to describe the salient features of the surgical needle in the reference image; a conversion module for perspective transformation of the multiple second feature points to the plane of the reference image based on the multiple second feature points and the multiple first feature points, to obtain multiple third feature points of the perspective-transformed target fundus needle insertion image; and a determination module for determining the needle insertion depth of the surgical needle in the target fundus needle insertion image based on the multiple third feature points and the multiple first feature points. In this way, the fundus needle insertion images from different shooting angles are transformed to the same shooting angle as the reference image, and the ratio of image pixels to actual distance is unified, which improves the accuracy of needle insertion depth measurement in fundus needle insertion images.

[0176] Based on the same application concept, please refer to Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application, such as... Figure 8 As shown, the electronic device 800 includes a processor 810, a memory 820, and a bus 830.

[0177] The memory 820 stores machine-readable instructions executable by the processor 810. When the electronic device 800 is running, the processor 810 and the memory 820 communicate through the bus 830. When the machine-readable instructions are executed by the processor 810, they perform the steps of the method for determining the needle depth in the fundus needle image provided in the above embodiment. For specific implementation, please refer to the method embodiment, which will not be repeated here.

[0178] Based on the same concept, this application also provides a computer-readable storage medium storing a computer program. When the computer program is run by a processor, it executes the steps of the method for determining the needle insertion depth in the fundus needle insertion image provided in the above embodiment. For specific implementation details, please refer to the method embodiment, which will not be repeated here.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described apparatus and unit can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0180] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0181] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] In addition, the functional units in the embodiments provided in this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0183] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0184] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0185] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The protection scope of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application. All should be covered within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to determine a method for determining the needle insertion depth in a fundus needle insertion image. The method includes: A target fundus needle insertion image is acquired, and multiple first key points of the surgical needle in the target fundus needle insertion image are determined based on the FAST corner detection algorithm; the target fundus needle insertion image is an image acquired during the fundus needle insertion surgery. Based on the plurality of first key points, the plurality of first feature points of the reference image, and the feature point matching algorithm, a plurality of second feature points of the target fundus needle insertion image are determined; the plurality of second feature points are points that match the plurality of first feature points among the plurality of first key points; the reference image is a pre-acquired image before the surgical needle is inserted; the plurality of first feature points are used to describe the salient features of the surgical needle in the reference image; Based on the plurality of second feature points and the plurality of first feature points, the plurality of second feature points are perspective-transformed to the plane where the reference image is located to obtain a plurality of third feature points of the target fundus needle insertion image after perspective transformation. Based on the plurality of third feature points and the plurality of first feature points, the insertion depth of the surgical needle in the target fundus needle insertion image is determined; The step of determining the insertion depth of the surgical needle in the target fundus needle image based on the plurality of third feature points and the plurality of first feature points includes: Multiple fourth feature points are selected from the plurality of first feature points to describe the length of the surgical needle from the tip to the tip of the needle, and the length of the surgical needle in the reference image is calculated based on the coordinates of the plurality of fourth feature points in the reference image coordinate system. Based on the coordinates of the third feature points corresponding to the plurality of fourth feature points in the fluoroscopically transformed target fundus needle insertion image, the length of the surgical needle in the fluoroscopically transformed target fundus needle insertion image is calculated. The difference between the length of the surgical needle in the reference image and the length in the target fundus needle insertion image after perspective transformation is determined as the insertion depth of the surgical needle in the target fundus needle insertion image.

2. The electronic device according to claim 1, characterized in that, The reference image is determined according to the following steps: Based on the FAST corner detection algorithm, multiple secondary key points of the surgical needle in the reference image were determined. Based on preset filtering rules, multiple target points are selected from the multiple second key points, and the multiple target points are determined as multiple first feature points of the reference image; the multiple target points are second key points at the significant feature positions of the surgical needle in the reference image.

3. The electronic device according to claim 1, characterized in that, The feature point matching algorithm includes the ORB algorithm. The step of determining multiple second feature points of the target fundus needle insertion image based on the multiple first key points, multiple first feature points of the reference image, and the feature point matching algorithm includes: For each first key point in the target fundus needle insertion image, the orientation of the first key point is determined based on the gray centroid of the first key point; Based on the BRIEF algorithm and the orientation of the first key point, the feature descriptor of the first key point is determined; Based on the feature descriptor of the first key point and the feature descriptor of each first feature point, determine whether there is a first feature point among the plurality of first feature points that matches the first key point; If so, the first key point is determined as the second feature point of the target fundus needle insertion image.

4. The electronic device according to claim 3, characterized in that, The determination of whether there is a first feature point matching the first key point among the plurality of second feature points based on the feature descriptor of the first key point and the feature descriptors of each first feature point includes: For each first key point in the target fundus needle insertion image, calculate the Hamming distance between the feature descriptor of the first key point and the feature descriptor of each first feature point; If the minimum value in the Hamming distance is less than or equal to the preset minimum Hamming distance threshold, the first feature point corresponding to the minimum value in the Hamming distance is determined as the first feature point that matches the first key point; If the minimum value of the Hamming distance is greater than the preset minimum Hamming distance threshold, it is determined that there is no first feature point among the plurality of first feature points that matches the first key point.

5. The electronic device according to claim 1, characterized in that, The step of perspective transforming the plurality of second feature points onto the plane of the reference image based on the plurality of second feature points and the plurality of first feature points includes: At least four sets of corresponding feature points are selected from the plurality of second feature points and the plurality of first feature points. Based on the coordinates of the at least four sets of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system, the perspective transformation matrix between the target fundus needle insertion image coordinate system and the reference image coordinate system is calculated. The perspective transformation matrix is ​​then determined as the coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located. Based on the coordinate mapping relationship and the plurality of second feature points, the coordinates of the plurality of second feature points in the reference image coordinate system are calculated, and the plurality of second feature points are perspective transformed to the plane where the reference image is located.

6. The electronic device according to claim 1, characterized in that, The feature point matching algorithm includes a scale-invariant feature transformation algorithm or an accelerated robust feature algorithm.

7. A device for determining the needle insertion depth in a fundus needle insertion image, characterized in that, The device for determining the needle insertion depth in the fundus needle insertion image includes: The acquisition module is used to acquire the target fundus needle insertion image and, based on the FAST corner detection algorithm, determine multiple first key points of the surgical needle in the target fundus needle insertion image; the target fundus needle insertion image is an image acquired during the fundus needle insertion surgery. A matching module is used to determine multiple second feature points of the target fundus needle insertion image based on the multiple first key points, multiple first feature points of the reference image, and a feature point matching algorithm; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image before the surgical needle is inserted; the multiple first feature points are used to describe the salient features of the surgical needle in the reference image; The conversion module is used to perform perspective transformation of the plurality of second feature points to the plane where the reference image is located, based on the plurality of second feature points and the plurality of first feature points, to obtain a plurality of third feature points of the perspective-transformed target fundus needle insertion image; The determination module is used to determine the insertion depth of the surgical needle in the target fundus needle insertion image based on the plurality of third feature points and the plurality of first feature points; When determining the insertion depth of the surgical needle in the target fundus needle insertion image based on the plurality of third feature points and the plurality of first feature points, the determining module is specifically used for: Multiple fourth feature points are selected from the plurality of first feature points to describe the length of the surgical needle from the tip to the tip of the needle, and the length of the surgical needle in the reference image is calculated based on the coordinates of the plurality of fourth feature points in the reference image coordinate system. Based on the coordinates of the third feature points corresponding to the plurality of fourth feature points in the fluoroscopically transformed target fundus needle insertion image, the length of the surgical needle in the fluoroscopically transformed target fundus needle insertion image is calculated. The difference between the length of the surgical needle in the reference image and the length in the target fundus needle insertion image after perspective transformation is determined as the insertion depth of the surgical needle in the target fundus needle insertion image.

Images