Method and device for determining needle insertion depth in fundus needle insertion image, equipment and medium

Abstract

The invention provides a method, device and equipment for determining the needle insertion depth in a fundus needle insertion image and a medium, and the method comprises the steps: obtaining a target fundus needle insertion image, and determining a plurality of first key points of a surgical needle in the target fundus needle insertion image based on a FAST corner detection algorithm; determining a plurality of second feature points of the target fundus needle insertion image based on the plurality of first key points, a plurality of first feature points of the reference image and a feature point matching algorithm; based on the plurality of second feature points and the first feature points, carrying out perspective transformation on the plurality of second feature points to a plane where the reference image is located, and obtaining a plurality of third feature points of the target fundus needle insertion image after perspective transformation; and based on the plurality of third feature points and the first feature point, determining the needle insertion depth of the surgical needle in the target fundus needle insertion image. The fundus needle insertion images of different shooting visual angles are converted to the same shooting visual angle as the reference image, the proportion of image pixels to the actual distance is unified, and the accuracy of needle insertion depth measurement in the fundus needle insertion images is improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a method, device, equipment and medium for determining a needle insertion depth in a fundus needle insertion image. Background Art

[0002] Needle insertion depth is an important evaluation indicator for the clinical registration of ophthalmic surgical robots, which is used to assess whether ophthalmic surgical robots are more stable than human hands.

[0003] The existing needle insertion depth measurement method manually marks the feature points of the surgical needle in the fundus needle insertion image, and manually measures the needle insertion depth of the surgical needle in the fundus needle insertion image. The measurement error in the fundus needle insertion images under different shooting angles is large, and the accuracy of the needle insertion depth measurement in the fundus needle insertion image is low. Summary of the invention

[0004] In view of this, the embodiments of the present application at least provide a method, device, equipment and medium for determining the needle insertion depth in the fundus needle insertion image, transforming the fundus needle insertion images of different shooting angles to the same shooting angle as the reference image, unifying the ratio of image pixels to actual distance, and improving the accuracy of needle insertion depth measurement in the fundus needle insertion image.

[0005] This application mainly includes the following aspects:

[0006] In a first aspect, an embodiment of the present application provides a method for determining a needle insertion depth in a fundus needle insertion image, the method comprising:

[0007] Acquire a target fundus needle insertion image, and determine multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during the fundus needle insertion surgery;

[0008] Based on the multiple first key points, the multiple first feature points of the reference image and the feature point matching algorithm, multiple second feature points of the target fundus needle insertion image are determined; the multiple second feature points are points among the multiple first key points that match the multiple first feature points; the reference image is a pre-acquired image of the surgical needle before the needle is inserted; the multiple first feature points are used to describe the significant features of the surgical needle in the reference image;

[0009] Based on the plurality of second feature points and the plurality of first feature points, perspective transforming the plurality of second feature points to the plane where the reference image is located, to obtain a plurality of third feature points of the target fundus needle insertion image after perspective transformation;

[0010] Based on the plurality of third feature points and the plurality of first feature points, the insertion depth of the surgical needle in the target fundus needle insertion image is determined.

[0011] In a second aspect, an embodiment of the present application further provides a device for determining a needle insertion depth in a fundus needle insertion image, wherein the device for determining a needle insertion depth in a fundus needle insertion image comprises:

[0012] An acquisition module is used to acquire a target fundus needle insertion image and determine a plurality of first key points of the surgical needle in the target fundus needle insertion image based on a FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during the fundus needle insertion surgery;

[0013] A matching module, for determining a plurality of second feature points of the target fundus needle insertion image based on the plurality of first key points, a plurality of first feature points of the reference image, and a feature point matching algorithm; the plurality of second feature points are points among the plurality of first key points that match the plurality of first feature points; the reference image is a pre-acquired image of the surgical needle before the needle is inserted; the plurality of first feature points are used to describe the significant features of the surgical needle in the reference image;

[0014] A conversion module, configured to perspectively transform the plurality of second feature points to the plane where the reference image is located based on the plurality of second feature points and the plurality of first feature points, so as to obtain a plurality of third feature points of the target fundus needle insertion image after perspective transformation;

[0015] A determination module is used to determine the insertion depth of the surgical needle in the target fundus needle insertion image based on the multiple third feature points and the multiple first feature points.

[0016] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for determining the needle insertion depth in the fundus needle insertion image as described above.

[0017] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for determining the needle insertion depth in the fundus needle insertion image as described above are executed.

[0018] The embodiments of the present application provide a method, device, equipment and medium for determining the needle insertion depth in a fundus needle insertion image, wherein the method includes: obtaining a target fundus needle insertion image, and determining multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during the fundus needle insertion surgery; based on the multiple first key points, multiple first feature points of the reference image and a feature point matching algorithm, determining multiple second feature points of the target fundus needle insertion image; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image of the surgical needle before insertion; the multiple first feature points are used to describe the significant features of the surgical needle in the reference image; based on the multiple second feature points and the multiple first feature points, the multiple second feature points are perspective transformed to the plane where the reference image is located to obtain multiple third feature points of the target fundus needle insertion image after the perspective transformation; based on the multiple third feature points and the multiple first feature points, determining the needle insertion depth of the surgical needle in the target fundus needle insertion image. In this way, the fundus needle insertion images of different shooting angles are transformed to the same shooting angle as the reference image, the ratio of image pixels to actual distance is unified, and the accuracy of needle insertion depth measurement in the fundus needle insertion images is improved.

[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flow chart showing a method for determining the needle insertion depth in a fundus needle insertion image provided by an embodiment of the present application is shown;

[0022] Figure 2 A schematic diagram of a surgical needle in an embodiment of the present application at a first shooting angle is shown;

[0023] Figure 3 A schematic diagram of a surgical needle in an embodiment of the present application at a second shooting angle is shown;

[0024] Figure 4 A schematic diagram of a reference image in an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram of a target fundus needle insertion image in an embodiment of the present application is shown;

[0026] Figure 6 A schematic diagram of a target fundus needle insertion image after perspective transformation in an embodiment of the present application is shown;

[0027] Figure 7 A functional module diagram of a device for determining a needle insertion depth in a fundus needle insertion image provided by an embodiment of the present application is shown;

[0028] Figure 8 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0030] The following is a detailed description of a method for determining the needle insertion depth in a fundus needle insertion image provided by an embodiment of the present application. Figure 1 , Figure 1 This is a flow chart of a method for determining the needle insertion depth in a fundus needle insertion image provided by an embodiment of the present application. Figure 1 As shown, the method includes:

[0031] S101, obtaining a target fundus needle insertion image, and determining multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during the fundus needle insertion surgery.

[0032] In this step, the target fundus needle insertion image is first obtained. In the embodiment of the present application, the target fundus needle insertion image is an image collected during the fundus needle insertion surgery. Specifically, the images during the surgery are obtained by a microscope or endoscope. In order to ensure image quality and measurement accuracy, it is necessary to use high-resolution camera equipment and reduce image distortion and noise. Specifically, after obtaining the target fundus needle insertion image, the target fundus needle insertion image can also be pre-processed, including steps such as denoising, contrast enhancement and distortion correction, so as to improve the clarity of the target fundus needle insertion image.

[0033] After acquiring the target fundus needle insertion image, multiple first key points of the surgical needle in the target fundus needle insertion image are determined based on the Features From Accelerated Segment Test (FAST) corner point detection algorithm. Among them, the FAST corner point detection algorithm is a feature point detection method used in computer vision, which can effectively detect corner points in the image as key points / feature points for image feature extraction. In this step, multiple first key points of the surgical needle in the target fundus needle insertion image are extracted by the FAST corner point detection algorithm.

[0034] Specifically, the principle of the FAST corner detection algorithm is to select a pixel p from the image, whose gray value is I p , set a preset threshold t, and draw a discretized Bresenham circle with a radius of 3 using the Bresenham arc algorithm with the pixel as the center. There are 16 pixels on the boundary of the circle. If the grayscale value of n consecutive pixels on the Bresenham circle is less than I p -t or greater than I p +t, then this point can be judged as a corner point (the value of n can be 12 or 9).

[0035] The FAST corner detection algorithm first needs to select multiple images of the scene to be detected, select an appropriate threshold n (n<12), and extract multiple key points / feature points as training data.

[0036] For a pixel point p on an image, the pixel points on its surrounding neighborhood circle with positions x, x∈{1,…,16} are represented as p→x. The following judgment formula can be used to divide the point p→x into three categories:

[0037]

[0038] Let P be the set of all pixels in the training image set. Any position x in the neighborhood circle around pixel p can divide the set P into three parts P d , P s and P b , where P d The definition of P is as follows: s and P b Definition and P d similar:

[0039] P d ={p∈P∶S p→s =d};

[0040] In other words, for any given position x, it can classify all points in the image into three categories. The first category P d Includes all pixels at position x that are darker than the center pixel p under threshold t, the second category P s Includes all pixels at position x that are close to the center pixel p under threshold t, P b Includes all pixels at position x that are brighter than the center pixel p under threshold t.

[0041] Define a bool variable K for each selected pixel p p , if p is a corner point, then K p True if the value is true, otherwise it is false.

[0042] The extracted key point / feature point set is trained, and a decision tree is established using the ID3 algorithm. The decision tree is divided by the x-th pixel point. Specifically, the ID3 algorithm is a greedy algorithm used to construct a decision tree. The ID3 algorithm originated from the Concept Learning System (CLS), and uses the rate of decrease of information entropy as the criterion for selecting test attributes. That is, at each node, the attribute with the highest information gain that has not been used for division is selected as the division criterion, and then this process is continued until the generated decision tree can perfectly classify the training examples. For the set P, the entropy value is:

[0043]

[0044] Where c is the number of corner points, is the number of non-corner points. The information gain obtained is:

[0045] ΔH=H(P)-H(P d )-H(P s )-H(P b );

[0046] Select the position with the maximum information gain for division and obtain a decision tree.

[0047] In the case where there are multiple feature points in the vicinity, non-maximal suppression is required. Each detected corner point is given a quantized value V, and then the V values ​​of adjacent corner points are compared to retain the point with the largest V value in the local neighborhood. The V value can be defined as the sum of the absolute grayscale differences between the key point / feature point and the 16 neighboring pixels, that is:

[0048]

[0049] In the above formula, S bright The gray value of the 16 neighboring pixels is greater than I p+t is a set of pixels, and S dark Indicates those gray values ​​less than I p -t is a collection of pixels.

[0050] Through the FAST corner detection algorithm, multiple first key points of the surgical needle in the target fundus needle insertion image can be identified.

[0051] S102, based on multiple first key points, multiple first feature points of the reference image and a feature point matching algorithm, determine multiple second feature points of the target fundus needle insertion image; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image of the surgical needle before insertion; the multiple first feature points are used to describe the significant features of the surgical needle in the reference image.

[0052] In this step, a feature matching algorithm is used to match multiple first key points of the surgical needle in the target fundus needle insertion image with multiple first feature points of the reference image, and the first key points that are successfully matched are used as second feature points, marked in the target fundus needle insertion image, and determined as multiple second feature points of the target fundus needle insertion image. Among them, the reference image is a pre-acquired image of the surgical needle before the needle is inserted. Specifically, the fundus needle insertion image includes an image before the surgical needle is inserted and an image after the surgical needle is inserted. The image before the surgical needle is inserted is further subdivided into an image in which the needle tip is close to the retina but does not reach the retina, an image in which the needle tip just reaches the retina but does not pierce the retina, and an image in which the needle tip is withdrawn away from the retina. In the embodiment of the present application, the first clear image in which the needle tip is close to the retina but does not reach the retina in the image before the surgical needle is inserted is selected as the reference image. Multiple first feature points are used to describe the significant features of the surgical needle in the reference image, such as the needle tip of the surgical needle, the inflection point of the surgical needle bending, the connection point between the end of the surgical needle and the robotic arm, and the feature mark on the robotic arm.

[0053] S103, based on the plurality of second feature points and the plurality of first feature points, perspective transform the plurality of second feature points to the plane where the reference image is located, to obtain a plurality of third feature points of the target fundus needle insertion image after perspective transformation.

[0054] In this step, due to the complexity of the human eye structure and the limitations of the surgical environment, the images of the fundus needle are often taken from different angles and heights. Therefore, the resolution of the shooting angle is different, and the number of pixels of the surgical needle in different images is different. For details, please refer to Figure 2 and Figure 3 , Figure 2 is a schematic diagram of a surgical needle in an embodiment of the present application at a first shooting angle, Figure 3 Schematic diagram of a surgical needle in a second shooting angle in an embodiment of the present application. Figure 2 and Figure 3 As shown, Figure 2 and Figure 3 The surgical needles in the figure are the same surgical needle, and the length is L. However, due to the different shooting angle resolutions, the number of pixels of the surgical needle in different images is different. Therefore, we need to unify the fundus needle insertion images to the same perspective and then perform pixel statistics to facilitate the subsequent calculation of the needle insertion depth. Specifically, based on the multiple second feature points in the target fundus needle insertion image and the multiple first feature points in the reference image, a mapping relationship between the two image coordinate systems is established, and the multiple second feature points in the perspective of the target fundus needle insertion image are unified to the plane of the reference image through perspective transformation, and multiple third feature points of the target fundus needle insertion image after perspective transformation are obtained.

[0055] S104, determining the insertion depth of the surgical needle in the target fundus needle insertion image based on the plurality of third feature points and the plurality of first feature points.

[0056] In the embodiment of the present application, since the part of the surgical needle that pierces the retina will not be displayed in the fundus needle insertion image, the insertion depth of the surgical needle in the target fundus needle insertion image can be obtained by comparing the length of the surgical needle in the reference image and the target fundus needle insertion image after perspective transformation. The length of the surgical needle in the reference image and the target fundus needle insertion image after perspective transformation can be calculated based on the multiple third feature points and the multiple first feature points.

[0057] Further, a plurality of first feature points of the reference image are determined according to the following steps:

[0058] Step a1: determining multiple second key points of the surgical needle in the reference image based on the FAST corner point detection algorithm.

[0059] In this step, the FAST corner detection algorithm can be used to identify multiple second key points of the surgical needle on the reference image. Specifically, the description of extracting key points by the FAST corner detection algorithm in step a1 can refer to the description in S101, and can achieve the same technical effect, which is not repeated here.

[0060] Step a2, based on preset screening rules, screen out multiple target points from multiple second key points, and determine the multiple target points as multiple first feature points of the reference image; the multiple target points are second key points at the significant feature positions of the surgical needle in the reference image.

[0061] In this step, after identifying multiple second key points of the surgical needle in the reference image, the second key points at the significant feature positions of the surgical needle in the reference image are selected from the multiple second key points according to the preset screening rules as multiple first feature points of the reference image, and a mask (MASK) of the surgical needle is generated, so that the target fundus needle insertion image can extract structural features similar to the mask in the image based on the mask, that is, the surgical needle in the needle insertion image at the significant feature position of the surgical needle in the target fundus reference image. The preset screening rules can be manually annotated and screened second key points, or a neural network algorithm can be used to identify the significant features of the surgical needle in the reference image for algorithmic annotation, which is not specifically limited here.

[0062] In a possible implementation, the feature point matching algorithm includes an ORB algorithm. In S102, based on the multiple first key points, the multiple first feature points of the reference image, and the feature point matching algorithm, multiple second feature points of the target fundus needle insertion image are determined, specifically including:

[0063] Step b1, for each first key point of the target fundus needle insertion image, determine the direction of the first key point according to the grayscale centroid of the first key point.

[0064] In the embodiment of the present application, the feature point matching algorithm adopts the OrientedFAST and Rotated BRIEF (ORB) algorithm. Since the corner points detected by the FAST corner point detection algorithm, that is, each first key point of the target fundus needle insertion image, do not have directionality, the ORB algorithm assigns a direction to each first key point by calculating the grayscale centroid of the pixel point.

[0065] Specifically, the direction vector is calculated as follows:

[0066] Compute the moments of an image in the neighborhood of a keypoint:

[0067] m pq =∑ x,y x p y q I(x,y);

[0068] Among them, I(x,y) represents the grayscale value of the pixel, (x,y) is the coordinate in the neighborhood of the key point, and p and q are non-negative integers.

[0069] The center of mass position is calculated by the following formula:

[0070]

[0071] Among them, m 00 is the zero-order moment, representing the grayscale sum of all pixels in the neighborhood, m 10 and m01 are the first order moments about x and y respectively.

[0072] The angle θ of the direction vector is:

[0073]

[0074] This way, each first keypoint is assigned a direction.

[0075] Step b2: determining a feature descriptor of the first key point according to the BRIEF algorithm and the direction of the first key point.

[0076] In this step, a feature descriptor of each first key point is determined according to a Binary Robust Independent Elementary Features (BRIEF) algorithm and the direction of each first key point calculated in step a1.

[0077] Among them, BRIEF is a binary feature descriptor generated by comparing the grayscale values ​​of randomly selected pixel pairs in the neighborhood of key points / feature points.

[0078] Specifically, for each key point / feature point, a set of pixel pairs is randomly selected in its neighborhood; the grayscale values ​​of each pair of pixels are compared. If the grayscale value of the first pixel is less than the grayscale value of the second pixel, the comparison result of the pixel pair is 1, otherwise it is 0. The comparison results of all pixel pairs are concatenated to form a binary string, which is the feature descriptor of each key point / feature point. In order to enhance the rotation invariance of the BRIEF feature descriptor, ORB modifies BRIEF so that its feature descriptor rotates with the direction of the feature point.

[0079] Step b3: Based on the feature descriptor of the first key point and the feature descriptors of each first feature point, it is determined whether there is a first feature point matching the first key point among the plurality of first feature points.

[0080] In this step, the method for determining the feature descriptor of each first feature point is the same as the method for determining the feature descriptor of each first key point. The specific implementation method can refer to the description in step b2, and the same technical effect can be achieved, so it will not be repeated here.

[0081] The feature descriptor of each first key point is compared with the feature descriptors of each first feature point to determine the similarity between the two feature descriptors. If the similarity is high, the first feature point of the reference image can be matched with the key point of the target fundus needle insertion image.

[0082] Step b4: If yes, determine the first key point as the second feature point of the target fundus needle insertion image.

[0083] In this step, if the feature descriptor of the first key point matches the feature descriptor of the first feature point, the first key point is determined as the second feature point of the target fundus needle insertion image. Through the above steps, the ORB algorithm can effectively detect and describe the feature points in the image and achieve feature point matching.

[0084] Furthermore, the feature point matching algorithm may also include a scale-invariant feature transformation algorithm or an accelerated robust feature algorithm.

[0085] In the embodiment of the present application, in addition to the ORB algorithm, the feature point matching algorithm may also use a scale-invariant feature transform (SIFT) algorithm or a speed up robust feature (SURF) algorithm to achieve feature point matching.

[0086] Furthermore, step b3 specifically includes:

[0087] Step c1, for each first key point of the target fundus needle insertion image, calculate the Hamming distance between the feature descriptor of the first key point and the feature descriptors of each first feature point.

[0088] In this step, since the BRIEF feature descriptor is a binary string, the similarity between the feature descriptors can be determined by calculating the Hamming distance between the feature descriptor of the first key point and the feature descriptors of each first feature point. The Hamming distance is a measure of the difference between two strings, which is defined as the number of different bits at each position in the two strings. The larger the Hamming distance between the two strings, the greater the difference. Conversely, the smaller the Hamming distance, the smaller the difference, and the more similar they are.

[0089] Step c2: if the minimum value in the Hamming distance is less than or equal to a preset minimum Hamming distance threshold, determine the first feature point corresponding to the minimum value in the Hamming distance as the first feature point matching the first key point.

[0090] In this step, if the minimum value in the Hamming distance is less than or equal to the preset minimum Hamming distance threshold, it means that the similarity of the first feature point most similar to the first key point meets the matching requirements of the feature point, and the first feature point can be determined as the first feature point matching the first key point. The preset minimum Hamming distance threshold is the minimum Hamming distance that can achieve key point matching determined according to the actual measurement accuracy.

[0091] Step c3: if the minimum value of the Hamming distance is greater than a preset minimum Hamming distance threshold, it is determined that there is no first feature point matching the first key point among the plurality of first feature points.

[0092] In this step, if the minimum value in the Hamming distance is greater than the preset minimum Hamming distance threshold, it means that there is no first feature point matching the first key point in the first feature points, that is, the first key point is not a key point at the significant feature position of the surgical needle.

[0093] Further, in S103, based on the plurality of second feature points and the plurality of first feature points, perspective transforming the plurality of second feature points to the plane where the reference image is located specifically includes:

[0094] Step d1, select at least four groups of corresponding feature points from multiple second feature points and multiple first feature points, calculate the perspective transformation matrix of the target fundus needle insertion image coordinate system and the reference image coordinate system according to the coordinates of the at least four groups of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system, and determine the perspective transformation matrix as the coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located.

[0095] In this step, since the multiple second feature points of the target fundus needle insertion image have corresponding first feature points in the reference image, and the two correspond to the same position of the surgical needle, the coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located can be established based on the correspondence between the multiple second feature points and the multiple first feature points.

[0096] In the embodiment of the present application, at least four groups of corresponding feature points are selected from the plurality of second feature points and the plurality of first feature points, and the perspective transformation matrix of the target fundus needle insertion image coordinate system and the reference image coordinate system is calculated according to the coordinates of the at least four groups of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system. Specifically, since the perspective transformation matrix can be completely solved with four groups of feature points, the present application uses four groups of feature points as an example, and the following are the specific steps for solving the perspective transformation matrix:

[0097] (0,0)→(x0,y0),(1,0)→(x1,y1),(1,1)→(x2,y2),(0,1)→(x3,y3);

[0098] Among them, (0,0)(1,0)(1,1)(0,1) are the coordinates of the four second feature points selected in the target fundus needle insertion image coordinate system; (x0,y0)(x1,y1)(x2,y2)(x3,y3) are the coordinates of the four first feature points corresponding to the four second feature points in the reference image coordinate system;

[0099] The coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located can be expressed as:

[0100]

[0101] Among them, u and v, ω are the coordinates of the target fundus needle image. Specifically, the perspective transformation matrix is a 3×3 matrix used to transform the point (u,v) in the target fundus needle image to the point (x,y) in the reference image, where In the secondary coordinate system, these points are represented as (u, v, 1) and (x, y, ω). The perspective transformation matrix H can be split into four parts. Specifically, the perspective transformation matrix

[0102] Among them, the first part Used to represent image linear transformation, including image scaling and rotation operations;

[0103] Part 2 Perspective transformation operations used to represent images;

[0104] The third part T3 = [a 31 a 32 ] is used to represent the translation operation of the image;

[0105] Part IVa 33 To achieve full scale transformation, a is usually set to simplify calculations. 33 =1.

[0106] Since a 33 =1, the perspective transformation matrix has a total of 8 unknown parameters, so 4 sets of coordinate pairs, that is, 8 equations are needed to solve, so the expression of the transformed (x, y) is:

[0107]

[0108] According to the above formula transformation, the following corresponding relationship is obtained:

[0109] x0=a 31 ;

[0110] x1=a 11 +a 31 -a 13 x1;

[0111] x2=a 11 +a 21 +a 31 -a 13 x2-a 23 x2;

[0112] x3=a 21 +a 31 -a 23 x3;

[0113] y0=a 32 ;

[0114] y1=a 12 +a 32 -a 13 y1;

[0115] y2=a 12 +a 22 +a 32 -a 13 y2-a 23 y2;

[0116] y3=a 22 +a 32 -a 23 y3;

[0117] Among them, several auxiliary variables are defined:

[0118] Δx1=x1-x2; Δx2=x3-x2; Δx3=x0-x1+x2-x3;

[0119] Δy1=y1-y2; Δy2=y3-y2; Δy3=y0-y1+y2-y3;

[0120] In the perspective transformation process, when Δx3 and Δy3 are both 0, it means that the plane where the target fundus needle image is located is parallel to the plane where the reference image is located. This is an affine transformation, and the specific parameters of the perspective transformation matrix are:

[0121] a 11 =x1-x0;

[0122] a 21 =x2-x1;

[0123] a 31 = x0;

[0124] a 12 =y1-y0;

[0125] a 22 =y2-y1;

[0126] a 32 =y0;

[0127] a 13 =0;

[0128] a 23 =0.

[0129] In the perspective transformation process, when Δx3 and Δy3 are not both 0, it is a perspective transformation, and the specific parameters of the perspective transformation matrix are:

[0130] a 11 =x1-x0+a 12 x1;

[0131] a 21 =x3-x0+x 12 x2;

[0132] a 31 = x0;

[0133] a 12 =y1-y y +a 13 y1;

[0134] a 22 =y3-y0+a 23 y3;

[0135] a 32 =y0;

[0136]

[0137] Step d2, based on the coordinate mapping relationship and the plurality of second feature points, calculate the coordinates of the plurality of second feature points in the reference image coordinate system, and perspective transform the plurality of second feature points to the plane where the reference image is located.

[0138] In this step, according to the coordinate mapping relationship solved in step d1, coordinate transformation is performed on each second feature point in the target fundus needle insertion image, and multiple second feature points are perspectively transformed to the plane where the reference image is located.

[0139] Furthermore, S104 specifically includes:

[0140] Step e1, selecting a plurality of fourth feature points for describing the distance from the needle end to the needle tip of the surgical needle from the plurality of first feature points, and calculating the length of the surgical needle in the reference image according to the coordinates of the plurality of fourth feature points in the reference image coordinate system.

[0141] In this step, see Figure 4 , Figure 4 Schematic diagram of a reference image in an embodiment of the present application. Figure 4 As shown, a plurality of fourth feature points for describing the distance from the needle end to the needle tip of the surgical needle are selected from the plurality of first feature points marked on the reference image, and the length of the surgical needle in the reference image is calculated by a multi-point fitting curve calculation method based on the coordinates of the plurality of fourth feature points in the reference image coordinate system.

[0142] Since the image of the surgical needle for fundus needle surgery is not a straight line, it is necessary to use a multi-point fitting curve algorithm to determine the best fitting curve based on multiple fourth feature points used to describe the distance from the needle end to the needle tip, using tools such as the least squares method and the nonlinear least squares method. This way, the fitting curve can be as close as possible to the actual curve of the surgical needle, thereby improving the accuracy of calculating the length of the surgical needle in the reference image.

[0143] Step e2, calculating the length of the surgical needle in the target fundus needle insertion image after perspective transformation according to the coordinates of the third feature points corresponding to the plurality of fourth feature points in the target fundus needle insertion image after perspective transformation.

[0144] See also Figure 5 and Figure 6 , Figure 5 A schematic diagram of a target fundus needle insertion image in an embodiment of the present application; Figure 6 Schematic diagram of the target fundus needle insertion image after perspective transformation in the embodiment of the present application. Figure 5 and Figure 6 As shown, multiple second feature points of the target fundus needle insertion image are perspective transformed, so that the feature points at different angles and heights are transformed to the same height and angle as the reference image, thereby obtaining multiple third feature points in the target fundus needle insertion image after perspective transformation.

[0145] In this step, since it is necessary to calculate the length of the surgical needle in the target fundus needle insertion image after perspective transformation, the feature points corresponding to the fourth feature points are obtained from the multiple third feature points, that is, the multiple feature points in the transformed target fundus needle insertion image used to describe the area from the needle end to the needle tip of the surgical needle. The coordinates of these third feature points in the target fundus needle insertion image after perspective transformation are used to calculate the length of the surgical needle in the target fundus needle insertion image after perspective transformation through the calculation method of multi-point fitting curve. The description of the calculation method of multi-point fitting curve can refer to the description in step e2, and the same technical effect can be achieved, so it will not be repeated here.

[0146] Step e3, determining the difference between the length of the surgical needle in the reference image and the length in the target fundus needle insertion image after perspective transformation as the insertion depth of the surgical needle in the target fundus needle insertion image.

[0147] In this step, the difference between the length of the surgical needle in the reference image calculated by the multi-point fitting curve calculation method and the length in the target fundus needle insertion image after perspective transformation is determined as the insertion depth of the surgical needle in the target fundus needle insertion image. Specifically, the calculated insertion depth can be accurate to within 3 pixels of the true depth, and since the calculated insertion depth is a pixel length, it needs to be divided by the resolution of the reference image to obtain the actual length of the insertion depth of the surgical needle in the target fundus needle insertion image.

[0148] An embodiment of the present application provides a method for determining the depth of a surgical needle in a fundus needle insertion image, comprising: acquiring a target fundus needle insertion image, and determining multiple first key points of a surgical needle in the target fundus needle insertion image based on a FAST corner point detection algorithm; the target fundus needle insertion image is an image acquired during a fundus needle insertion operation; based on multiple first key points, multiple first feature points of a reference image, and a feature point matching algorithm, determining multiple second feature points of the target fundus needle insertion image; the multiple second feature points are points that match multiple first feature points among multiple first key points; the reference image is a pre-acquired image of the surgical needle before insertion; the multiple first feature points are used to describe the significant features of the surgical needle in the reference image; based on the multiple second feature points and the multiple first feature points, the multiple second feature points are perspectively transformed to the plane where the reference image is located to obtain multiple third feature points of the target fundus needle insertion image after perspective transformation; based on the multiple third feature points and the multiple first feature points, determining the depth of the surgical needle in the target fundus needle insertion image. In this way, the fundus needle insertion images of different shooting angles are transformed to the same shooting angle as the reference image, the ratio of image pixels to actual distance is unified, and the accuracy of needle insertion depth measurement in the fundus needle insertion images is improved.

[0149] Based on the same application concept, the embodiments of the present application also provide a device for determining the needle insertion depth in a fundus needle insertion image that corresponds to the method for determining the needle insertion depth in a fundus needle insertion image provided in the above embodiments. Since the principle of solving the problem by the device in the embodiments of the present application is similar to the method for determining the needle insertion depth in a fundus needle insertion image in the above embodiments of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0150] See also Figure 7 , Figure 7 This is a functional module diagram of a device for determining the needle insertion depth in a fundus needle insertion image provided by an embodiment of the present application. Figure 7 As shown, the device 700 for determining the needle insertion depth in the fundus needle insertion image includes:

[0151] The acquisition module 710 is used to acquire the target fundus needle insertion image and determine multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during the fundus needle insertion surgery.

[0152] The matching module 720 is used to determine multiple second feature points of the target fundus needle insertion image based on multiple first key points, multiple first feature points of the reference image and a feature point matching algorithm; the multiple second feature points are points that match the multiple first feature points among the multiple first key points; the reference image is a pre-acquired image of the surgical needle before insertion; and the multiple first feature points are used to describe the significant features of the surgical needle in the reference image.

[0153] The conversion module 730 is used to perspectively transform the multiple second feature points to the plane where the reference image is located based on the multiple second feature points and the multiple first feature points, so as to obtain multiple third feature points of the target fundus needle insertion image after perspective transformation.

[0154] The determination module 740 is used to determine the insertion depth of the surgical needle in the target fundus needle insertion image based on the multiple third feature points and the multiple first feature points.

[0155] Furthermore, the matching module 720 is used to determine a plurality of first feature points of the reference image according to the following steps:

[0156] Based on the FAST corner detection algorithm, multiple second key points of the surgical needle in the reference image are determined;

[0157] Based on preset screening rules, multiple target points are screened out from multiple second key points, and the multiple target points are determined as multiple first feature points of the reference image; the multiple target points are second key points at significant feature positions of the surgical needle in the reference image.

[0158] In a possible implementation, the feature point matching algorithm includes an ORB algorithm. When the matching module 720 is used to determine a plurality of second feature points of the target fundus needle insertion image based on a plurality of first key points, a plurality of first feature points of the reference image, and the feature point matching algorithm, the matching module 720 is specifically used to:

[0159] For each first key point of the target fundus needle insertion image, determine the direction of the first key point according to the grayscale centroid of the first key point;

[0160] Determine the feature descriptor of the first key point according to the BRIEF algorithm and the direction of the first key point;

[0161] Based on the feature descriptor of the first key point and the feature descriptors of each first feature point, determining whether there is a first feature point matching the first key point among the plurality of first feature points;

[0162] If so, the first key point is determined as the second feature point of the target fundus needle insertion image.

[0163] Further, when the matching module 720 is used to determine whether there is a first feature point matching the first key point among the multiple first feature points based on the feature descriptor of the first key point and the feature descriptors of each first feature point, the matching module 720 is specifically used to:

[0164] For each first key point of the target fundus needle insertion image, calculate the Hamming distance between the feature descriptor of the first key point and the feature descriptors of each first feature point;

[0165] If the minimum value in the Hamming distance is less than or equal to a preset minimum Hamming distance threshold, the first feature point corresponding to the minimum value in the Hamming distance is determined as the first feature point matching the first key point;

[0166] If the minimum value of the Hamming distance is greater than a preset minimum Hamming distance threshold, it is determined that there is no first feature point matching the first key point among the plurality of first feature points.

[0167] Further, when the conversion module 730 is used to perspectively transform the plurality of second feature points to the plane where the reference image is located based on the plurality of second feature points and the plurality of first feature points, the conversion module 730 is specifically used to:

[0168] Select at least four groups of corresponding feature points from the plurality of second feature points and the plurality of first feature points, calculate the perspective transformation matrix of the target fundus needle insertion image coordinate system and the reference image coordinate system according to the coordinates of the at least four groups of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system, and determine the perspective transformation matrix as a coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located;

[0169] Based on the coordinate mapping relationship and the plurality of second feature points, coordinates of the plurality of second feature points in the reference image coordinate system are calculated, and the plurality of second feature points are perspectively transformed to the plane where the reference image is located.

[0170] Further, when the determination module 740 is used to determine the insertion depth of the surgical needle in the target fundus needle insertion image based on the plurality of third feature points and the plurality of first feature points, the determination module 740 is specifically used to:

[0171] Selecting a plurality of fourth feature points for describing the distance from the needle end to the needle tip of the surgical needle from the plurality of first feature points, and calculating the length of the surgical needle in the reference image according to the coordinates of the plurality of fourth feature points in the reference image coordinate system;

[0172] Calculating the length of the surgical needle in the target fundus needle insertion image after perspective transformation according to the coordinates of the third feature points corresponding to the plurality of fourth feature points in the target fundus needle insertion image after perspective transformation;

[0173] The difference between the length of the surgical needle in the reference image and the length in the target fundus needle insertion image after perspective transformation is determined as the insertion depth of the surgical needle in the target fundus needle insertion image.

[0174] Furthermore, the feature point matching algorithm includes a scale-invariant feature transformation algorithm or an accelerated robust feature algorithm.

[0175] The embodiment of the present application provides a device for determining the depth of a surgical needle in a fundus needle insertion image, comprising: an acquisition module, used to acquire a target fundus needle insertion image, and determine multiple first key points of a surgical needle in the target fundus needle insertion image based on a FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during a fundus needle insertion operation; a matching module, used to determine multiple second feature points of the target fundus needle insertion image based on multiple first key points, multiple first feature points of a reference image, and a feature point matching algorithm; the multiple second feature points are points that match multiple first feature points among multiple first key points; the reference image is a pre-acquired image of the surgical needle before insertion; the multiple first feature points are used to describe the significant features of the surgical needle in the reference image; a conversion module, used to perspective transform the multiple second feature points to the plane where the reference image is located based on the multiple second feature points and the multiple first feature points, to obtain multiple third feature points of the target fundus needle insertion image after perspective transformation; a determination module, used to determine the depth of the surgical needle in the target fundus needle insertion image based on the multiple third feature points and the multiple first feature points. In this way, the fundus needle insertion images of different shooting angles are transformed to the same shooting angle as the reference image, the ratio of image pixels to actual distance is unified, and the accuracy of needle insertion depth measurement in the fundus needle insertion images is improved.

[0176] Based on the same application idea, please refer to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the electronic device 800 includes a processor 810 , a memory 820 and a bus 830 .

[0177] The memory 820 stores machine-readable instructions executable by the processor 810. When the electronic device 800 is running, the processor 810 communicates with the memory 820 through the bus 830. When the processor 810 is running, the machine-readable instructions are executed by the steps of the method for determining the needle insertion depth in the fundus needle insertion image provided in the above embodiment. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0178] Based on the same application concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for determining the needle insertion depth in the fundus needle insertion image provided in the above embodiment are executed. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0179] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0180] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0181] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0183] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0184] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0185] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for determining the needle insertion depth in a fundus needle insertion image, characterized in that: The method comprises: Acquire a target fundus needle insertion image, and determine multiple first key points of the surgical needle in the target fundus needle insertion image based on the FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during the fundus needle insertion surgery; Based on the multiple first key points, the multiple first feature points of the reference image and the feature point matching algorithm, multiple second feature points of the target fundus needle insertion image are determined; the multiple second feature points are points among the multiple first key points that match the multiple first feature points; the reference image is a pre-acquired image of the surgical needle before the needle is inserted; the multiple first feature points are used to describe the significant features of the surgical needle in the reference image; Based on the plurality of second feature points and the plurality of first feature points, perspective transforming the plurality of second feature points to the plane where the reference image is located, to obtain a plurality of third feature points of the target fundus needle insertion image after perspective transformation; Based on the plurality of third feature points and the plurality of first feature points, the insertion depth of the surgical needle in the target fundus needle insertion image is determined.

2. The method for determining the needle insertion depth in the fundus needle insertion image according to claim 1, characterized in that: Determine a plurality of first feature points of the reference image according to the following steps: Based on the FAST corner detection algorithm, multiple second key points of the surgical needle in the reference image are determined; Based on preset screening rules, multiple target points are screened out from the multiple second key points, and the multiple target points are determined as multiple first feature points of the reference image; the multiple target points are second key points of the significant feature positions of the surgical needle in the reference image.

3. The method for determining the needle insertion depth in the fundus needle insertion image according to claim 1, characterized in that: The feature point matching algorithm includes an ORB algorithm, and the determining of the plurality of second feature points of the target fundus needle insertion image based on the plurality of first key points, the plurality of first feature points of the reference image, and the feature point matching algorithm includes: For each first key point of the target fundus needle insertion image, determine the direction of the first key point according to the grayscale centroid of the first key point; Determine a feature descriptor of the first key point according to the BRIEF algorithm and the direction of the first key point; Based on the feature descriptor of the first key point and the feature descriptors of each first feature point, determining whether there is a first feature point matching the first key point among the plurality of first feature points; If so, the first key point is determined as the second feature point of the target fundus needle insertion image.

4. The method for determining the needle insertion depth in the fundus needle insertion image according to claim 3, characterized in that: The determining, based on the feature descriptor of the first key point and the feature descriptors of each first feature point, whether there is a first feature point matching the first key point among the plurality of second feature points comprises: For each first key point of the target fundus needle insertion image, calculating the Hamming distance between the feature descriptor of the first key point and the feature descriptors of each first feature point; If the minimum value in the Hamming distance is less than or equal to a preset minimum Hamming distance threshold, determining the first feature point corresponding to the minimum value in the Hamming distance as the first feature point matching the first key point; If the minimum value of the Hamming distance is greater than a preset minimum Hamming distance threshold, it is determined that there is no first feature point matching the first key point among the plurality of first feature points.

5. The method for determining the needle insertion depth in the fundus needle insertion image according to claim 1, characterized in that: The perspective transformation of the plurality of second feature points to the plane where the reference image is located based on the plurality of second feature points and the plurality of first feature points comprises: Select at least four groups of corresponding feature points from the plurality of second feature points and the plurality of first feature points, calculate the perspective transformation matrix of the target fundus needle insertion image coordinate system and the reference image coordinate system according to the coordinates of the at least four groups of corresponding feature points in the target fundus needle insertion image coordinate system and the reference image coordinate system, and determine the perspective transformation matrix as a coordinate mapping relationship between the plane where the target fundus needle insertion image is located and the plane where the reference image is located; Based on the coordinate mapping relationship and the plurality of second feature points, coordinates of the plurality of second feature points in the reference image coordinate system are calculated, and the plurality of second feature points are perspectively transformed to the plane where the reference image is located.

6. The method for determining the needle insertion depth in the fundus needle insertion image according to claim 1, characterized in that: The step of determining the insertion depth of the surgical needle in the target fundus needle insertion image based on the plurality of third feature points and the plurality of first feature points includes: Selecting a plurality of fourth feature points for describing the distance from the needle end to the needle tip of the surgical needle from the plurality of first feature points, and calculating the length of the surgical needle in the reference image according to the coordinates of the plurality of fourth feature points in the reference image coordinate system; Calculating the length of the surgical needle in the target fundus needle insertion image after perspective transformation according to the coordinates of the third feature points corresponding to the plurality of fourth feature points in the target fundus needle insertion image after perspective transformation; The difference between the length of the surgical needle in the reference image and the length in the target fundus needle insertion image after perspective transformation is determined as the insertion depth of the surgical needle in the target fundus needle insertion image.

7. The method for determining the needle insertion depth in the fundus needle insertion image according to claim 1, characterized in that: The feature point matching algorithm includes a scale-invariant feature transformation algorithm or an accelerated robust feature algorithm.

8. A device for determining the needle insertion depth in a fundus needle insertion image, characterized in that: The device for determining the needle insertion depth in the fundus needle insertion image comprises: An acquisition module is used to acquire a target fundus needle insertion image and determine a plurality of first key points of the surgical needle in the target fundus needle insertion image based on a FAST corner point detection algorithm; the target fundus needle insertion image is an image collected during the fundus needle insertion surgery; A matching module, for determining a plurality of second feature points of the target fundus needle insertion image based on the plurality of first key points, a plurality of first feature points of the reference image, and a feature point matching algorithm; the plurality of second feature points are points among the plurality of first key points that match the plurality of first feature points; the reference image is a pre-acquired image of the surgical needle before the needle is inserted; the plurality of first feature points are used to describe the significant features of the surgical needle in the reference image; A conversion module, configured to perspectively transform the plurality of second feature points to the plane where the reference image is located based on the plurality of second feature points and the plurality of first feature points, so as to obtain a plurality of third feature points of the target fundus needle insertion image after perspective transformation; A determination module is used to determine the insertion depth of the surgical needle in the target fundus needle insertion image based on the multiple third feature points and the multiple first feature points.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for determining the needle insertion depth in the fundus needle insertion image as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for determining the needle insertion depth in a fundus needle insertion image according to any one of claims 1 to 7 are executed.

Images