Needle inserting positioning and calculating device and method in ophthalmologic operation

Abstract

The invention discloses an ophthalmologic intraoperative needle insertion positioning and calculating device and method, and the method comprises the steps: carrying out the nicking processing of a thick tube of an original injection needle based on mark information, obtaining a target injection needle, obtaining the needle insertion video data of the target injection needle, carrying out the image frame processing, further dividing an image frame set, and carrying out the image frame processing. And registration is carried out based on the mark points, so that accurate needle point positioning and needle insertion depth calculation can be realized. According to the method, accurate alignment and positioning can be carried out at different needle inserting moments through key point detection and an affine transformation principle, so that an efficient and reliable needle inserting depth measuring means is provided, and the needle inserting precision of an injection needle in an ophthalmic operation is improved.

Description

Technical Field

[0001] This invention relates to the field of needle insertion positioning technology, and in particular to a device and method for needle insertion positioning and calculation during ophthalmic surgery. Background Technology

[0002] In retinal injection surgery, the depth of needle insertion is a key indicator affecting treatment efficacy and safety. Too deep an insertion may damage the retina or choroid; too shallow an insertion may impair precise drug release. Currently, depth estimation primarily relies on analyzing surgical images captured by a microscopic imaging system and marking reference points for the injection needle within the images for calculation. However, the smooth, sparse features of the injection needle make feature extraction difficult, and it is also susceptible to the effects of lighting, imaging angle, and tissue reflection, making precise needle marking challenging and consequently affecting the accuracy of depth prediction.

[0003] Existing needle insertion positioning and calculation methods involve setting marks on the outer sheath. During operation, the marks on the outer sheath are read, and the depth of the outer sheath insertion into the endoscope is predicted based on the marks and the length of the forceps channel. However, the needle body is nested within the outer sheath, which occupies additional space and limits its application scenarios. Furthermore, this method only assesses the depth of the injection needle insertion into the endoscope and cannot assess the needle insertion distance from the injection site to the point of penetration. Summary of the Invention

[0004] This invention provides an ophthalmic intraoperative needle insertion positioning and calculation device and method to accurately locate the distance from the injection site to the needle insertion point.

[0005] To solve the above-mentioned technical problems, the present invention provides an ophthalmic intraoperative needle insertion positioning and calculation device, comprising: a scoring module, an image acquisition module, a key point detection module, a registration module, and a calculation module;

[0006] The scoring module is used to acquire marking information and score the coarse tube of the original injection needle based on the marking information to obtain the target injection needle;

[0007] The image acquisition module is used to acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data to obtain an image frame set, and divide the image frame set according to a preset filtering rule to obtain a reference frame and a frame set to be aligned.

[0008] The key point detection module is used to preset a key point detection network, and perform feature point detection on the reference frame and the set of frames to be aligned based on the key point detection network to obtain key points of the reference frame and key points of each frame to be aligned; the key points of the reference frame and the key points of each frame to be aligned all include marker points, pinpoints and boundary points.

[0009] The registration module is used to register the reference frame and the current frame to be aligned based on the marker points for each frame to be aligned in the set of frames to be aligned, and to obtain the transformation matrix of the reference frame and the current frame to be aligned based on the affine transformation principle and the marker points.

[0010] The calculation module is used to calculate the needle insertion depth at the corresponding moment of the current frame to be aligned based on the transformation matrix, the needle tip, and the intersection point, thereby completing the needle insertion positioning and calculation.

[0011] This invention uses a scoring module to acquire the target injection needle and an image acquisition module to obtain injection video data and perform image frame processing. The image frame set is further divided to obtain a reference frame and a frame to be aligned. A key point detection module then detects key points in each image frame, enabling the registration module to perform registration based on these marker points. This achieves precise needle tip positioning and injection depth calculation. Through key point detection and affine transformation principles, this device can accurately align and position the needle at different injection points, providing an efficient and reliable method for measuring injection depth and improving the accuracy of injection needle insertion in ophthalmic surgery.

[0012] Furthermore, the image acquisition module is used to acquire needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data to obtain an image frame set; and divide the image frame set according to a preset filtering rule to obtain a reference frame and a set of frames to be aligned, including:

[0013] Acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data, and obtain an initial image set;

[0014] Images from the initial image set are extracted based on windows of preset size and position to obtain an image frame set;

[0015] Based on a preset recognition algorithm, any image in the image frame set that has needle tip elements and thick pipe scratch marking elements is determined as the reference frame, and all image frames after the reference frame are determined as frames to be aligned, thus generating a set of frames to be aligned.

[0016] Based on a preset recognition algorithm, any image in the target image frame set that has needle tip elements and thick pipe scratch marking elements is determined as the reference frame, and all image frames after the reference frame are determined as frames to be aligned, thus generating a set of frames to be aligned.

[0017] This invention determines the reference frame through preset filtering rules and filters image frames according to their order and characteristics. This effectively optimizes image frame selection, ensures the accuracy of the reference frame, and thus improves the accuracy of subsequent image registration and needle depth calculation. It reduces errors in image processing and improves image processing efficiency, providing a more precise technical guarantee for needle positioning.

[0018] Furthermore, the keypoint detection module is used to preset a keypoint detection network, and perform feature point detection on the reference frame and the set of frames to be aligned based on the keypoint detection network to obtain keypoints of the reference frame and keypoints of each frame to be aligned, including:

[0019] The key point detection network is trained based on historical image data to obtain the target key point detection network;

[0020] Based on the target key point detection network, feature point detection is performed on the reference frame and the set of frames to be aligned to obtain the reference frame marker point, the reference frame tip point and the reference frame boundary point, the frame to be aligned marker point, the frame to be aligned tip point and the frame to be aligned boundary point.

[0021] This invention trains the network using historical image data, thereby improving the accuracy and precision of key point detection. By detecting feature points on the reference frame and the set of frames to be aligned, important information such as the needle tip, marker points, and boundary points can be effectively obtained, providing accurate feature data for subsequent needle insertion depth calculation. This enhances the correspondence between image frames, making needle insertion positioning more precise and further improving the controllability and safety of the surgery.

[0022] Furthermore, the key point detection network includes a detection head, a feature fusion layer, and a backbone network;

[0023] The backbone network is used to extract multi-scale semantic features from the input image; the feature fusion layer is used to fuse the multi-scale semantic features to generate fused features; the detection head is used to identify the fused features and obtain the location of key points.

[0024] Furthermore, the marker points include a plurality of points; the registration module is used to register the reference frame and the current frame to be aligned based on the marker points for each frame to be aligned in the set of frames to be aligned, and to obtain the transformation matrix of the reference frame and the current frame to be aligned based on the affine transformation principle and the marker points, including:

[0025] For each frame to be aligned in the set of frames to be aligned, each point of the current frame to be aligned is registered one by one based on each point of the reference frame marker.

[0026] Based on the principle of affine transformation, each point of the current frame to be aligned is mapped to the reference frame to generate a transformation matrix.

[0027] This invention achieves high-precision registration between the reference frame and the current frame to be aligned by using the affine transformation principle for precise registration and generating a transformation matrix based on the one-to-one correspondence between marker points. This ensures a more accurate correspondence between image frames, thereby improving the accuracy and reliability of needle insertion positioning. Precise registration and transformation ensure consistency between different image frames, providing a solid technical foundation for accurate needle insertion depth calculation.

[0028] Furthermore, the calculation module is used to calculate the needle insertion depth at the corresponding moment of the current frame to be aligned based on the transformation matrix, the needle tip point, and the intersection point, to complete the needle insertion positioning and calculation, including:

[0029] The reference frame tip point is obtained as the first tip coordinate, and the reference frame boundary point is obtained as the first boundary coordinate;

[0030] Based on the transformation matrix, obtain the second needle tip coordinates and the second boundary coordinates of the needle tip and boundary point of the current frame to be aligned in the reference frame coordinate system;

[0031] The needle insertion depth is calculated based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete the needle insertion positioning and calculation.

[0032] This invention obtains the coordinates of the needle tip and the intersection point of the reference frame and the frame to be aligned, and combines them with the principle of affine transformation. This method can achieve accurate needle insertion depth calculation, ensuring positioning accuracy during the needle insertion process.

[0033] Furthermore, the step of calculating the needle insertion depth based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete the needle insertion positioning and calculation includes:

[0034] Calculate the first pixel distance from the reference frame tip point to the reference frame boundary point in the reference frame coordinate system based on the first tip coordinates and the first boundary coordinates.

[0035] The second pixel distance from the tip of the current frame to the boundary of the current frame to be aligned is calculated based on the second tip coordinate and the second boundary coordinate in the reference frame coordinate system.

[0036] The needle insertion depth is calculated based on the actual distance from the tip of the target injection needle to the junction point, the first pixel distance, and the second pixel distance, thus completing the needle insertion positioning and calculation.

[0037] Furthermore, after extracting images from the initial image set using a window of preset size and position to obtain an image frame set, the process further includes:

[0038] The image frame set is sorted according to timestamp order to obtain the target image frame set.

[0039] Secondly, the present invention provides a method for intraoperative needle placement and calculation in ophthalmic surgery, including:

[0040] Obtain the marking information, and perform a scoring process on the coarse tube of the original injection needle based on the marking information to obtain the target injection needle;

[0041] The needle insertion video data of the target injection needle is acquired, and the needle insertion video data is processed into image frames to obtain an image frame set; and the image frame set is divided according to a preset filtering rule to obtain a reference frame and a frame set to be aligned.

[0042] A preset key point detection network is used to detect feature points on the reference frame and the set of frames to be aligned, thereby obtaining key points of the reference frame and key points of each frame to be aligned; the key points of the reference frame and the key points of each frame to be aligned all include marker points, pinpoints and boundary points.

[0043] For each frame to be aligned in the set of frames to be aligned, the reference frame and the current frame to be aligned are registered based on the marker points, and the transformation matrix of the reference frame and the current frame to be aligned is obtained based on the affine transformation principle and the marker points.

[0044] Based on the transformation matrix, the needle tip, and the intersection point, the needle insertion depth at the corresponding moment of the current frame to be aligned is calculated to complete the needle insertion positioning and calculation.

[0045] Furthermore, the step of acquiring the needle insertion video data of the target injection needle, performing image frame processing on the needle insertion video data to obtain an image frame set, and dividing the image frame set based on a preset filtering rule to obtain a reference frame and a set of frames to be aligned includes:

[0046] Acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data, and obtain an initial image set;

[0047] Images from the initial image set are extracted based on windows of preset size and position to obtain an image frame set;

[0048] Based on a preset recognition algorithm, any image in the image frame set that has needle tip elements and thick pipe scratch marking elements is determined as the reference frame, and all image frames after the reference frame are determined as frames to be aligned, thus generating a set of frames to be aligned. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of an ophthalmic intraoperative needle insertion positioning and calculation device provided in an embodiment of the present invention;

[0050] Figure 2 A schematic diagram of a primitive injection needle provided for an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of a target injection needle provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of the structure of a key point detection network provided in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of needle insertion provided in an embodiment of the present invention. Detailed Implementation

[0054] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0055] The terms "first" and "second," etc., in the specification, claims, and drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0056] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0057] Example 1

[0058] See Figure 1 , Figure 1 This is a schematic diagram of a device for intraoperative needle placement and calculation in ophthalmology provided in an embodiment of the present invention. The present invention also provides a method for intraoperative needle placement and calculation in ophthalmology, including a notching module 101, an image acquisition module 102, a key point detection module 103, a registration module 104, and a calculation module 105, as detailed below:

[0059] The scoring module 101 is used to acquire marking information and score the coarse tube of the original injection needle based on the marking information to obtain the target injection needle;

[0060] Please refer to Figure 2 , Figure 2 This is a schematic diagram of a primitive injection needle provided for an embodiment of the present invention.

[0061] In this embodiment, the original injection needle includes a needle tip, a thin tube, a thick tube, and the junction between the two.

[0062] In this embodiment, since the original injection needle body is smooth and has sparse features, several marks (such as straight lines, triangles, rectangles, pentagrams, etc.) are etched on the surface of the needle body to enhance the recognizability of the needle body features under imaging, optimize the selection of ideal marker points, and avoid additional volume occupation; moreover, etching does not require specific distance information, reducing the processing difficulty.

[0063] Please refer to Figure 3 , Figure 3 This is a schematic diagram of a target injection needle provided in an embodiment of the present invention.

[0064] In this embodiment, marking information is obtained, which includes marking elements and corresponding marking point information. Marking elements include lines, triangles, rectangles, pentagrams, etc. If the marking element is a line, the two endpoints of the line are the marking point information; if it is a triangle, the three vertices are the marking point information.

[0065] In this embodiment, the marker point information refers to the marker points that can be identified by the subsequent key point detection network.

[0066] In one embodiment, a single injection needle may have multiple marker elements.

[0067] The image acquisition module 102 is used to acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data to obtain an image frame set, and divide the image frame set according to a preset filtering rule to obtain a reference frame and a frame set to be aligned.

[0068] In this embodiment, the image acquisition module is used to acquire needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data to obtain an image frame set; and divide the image frame set according to a preset filtering rule to obtain a reference frame and a set of frames to be aligned, including:

[0069] Acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data, and obtain an initial image set;

[0070] Images from the initial image set are extracted based on windows of preset size and position to obtain an image frame set;

[0071] Based on a preset recognition algorithm, any image in the image frame set that has needle tip elements and thick pipe scratch marking elements is determined as the reference frame, and all image frames after the reference frame are determined as frames to be aligned, thus generating a set of frames to be aligned.

[0072] In this embodiment, after extracting images from the initial image set using a window of preset size and position to obtain an image frame set, the method further includes:

[0073] The image frame set is sorted according to timestamp order to obtain the target image frame set.

[0074] In this embodiment, firstly, the insertion video data of the target injection needle is acquired, and then image frame processing is performed on the video data. Specifically, the insertion video data is recorded using a high frame rate acquisition device, and then frame extraction is performed using an image processing algorithm to obtain an initial image set. To ensure the validity and accuracy of the image frames, each frame in the initial image set is extracted using a preset window size and position. The extraction process depends on the features and position of the image content to ensure that key information related to the insertion process can be obtained. After extraction, the extracted image frames are sorted according to timestamps to obtain the target image frame set. Next, based on a preset recognition algorithm, one image is selected from the target image frame set as a reference frame. This image needs to have clearly identifiable needle tip elements and coarse tube scratch marking elements to ensure that the image can serve as a reference standard for the subsequent needle insertion positioning process. To ensure the accuracy of the reference frame, the image recognition algorithm automatically checks whether the features of the needle tip and coarse tube scratch are present in each frame image. Only images that meet the preset feature conditions are selected as reference frames.

[0075] In this embodiment, a reference frame is determined by a preset filtering rule, and the image frames are filtered according to their order and features. This effectively optimizes the selection of image frames, ensures the accuracy of the reference frame, and thus improves the accuracy of subsequent image registration and needle depth calculation. It reduces errors in image processing, improves image processing efficiency, and provides a more precise technical guarantee for needle positioning.

[0076] The key point detection module 103 is used to preset a key point detection network, and perform feature point detection on the reference frame and the set of frames to be aligned based on the key point detection network to obtain key points of the reference frame and key points of each frame to be aligned; the key points of the reference frame and the key points of each frame to be aligned all include marker points, pinpoints and boundary points.

[0077] In this embodiment, the keypoint detection module is used to preset a keypoint detection network, and perform feature point detection on the reference frame and the set of frames to be aligned based on the keypoint detection network to obtain keypoints of the reference frame and keypoints of each frame to be aligned, including:

[0078] The key point detection network is trained based on historical image data to obtain the target key point detection network;

[0079] Based on the target key point detection network, feature point detection is performed on the reference frame and the set of frames to be aligned to obtain the reference frame marker point, the reference frame tip point and the reference frame boundary point, the frame to be aligned marker point, the frame to be aligned tip point and the frame to be aligned boundary point.

[0080] Please refer to Figure 4 , Figure 4 This is a schematic diagram of a key point detection network provided in an embodiment of the present invention.

[0081] In this embodiment, the key point detection network includes a detection head, a feature fusion layer, and a backbone network;

[0082] The backbone network is used to extract multi-scale semantic features from the input image; the feature fusion layer is used to fuse the multi-scale semantic features to generate fused features; the detection head is used to identify the fused features and obtain the location of key points.

[0083] In this embodiment, a deep learning-based keypoint detection network is used to process images during the needle insertion process in ophthalmic surgery. Key feature points are accurately extracted from the reference frame and the frame to be aligned, including the needle tip, the midpoint of the thick-thin junction, and the edge of the puncture mark. The network consists of three main parts: the backbone, the feature fusion layer, and the detection head.

[0084] In this embodiment, the backbone network is responsible for extracting multi-scale semantic features from the image. The backbone network typically consists of convolutional layers and lightweight residual modules, enabling the network to efficiently capture detailed and structural features in the image, such as needle tips, thin-thickness boundaries, and scratches. The backbone network can accurately distinguish and extract detailed features at different scales in the image.

[0085] In this embodiment, the feature fusion layer (Neck) uses structures such as FPN (Feature Pyramid Network) and PAN (Path Aggregation Network) to fuse feature maps from different scales. This process significantly enhances the network's ability to perceive key points and fine-grained targets. By fusing features from multiple scales, the network can identify and locate pinpoints, midpoints at the boundary between thick and thin lines, and edge points of scratches at different viewpoints and details, maintaining high accuracy even in complex image scenes.

[0086] In this embodiment, the detection head is the core part of the network, responsible for regressing the specific location of feature points and their visibility confidence. Through this detection head, the network can accurately determine every key point in the image, including pinpoints, midpoints at the junction of thick and thin lines, and edge points of scratches. The regression task outputs the two-dimensional coordinates of each key point, along with its visibility confidence, ensuring that the localization results for each target in the image have high reliability.

[0087] In this embodiment, after processing by the keypoint detection network, a series of point sets are obtained. These point sets are used to describe the two-dimensional coordinates of the pinpoint, the midpoint of the thick-thin boundary, and the edge of the scratch in the image. Here, the pinpoint of the reference frame is defined as A, the boundary point as B, and the marker point as P = {C1, C2, ..., C}. n}, Frame T to be aligned i The needle tip is A′, the intersection point is B′, and the marked point is P′={C1′,C2′,…,C n ′}, where n≥3. In this embodiment, the boundary point is the midpoint of the junction between the thin and thick tubes of the injection needle.

[0088] In this embodiment, the keypoint detection network is trained based on historical image data, thereby ensuring that it can adapt to different needle insertion scenarios and extract accurate feature points. The training process not only enhances the network's generalization ability but also enables it to handle challenges under different image qualities and viewpoints.

[0089] In this embodiment, feature points of the reference frame and the frame to be aligned can be accurately detected and extracted, providing solid data support for subsequent image registration and needle insertion depth calculation. This method effectively improves the accuracy of the needle insertion process and provides a highly reliable technical guarantee for needle insertion positioning in ophthalmic surgery.

[0090] In this embodiment, historical image data is used to train the network, thereby improving the accuracy and precision of key point detection. By performing feature point detection on the reference frame and the set of frames to be aligned, important information such as the needle tip, marker points, and boundary points can be effectively obtained, providing accurate feature data for subsequent needle insertion depth calculation. This enhances the correspondence between image frames, making needle insertion positioning more precise and further improving the controllability and safety of the surgery.

[0091] The registration module 104 is used to register the reference frame and the current frame to be aligned based on the marker points for each frame to be aligned in the set of frames to be aligned, and to obtain the transformation matrix of the reference frame and the current frame to be aligned based on the affine transformation principle and the marker points.

[0092] In this embodiment, the marker points include a plurality of points; the registration module is used to register the reference frame and the current frame to be aligned based on the marker points for each frame to be aligned in the set of frames to be aligned, and to obtain the transformation matrix of the reference frame and the current frame to be aligned based on the affine transformation principle and the marker points, including:

[0093] For each frame to be aligned in the set of frames to be aligned, each point of the current frame to be aligned is registered one by one based on each point of the reference frame marker.

[0094] Based on the principle of affine transformation, each point of the current frame to be aligned is mapped to the reference frame to generate a transformation matrix.

[0095] In this embodiment, the reference frame T0 point set A, B, P is obtained, and the frame to be aligned T is... i Point set A′, B′, P′; key point matching is performed using the marked points, i.e., P and P′ are registered; simultaneously, according to the principle of affine transformation, the aligned frame T is... i The point P′ is mapped to the point P of T0 to obtain the affine transformation matrix M, where M is a 2x3 matrix; this implementation method uses the cv2.getAffineTransform() function of the OpenCV package;

[0096] In this embodiment, the cv2.getAffineTransform() function from the OpenCV package is used to calculate the transformation matrix. This function requires the point set of the reference frame and the point set of the frame to be aligned as input, and optimizes it using the least squares method to calculate the optimal affine transformation matrix M, thereby achieving the registration between the frame to be aligned and the reference frame.

[0097] In this embodiment, the calculated affine transformation matrix M maps the marker points in the frame to be aligned to the coordinate system of the reference frame. This achieves image registration, ensuring that the shape, position, and scale of the frame to be aligned are consistent with the reference frame. Finally, the marker points of all frames to be aligned will undergo corresponding transformations based on this transformation matrix, ensuring consistency of all frames in spatial coordinates and providing accurate data support for needle depth calculation and positioning.

[0098] In this embodiment, precise registration of the frames to be aligned is achieved by using the affine transformation principle and generating a transformation matrix based on the one-to-one correspondence between marker points. This ensures high-precision registration between the reference frame and the current frame to be aligned, guaranteeing a more accurate correspondence between image frames and thus improving the accuracy and reliability of needle insertion positioning. Precise registration and transformation ensure consistency between different image frames, providing a solid technical foundation for accurate calculation of needle insertion depth.

[0099] The calculation module 105 is used to calculate the needle insertion depth at the corresponding moment of the current frame to be aligned, based on the transformation matrix, the needle tip, and the intersection point, thereby completing the needle insertion positioning and calculation.

[0100] In this embodiment, the calculation module is used to calculate the needle insertion depth at the corresponding moment of the current frame to be aligned based on the transformation matrix, the needle tip point, and the intersection point, thereby completing the needle insertion positioning and calculation, including:

[0101] The reference frame tip point is obtained as the first tip coordinate, and the reference frame boundary point is obtained as the first boundary coordinate;

[0102] Based on the transformation matrix, obtain the second needle tip coordinates and the second boundary coordinates of the needle tip and boundary point of the current frame to be aligned in the reference frame coordinate system;

[0103] The needle insertion depth is calculated based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete the needle insertion positioning and calculation.

[0104] In this embodiment, key points are extracted from the reference frame, including the needle tip and the midpoint (boundary point) at the junction of the thick and thin sections. Here, the needle tip in the reference frame is defined as A, and the boundary point as B. These two points represent critical positions during the needle insertion process, namely the position of the needle tip and the midpoint of the thick tube groove, and they play a crucial role in calculating the needle insertion depth.

[0105] In this embodiment, the step of calculating the needle insertion depth based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete the needle insertion positioning and calculation includes:

[0106] Calculate the first pixel distance from the reference frame tip point to the reference frame boundary point in the reference frame coordinate system based on the first tip coordinates and the first boundary coordinates.

[0107] The second pixel distance from the tip of the current frame to the boundary of the current frame to be aligned is calculated based on the second tip coordinate and the second boundary coordinate in the reference frame coordinate system.

[0108] The needle insertion depth is calculated based on the actual distance from the tip of the target injection needle to the junction point, the first pixel distance, and the second pixel distance, thus completing the needle insertion positioning and calculation.

[0109] In this embodiment, a transformation matrix M from the frame to be aligned to the reference frame is obtained through the aforementioned affine transformation process. Using this matrix, the pinpoints and boundary points in the frame to be aligned are transformed into the coordinate system of the reference frame. During this process, the pinpoint A′ and boundary point B′ in the frame to be aligned are transformed into their corresponding points A″ and B″ in the coordinate system of the reference frame. Specifically, through matrix multiplication, the point set of the frame to be aligned is mapped to the coordinate system of the reference frame. The formula for this mapping is:

[0110]

[0111] Where (x, y) are the coordinates of a point in the frame to be aligned, and (x′, y′) are the coordinates of a point in the transformed reference frame coordinate system. Through this transformation, the coordinates of the tip and the intersection point of the frame to be aligned are accurately converted.

[0112] In this embodiment, after image registration is completed, the needle depth estimation module uses the reference frame T0 point set A and B, and the frame to be aligned T0 as reference frames. i The needle insertion depth is calculated based on the actual distance lr from the needle tip to the boundary point, given A″ and B″ in the reference frame T0 coordinate system. The principle is as follows:

[0113]

[0114] In this embodiment, the pixel distance dis from point A to point B in the reference frame T0 is calculated according to the Euclidean distance formula. AB Then, calculate the frame T to be aligned according to the Euclidean distance formula. i The distance between pixels A″ and B″ in the reference frame T0 coordinate system is dis A″B″ Therefore, the actual distance represented by each pixel is calculated as r = lr / dis. AB Then calculate the actual needle advance distance real = (dis AB -dis A″B″ )*r.

[0115] Please refer to Figure 5 , Figure 5 This is a schematic diagram of needle insertion provided in an embodiment of the present invention.

[0116] In this embodiment, the pixel distance between the needle tip and the boundary point in the reference frame coordinate system is obtained by calculating the Euclidean distance between them. This distance represents the relative position of the needle tip and the boundary point in the image. This pixel distance reflects the spatial relationship between the needle tip and the boundary point and is the basis for calculating the needle depth. Therefore, based on the transformation matrix in the reference frame coordinate system, the coordinates of the needle tip and the boundary point in the frame to be aligned are mapped to the coordinate system of the reference frame, and the pixel distance between the needle tip and the boundary point in the frame to be aligned is calculated. This calculation is also based on the Euclidean distance formula, taking into account the transformed coordinates, to obtain the pixel distance between the needle tip and the boundary point in the frame to be aligned.

[0117] In this embodiment, to convert pixel distance into actual physical distance, a calibration process is used to determine the actual distance of each pixel in physical space. This calibration process is obtained using known standard-sized images or actual measurements, enabling the conversion between pixel distances in the image and actual physical distances. Then, by multiplying the pixel distances of the needle tip and the junction point in the frame to be aligned with the actual physical distance corresponding to each pixel, the actual needle insertion depth can be calculated. This needle insertion depth is the actual physical distance from the needle tip to the midpoint of the junction between the thick and thin sections, representing the actual puncture depth of the needle during surgery.

[0118] In this embodiment, by obtaining the coordinates of the needle tip and the intersection point of the reference frame and the frame to be aligned, and combining the principle of affine transformation, this method can achieve accurate needle insertion depth calculation, ensuring positioning accuracy during the needle insertion process.

[0119] In this embodiment, by acquiring the video data of the target injection needle insertion and performing image frame processing, further dividing the image frame set, and registering it based on marker points, accurate needle tip positioning and insertion depth calculation can be achieved. This method, through key point detection and affine transformation principles, can accurately align and position the needle at different insertion points, thus providing an efficient and reliable means of measuring insertion depth and improving the insertion accuracy of injection needles in ophthalmic surgery.

[0120] This invention also provides a method for intraoperative needle placement and calculation in ophthalmology, including:

[0121] Obtain the marking information, and perform a scoring process on the coarse tube of the original injection needle based on the marking information to obtain the target injection needle;

[0122] The needle insertion video data of the target injection needle is acquired, and the needle insertion video data is processed into image frames to obtain an image frame set; and the image frame set is divided according to a preset filtering rule to obtain a reference frame and a frame set to be aligned.

[0123] A preset key point detection network is used to detect feature points on the reference frame and the set of frames to be aligned, thereby obtaining key points of the reference frame and key points of each frame to be aligned; the key points of the reference frame and the key points of each frame to be aligned all include marker points, pinpoints and boundary points.

[0124] For each frame to be aligned in the set of frames to be aligned, the reference frame and the current frame to be aligned are registered based on the marker points, and the transformation matrix of the reference frame and the current frame to be aligned is obtained based on the affine transformation principle and the marker points.

[0125] Based on the transformation matrix, the needle tip, and the intersection point, the needle insertion depth at the corresponding moment of the current frame to be aligned is calculated to complete the needle insertion positioning and calculation.

[0126] In this embodiment, the step of acquiring the needle insertion video data of the target injection needle, performing image frame processing on the needle insertion video data to obtain an image frame set, and dividing the image frame set based on a preset filtering rule to obtain a reference frame and a set of frames to be aligned includes:

[0127] Acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data, and obtain an initial image set;

[0128] Images from the initial image set are extracted based on windows of preset size and position to obtain an image frame set;

[0129] Based on a preset recognition algorithm, any image in the image frame set that has needle tip elements and thick pipe scratch marking elements is determined as the reference frame, and all image frames after the reference frame are determined as frames to be aligned, thus generating a set of frames to be aligned.

[0130] In this embodiment, the preset keypoint detection network performs feature point detection on the reference frame and the set of frames to be aligned, and obtains keypoints of the reference frame and keypoints of each frame to be aligned, including:

[0131] The key point detection network is trained based on historical image data to obtain the target key point detection network;

[0132] Based on the target key point detection network, feature point detection is performed on the reference frame and the set of frames to be aligned to obtain the reference frame marker point, the reference frame tip point and the reference frame boundary point, the frame to be aligned marker point, the frame to be aligned tip point and the frame to be aligned boundary point.

[0133] In this embodiment, the key point detection network includes a detection head, a feature fusion layer, and a backbone network;

[0134] The backbone network is used to extract multi-scale semantic features from the input image; the feature fusion layer is used to fuse the multi-scale semantic features to generate fused features; the detection head is used to identify the fused features and obtain the location of key points.

[0135] In this embodiment, the marker points include a plurality of points; for each frame to be aligned in the set of frames to be aligned, the registration of the reference frame and the current frame to be aligned is performed based on the marker points, and the transformation matrix of the reference frame and the current frame to be aligned is obtained based on the affine transformation principle and the marker points, including:

[0136] For each frame to be aligned in the set of frames to be aligned, each point of the current frame to be aligned is registered one by one based on each point of the reference frame marker.

[0137] Based on the principle of affine transformation, each point of the current frame to be aligned is mapped to the reference frame to generate a transformation matrix.

[0138] In this embodiment, the step of calculating the needle insertion depth at the corresponding moment of the current frame to be aligned based on the transformation matrix, the needle tip point, and the intersection point, to complete the needle insertion positioning and calculation, includes:

[0139] The reference frame tip point is obtained as the first tip coordinate, and the reference frame boundary point is obtained as the first boundary coordinate;

[0140] Based on the transformation matrix, obtain the second needle tip coordinates and the second boundary coordinates of the needle tip and boundary point of the current frame to be aligned in the reference frame coordinate system;

[0141] The needle insertion depth is calculated based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete the needle insertion positioning and calculation.

[0142] In this embodiment, the step of calculating the needle insertion depth based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete the needle insertion positioning and calculation includes:

[0143] Calculate the first pixel distance from the reference frame tip point to the reference frame boundary point in the reference frame coordinate system based on the first tip coordinates and the first boundary coordinates.

[0144] The second pixel distance from the tip of the current frame to the boundary of the current frame to be aligned is calculated based on the second tip coordinate and the second boundary coordinate in the reference frame coordinate system.

[0145] The needle insertion depth is calculated based on the actual distance from the tip of the target injection needle to the junction point, the first pixel distance, and the second pixel distance, thus completing the needle insertion positioning and calculation.

[0146] In this embodiment, after extracting images from the initial image set using a window of preset size and position to obtain an image frame set, the method further includes:

[0147] The image frame set is sorted according to timestamp order to obtain the target image frame set.

[0148] In this embodiment of the invention, a terminal device is also provided, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements the above-described method for intraoperative needle placement and calculation in ophthalmology.

[0149] In this embodiment of the invention, a computer-readable storage medium is also provided, which includes a stored computer program, wherein the computer program controls the device where the computer-readable storage medium is located to execute the above-described ophthalmic intraoperative needle insertion positioning and calculation method when it is running.

[0150] For example, a computer program can be divided into one or more modules, one or more of which are stored in memory and executed by a processor to perform the present invention. The one or more modules can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a terminal device.

[0151] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor, memory, and display. Those skilled in the art will understand that the above components are merely examples of terminal devices and do not constitute a limitation on the terminal device. It may include more or fewer components, or combinations of certain components, or different components. For example, the terminal device may also include input / output devices, network access devices, buses, etc.

[0152] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device through various interfaces and lines.

[0153] Memory can be used to store computer programs and / or modules. The processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as sound playback, text conversion, etc.), etc.; the data storage area can store data created based on the use of the mobile phone (such as audio data, text message data, etc.). In addition, memory can include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0154] The module for needle placement and calculation during ophthalmic surgery, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. Those skilled in the art can understand and implement this without any inventive effort.

[0155] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A device for intraoperative needle insertion positioning and calculation in ophthalmology, characterized in that, include: The module includes a scoring module, an image acquisition module, a key point detection module, a registration module, and a calculation module. The scoring module is used to acquire marking information and score the coarse tube of the original injection needle based on the marking information to obtain the target injection needle; The image acquisition module is used to acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data to obtain an image frame set, and divide the image frame set according to a preset filtering rule to obtain a reference frame and a frame set to be aligned. The key point detection module is used to preset a key point detection network, and perform feature point detection on the reference frame and the set of frames to be aligned based on the key point detection network to obtain key points of the reference frame and key points of each frame to be aligned. The key points of the reference frame and each key point of the frame to be aligned include marker points, pin points, and boundary points; The registration module is used to register the reference frame and the current frame to be aligned based on the marker points for each frame to be aligned in the set of frames to be aligned, and to obtain the transformation matrix of the reference frame and the current frame to be aligned based on the affine transformation principle and the marker points. The calculation module is used to calculate the needle insertion depth at the corresponding moment of the current frame to be aligned based on the transformation matrix, the needle tip, and the intersection point, thereby completing the needle insertion positioning and calculation.

2. The ophthalmic intraoperative needle insertion positioning and calculation device as described in claim 1, characterized in that, The image acquisition module is used to acquire the insertion video data of the target injection needle, perform image frame processing on the insertion video data to obtain an image frame set; and divide the image frame set according to a preset filtering rule to obtain a reference frame and a set of frames to be aligned, including: Acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data, and obtain an initial image set; Images from the initial image set are extracted based on windows of preset size and position to obtain an image frame set; Based on a preset recognition algorithm, any image in the image frame set that has needle tip elements and thick pipe scratch marking elements is determined as the reference frame, and all image frames after the reference frame are determined as frames to be aligned, thus generating a set of frames to be aligned.

3. The ophthalmic intraoperative needle insertion positioning and calculation device as described in claim 2, characterized in that, The keypoint detection module is used to preset a keypoint detection network, and perform feature point detection on the reference frame and the set of frames to be aligned based on the keypoint detection network to obtain keypoints of the reference frame and keypoints of each frame to be aligned, including: The key point detection network is trained based on historical image data to obtain the target key point detection network; Based on the target key point detection network, feature point detection is performed on the reference frame and the set of frames to be aligned to obtain the reference frame marker point, the reference frame tip point and the reference frame boundary point, the frame to be aligned marker point, the frame to be aligned tip point and the frame to be aligned boundary point.

4. The ophthalmic intraoperative needle insertion positioning and calculation device as described in claim 3, characterized in that, The keypoint detection network includes a detection head, a feature fusion layer, and a backbone network; The backbone network is used to extract multi-scale semantic features from the input image; the feature fusion layer is used to fuse the multi-scale semantic features to generate fused features; the detection head is used to identify the fused features and obtain the location of key points.

5. The ophthalmic intraoperative needle insertion positioning and calculation device as described in claim 3, characterized in that, The marker points include a plurality of points; the registration module is used to register the reference frame and the current frame to be aligned based on the marker points for each frame to be aligned in the set of frames to be aligned, and to obtain the transformation matrix of the reference frame and the current frame to be aligned based on the affine transformation principle and the marker points, including: For each frame to be aligned in the set of frames to be aligned, each point of the current frame to be aligned is registered one by one based on each point of the reference frame marker. Based on the principle of affine transformation, each point of the current frame to be aligned is mapped to the reference frame to generate a transformation matrix.

6. The ophthalmic intraoperative needle insertion positioning and calculation device as described in claim 5, characterized in that, The calculation module is used to calculate the needle insertion depth at the corresponding moment of the current frame to be aligned based on the transformation matrix, the needle tip point, and the intersection point, to complete the needle insertion positioning and calculation, including: The reference frame tip point is obtained as the first tip coordinate, and the reference frame boundary point is obtained as the first boundary coordinate; Based on the transformation matrix, obtain the second needle tip coordinates and the second boundary coordinates of the needle tip and boundary point of the current frame to be aligned in the reference frame coordinate system; The needle insertion depth is calculated based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete the needle insertion positioning and calculation.

7. The ophthalmic intraoperative needle insertion positioning and calculation device as described in claim 6, characterized in that, The process of calculating the needle insertion depth based on the first needle tip coordinates, the first boundary coordinates, the second needle tip coordinates, and the second boundary coordinates to complete needle insertion positioning and calculation includes: Calculate the first pixel distance from the reference frame tip point to the reference frame boundary point in the reference frame coordinate system based on the first tip coordinates and the first boundary coordinates. The second pixel distance from the tip of the current frame to the boundary of the current frame to be aligned is calculated based on the second tip coordinate and the second boundary coordinate in the reference frame coordinate system. The needle insertion depth is calculated based on the actual distance from the tip of the target injection needle to the junction point, the first pixel distance, and the second pixel distance, thus completing the needle insertion positioning and calculation.

8. The ophthalmic intraoperative needle insertion positioning and calculation device as described in claim 7, characterized in that, After extracting images from the initial image set through a window based on a preset size and position to obtain an image frame set, the method further includes: The image frame set is sorted according to timestamp order to obtain the target image frame set.

9. A method for intraoperative needle insertion positioning and calculation in ophthalmology, characterized in that, include: Obtain the marking information, and perform a scoring process on the coarse tube of the original injection needle based on the marking information to obtain the target injection needle; The needle insertion video data of the target injection needle is acquired, and the needle insertion video data is processed into image frames to obtain an image frame set; and the image frame set is divided according to a preset filtering rule to obtain a reference frame and a frame set to be aligned. A preset key point detection network is used to perform feature point detection on the reference frame and the set of frames to be aligned, thereby obtaining the key points of the reference frame and the key points of each frame to be aligned. The key points of the reference frame and each key point of the frame to be aligned include marker points, pin points, and boundary points; For each frame to be aligned in the set of frames to be aligned, the reference frame and the current frame to be aligned are registered based on the marker points, and the transformation matrix of the reference frame and the current frame to be aligned is obtained based on the affine transformation principle and the marker points. Based on the transformation matrix, the needle tip, and the intersection point, the needle insertion depth at the corresponding moment of the current frame to be aligned is calculated to complete the needle insertion positioning and calculation.

10. The method for intraoperative needle placement and calculation in ophthalmology as described in claim 9, characterized in that, The process of acquiring the injection video data of the target injection needle, performing image frame processing on the injection video data to obtain an image frame set, and dividing the image frame set according to a preset filtering rule to obtain a reference frame and a set of frames to be aligned includes: Acquire the needle insertion video data of the target injection needle, perform image frame processing on the needle insertion video data, and obtain an initial image set; Images from the initial image set are extracted based on windows of preset size and position to obtain an image frame set; Based on a preset recognition algorithm, any image in the image frame set that has needle tip elements and thick pipe scratch marking elements is determined as the reference frame, and all image frames after the reference frame are determined as frames to be aligned, thus generating a set of frames to be aligned.

Images