Method, device, electronic device and storage medium for determining retinal needle insertion depth

Abstract

The present application provides a method, device, electronic device and storage medium for determining the depth of retinal needle insertion, including: performing target detection on a retinal surgical video to identify the three-dimensional coordinate information of key feature points of the needle body during movement; performing outlier removal processing on multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, clustering the multiple target three-dimensional coordinate information to determine the center of mass coordinate information under each cluster; sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to timestamp information to determine the valid three-dimensional coordinate information under each cluster, and determining the needle insertion status based on the inter-cluster distance between any two center of mass coordinate information; updating the center of mass coordinate information of the cluster corresponding to the needle insertion status to determine the updated center of mass coordinate information, and determining the actual insertion depth of the needle body based on the updated center of mass coordinate information and the camera internal parameters, thereby improving the accuracy of the needle insertion depth determination.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a method, device, electronic device, and storage medium for determining the depth of a retinal needle insertion. Background Art

[0002] In ophthalmic surgery, accurate needle insertion depth prediction is a key technology that impacts surgical success rates and patient recovery quality. However, due to the unique nature of ophthalmic surgery, needle insertion depth prediction faces multiple challenges in practice. A common solution is to directly predict two key points, the needle body and the needle tip, using visual recognition algorithms such as neural networks. This, combined with the needle insertion angle, calculates the insertion depth using a three-dimensional distance formula. However, existing technologies still suffer from limitations such as insufficient accuracy and robustness when faced with image blur and point drift. More reliable methods are urgently needed to optimize point pair selection and needle insertion depth prediction. Furthermore, some solutions use the actual pixel length of the needle body as the basis for depth calculation. However, the needle insertion operation takes up relatively short time during the entire surgical process, resulting in a large number of interfering frames in the surgical video that are unrelated to the insertion. This interfering information can make it difficult to clearly associate the previous and next positions of the needle body. Specifically, when the needle body appears in the same position during insertion, its temporal characteristics may change. This makes it difficult for the system to determine whether the current state of the area is "just completed" or "stuck or jittering after the insertion." This temporal variation significantly increases the complexity and uncertainty of needle insertion depth prediction. Therefore, how to accurately determine the retinal needle insertion depth has become a technical issue that cannot be underestimated. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, device, electronic device and storage medium for determining the retinal needle insertion depth. By sorting and clustering the target three-dimensional coordinate information, it is possible to accurately identify timestamps unrelated to needle insertion, reduce the interference of irrelevant information on the results, and distinguish the needle insertion status to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0004] The present invention provides a method for determining the depth of a retinal needle insertion, the method comprising:

[0005] The YOLO model is used to detect objects in retinal surgical videos and identify the three-dimensional coordinates of key feature points during the needle's motion.

[0006] performing outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, performing clustering processing on the plurality of target three-dimensional coordinate information to determine centroid coordinate information of each cluster;

[0007] Sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to the timestamp information to determine the valid three-dimensional coordinate information under each cluster, and determining the insertion status of the needle body based on the inter-cluster distance between any two centroid coordinate information;

[0008] The centroid coordinate information of the cluster corresponding to the insertion state of the needle body is updated to determine the updated centroid coordinate information, and the actual insertion depth of the needle body is determined based on the updated centroid coordinate information and the camera internal parameters.

[0009] In one possible implementation, the YOLO model is used to perform target detection on the retinal surgical video to identify the three-dimensional coordinate information of key feature points of the needle during its motion, including:

[0010] Performing target detection on the retinal surgical video based on the YOLO model to identify the moving image area of the needle body;

[0011] Extracting two-dimensional coordinate information of key feature points of the needle body in the motion image area;

[0012] Coordinate integration is performed based on the time stamp information corresponding to the motion image region and the two-dimensional coordinate information of the key feature point to determine the three-dimensional coordinate information of the key feature point.

[0013] In a possible implementation manner, clustering the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster includes:

[0014] Randomly select the initial cluster number target three-dimensional coordinate information as the initial centroid coordinate information;

[0015] For each target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each initial centroid coordinate information; if the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information;

[0016] Based on the three-dimensional coordinate information of multiple targets under each cluster, the new centroid coordinate information of each cluster is re-determined until the difference between the new centroid coordinate information and the corresponding initial centroid coordinate information is less than a preset threshold, and the determination of the new centroid coordinate information is stopped;

[0017] A plurality of initial cluster numbers are set, and the target cluster number is determined based on a plurality of centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and the centroid coordinate information under each cluster under the target cluster number is determined.

[0018] In a possible implementation, determining the number of target clusters based on the plurality of centroid coordinate information under each initial number of clusters and the corresponding target three-dimensional coordinate information includes:

[0019] Determine the sum of squares of the distances between the target three-dimensional coordinate information of multiple clusters and the corresponding centroid coordinate information under each initial number of clusters;

[0020] Detect whether the second derivative of the sum of squares of the distances for each initial number of clusters is negative;

[0021] If yes, the initial number of clusters is used as the target number of clusters.

[0022] In a possible implementation manner, determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information includes:

[0023] Determine the inter-cluster distance between any two centroid coordinates based on the Euclidean distance calculation formula;

[0024] The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the pre-needle insertion state of the needle body, the second cluster is determined as the post-needle insertion state of the needle body, and the third cluster is determined as the stable needle insertion state of the needle body; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

[0025] In a possible implementation, updating the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body to determine the updated centroid coordinate information includes:

[0026] determining updated centroid coordinate information of the first cluster based on valid three-dimensional coordinate information after a preset timestamp in the first cluster corresponding to the pre-needle insertion state;

[0027] Based on the valid three-dimensional coordinate information after the preset timestamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

[0028] In one possible implementation, determining the actual insertion depth of the needle body based on the updated center of mass coordinate information and camera intrinsic parameters includes:

[0029] Determining difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster;

[0030] The difference coordinate information and the camera intrinsic parameter are converted into pixel distance to determine the actual needle insertion depth.

[0031] The present application also provides a device for determining the depth of a retinal needle insertion, the device comprising:

[0032] The needle recognition module is used to detect targets in retinal surgical videos based on the YOLO model and identify the three-dimensional coordinate information of key feature points of the needle during its movement;

[0033] A clustering module is used to perform outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, perform clustering processing on the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster;

[0034] A needle insertion status recognition module is used to sort and filter the multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the needle insertion status of the needle body based on the inter-cluster distance between any two centroid coordinate information;

[0035] The depth determination module is used to update the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determine the updated centroid coordinate information, and determine the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters.

[0036] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for determining the retinal needle insertion depth as described above are performed.

[0037] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for determining the depth of retinal needle insertion as described above are executed.

[0038] The embodiments of the present application provide a method, device, electronic device and storage medium for determining the retinal needle insertion depth. The determination method includes: performing target detection on the retinal surgical video based on the YOLO model to identify the three-dimensional coordinate information of the key feature points of the needle body during the movement; performing outlier removal processing on the multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, clustering the multiple target three-dimensional coordinate information to determine the center of mass coordinate information under each cluster; sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to timestamp information to determine the valid three-dimensional coordinate information under each cluster, and determining the insertion state of the needle body based on the inter-cluster distance between any two center of mass coordinate information; updating the center of mass coordinate information of the cluster corresponding to the insertion state of the needle body to determine the updated center of mass coordinate information, and determining the actual insertion depth of the needle body based on the updated center of mass coordinate information and the camera internal parameters. By sorting and clustering the target three-dimensional coordinate information, it is possible to accurately identify timestamps unrelated to needle insertion, reduce the interference of irrelevant information on the results, and distinguish the needle insertion status to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0039] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A flowchart of a method for determining the depth of a retinal needle insertion provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of a method for determining the depth of a retinal needle insertion provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a device for determining the depth of a retinal needle insertion provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0046] First, the application scenarios to which this application is applicable are introduced. This application can be applied in the field of medical image processing technology.

[0047] Research has found that accurate needle insertion depth prediction in ophthalmic surgery is a key technology that impacts surgical success rates and patient recovery quality. However, due to the unique nature of ophthalmic surgery, needle insertion depth prediction faces multiple challenges in practice. A common solution is to directly predict two key points, the needle body and the needle tip, using visual recognition algorithms such as neural networks. This, combined with the needle insertion angle, calculates the insertion depth using a three-dimensional distance formula. However, existing technologies still suffer from limitations such as insufficient accuracy and robustness when faced with image blur and point drift. More reliable methods are urgently needed to optimize point pair selection and needle insertion depth prediction. Furthermore, some solutions use the actual pixel length of the needle body as the basis for depth calculation. However, the needle insertion operation takes up relatively short time during the entire surgical process, resulting in a large number of interfering frames in the surgical video that are unrelated to the insertion. This interfering information can make it difficult to clearly associate the previous and next positions of the needle body. Specifically, when the needle body appears in the same position during insertion, its temporal characteristics may change. This makes it difficult for the system to determine whether the current state of the area is "just completed" or "stuck or jittering after the insertion." This temporal variation significantly increases the complexity and uncertainty of needle insertion depth prediction. Therefore, how to accurately determine the retinal needle insertion depth has become a technical issue that cannot be underestimated.

[0048] Based on this, an embodiment of the present application provides a method for determining the depth of retinal needle insertion. By sorting and clustering the target three-dimensional coordinate information, it can accurately identify timestamps unrelated to needle insertion, reduce the interference of irrelevant information on the results, and distinguish the needle insertion status to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0049] See also Figure 1 , Figure 1This is a flow chart of a method for determining the depth of a retinal needle insertion provided in an embodiment of the present application. Figure 1 As shown in , the determination method provided in the embodiment of the present application includes:

[0050] S101: Perform target detection on retinal surgical videos based on the YOLO model to identify the three-dimensional coordinate information of key feature points during the needle's movement.

[0051] In this step, target detection is performed on the retinal surgical video based on the YOLO model to identify the three-dimensional coordinate information of the key feature points of the needle during its movement.

[0052] Here, a high-resolution camera is used to record images or videos in real time during the procedure, ensuring that details of the needle and its surroundings are captured.

[0053] The YOLO (You Only Look Once) model is a real-time target detection algorithm. Its core idea is to transform target detection into a regression problem. Based on a separate end-to-end network, it completes the process from inputting the original image to outputting the object's location and category. For example, it can detect an object in an image and identify its location in the image.

[0054] In one possible implementation, the YOLO model is used to perform target detection on the retinal surgical video to identify the three-dimensional coordinate information of key feature points of the needle during its motion, including:

[0055] A: Based on the YOLO model, target detection is performed on the retinal surgery video to identify the moving image area of the needle.

[0056] Here, target detection is performed on the retinal surgery video based on the YOLO model to identify the moving image area of the needle body.

[0057] B: Extracting the two-dimensional coordinate information of the key feature points of the needle body in the motion image area.

[0058] Here, the two-dimensional coordinate information of the key feature points of the needle body is extracted from the motion image area using the MMPose tool ( x,y Among them, the key feature points include the needle tip, the connection between the needle body and other feature points related to the needle body.

[0059] C: performing coordinate integration based on the timestamp information corresponding to the motion image region and the two-dimensional coordinate information to determine the three-dimensional coordinate information of the key feature point.

[0060] Here, according to the time stamp information corresponding to the motion image area ( z) and the two-dimensional coordinate information of the key feature points to determine the three-dimensional coordinate information of the key feature points ( x 、 y 、 z ).

[0061] Among them, the traditional method's reliance on time sequence often leads to time sequence errors due to image blur, frame drift or inconsistent time information, which in turn affects the accuracy and stability of the model. z The introduction of the axis forms a three-dimensional coordinate system that combines the plane coordinates and time information of each key feature point. This method effectively avoids the timing errors caused by image blur or point drift, making the classification of the needle body state more robust, rather than relying solely on a single frame of data when the image is blurred. This improves the robustness of the system and reduces the risk of misjudgment.

[0062] S102: performing outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, performing clustering processing on the plurality of target three-dimensional coordinate information to determine centroid coordinate information of each cluster.

[0063] In this step, outlier removal processing is performed on the multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, and clustering processing is performed on the multiple target three-dimensional coordinate information to determine the centroid coordinate information of each cluster.

[0064] Here, removing outliers in the data during the data preprocessing stage is a key step to ensure the reliability and stability of the input data, aiming to reduce the errors caused by noise points or drift points. Through density-based clustering methods, points that do not belong to any major data group can be effectively identified and eliminated. The distance θ between each point and other points is calculated (the distance θ here is the distance after coordinate normalization). x,y,z dimensional distance) and the minimum number of neighbors n to filter data points. First, for each data point, define a neighborhood range S, which represents the area around the point. Then, within the range of S, when the number of point pairs that satisfy the distance between the point dot and other points is less than θ exceeds n, the point is judged to belong to a high-density area, and the point is set to be a core point, which is located in a dense area. Next, by checking other points in the neighborhood of the core point, it can be determined whether they are boundary points (insufficient points but belong to the neighborhood of the core point) or noise points (neither in the neighborhood of any core point nor have enough neighbors). Boundary points are considered to be part of the current data population, while noise points are eliminated.

[0065] In a possible implementation manner, clustering the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster includes:

[0066] a: Randomly select the initial number of clusters of target three-dimensional coordinate information as the initial centroid coordinate information.

[0067] Here, the initial cluster number k is set to start from 2, and k target three-dimensional coordinate information is randomly selected as the centroid coordinate information 、 .

[0068] b: For each target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each initial centroid coordinate information. If the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information.

[0069] Among them, each target three-dimensional coordinate information is traversed, the distance between the target three-dimensional coordinate information and each initial centroid coordinate information is calculated, and the target three-dimensional coordinate information is assigned to the cluster corresponding to the nearest initial centroid coordinate information.

[0070] Here, the target 3D coordinate information is assigned to the cluster corresponding to the nearest initial centroid coordinate information using the following formula:

[0071]

[0072] in, For the target three-dimensional coordinate information Perform cluster allocation, For the j The initial center of mass coordinate information.

[0073] c: Based on the three-dimensional coordinate information of multiple targets under each cluster, the new center of mass coordinate information of each cluster is re-determined until the difference between the new center of mass coordinate information and the corresponding initial center of mass coordinate information is less than a preset threshold, and the determination of the new center of mass coordinate information is stopped.

[0074] Here, the new centroid coordinate information of each cluster is re-determined by the following formula:

[0075]

[0076] in, For the j New centroid coordinate information, For the j clusters, for The next i The three-dimensional coordinate information of the target.

[0077] d: Set multiple groups of initial cluster numbers, determine the target cluster number based on multiple centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and determine the centroid coordinate information under each cluster under the target cluster number.

[0078] Here, after completing the above process, adjust the k value (generally set to k+1), repeat the above operation m times, and obtain m groups of centroid coordinate information under different k values. According to the multiple centroid coordinate information under each initial number of clusters and the corresponding target three-dimensional coordinate information, the target cluster number is determined, and the centroid coordinate information of each cluster under the target number of clusters is determined according to the above steps.

[0079] In a possible implementation, determining the number of target clusters based on the plurality of centroid coordinate information under each initial number of clusters and the corresponding target three-dimensional coordinate information includes:

[0080] Determine the sum of the squares of the distances from the target three-dimensional coordinate information of multiple clusters to the corresponding centroid coordinate information under each initial cluster number; detect whether the second derivative of the sum of the squares of the distances for each initial cluster number is negative; if so, use the initial cluster number as the target cluster number.

[0081] Here, the sum of squared distances of the initial number of clusters is determined by the following formula:

[0082]

[0083] Among them, Inertia is the initial number of clusters k The sum of squared distances, for The centroid coordinate information of the cluster, x for The target three-dimensional coordinate information of the cluster. When the second derivative of the inertia is negative, that is, when it becomes flat, determine the k value at this time and record the k value at this time as the target number of clusters.

[0084] Among them, traditional clustering methods usually require manual setting of the number of clusters k, which may not be applicable when facing different operating scenarios or data sets, resulting in poor clustering results. However, this application automatically determines the optimal number of clusters k through analysis of Inertia, avoiding the inadaptability of manually setting the k value and improving the accuracy and adaptability of the clustering results. Dynamically determining the k value can optimize the clustering results based on the actual data situation, allowing the system to adapt to different surgical scenarios and improving the quality and accuracy of clustering.

[0085] In traditional techniques, interfering frames (such as irrelevant background changes or unstable images in the video) often prevent accurate correlation between the positions before and after needle insertion, affecting depth prediction. However, this application uses a clustering algorithm combined with timestamps for data filtering, effectively filtering out data points that fall within the incorrect time range. This technology verifies the temporal order of data points during clustering, thereby removing interfering data and accurately identifying frames unrelated to needle insertion, reducing the impact of irrelevant information on the results and improving the accuracy of depth prediction.

[0086] S103: sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determining the valid three-dimensional coordinate information under each cluster, and determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information.

[0087] In this step, the multiple target three-dimensional coordinate information under each cluster is sorted and filtered according to the timestamp information to determine the valid three-dimensional coordinate information under each cluster, and the insertion status of the needle body is determined based on the inter-cluster distance between any two centroid coordinate information.

[0088] The needle insertion state of the needle body includes a state before the needle insertion, a state after the needle insertion, and a stable needle insertion state.

[0089] Here, the centroid coordinates of each cluster obtained from clustering are sorted by frame time sequence, combining the time series information of the video frames. Points that do not conform to the time series (i.e., points that were incorrectly assigned from the previous time range to the next) are discarded. This time series sorting allows for further analysis of the dynamic behavior and changes of the needle during surgery.

[0090] In a possible implementation manner, determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information includes:

[0091] (1): Determine the inter-cluster distance between any two centroid coordinate information based on the Euclidean distance calculation formula.

[0092] Here, the time factor of the frame is ignored in the process of measuring the distance between clusters, and only the relative position of the needle on the image plane is considered. Specifically, the calculation of the inter-cluster distance is based on the Euclidean distance between the coordinate position information of each centroid.

[0093] (2): The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the state before the needle is inserted, the second cluster is determined as the state after the needle is inserted, and the third cluster is determined as the stable state of the needle; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

[0094] Here, by analyzing the inter-cluster distances in all frames, we find the maximum spacing of the needle body on the plane coordinates (x, y). This step lays the foundation for judging the needle insertion status by screening the needle body's motion range. Based on the changes in inter-cluster distances and frame sequence information, we can effectively judge the current needle insertion status. Specifically, the maximum inter-cluster distance is regarded as a sign of needle insertion action. Assume that the distance (x, y) is the cluster With cluster The distance between clusters Indicates the state before the needle is inserted, cluster Indicates the state after needle insertion, and cluster The needle is in a stable state.

[0095] Here, traditional methods have difficulty dealing with complex situations such as stagnation or jitter when dealing with the dynamic changes of the needle body during the insertion process, which increases the difficulty and uncertainty of prediction. However, this application combines time series sorting and cluster analysis to accurately identify the states of "needle insertion has just been completed" and "stagnation or jitter after needle insertion." Through clear needle insertion state classification and time series analysis, the system can distinguish different dynamic behaviors during the needle insertion stage, thereby more accurately describing the state of the needle body and improving the prediction accuracy of needle insertion depth.

[0096] S104: updating the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determining the updated centroid coordinate information, and determining the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera intrinsic parameters.

[0097] In this step, the centroid coordinate information of the cluster corresponding to the insertion state of the needle body is updated to determine the updated centroid coordinate information. Based on the updated centroid coordinate information and the camera internal parameters, the actual insertion depth of the needle body is determined.

[0098] In a possible implementation, updating the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body to determine the updated centroid coordinate information includes:

[0099] I: Based on the valid three-dimensional coordinate information after the preset timestamp in the first cluster corresponding to the pre-needle insertion state, determine the updated centroid coordinate information of the first cluster.

[0100] Here, the updated centroid coordinate information of the first cluster is determined based on the valid three-dimensional coordinate information after the preset timestamp in the first cluster corresponding to the state before needle insertion.

[0101] Among them, according to the first cluster The updated centroid coordinate information of the first cluster is determined based on the valid three-dimensional coordinate information of the last quarter of the needle insertion phase.

[0102] II: Based on the valid three-dimensional coordinate information after the preset timestamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

[0103] Here, the updated centroid coordinate information of the third cluster is determined based on the valid three-dimensional coordinate information after the preset time stamp in the third cluster corresponding to the needle insertion stable state.

[0104] Among them, according to the third cluster The updated centroid coordinate information of the third cluster is determined based on the valid three-dimensional coordinate information of the last quarter of the needle insertion stable phase.

[0105] In one possible implementation, determining the actual insertion depth of the needle body based on the updated center of mass coordinate information and camera intrinsic parameters includes:

[0106] i: Determine difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster.

[0107] Here, the difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster is determined.

[0108] ii: Perform pixel distance conversion processing on the difference coordinate information and the camera intrinsic parameter to determine the actual needle insertion depth.

[0109] Here, pixel distance conversion is performed on the difference coordinate information and the camera intrinsic parameters to determine the actual needle insertion depth.

[0110] Among them, based on the plane coordinates (x, y) and time series, the recalculated aggregate

[0111] After the class center and Calculate the pixel distance of the needle insertion. Combining the needle's 3D coordinates (x, y, z) with the camera's intrinsic parameters, the pixel distance is converted to the actual insertion depth. This result can be directly used to assess the insertion depth and position of the needle during surgery, providing precise spatial information to support surgical procedures.

[0112] For further information, see Figure 2 , Figure 2 This is a schematic diagram of a method for determining the depth of retinal needle insertion provided in an embodiment of the present application. Figure 2As shown in , the two-dimensional coordinate information (x, y) and frame coordinates (z) of the key feature points of the needle during movement are identified according to the YOLO model, and the two-dimensional coordinate information and frame coordinates are combined to construct three-dimensional coordinate information. The three-dimensional coordinate information is processed for outliers and Kmeans clustering method to obtain the three-dimensional coordinate information under each cluster, and the inter-class distance of the centroid coordinate information under each cluster in the (x, y) dimension is measured. The longest inter-class distance in the (x, y) dimension is screened out to determine the needle insertion status, and the cluster center point processing is performed on the cluster corresponding to the needle insertion status. Then, the needle insertion pixel distance between the updated centroid coordinate information and the camera intrinsic parameters are used to determine the actual needle insertion depth of the needle.

[0113] In this application, by combining timestamps with the relative position of the image plane, not only does this effectively overcome the effects of timing errors and interfering frames in existing technologies, but it also employs dynamic optimization in cluster analysis, significantly improving the accuracy and robustness of needle depth prediction. Compared to traditional methods, this application offers greater flexibility and adaptability, more accurately capturing the dynamic changes of the needle during surgery, significantly improving prediction reliability and processing efficiency.

[0114] The present application provides a method for determining the depth of a retinal needle insertion, the method comprising: performing target detection on a retinal surgical video based on a YOLO model to identify three-dimensional coordinate information of key feature points of a needle during its movement; performing outlier removal processing on multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information; clustering the multiple target three-dimensional coordinate information to determine the centroid coordinate information of each cluster; sorting and filtering the multiple target three-dimensional coordinate information of each cluster according to timestamp information to determine the valid three-dimensional coordinate information of each cluster; determining the insertion state of the needle based on the inter-cluster distance between any two centroid coordinate information; updating the centroid coordinate information of the cluster corresponding to the insertion state of the needle to determine the updated centroid coordinate information; and determining the actual insertion depth of the needle based on the updated centroid coordinate information and camera intrinsic parameters. By sorting and clustering the target three-dimensional coordinate information, timestamps unrelated to the needle insertion can be accurately identified, reducing the interference of irrelevant information on the results, and distinguishing the needle insertion state to more accurately describe the needle insertion state, thereby improving the accuracy of needle insertion depth determination.

[0115] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a device for determining the depth of retinal needle insertion provided in an embodiment of the present application. Figure 3 As shown in , the device 300 for determining the retinal needle insertion depth includes:

[0116] The needle recognition module 310 is used to perform target detection on the retinal surgical video based on the YOLO model and identify the three-dimensional coordinate information of key feature points of the needle during its movement;

[0117] A clustering module 320 is configured to perform outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, and to perform clustering processing on the plurality of target three-dimensional coordinate information to determine centroid coordinate information of each cluster;

[0118] A needle insertion status recognition module is used to sort and filter the multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the needle insertion status of the needle body based on the inter-cluster distance between any two centroid coordinate information;

[0119] The depth determination module is used to update the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determine the updated centroid coordinate information, and determine the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters.

[0120] Furthermore, when the needle body recognition module 310 is used to perform target detection on the retinal surgical video based on the YOLO model and identify the three-dimensional coordinate information of key feature points of the needle body during the movement, the needle body recognition module 310 is specifically used to:

[0121] Performing target detection on the retinal surgical video based on the YOLO model to identify the moving image area of the needle body;

[0122] Extracting two-dimensional coordinate information of key feature points of the needle body in the motion image area;

[0123] Coordinate integration is performed based on the time stamp information corresponding to the motion image region and the two-dimensional coordinate information of the key feature point to determine the three-dimensional coordinate information of the key feature point.

[0124] Furthermore, when the clustering module 320 is used to perform clustering processing on the plurality of target three-dimensional coordinate information and determine the centroid coordinate information of each cluster, the clustering module 320 is specifically used to:

[0125] Randomly select the initial cluster number target three-dimensional coordinate information as the initial centroid coordinate information;

[0126] For each target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each initial centroid coordinate information; if the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information;

[0127] Based on the three-dimensional coordinate information of multiple targets under each cluster, the new centroid coordinate information of each cluster is re-determined until the difference between the new centroid coordinate information and the corresponding initial centroid coordinate information is less than a preset threshold, and the determination of the new centroid coordinate information is stopped;

[0128] A plurality of initial cluster numbers are set, and the target cluster number is determined based on a plurality of centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and the centroid coordinate information under each cluster under the target cluster number is determined.

[0129] Furthermore, when the clustering module 320 is used to determine the number of target clusters based on the plurality of centroid coordinate information under each initial number of clusters and the corresponding target three-dimensional coordinate information, the clustering module 320 is specifically used to:

[0130] Determine the sum of squares of the distances between the target three-dimensional coordinate information of multiple clusters and the corresponding centroid coordinate information under each initial number of clusters;

[0131] Detect whether the second derivative of the sum of squares of the distances for each initial number of clusters is negative;

[0132] If yes, the initial number of clusters is used as the target number of clusters.

[0133] Furthermore, when the needle insertion state recognition module 330 is used to determine the needle insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information, the needle insertion state recognition module 330 is specifically used to:

[0134] Determine the inter-cluster distance between any two centroid coordinates based on the Euclidean distance calculation formula;

[0135] The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the pre-needle insertion state of the needle body, the second cluster is determined as the post-needle insertion state of the needle body, and the third cluster is determined as the stable needle insertion state of the needle body; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

[0136] Furthermore, when the depth determination module 340 is used to update the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body and determine the updated centroid coordinate information, the depth determination module 340 is specifically used to:

[0137] determining updated centroid coordinate information of the first cluster based on valid three-dimensional coordinate information after a preset timestamp in the first cluster corresponding to the pre-needle insertion state;

[0138] Based on the valid three-dimensional coordinate information after the preset timestamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

[0139] Furthermore, when the depth determination module 340 is used to determine the actual insertion depth of the needle body based on the updated center of mass coordinate information and the camera intrinsic parameters, the depth determination module 340 is specifically used to:

[0140] Determining difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster;

[0141] The difference coordinate information and the camera intrinsic parameter are converted into pixel distance to determine the actual needle insertion depth.

[0142] An embodiment of the present application provides a device for determining the depth of retinal needle insertion, which includes: a needle body recognition module, which is used to perform target detection on retinal surgical videos based on a YOLO model, and identify three-dimensional coordinate information of key feature points of the needle body during movement; a clustering module, which is used to remove outliers on multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, cluster the multiple target three-dimensional coordinate information, and determine the center of mass coordinate information under each cluster; a needle insertion state recognition module, which is used to sort and filter multiple target three-dimensional coordinate information under each cluster according to timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the insertion state of the needle body based on the inter-cluster distance between any two center of mass coordinate information; a depth determination module, which is used to update the center of mass coordinate information of the cluster corresponding to the insertion state of the needle body, determine the updated center of mass coordinate information, and determine the actual insertion depth of the needle body based on the updated center of mass coordinate information and camera internal parameters. By sorting and clustering the target three-dimensional coordinate information, it is possible to accurately identify timestamps unrelated to needle insertion, reduce the interference of irrelevant information on the results, and distinguish the needle insertion status to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0143] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in FIG, the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0144] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 as well as Figure 2The specific implementation of the steps of the method for determining the retinal needle insertion depth in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0145] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The specific implementation of the steps of the method for determining the retinal needle insertion depth in the method embodiment shown can be found in the method embodiment and will not be repeated here.

[0146] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0147] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0148] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0149] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0150] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0151] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for determining the depth of retinal needle insertion, characterized in that: The determination method includes: The YOLO model is used to detect objects in retinal surgical videos and identify the three-dimensional coordinates of key feature points during the needle's motion. performing outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, performing clustering processing on the plurality of target three-dimensional coordinate information to determine centroid coordinate information of each cluster; Sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to the timestamp information to determine the valid three-dimensional coordinate information under each cluster, and determining the insertion status of the needle body based on the inter-cluster distance between any two centroid coordinate information; updating the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determining updated centroid coordinate information, and determining the actual insertion depth of the needle body based on the updated centroid coordinate information and camera intrinsic parameters; The YOLO model is used to perform target detection on the retinal surgical video and identify the three-dimensional coordinate information of key feature points of the needle during movement, including: Performing target detection on the retinal surgical video based on the YOLO model to identify the moving image area of the needle body; Extracting two-dimensional coordinate information of key feature points of the needle body in the motion image area; Coordinate integration is performed based on the time stamp information corresponding to the motion image region and the two-dimensional coordinate information of the key feature point to determine the three-dimensional coordinate information of the key feature point.

2. The determination method according to claim 1, characterized in that The clustering process of the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster includes: Randomly select the initial cluster number target three-dimensional coordinate information as the initial centroid coordinate information; For each target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each initial centroid coordinate information; if the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information; Based on the three-dimensional coordinate information of multiple targets under each cluster, the new centroid coordinate information of each cluster is re-determined until the difference between the new centroid coordinate information and the corresponding initial centroid coordinate information is less than a preset threshold, and the determination of the new centroid coordinate information is stopped; A plurality of initial cluster numbers are set, and the target cluster number is determined based on a plurality of centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and the centroid coordinate information under each cluster under the target cluster number is determined.

3. The determination method according to claim 2, characterized in that: The determining of the number of target clusters based on the plurality of centroid coordinate information under each initial number of clusters and the corresponding target three-dimensional coordinate information includes: Determine the sum of squares of the distances between the target three-dimensional coordinate information of multiple clusters and the corresponding centroid coordinate information under each initial number of clusters; Detect whether the second derivative of the sum of squares of the distances for each initial number of clusters is negative; If yes, the initial number of clusters is used as the target number of clusters.

4. The determination method according to claim 1, characterized in that The step of determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information includes: Determine the inter-cluster distance between any two centroid coordinates based on the Euclidean distance calculation formula; The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the pre-needle insertion state of the needle body, the second cluster is determined as the post-needle insertion state of the needle body, and the third cluster is determined as the stable needle insertion state of the needle body; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

5. The determination method according to claim 1, characterized in that: Updating the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body to determine the updated centroid coordinate information includes: determining updated centroid coordinate information of the first cluster based on valid three-dimensional coordinate information after a preset timestamp in the first cluster corresponding to the pre-needle insertion state; Based on the valid three-dimensional coordinate information after the preset timestamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

6. The determination method according to claim 5, characterized in that: The method of determining the actual insertion depth of the needle body based on the updated center of mass coordinate information and the camera internal parameters includes: Determining difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster; The difference coordinate information and the camera intrinsic parameter are converted into pixel distance to determine the actual needle insertion depth.

7. A device for determining the depth of retinal needle insertion, characterized in that: The determining device comprises: The needle recognition module is used to detect targets in retinal surgical videos based on the YOLO model and identify the three-dimensional coordinate information of key feature points of the needle during its movement; A clustering module is used to perform outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, perform clustering processing on the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster; A needle insertion status recognition module is used to sort and filter the multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the needle insertion status of the needle body based on the inter-cluster distance between any two centroid coordinate information; a depth determination module, configured to update the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determine the updated centroid coordinate information, and determine the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera intrinsic parameters; When the needle body recognition module is used to perform target detection on the retinal surgical video based on the YOLO model and identify the three-dimensional coordinate information of the key feature points of the needle body during the movement, the needle body recognition module is specifically used to: Performing target detection on the retinal surgical video based on the YOLO model to identify the moving image area of the needle body; Extracting two-dimensional coordinate information of key feature points of the needle body in the motion image area; Coordinate integration is performed based on the time stamp information corresponding to the motion image region and the two-dimensional coordinate information of the key feature point to determine the three-dimensional coordinate information of the key feature point.

8. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus. When the processor is running, the machine-readable instructions execute the steps of the method for determining the retinal needle insertion depth as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for determining the retinal needle insertion depth according to any one of claims 1 to 6 are executed.

Images