Method and device for determining retina needle insertion depth, electronic equipment and storage medium

Abstract

The invention provides a retina needle insertion depth determination method and device, electronic equipment and a storage medium, and the method comprises the steps: carrying out the target detection of a retina operation video, and recognizing the three-dimensional coordinate information of a key feature point of a needle body in a movement process; performing outlier removal processing on the multiple pieces of three-dimensional coordinate information to obtain multiple pieces of target three-dimensional coordinate information, performing clustering processing on the multiple pieces of target three-dimensional coordinate information, and determining centroid coordinate information under each cluster; sorting and filtering the multiple pieces of target three-dimensional coordinate information under each cluster according to timestamp information, determining effective three-dimensional coordinate information under each cluster, and determining a needle feeding state based on an inter-cluster distance between any two pieces of mass center coordinate information; according to the method, the center-of-mass coordinate information of the cluster corresponding to the needle inserting state is updated, the updated center-of-mass coordinate information is determined, the actual needle inserting depth of the needle body is determined based on the updated center-of-mass coordinate information and the internal reference of the camera, and the accuracy of determining the needle inserting depth is improved.

Description

Technical Field

[0001] The present application relates to the field of medical image processing technology, and in particular to a method, device, electronic device and storage medium for determining the depth of a retinal needle insertion. Background Art

[0002] In ophthalmic surgery, accurate prediction of needle insertion depth is one of the key technologies that affect the success rate of surgery and the quality of postoperative recovery of patients. However, due to the uniqueness of ophthalmic surgery scenarios, the prediction of needle insertion depth faces multiple challenges in practice. A common solution is to directly predict the two key points of the needle body and the needle tip through visual recognition algorithms such as neural networks, and combine the needle insertion angle of the needle body to calculate the needle insertion depth using a three-dimensional space distance formula. However, when faced with problems such as blurred images and point drift, the existing technology still has defects such as insufficient accuracy and weak robustness, and a more reliable method is urgently needed to optimize the point pair selection and needle insertion depth prediction process. In addition, in some technical solutions, the actual pixel length of the needle body is used as the basis for depth calculation. However, the needle insertion operation takes a short time in the entire surgical process, and a large number of interference frames unrelated to the needle insertion are formed in the surgical video. These interference information may make it impossible to clearly associate the front and back positions of the needle body. Specifically, when the needle body appears in the same position during the needle insertion process, its front and back timing characteristics may change. This makes it impossible for the system to determine whether the state of the current area is "just completed needle insertion" or "stagnation or jitter after needle insertion is completed." This problem of time series variation significantly increases the complexity and uncertainty of needle insertion depth prediction. Therefore, how to accurately determine the retinal needle insertion depth has become a technical issue that cannot be underestimated. Summary of the invention

[0003] In view of this, the purpose of the present application is to provide a method, device, electronic device and storage medium for determining the retinal needle insertion depth. By sorting and clustering the target three-dimensional coordinate information, it is possible to accurately identify timestamps that are not related to needle insertion, reduce the interference of irrelevant information on the results, and distinguish the needle insertion status to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0004] The present application embodiment provides a method for determining the depth of a retinal needle insertion, the method comprising: Based on the YOLO model, target detection is performed on retinal surgical videos to identify the three-dimensional coordinate information of key feature points of the needle during its movement. Performing outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, and performing clustering processing on the plurality of target three-dimensional coordinate information to determine centroid coordinate information under each cluster; The three-dimensional coordinate information of multiple targets under each cluster is sorted and filtered according to the timestamp information, the valid three-dimensional coordinate information under each cluster is determined, and the insertion state of the needle body is determined based on the inter-cluster distance between any two centroid coordinate information; The centroid coordinate information of the cluster corresponding to the insertion state of the needle body is updated to determine the updated centroid coordinate information, and the actual insertion depth of the needle body is determined based on the updated centroid coordinate information and the camera internal parameters.

[0005] In a possible implementation, the target detection of the retinal surgical video based on the YOLO model and the identification of the three-dimensional coordinate information of the key feature points of the needle during the movement process include: Performing target detection on the retinal surgical video based on the YOLO model to identify the moving image area of ​​the needle body; Extracting two-dimensional coordinate information of key feature points of the needle body in the motion image area; Coordinate integration is performed based on the time stamp information corresponding to the motion image area and the two-dimensional coordinate information of the key feature point to determine the three-dimensional coordinate information of the key feature point.

[0006] In a possible implementation manner, clustering the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster includes: Randomly select the initial cluster number target three-dimensional coordinate information as the initial centroid coordinate information; For each of the target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each of the initial centroid coordinate information, and if the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information; Based on the three-dimensional coordinate information of multiple targets under each cluster, new centroid coordinate information of each cluster is re-determined until the difference between the new centroid coordinate information and the corresponding initial centroid coordinate information is less than a preset threshold, and the determination of the new centroid coordinate information is stopped; A plurality of groups of initial cluster numbers are set, and the target cluster number is determined based on a plurality of centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and the centroid coordinate information under each cluster under the target cluster number is determined.

[0007] In a possible implementation manner, determining the number of target clusters based on a plurality of centroid coordinate information under each initial number of clusters and corresponding target three-dimensional coordinate information includes: Determine the sum of squares of distances from target three-dimensional coordinate information of multiple clusters to corresponding centroid coordinate information under each initial number of clusters; Detect whether the second derivative of the sum of squares of the distances for each initial cluster number is negative; If yes, the initial number of clusters is used as the target number of clusters.

[0008] In a possible implementation manner, determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information includes: Based on the Euclidean distance calculation formula, the inter-cluster distance between any two centroid coordinate information is determined; The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the state of the needle body before insertion, the second cluster is determined as the state of the needle body after insertion, and the third cluster is determined as the stable state of the needle body; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

[0009] In a possible implementation manner, updating the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body to determine the updated centroid coordinate information includes: Determining updated centroid coordinate information of the first cluster based on valid three-dimensional coordinate information after a preset timestamp in the first cluster corresponding to the pre-needle insertion state; Based on the valid three-dimensional coordinate information after the preset time stamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

[0010] In a possible implementation manner, determining the actual insertion depth of the needle body based on the updated center of mass coordinate information and the camera internal parameters includes: Determining difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster; The difference coordinate information and the camera internal parameters are subjected to pixel distance conversion processing to determine the actual needle insertion depth.

[0011] The present application also provides a device for determining the depth of a retinal needle insertion, the device comprising: The needle recognition module is used to detect targets in retinal surgical videos based on the YOLO model and identify the three-dimensional coordinate information of key feature points of the needle during its movement; A clustering module is used to perform outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, perform clustering processing on the plurality of target three-dimensional coordinate information to determine the centroid coordinate information under each cluster; A needle insertion state recognition module is used to sort and filter multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the needle insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information; The depth determination module is used to update the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determine the updated centroid coordinate information, and determine the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters.

[0012] An embodiment of the present application also provides an electronic device, including: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the method for determining the retinal needle insertion depth as described above are performed.

[0013] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for determining the retinal needle insertion depth as described above are executed.

[0014] The embodiments of the present application provide a method, device, electronic device and storage medium for determining the retinal needle insertion depth. The determination method includes: performing target detection on the retinal surgical video based on the YOLO model to identify the three-dimensional coordinate information of the key feature points of the needle body during the movement; performing outlier removal processing on multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, clustering the multiple target three-dimensional coordinate information to determine the centroid coordinate information under each cluster; sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to timestamp information to determine the valid three-dimensional coordinate information under each cluster, and determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information; updating the centroid coordinate information of the cluster corresponding to the insertion state of the needle body to determine the updated centroid coordinate information, and determining the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters. By sorting and clustering the target three-dimensional coordinate information, timestamps unrelated to needle insertion can be accurately identified, reducing the interference of irrelevant information on the results, and the needle insertion status can be distinguished to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0015] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 A flow chart of a method for determining the depth of a retinal needle insertion provided in an embodiment of the present application; Figure 2 A schematic diagram of a method for determining the depth of a retinal needle insertion provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a device for determining the depth of a retinal needle insertion provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0019] First, the application scenarios to which the present application is applicable are introduced. The present application can be applied in the field of medical image processing technology.

[0020] Research has found that in ophthalmic surgery, accurate prediction of needle insertion depth is one of the key technologies that affect the success rate of surgery and the quality of postoperative recovery of patients. However, due to the uniqueness of ophthalmic surgery scenarios, the prediction of needle insertion depth faces multiple challenges in practice. A common solution is to directly predict the two key points of the needle body and the needle tip through visual recognition algorithms such as neural networks, and combine the needle insertion angle of the needle body to calculate the needle insertion depth using a three-dimensional space distance formula. However, when faced with problems such as blurred images and point drift, the existing technology still has defects such as insufficient accuracy and weak robustness, and a more reliable method is urgently needed to optimize the point pair selection and needle insertion depth prediction process. In addition, in some technical solutions, the actual pixel length of the needle body is used as the basis for depth calculation. However, the needle insertion operation takes a short time in the entire surgical process, and a large number of interference frames unrelated to the needle insertion are formed in the surgical video. These interference information may make it impossible to clearly associate the front and back positions of the needle body. Specifically, when the needle body appears in the same position during the needle insertion process, its front and back timing characteristics may change. This makes it impossible for the system to determine whether the state of the current area is "just completed needle insertion" or "stagnation or jitter after needle insertion is completed." This problem of time series variation significantly increases the complexity and uncertainty of needle insertion depth prediction. Therefore, how to accurately determine the retinal needle insertion depth has become a technical issue that cannot be underestimated.

[0021] Based on this, an embodiment of the present application provides a method for determining the retinal needle insertion depth. By sorting and clustering the target three-dimensional coordinate information, it can accurately identify timestamps that are not related to needle insertion, reduce the interference of irrelevant information on the results, and distinguish the needle insertion status to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0022] See also Figure 1 , Figure 1 This is a flow chart of a method for determining the depth of a retinal needle insertion provided in an embodiment of the present application. Figure 1 As shown in , the determination method provided in the embodiment of the present application includes: S101: Perform target detection on retinal surgical videos based on the YOLO model to identify the three-dimensional coordinate information of key feature points of the needle during movement.

[0023] In this step, target detection is performed on the retinal surgical video based on the YOLO model to identify the three-dimensional coordinate information of key feature points of the needle during its movement.

[0024] Here, a high-resolution camera is used to record images or videos in real time during the procedure, ensuring that details of the needle and its surroundings are captured.

[0025] Among them, the YOLO model (You Only Look Once) is a real-time target detection algorithm. The core idea is to transform target detection into a regression problem, and based on a separate end-to-end network, complete the output from the input of the original image to the object position and category, such as detecting an object in an image and identifying its location in the image.

[0026] In a possible implementation, the target detection of the retinal surgical video based on the YOLO model and the identification of the three-dimensional coordinate information of the key feature points of the needle during the movement process include: A: Based on the YOLO model, target detection is performed on the retinal surgery video to identify the moving image area of ​​the needle body.

[0027] Here, target detection is performed on the retinal surgical video based on the YOLO model to identify the moving image area of ​​the needle.

[0028] B: Extracting the two-dimensional coordinate information of the key feature points of the needle body in the motion image area.

[0029] Here, the two-dimensional coordinate information of the key feature points of the needle body is extracted from the motion image area using the MMPose tool ( x,y ). Among them, the key feature points include the needle tip, the connection between the needle body and other feature points related to the needle body.

[0030] C: performing coordinate integration based on the timestamp information corresponding to the motion image area and the two-dimensional coordinate information to determine the three-dimensional coordinate information of the key feature point. Here, according to the time stamp information corresponding to the moving image area ( z ) and the two-dimensional coordinate information of the key feature points to determine the three-dimensional coordinate information of the key feature points ( x , y , z ).

[0031] Among them, the dependence of traditional methods on timing often leads to timing errors caused by image blur, frame drift or inconsistent time information, which in turn affects the accuracy and stability of the model. z The axis is introduced to form a three-dimensional coordinate system, which combines the plane coordinates and time information of each key feature point. This method effectively avoids the timing error caused by image blur or point drift, making the classification of the needle state more robust, and no longer relying solely on a single frame of data when the image is blurred, thereby improving the robustness of the system and reducing the risk of misjudgment.

[0032] S102: performing outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, performing clustering processing on the plurality of target three-dimensional coordinate information to determine centroid coordinate information under each cluster.

[0033] In this step, outlier removal processing is performed on the multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, and clustering processing is performed on the multiple target three-dimensional coordinate information to determine the centroid coordinate information under each cluster.

[0034] Here, removing outliers in the data during the data preprocessing stage is a key step to ensure the reliability and stability of the input data, aiming to reduce the errors caused by noise points or drift points. Through density-based clustering methods, points that do not belong to any major data group can be effectively identified and removed. The distance θ between each point and other points is calculated (the distance θ here is the distance after the coordinates are normalized). x,y,z dimensional distance) and the minimum number of neighbors n to filter data points. First, for each data point, define a neighborhood range S, which represents the area around the point. Then within the range S, when the number of point pairs that satisfy the distance between the point dot and other points is less than θ exceeds n, the point is judged to belong to a high-density area, and the point is set to be a core point, which is located in a dense area. Next, by checking other points in the neighborhood of the core point, it can be determined whether they are boundary points (not enough points but belong to the neighborhood of the core point) or noise points (neither in the neighborhood of any core point nor have enough neighbors). Boundary points are considered to be part of the current data population, while noise points are eliminated. In a possible implementation manner, clustering the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster includes: a: Randomly select the initial cluster number target three-dimensional coordinate information as the initial centroid coordinate information.

[0035] Here, the initial cluster number k is set to start from 2, and k target 3D coordinate information is randomly selected as the centroid coordinate information , .

[0036] b: For each target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each initial centroid coordinate information; if the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information.

[0037] Among them, each target three-dimensional coordinate information is traversed, the distance between the target three-dimensional coordinate information and each initial centroid coordinate information is calculated, and the target three-dimensional coordinate information is allocated to the cluster corresponding to the nearest initial centroid coordinate information.

[0038] Here, the target 3D coordinate information is assigned to the cluster corresponding to the nearest initial centroid coordinate information through the following formula:

[0039] in, For the target three-dimensional coordinate information Perform cluster allocation, For the j The initial center of mass coordinate information.

[0040] c: Based on the three-dimensional coordinate information of multiple targets under each cluster, the new centroid coordinate information of each cluster is re-determined until the difference between the new centroid coordinate information and the corresponding initial centroid coordinate information is less than a preset threshold, and the determination of the new centroid coordinate information is stopped.

[0041] Here, the new centroid coordinate information of each cluster is re-determined by the following formula:

[0042] in, For the j New centroid coordinate information, For the j Clusters, for The next i The three-dimensional coordinate information of the target.

[0043] d: Set multiple groups of initial cluster numbers, determine the target cluster number based on multiple centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and determine the centroid coordinate information under each cluster under the target cluster number.

[0044] Here, after completing the above process, adjust the k value (generally set to k+1), repeat the above operation m times, and obtain m groups of centroid coordinate information under different k values. According to the multiple centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, the target cluster number is determined, and the centroid coordinate information of each cluster under the target cluster number is determined according to the above steps.

[0045] In a possible implementation manner, determining the number of target clusters based on a plurality of centroid coordinate information under each initial number of clusters and corresponding target three-dimensional coordinate information includes: Determine the sum of squares of distances from target three-dimensional coordinate information of multiple clusters to corresponding centroid coordinate information under each initial cluster number; detect whether the second derivative of the sum of squares of distances for each initial cluster number is negative; if so, use the initial cluster number as the target cluster number.

[0046] Here, the sum of squared distances of the initial number of clusters is determined by the following formula:

[0047] Among them, Inertia is the initial number of clusters k The sum of squared distances, for The centroid coordinate information of the cluster, x for The target three-dimensional coordinate information of the cluster. When the second-order derivative of Inertia is negative, that is, it becomes flat, determine the k value at this time and record the k value at this time as the target cluster number.

[0048] Among them, traditional clustering methods usually require manual setting of the number of clusters k, which may not be applicable when facing different operation scenarios or data sets, resulting in poor clustering results. However, this application automatically determines the optimal number of clusters k through analysis of Inertia, avoiding the inadaptability of manually setting the k value and improving the accuracy and adaptability of the clustering results. Dynamically determining the k value can optimize the clustering results according to the actual data situation, allowing the system to adapt to different surgical scenarios and improving the quality and accuracy of clustering.

[0049] Here, in traditional technology, interference frames (such as irrelevant background changes or unstable images in the video) often lead to the inability to accurately associate the positions before and after the needle is inserted, affecting the depth prediction. However, this application uses a clustering algorithm combined with a timestamp for data screening, which can effectively filter out data points within the wrong time range. Through this technology, the system can verify the time sequence of data points when clustering, thereby removing interference data, accurately identifying frames unrelated to needle insertion, reducing the interference of irrelevant information on the results, and improving the accuracy of depth prediction.

[0050] S103: sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determining the valid three-dimensional coordinate information under each cluster, and determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information.

[0051] In this step, the three-dimensional coordinate information of multiple targets under each cluster is sorted and filtered according to the timestamp information, the valid three-dimensional coordinate information under each cluster is determined, and the insertion state of the needle body is determined according to the inter-cluster distance between any two centroid coordinate information.

[0052] The needle insertion state of the needle body includes a state before the needle insertion, a state after the needle insertion, and a stable needle insertion state.

[0053] Here, combined with the time series information of the video frame, the centroid coordinate information of each cluster obtained by clustering is sorted in the time order of the frame. For points that do not conform to the time series, that is, points that are incorrectly assigned to the next time range in the previous time range, these points are discarded. The dynamic behavior and changes of the needle during the operation can be further analyzed by sorting the time series.

[0054] In a possible implementation manner, determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information includes: (1): Determine the inter-cluster distance between any two centroid coordinate information based on the Euclidean distance calculation formula.

[0055] Here, the time factor of the frame is ignored in the process of measuring the distance between clusters, and only the relative position of the needle on the image plane is concerned. Specifically, the calculation of the distance between clusters is based on the Euclidean distance between the coordinate position information of each centroid.

[0056] (2): The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the state of the needle body before insertion, the second cluster is determined as the state of the needle body after insertion, and the third cluster is determined as the stable state of the needle body; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

[0057] Here, by analyzing the inter-cluster distances in all frames, we find the maximum spacing of the needle body on the plane coordinates (x, y). This step lays the foundation for judging the state of needle insertion by screening the range of motion of the needle body. Based on the change of inter-cluster distances and frame sequence information, we can effectively judge the current state of needle insertion. Specifically, the maximum inter-cluster distance is regarded as a sign of needle insertion action. Assume that the distance (x, y) is the cluster With cluster The distance between clusters Indicates the state before needle insertion, cluster Indicates the state after needle insertion, and cluster The needle insertion is stable.

[0058] Here, the traditional method has difficulty dealing with complex situations such as stagnation or jitter when dealing with the dynamic changes of the needle body during the needle insertion process, which increases the difficulty and uncertainty of the prediction. However, the present application combines time series sorting and cluster analysis to accurately identify the states of "just completed needle insertion" and "stagnation or jitter after needle insertion". Through clear needle insertion state classification and time series analysis, the system can distinguish different dynamic behaviors during the needle insertion stage, thereby more accurately describing the state of the needle body and improving the prediction accuracy of the needle insertion depth.

[0059] S104: updating the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determining the updated centroid coordinate information, and determining the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters.

[0060] In this step, the centroid coordinate information of the cluster corresponding to the insertion state of the needle body is updated to determine the updated centroid coordinate information, and the actual insertion depth of the needle body is determined based on the updated centroid coordinate information and the camera internal parameters.

[0061] In a possible implementation manner, updating the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body to determine the updated centroid coordinate information includes: I: Based on the valid three-dimensional coordinate information after the preset timestamp in the first cluster corresponding to the state before needle insertion, the updated centroid coordinate information of the first cluster is determined.

[0062] Here, the updated centroid coordinate information of the first cluster is determined according to the valid three-dimensional coordinate information after the preset timestamp in the first cluster corresponding to the state before needle insertion.

[0063] Among them, according to the first cluster The updated centroid coordinate information of the first cluster is determined based on the valid three-dimensional coordinate information of the last quarter of the needle insertion phase.

[0064] II: Based on the valid three-dimensional coordinate information after the preset timestamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

[0065] Here, the updated centroid coordinate information of the third cluster is determined according to the valid three-dimensional coordinate information after the preset time stamp in the third cluster corresponding to the needle insertion stable state.

[0066] Among them, according to the third cluster The updated centroid coordinate information of the third cluster is determined based on the valid three-dimensional coordinate information of the last quarter of the needle insertion stable phase.

[0067] In a possible implementation manner, determining the actual insertion depth of the needle body based on the updated center of mass coordinate information and the camera internal parameters includes: i: Determine the difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster.

[0068] Here, the difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster is determined.

[0069] ii: Perform pixel distance conversion processing on the difference coordinate information and the camera internal parameters to determine the actual needle insertion depth.

[0070] Here, the difference coordinate information and the camera internal parameters are converted into pixel distance to determine the actual needle insertion depth.

[0071] In this case, based on the plane coordinates (x, y) and time series, the recalculated aggregate After the class center and Calculate the pixel distance of the needle. Combine the three-dimensional coordinates (x, y, z) of the needle and the camera internal parameters to convert the pixel distance of the needle into the actual needle depth. This result can be directly used to evaluate the depth and position of the needle insertion during surgery, providing accurate spatial information support for surgical operations.

[0072] For further information, see Figure 2 , Figure 2 A schematic diagram of a method for determining the depth of a retinal needle insertion provided in an embodiment of the present application. Figure 2 As shown in , the two-dimensional coordinate information (x, y) and frame coordinates (z) of the key feature points of the needle body during movement are identified according to the YOLO model, and the two-dimensional coordinate information and the frame coordinates are combined to construct the three-dimensional coordinate information. The three-dimensional coordinate information is processed by outliers and Kmeans clustering method to obtain the three-dimensional coordinate information under each cluster, and the inter-class distance of the centroid coordinate information under each cluster in the (x, y) dimension is measured, and the longest inter-class distance in the (x, y) dimension is screened out to determine the needle insertion state, and the cluster center point processing is performed on the cluster corresponding to the needle insertion state. Then, the needle insertion pixel distance between the updated centroid coordinate information and the camera internal parameters are determined to determine the actual insertion depth of the needle body.

[0073] In this application, by introducing the combination of timestamp and relative position of the image plane, not only the influence of timing error and interference frame in the prior art is effectively overcome, but also the dynamic optimization method is adopted in cluster analysis, which greatly improves the accuracy and robustness of needle depth prediction. Compared with traditional methods, this application has higher flexibility and adaptability, can more accurately capture the dynamic changes of the needle body during surgery, and significantly improves the reliability of prediction and processing efficiency.

[0074] The present application provides a method for determining the depth of retinal needle insertion, the method comprising: performing target detection on the surgical video of the retina based on the YOLO model, identifying the three-dimensional coordinate information of the key feature points of the needle body during the movement; performing outlier removal processing on multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, clustering the multiple target three-dimensional coordinate information, and determining the centroid coordinate information under each cluster; sorting and filtering the multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determining the effective three-dimensional coordinate information under each cluster, and determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information; updating the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determining the updated centroid coordinate information, and determining the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters. By sorting and clustering the target three-dimensional coordinate information, it is possible to accurately identify timestamps unrelated to the insertion of the needle, reduce the interference of irrelevant information on the results, and distinguish the insertion state of the needle body so as to more accurately describe the insertion state of the needle body, thereby improving the accuracy of the determination of the insertion depth.

[0075] See also Figure 3 , Figure 3 This is a schematic diagram of the structure of a device for determining the depth of retinal needle insertion provided by an embodiment of the present application. Figure 3 As shown in , the device 300 for determining the retinal needle insertion depth includes: The needle body recognition module 310 is used to perform target detection on the retinal surgery video based on the YOLO model and recognize the three-dimensional coordinate information of key feature points of the needle body during movement; The clustering module 320 is used to perform outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, and perform clustering processing on the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster; A needle insertion state recognition module is used to sort and filter multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the needle insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information; The depth determination module is used to update the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determine the updated centroid coordinate information, and determine the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters.

[0076] Furthermore, when the needle body recognition module 310 is used to perform target detection on the retinal surgical video based on the YOLO model and identify the three-dimensional coordinate information of the key feature points of the needle body during the movement, the needle body recognition module 310 is specifically used to: Performing target detection on the retinal surgical video based on the YOLO model to identify the moving image area of ​​the needle body; Extracting two-dimensional coordinate information of key feature points of the needle body in the motion image area; Coordinate integration is performed based on the time stamp information corresponding to the motion image area and the two-dimensional coordinate information of the key feature point to determine the three-dimensional coordinate information of the key feature point.

[0077] Furthermore, when the clustering module 320 is used to perform clustering processing on the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster, the clustering module 320 is specifically used to: Randomly select the initial cluster number target three-dimensional coordinate information as the initial centroid coordinate information; For each of the target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each of the initial centroid coordinate information, and if the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information; Based on the three-dimensional coordinate information of multiple targets under each cluster, new centroid coordinate information of each cluster is re-determined until the difference between the new centroid coordinate information and the corresponding initial centroid coordinate information is less than a preset threshold, and the determination of the new centroid coordinate information is stopped; A plurality of groups of initial cluster numbers are set, and the target cluster number is determined based on a plurality of centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and the centroid coordinate information under each cluster under the target cluster number is determined.

[0078] Furthermore, when the clustering module 320 is used to determine the number of target clusters based on the plurality of centroid coordinate information under each initial number of clusters and the corresponding target three-dimensional coordinate information, the clustering module 320 is specifically used to: Determine the sum of squares of distances from target three-dimensional coordinate information of multiple clusters to corresponding centroid coordinate information under each initial number of clusters; Detect whether the second derivative of the sum of squares of the distances for each initial cluster number is negative; If yes, the initial number of clusters is used as the target number of clusters.

[0079] Furthermore, when the needle insertion state recognition module 330 is used to determine the needle insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information, the needle insertion state recognition module 330 is specifically used to: Based on the Euclidean distance calculation formula, the inter-cluster distance between any two centroid coordinate information is determined; The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the state of the needle body before insertion, the second cluster is determined as the state of the needle body after insertion, and the third cluster is determined as the stable state of the needle body; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

[0080] Furthermore, when the depth determination module 340 is used to update the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body and determine the updated centroid coordinate information, the depth determination module 340 is specifically used to: Determining updated centroid coordinate information of the first cluster based on valid three-dimensional coordinate information after a preset timestamp in the first cluster corresponding to the pre-needle insertion state; Based on the valid three-dimensional coordinate information after the preset time stamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

[0081] Furthermore, when the depth determination module 340 is used to determine the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters, the depth determination module 340 is specifically used to: Determining difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster; The difference coordinate information and the camera internal parameters are subjected to pixel distance conversion processing to determine the actual needle insertion depth.

[0082] An embodiment of the present application provides a device for determining the depth of retinal needle insertion, the determination device comprising: a needle body recognition module, used to perform target detection on a retinal surgical video based on a YOLO model, and identify the three-dimensional coordinate information of key feature points of the needle body during movement; a clustering module, used to perform outlier removal processing on multiple three-dimensional coordinate information to obtain multiple target three-dimensional coordinate information, cluster the multiple target three-dimensional coordinate information, and determine the centroid coordinate information under each cluster; a needle insertion state recognition module, used to sort and filter the multiple target three-dimensional coordinate information under each cluster according to timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the needle insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information; a depth determination module, used to update the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body, determine the updated centroid coordinate information, and determine the actual needle insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters. By sorting and clustering the target three-dimensional coordinate information, timestamps unrelated to needle insertion can be accurately identified, reducing the interference of irrelevant information on the results, and the needle insertion status can be distinguished to more accurately describe the needle insertion status of the needle body, thereby improving the accuracy of needle insertion depth determination.

[0083] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0084] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 as well as Figure 2 The steps of the method for determining the retinal needle insertion depth in the method embodiment shown, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0085] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 as well as Figure 2 The steps of the method for determining the retinal needle insertion depth in the method embodiment shown, and the specific implementation method can be found in the method embodiment, which will not be repeated here.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0087] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0089] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0090] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0091] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for determining the depth of retinal needle insertion, characterized in that: The determination method comprises: Based on the YOLO model, target detection is performed on retinal surgical videos to identify the three-dimensional coordinate information of key feature points of the needle during its movement. Performing outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, and performing clustering processing on the plurality of target three-dimensional coordinate information to determine centroid coordinate information under each cluster; The three-dimensional coordinate information of multiple targets under each cluster is sorted and filtered according to the timestamp information, the valid three-dimensional coordinate information under each cluster is determined, and the insertion state of the needle body is determined based on the inter-cluster distance between any two centroid coordinate information; The centroid coordinate information of the cluster corresponding to the insertion state of the needle body is updated to determine the updated centroid coordinate information, and the actual insertion depth of the needle body is determined based on the updated centroid coordinate information and the camera internal parameters.

2. The determination method according to claim 1, characterized in that: The YOLO model is used to detect the target of the retinal surgical video and identify the three-dimensional coordinate information of the key feature points of the needle during the movement, including: Performing target detection on the retinal surgical video based on the YOLO model to identify the moving image area of ​​the needle body; Extracting two-dimensional coordinate information of key feature points of the needle body in the motion image area; Coordinate integration is performed based on the time stamp information corresponding to the motion image area and the two-dimensional coordinate information of the key feature point to determine the three-dimensional coordinate information of the key feature point.

3. The determination method according to claim 1, characterized in that: The clustering process of the plurality of target three-dimensional coordinate information to determine the centroid coordinate information of each cluster includes: Randomly select the initial cluster number target three-dimensional coordinate information as the initial centroid coordinate information; For each of the target three-dimensional coordinate information, determine the distance between the target three-dimensional coordinate information and each of the initial centroid coordinate information, and if the distance is less than a preset distance, assign the target three-dimensional coordinate information to the cluster corresponding to the initial centroid coordinate information; Based on the three-dimensional coordinate information of multiple targets under each cluster, new centroid coordinate information of each cluster is re-determined until the difference between the new centroid coordinate information and the corresponding initial centroid coordinate information is less than a preset threshold, and the determination of the new centroid coordinate information is stopped; A plurality of groups of initial cluster numbers are set, and the target cluster number is determined based on a plurality of centroid coordinate information under each initial cluster number and the corresponding target three-dimensional coordinate information, and the centroid coordinate information under each cluster under the target cluster number is determined.

4. The determination method according to claim 3, characterized in that: The step of determining the number of target clusters based on the plurality of centroid coordinate information under each initial number of clusters and the corresponding target three-dimensional coordinate information includes: Determine the sum of squares of distances from target three-dimensional coordinate information of multiple clusters to corresponding centroid coordinate information under each initial number of clusters; Detect whether the second derivative of the sum of squares of the distances for each initial cluster number is negative; If yes, the initial number of clusters is used as the target number of clusters.

5. The determination method according to claim 1, characterized in that: The step of determining the insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information includes: Based on the Euclidean distance calculation formula, the inter-cluster distance between any two centroid coordinate information is determined; The first cluster of the two clusters corresponding to the maximum inter-cluster distance is determined as the state of the needle body before insertion, the second cluster is determined as the state of the needle body after insertion, and the third cluster is determined as the stable state of the needle body; wherein the timestamps of the first cluster, the second cluster and the third cluster increase in sequence.

6. The determination method according to claim 1, characterized in that: Updating the centroid coordinate information of the cluster corresponding to the needle insertion state of the needle body to determine the updated centroid coordinate information includes: Determining updated centroid coordinate information of the first cluster based on valid three-dimensional coordinate information after a preset timestamp in the first cluster corresponding to the pre-needle insertion state; Based on the valid three-dimensional coordinate information after the preset time stamp in the third cluster corresponding to the needle insertion stable state, the updated centroid coordinate information of the third cluster is determined.

7. The determination method according to claim 6, characterized in that: The actual insertion depth of the needle body is determined based on the updated centroid coordinate information and the camera internal parameters, including: Determining difference coordinate information between the updated centroid coordinate information of the third cluster and the updated centroid coordinate information of the first cluster; The difference coordinate information and the camera internal parameters are subjected to pixel distance conversion processing to determine the actual needle insertion depth.

8. A device for determining the depth of retinal needle insertion, characterized in that: The determining device comprises: The needle recognition module is used to detect targets in retinal surgical videos based on the YOLO model and identify the three-dimensional coordinate information of key feature points of the needle during its movement; A clustering module is used to perform outlier removal processing on the plurality of three-dimensional coordinate information to obtain a plurality of target three-dimensional coordinate information, perform clustering processing on the plurality of target three-dimensional coordinate information to determine the centroid coordinate information under each cluster; A needle insertion state recognition module is used to sort and filter multiple target three-dimensional coordinate information under each cluster according to the timestamp information, determine the valid three-dimensional coordinate information under each cluster, and determine the needle insertion state of the needle body based on the inter-cluster distance between any two centroid coordinate information; The depth determination module is used to update the centroid coordinate information of the cluster corresponding to the insertion state of the needle body, determine the updated centroid coordinate information, and determine the actual insertion depth of the needle body based on the updated centroid coordinate information and the camera internal parameters.

9. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and the machine-readable instructions are executed by the processor to execute the steps of the method for determining the retinal needle insertion depth as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method for determining the retinal needle insertion depth as described in any one of claims 1 to 7 are executed.

Images