Laparoscopic surgery automatic focusing method and system based on surgery operation area
By using an autofocus method, fitting surgical operation area labels with deformation fields and Gaussian filtering, and combining deep learning technology, the problems of time-consuming and inaccurate traditional manual focusing methods are solved. This enables real-time and accurate focusing of the surgical operation area in laparoscopic surgery, improving the accuracy and efficiency of the surgery.
Patent Information
- Application Number
- CN202510872168.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-26
- Publication Date
- 2025-10-28
AI Technical Summary
Traditional manual focusing methods are time-consuming in laparoscopic surgery and are easily affected by the operator's subjective factors, resulting in inaccurate focusing and affecting the accuracy and efficiency of the surgical operation.
An automatic focusing method based on the surgical operation area is adopted. By acquiring video clips of surgical instruments, the query frame of the surgical operation area and the tip of the instrument are determined. The operation area label is fitted by deformation field and Gaussian filtering, and the laparoscopic focusing parameters are optimized in real time. Combined with deep learning technology, precise focusing is achieved.
It enables real-time and accurate focusing of the surgical area, reducing the risks caused by unclear vision and improving the accuracy and efficiency of surgery.
Smart Images
Figure CN120856964A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of surgical microscope technology, and in particular to an automatic focusing method and system for laparoscopic surgery based on the surgical operating area. Background Technology
[0002] With the continuous development of medical technology, laparoscopic surgery has been widely used in clinical practice due to its advantages such as minimal invasiveness and rapid recovery. However, during laparoscopic surgery, the clarity of the surgical field is crucial to the success of the procedure due to the limited surgical field. Traditional manual focusing methods are not only time-consuming but also easily affected by the operator's subjective factors, leading to inaccurate focusing. Therefore, developing a laparoscopic surgical system with automatic focusing to improve the accuracy and efficiency of surgical operations has become an urgent problem to be solved in the current medical field. Summary of the Invention
[0003] This application proposes an automatic focusing method and system for laparoscopic surgery based on the surgical operation area to overcome or at least partially overcome the shortcomings of the prior art.
[0004] This application discloses an automatic focusing method for laparoscopic surgery based on the surgical operation area, the method comprising:
[0005] Select surgical instrument video segments from real surgical videos, and determine the surgical operation area query frame and surgical instrument tip point based on the surgical instrument video segments;
[0006] Obtain the deformation field from the query frame to other frames;
[0007] Based on the deformation field, the position information of the tip of the surgical instrument in each frame of the surgical instrument video segment is obtained in the query frame, and the distribution point data of all surgical operation areas is obtained.
[0008] Based on the obtained distribution point data of the surgical operation area, Gaussian filtering is used to fit the distribution point data of all surgical operation areas to obtain the surgical operation area labels.
[0009] The surgical area detection model is trained based on the acquired surgical area labels, and the focus score of the surgical area is obtained in real time. The laparoscopic focusing parameters are continuously optimized and adjusted based on the score results.
[0010] In one embodiment, the step of selecting surgical instrument video segments based on real surgical videos and determining intermediate frames based on surgical instrument video segments further includes:
[0011] Input the real surgical video into the surgical instrument segmentation model to select the outline of the surgical instrument, and obtain the minimum bounding rectangle of the surgical instrument outline.
[0012] The outline of the surgical instrument is fitted with a minimum bounding box, and images of surgical instrument segments with more than N consecutive frames are selected as surgical instrument video segments using the minimum bounding box.
[0013] Each surgical instrument video segment needs to be processed individually, and the middle frame of each surgical instrument video segment is selected as the query frame.
[0014] In one embodiment, determining the tip of a surgical instrument based on a video clip of the surgical instrument includes:
[0015] Obtain the center point, width, height, and rotation angle of the smallest rectangle containing the outline of the surgical instrument;
[0016] Obtain the offset of the surgical instrument outline relative to the center point of the minimum rectangle;
[0017] Based on the offset of the minimum rectangle and the direction vector of the major axis, calculate the projection length of each point on the surgical instrument contour onto the major axis of the minimum rectangle, and select the point with the largest or smallest projection length.
[0018] By comparing the minimum distance from the point with the largest and smallest projection length to the edge pixels, the point with the largest minimum distance is selected as the tip of the surgical instrument. The image coordinates of the tip of the surgical instrument are the location of the surgical operation area.
[0019] In one embodiment, obtaining the deformation field from the query frame to other frames further includes:
[0020] The selected surgical instrument video clips are divided into two segments, one before and one after the query frame. Both segments contain the query frame.
[0021] The first part of the video is reversed to ensure that the query frame is the first frame, and the subsequent part of the video does not require additional processing.
[0022] The two video segments are input into the point tracking model to obtain the positions of the grid points in the query frame on other frames.
[0023] Based on the positions of the grid points in the query frame on other frames, obtain the deformation field from the query frame to other frames.
[0024] In one embodiment, the step of obtaining the position information of the surgical instrument tip in each frame of the surgical instrument video clip based on the deformation field, and obtaining the distribution point data of all surgical operation areas, further includes:
[0025] Project the tip of the surgical instrument in other frames of the surgical instrument video clip onto the query frame to obtain the position information of the tip of the surgical instrument in the query frame;
[0026] Based on the position information of the surgical instrument tip in the query frame, the surgical operation area of the query frame is obtained. Then, according to the deformation field matrix, the surgical operation area of the query frame is mapped onto other frames to obtain the position information of the surgical instrument tip in each frame of the surgical instrument video segment in the query frame, thus obtaining the distribution point data of all surgical operation areas.
[0027] In one embodiment, the step of using Gaussian filtering to fit the distribution point data of all surgical operation areas based on the acquired distribution point data of the surgical operation areas to obtain surgical operation area labels further includes:
[0028] Preprocess the data on the distribution points of the surgical operation area;
[0029] Gaussian blur is used to process the preprocessed surgical operation area distribution point data into a heat map, and the heat map is then normalized.
[0030] Conditional random fields are used to enhance the boundary quality of normalized heatmaps;
[0031] Set a filtering threshold to obtain labels for the surgical operation area.
[0032] In one embodiment, the step of training a surgical area detection model based on the acquired surgical area labels, acquiring a focus score for the surgical area in real time, and continuously optimizing and adjusting the laparoscopic focusing parameters based on the focus score results further includes:
[0033] The surgical operation area image is converted into a grayscale image, and then the Sobel operator is used to calculate the gradient of each pixel in the surgical operation area.
[0034] The average gradient magnitude of each pixel in the surgical area is used as the focus quality score.
[0035] In one embodiment, obtaining the deformation field from the query frame to other frames further includes:
[0036] The optimization of the deformation field includes the following steps:
[0037] Choose a loss function.
[0038] The loss function is used to measure the deformation field calculation grid point p′ and the point tracking prediction grid point p. tracking The difference between them; where the point tracking result of grid point p in other frames is p tracking The grid point p is transformed into a point p′ in other frames through the deformation field transformation, and the deformation field is optimized by minimizing the loss function.
[0039] In one embodiment, the boundary quality of the heatmap enhanced by conditional random fields further includes:
[0040] The normalized heatmap and the real surgical video are fed into a conditional random field. The color and spatial information of the real surgical video are used to improve the boundary of the heatmap and remove noisy areas.
[0041] This application also discloses an automatic focusing system for laparoscopic surgery based on the surgical operation area, an image acquisition module that selects surgical instrument video segments based on real surgical videos, and determines the surgical operation area query frame and the tip of the surgical instrument based on the surgical instrument video segments;
[0042] The deformation field acquisition module acquires the deformation field from the query frame to other frames;
[0043] The operation area acquisition module obtains the position information of the tip of the surgical instrument in each frame of the surgical instrument video clip based on the deformation field, and obtains the distribution point data of all surgical operation areas.
[0044] The operation area label acquisition module uses Gaussian filtering to fit the distribution of all surgical operation areas based on the acquired distribution point data of the surgical operation areas, and obtains the surgical operation area labels.
[0045] The parameter adaptive adjustment module trains the operation area detection model based on the acquired surgical operation area labels, obtains the focus score of the operation area in real time, and continuously optimizes and adjusts the laparoscopic focusing parameters based on the score results.
[0046] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:
[0047] This application enables real-time and accurate adjustment of focusing parameters, ensuring the surgical area remains in sharp focus at all times. This not only helps surgeons more accurately determine the location of lesions and the progress of the surgery, but also reduces surgical risks caused by unclear vision.
[0048] This application utilizes advanced image processing and deep learning technologies, possesses intelligent recognition capabilities, and can automatically identify key features of the surgical operation area to achieve precise focusing.
[0049] This application creates operation area labels based on real surgical videos, eliminating the need for additional manual corrections and ensuring the accuracy of the operation area labels. At the same time, it can be customized according to the habits and needs of different surgeons. Attached Figure Description
[0050] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0051] Figure 1This is a flowchart of the automatic focusing method for laparoscopic surgery based on the surgical operation area according to Embodiment 1 of this application;
[0052] Figure 2 This is a schematic diagram illustrating the determination of the circumscribed rectangle of the surgical instrument and the tip of the surgical instrument in this application;
[0053] Figure 3 This is a flowchart of the deformation field acquisition process;
[0054] Figure 4 This is a schematic diagram of the projection of the tip of a surgical instrument onto the query frame;
[0055] Figure 5 This is the step of obtaining the surgical procedure area label;
[0056] Figure 6 This is a schematic diagram of the automatic focusing system for laparoscopic surgery based on the surgical operation area according to Embodiment 2 of this application. Detailed Implementation
[0057] To make the purpose, technical solutions, and advantages of this application more clear, the technical solutions of this application will be clearly and completely described below in conjunction with the specific embodiments of this application and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.
[0058] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0059] Embodiment 1 of this application discloses an automatic focusing method for laparoscopic surgery based on the surgical operation area, from... Figure 1 It can be seen that this application includes at least steps S100 to S500:
[0060] S100: Select surgical instrument video segments based on real surgical videos, and determine the surgical operation area query frame and surgical instrument tip point based on the surgical instrument video segments.
[0061] like Figure 2 The diagram shown illustrates the determination of the circumscribed rectangle of the surgical instrument and the tip of the surgical instrument.
[0062] Specifically, it includes:
[0063] S110: Input the real surgical video into the surgical instrument segmentation model to select the surgical instrument outline and obtain the minimum bounding rectangle of the surgical instrument outline.
[0064] First, all images are fed into the surgical instrument segmentation model to obtain the segmentation results of the surgical instruments, and then their minimum bounding rectangle is fitted.
[0065] S120, fit the outline of the surgical instrument to a minimum rectangle, and select images of surgical instrument segments with more than N consecutive frames as surgical instrument video segments through the minimum rectangle.
[0066] Specifically: Select images of surgical instruments that have a minimum rectangle length greater than 1 / 10 of the length of the image in the actual surgical video and that are more than 200 consecutive frames long as surgical instrument video segments.
[0067] S130 Each surgical instrument video segment needs to be processed separately, and the middle frame of each surgical instrument video segment is selected as the query frame to provide a reference for subsequent deformation field calculation.
[0068] While fitting the minimum bounding box to the contour of the surgical instrument, the image coordinates of the tip of the surgical instrument are obtained as the location of the surgical operation area. The specific steps for determining the tip of the surgical instrument include:
[0069] S111 obtains the center point position of the minimum rectangle of the surgical instrument outline, the width and height of the rectangle, and the rotation angle.
[0070] Specifically: First, obtain the center point P of the smallest rectangle containing the outline of the surgical instrument. box The width and height (W) of the smallest rectangle box H box The direction of the major axis of the minimum rectangle is determined by the values of its width and height, and the direction vector of the major axis is calculated using the rotation angle. The calculation formula is as follows:
[0071]
[0072] In Formula 1.1, the parameters It is the direction vector of the major axis of the smallest rectangle.
[0073] S112 obtains the offset of the surgical instrument outline relative to the center point of the minimum rectangle;
[0074] Specifically: Next, obtain the contour offset (offset) of the surgical instrument's contour relative to the center point of the smallest rectangle.
[0075] offset = Contours - P box (0.2)
[0076] S113 calculates the projection length of each point on the surgical instrument contour onto the major axis of the minimum rectangle based on the offset and the direction vector of the major axis, and selects the point with the largest or smallest projection length.
[0077] Specifically: Based on the offset and the direction vector of the major axis, calculate the projection length of each point on the surgical instrument contour onto the major axis of the smallest rectangle, that is, calculate the contour offset and direction vector. The inner product innerProb is calculated using the following formula:
[0078]
[0079] The point where the inner product value (innerProb) is the maximum or minimum is the tip of the surgical instrument. These two points are first identified and marked as P. max and P min P max P is the point of maximum and the point of minimum. max and P min The formula is as follows:
[0080]
[0081] S114 compares the minimum distance from the point with the largest projection length to the edge pixel with the smallest projection length, and selects the point with the largest minimum distance as the tip of the surgical instrument.
[0082] Specifically: Obtain the distance matrix D from non-edge pixels to the nearest edge pixel using the binary edge map of the surgical instrument operation screen. The values in this distance matrix D represent the Euclidean distance from that pixel to the nearest edge pixel. Then, query P... max and P min The minimum distance from two points to the edge pixel is used to determine the tip point P of the instrument, with the point having the larger distance being the tip point P. tip The calculation formula is:
[0083]
[0084] S200, obtain the deformation field from the query frame to other frames.
[0085] This application aims to obtain the position information of the operating area in each frame of a surgical instrument video clip within a query frame. This requires acquiring the deformation field from each frame to the reference frame. Point tracking code typically initializes the point grid with the first frame. Therefore, the selected surgical instrument video clip needs to be divided into two segments, with the query frame as the boundary. The first segment needs to be reversed to ensure the query frame is the first frame, while the second segment requires no additional processing. The two segments are then fed into the point tracking network to obtain the positions of the grid points in the query frame on other frames. Finally, the deformation field is calculated based on the changes in the grid point positions between the two frames.
[0086] The deformation field acquisition process from the query frame to other frames is as follows: Figure 3 As shown, it specifically includes:
[0087] S210, the selected surgical instrument video segment is divided into two video segments with the query frame as the boundary, and both video segments contain the query frame.
[0088] S220: The first video is reversed to ensure that the query frame is the first frame, and the subsequent video does not require additional processing.
[0089] S230: Input the two video segments into the point tracking model respectively to obtain the position of the grid points in the query frame on other frames.
[0090] The point tracking model used in this application is a deep learning model, which can be applied to motion analysis and can also be extended to motion tracking of various objects. The point tracking model used in this application is the SpatialTracker model.
[0091] S240, based on the positions of the grid points in the query frame on other frames, obtain the deformation field from the query frame to other frames.
[0092] In this application, the size of the grid points in the query frame during point tracking is H×W, indicating that the grid points are evenly distributed across the query frame in H rows and W columns. The coordinates of each grid point p can be represented as (x, y), and the tracking result of point p in other frames is p. tracking Point p is transformed into a point p′ in another frame through the deformation field, and its calculation formula is as follows:
[0093] p′=p+f(p) (0.6)
[0094] Where f is the deformation field to be obtained. Since this application targets a two-dimensional image, the deformation field f(p) is also two-dimensional, and the package represents the horizontal X and vertical Y displacements of grid point p:
[0095] f(p)=(f x (p), f y (p)) (0.7)
[0096] This application S200 also includes optimization of the deformation field f(p).
[0097] Solving for the deformation field f(p) is an optimization problem, with the goal of aligning the grid points in the target frame with the predicted deformation field in the query frame. To this end, this application uses the following loss function to solve for the deformation field.
[0098] The loss function used in this application is the tracking loss function L. trackingUsed for measuring deformation field calculation grid point p′ and point tracking prediction grid point p tracking The difference between them, the point tracking result of point p in other frames is p tracking Point p is transformed into point p′ in other frames through the deformation field. The deformation field is optimized by minimizing this loss so that the deformation produced by the deformation field can align the prediction results. The loss function formula is as follows:
[0099]
[0100] Where N is the total number of grid points, i.e., N = H × W.
[0101] By iteratively optimizing the deformation field using the above loss function, the deformation fields of the query frame to other frames of the surgical instrument video segment are obtained.
[0102] S300: Based on the deformation field, obtain the position information of the tip of the surgical instrument in each frame of the surgical instrument video clip in the query frame, and obtain the distribution point data of all surgical operation areas.
[0103] Specifically, such as Figure 4 The image shown is a schematic diagram of the projection of the tip of a surgical instrument onto a query frame.
[0104] The S300 also includes:
[0105] S310 projects the tip of the surgical instrument in other frames of the surgical instrument video clip onto the query frame to obtain the position information of the tip of the surgical instrument in the query frame;
[0106] Specifically: After obtaining the deformation field from the query frame to other frames, it is necessary to project the tip of the surgical instrument in other frames of the surgical instrument video clip onto the query frame in order to obtain the distribution of the surgical operation area on the query frame, that is, the position information of the tip of the surgical instrument in the query frame.
[0107] Based on the position information of the surgical instrument tip in the query frame, S320 obtains the surgical operation area of the query frame. Then, according to the deformation field matrix, it maps the surgical operation area of the query frame onto other frames, thereby obtaining the position information of the surgical instrument tip in each frame of the surgical instrument video segment and obtaining the distribution point data of all surgical operation areas.
[0108] After obtaining the distribution points of the operation area in the query frame, these points are then mapped onto other frames based on the deformation field matrix, thus providing the distribution of the operation area for all images. The location of the tip of the surgical instrument is the location of the surgical operation area.
[0109] Based on the acquired distribution point data of the surgical operation area, S400 uses Gaussian filtering to fit the distribution of all surgical operation areas and obtain the surgical operation area labels.
[0110] The surgical operation area distribution obtained in step S300 above refers to the distribution point data of the surgical operation area. This distribution point data is a series of two-dimensional coordinate points, which need to be further converted into binary map labels for semantic segmentation before they can be used for network training. For example... Figure 5 As shown, Figure 5 The steps to obtain the surgical procedure area label are as follows:
[0111] S410, preprocess the data of the distribution points in the surgical operation area;
[0112] Specifically, the preprocessing of the surgical operation area distribution point data involves: removing sparse points in the surgical operation area distribution based on the neighborhood density.
[0113] S420, Gaussian blur is used to process the preprocessed surgical operation area distribution point data into a heat map, and the heat map is normalized;
[0114] Specifically: Gaussian blurring was used to process the distribution data of surgical operation area points into a heat map, and the heat map results were normalized to the range of 0 to 1.
[0115] S430 employs conditional random fields to enhance the boundary quality of normalized heatmaps;
[0116] Specifically, the normalized heatmap and the original surgical scene image (i.e., the real surgical video mentioned above) are fed into a conditional random field. The color and spatial information of the original image are used to improve the boundary of the heatmap and remove noisy areas.
[0117] This application uses Conditional Random Field (CRF) to optimize image segmentation results, a classic post-processing technique that effectively improves the accuracy and robustness of the segmentation results. Its core idea is to correct noise and boundary blurring in local predictions by modeling the global contextual relationships between pixels. This algorithm has been integrated into the Python library pydensecrf.
[0118] S440, set the filtering threshold and obtain the surgical operation area label.
[0119] The S500 trains a surgical area detection model based on the acquired surgical area labels, obtains the focus score of the surgical area in real time, and continuously optimizes and adjusts the laparoscopic focusing parameters based on the focus score results.
[0120] Specifically, the binary map labels of the surgical area obtained in step S400 are used to train the surgical area detection model. The surgical area detection model is a semantic segmentation model, which realizes automatic detection of the surgical area during the operation. Then, the laparoscopic focusing parameters are continuously adjusted, and the focusing effect of the laparoscopic lens on the surgical area is monitored in real time. The focusing parameters are continuously optimized through a feedback mechanism to ensure that the surgical area is always clear.
[0121] Based on the acquired surgical operation area labels, a surgical operation area detection model is trained, and the real-time focus score of the surgical operation area is further obtained, including:
[0122] S510 converts the surgical operation area image into a grayscale image, and then uses the Sobel operator to calculate the gradient of each pixel in the surgical operation area.
[0123] The focus quality scoring algorithm in this application is primarily based on the gradient information of the image. First, the image of the operation region is converted to grayscale, and then the Sobel operator is used to calculate the gradient of each pixel in the operation region.
[0124]
[0125] In the formula, G represents the magnitude of the gradient. x and G y These represent the gradients in the x and y directions, respectively.
[0126] S520 uses the average gradient magnitude of each pixel in the surgical operation area as the focus quality score.
[0127] The formula for calculating the focus quality score is as follows:
[0128]
[0129] Where M represents the total number of pixels in the surgical operation area, and G i This represents the gradient magnitude at the i-th pixel.
[0130] By following the steps and formulas described above, the focus quality score of the laparoscopic image can be calculated. This score is used for autofocus control; by adjusting the focus parameters to maximize the score, automatic focusing of the surgical area can be achieved.
[0131] like Figure 6 As shown, Embodiment 2 of this application discloses an automatic focusing system for laparoscopic surgery based on the surgical operation area. The system includes:
[0132] Image acquisition module 310 selects surgical instrument video segments based on real surgical videos, and determines the surgical operation area query frame and surgical instrument tip point based on the surgical instrument video segments;
[0133] The deformation field acquisition module 320 acquires the deformation field from the query frame to other frames;
[0134] The operation area acquisition module 330 acquires the position information of the tip of the surgical instrument in each frame of the surgical instrument video clip based on the deformation field, and acquires the distribution point data of all surgical operation areas.
[0135] The operation area label acquisition module 340 uses Gaussian filtering to fit the distribution of all surgical operation areas based on the acquired distribution point data of the surgical operation area to obtain the surgical operation area label.
[0136] The parameter adaptive adjustment module 350 trains the operation area detection model based on the acquired surgical operation area labels, obtains the focus score of the operation area in real time, and continuously optimizes and adjusts the laparoscopic focusing parameters based on the score results.
[0137] It should be noted that the aforementioned laparoscopic surgery autofocus system based on the surgical operation area can realize the aforementioned laparoscopic surgery autofocus system based on the surgical operation area, which will not be elaborated here.
[0138] This application enables real-time and accurate adjustment of focusing parameters, ensuring the surgical area remains in sharp focus at all times. This not only helps surgeons more accurately determine the location of lesions and the progress of the surgery, but also reduces surgical risks caused by unclear vision.
[0139] This application utilizes advanced image processing and deep learning technologies, possesses intelligent recognition capabilities, and can automatically identify key features of the surgical operation area to achieve precise focusing.
[0140] This application creates operation area labels based on real surgical videos, eliminating the need for additional manual corrections and ensuring the accuracy of the operation area labels. At the same time, it can be customized according to the habits and needs of different surgeons.
[0141] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0142] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A method for automatic focusing in laparoscopic surgery based on the surgical operating area, characterized in that, The method includes: Select surgical instrument video segments from real surgical videos, and determine the surgical operation area query frame and surgical instrument tip point based on the surgical instrument video segments; Obtain the deformation field from the query frame to other frames; Based on the deformation field, the position information of the tip of the surgical instrument in each frame of the surgical instrument video segment is obtained in the query frame, and the distribution point data of all surgical operation areas is obtained. Based on the obtained distribution point data of the surgical operation area, Gaussian filtering is used to fit the distribution point data of all surgical operation areas to obtain the surgical operation area labels. The surgical area detection model is trained based on the acquired surgical area labels, and the focus score of the surgical area is obtained in real time. The laparoscopic focusing parameters are continuously optimized and adjusted based on the score results.
2. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 1, characterized in that, The step of selecting surgical instrument video segments based on real surgical videos and determining intermediate frames based on surgical instrument video segments further includes: Input the real surgical video into the surgical instrument segmentation model to select the outline of the surgical instrument, and obtain the minimum bounding rectangle of the surgical instrument outline. The outline of the surgical instrument is fitted with a minimum bounding box, and images of surgical instrument segments with more than N consecutive frames are selected as surgical instrument video segments using the minimum bounding box. Each surgical instrument video segment needs to be processed individually, and the middle frame of each surgical instrument video segment is selected as the query frame.
3. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 2, characterized in that, Based on video footage of surgical instruments, the tip of the surgical instrument was identified as including: Obtain the center point, width, height, and rotation angle of the smallest rectangle containing the outline of the surgical instrument; Obtain the offset of the surgical instrument outline relative to the center point of the minimum rectangle; Based on the offset of the minimum rectangle and the direction vector of the major axis, calculate the projection length of each point on the surgical instrument contour onto the major axis of the minimum rectangle, and select the point with the largest or smallest projection length. By comparing the minimum distance from the point with the largest and smallest projection length to the edge pixels, the point with the largest minimum distance is selected as the tip of the surgical instrument. The image coordinates of the tip of the surgical instrument are the location of the surgical operation area.
4. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 1, characterized in that, The step of obtaining the deformation field from the query frame to other frames further includes: The selected surgical instrument video clips are divided into two segments, one before and one after the query frame. Both segments contain the query frame. The first part of the video is reversed to ensure that the query frame is the first frame, and the subsequent part of the video does not require additional processing. The two video segments are input into the point tracking model to obtain the positions of the grid points in the query frame on other frames. Based on the positions of the grid points in the query frame on other frames, obtain the deformation field from the query frame to other frames.
5. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 1, characterized in that, The step of obtaining the position information of the tip of the surgical instrument in each frame of the surgical instrument video clip based on the deformation field, and obtaining the distribution point data of all surgical operation areas, further includes: Project the tip of the surgical instrument in other frames of the surgical instrument video clip onto the query frame to obtain the position information of the tip of the surgical instrument in the query frame; Based on the position information of the surgical instrument tip in the query frame, the surgical operation area of the query frame is obtained. Then, according to the deformation field matrix, the surgical operation area of the query frame is mapped onto other frames to obtain the position information of the surgical instrument tip in each frame of the surgical instrument video segment in the query frame, thus obtaining the distribution point data of all surgical operation areas.
6. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 1, characterized in that, The step of using Gaussian filtering to fit the distribution point data of all surgical operation areas based on the acquired distribution point data of the surgical operation areas to obtain surgical operation area labels further includes: Preprocess the data on the distribution points of the surgical operation area; Gaussian blur is used to process the preprocessed surgical operation area distribution point data into a heat map, and the heat map is then normalized. Conditional random fields are used to enhance the boundary quality of normalized heatmaps; Set a filtering threshold to obtain labels for the surgical operation area.
7. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 1, characterized in that, The step of training a surgical area detection model based on the acquired surgical area labels, acquiring real-time focus scores for the surgical area, and continuously optimizing and adjusting the laparoscopic focusing parameters based on the focus score results further includes: The surgical operation area image is converted into a grayscale image, and then the Sobel operator is used to calculate the gradient of each pixel in the surgical operation area. The average gradient magnitude of each pixel in the surgical area is used as the focus quality score.
8. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 4, characterized in that, The process of obtaining the deformation field from the query frame to other frames also includes: The optimization of the deformation field includes the following steps: Choose a loss function. The loss function is used to measure the deformation field calculation grid point p′ and the point tracking prediction grid point p. tracking The difference between them; where the point tracking result of grid point p in other frames is p tracking The grid point p is transformed into a point p′ in other frames through the deformation field transformation, and the deformation field is optimized by minimizing the loss function.
9. The automatic focusing method for laparoscopic surgery based on the surgical operation area as described in claim 6, characterized in that, The boundary quality of heatmaps enhanced by conditional random fields further includes: The normalized heatmap and the real surgical video are fed into a conditional random field. The color and spatial information of the real surgical video are used to improve the boundary of the heatmap and remove noisy areas.
10. An automatic focusing system for laparoscopic surgery based on the surgical operating area, characterized in that, The image acquisition module selects surgical instrument video segments based on real surgical videos, and determines the surgical operation area query frame and the tip of the surgical instrument based on the surgical instrument video segments. The deformation field acquisition module acquires the deformation field from the query frame to other frames; The operation area acquisition module obtains the position information of the tip of the surgical instrument in each frame of the surgical instrument video clip based on the deformation field, and obtains the distribution point data of all surgical operation areas. The operation area label acquisition module uses Gaussian filtering to fit the distribution of all surgical operation areas based on the acquired distribution point data of the surgical operation areas, and obtains the surgical operation area labels. The parameter adaptive adjustment module trains the operation area detection model based on the acquired surgical operation area labels, obtains the focus score of the operation area in real time, and continuously optimizes and adjusts the laparoscopic focusing parameters based on the score results.