Laparoscope autonomous adjustment method, system and equipment based on visual saliency prediction

Abstract

The invention discloses a method, a system and equipment for automatically adjusting a laparoscope based on visual saliency prediction, and the method comprises the steps: collecting eye movement data of a doctor in an operation process, and calculating a fixation point position of the eye movement data in a laparoscope screen; on the basis of the obtained fixation point position, making a heat map label of the region of interest of an operation, and training a depth model by training the heat map label of the region of interest to realize automatic identification of the heat map of the region of interest; detecting the surgical instrument by using the instance segmentation model to obtain the tip position of the surgical instrument; calculating the position and the size of a cutting area based on the identified region-of-interest heat map and the tip position of the surgical instrument; and further calculating and constraining the size of the cutting area to finish the adjustment of the operation picture. According to the method, an area concerned by a doctor can be recognized in real time, and the area is intercepted from the laparoscope image and is automatically amplified to replace an original laparoscope image.

Description

Technical Field

[0001] This application relates to the field of microscope control technology, specifically to a method, system, and device for autonomous adjustment of laparoscopy based on visual saliency prediction. Background Technology

[0002] With the continuous advancement of medical technology, laparoscopic surgery has become one of the important methods of modern surgery. Compared with traditional open surgery, laparoscopic surgery has advantages such as less trauma, faster recovery, and shorter hospital stay, and is therefore widely used in various surgical procedures. During laparoscopic surgery, doctors need to obtain images of the abdominal cavity through a laparoscope camera, which serves as the basis for surgical operations. However, due to the limited field of view of laparoscopic images, and the fact that the areas requiring attention during surgery may change at any time, the assistant holding the laparoscope often needs to constantly adjust the position and angle of the laparoscope to obtain the best view. This process is not only time-consuming but may also affect the smooth progress of the surgery. In addition, the critical areas in laparoscopic images are often small, requiring doctors to concentrate for a long time, which can easily lead to visual fatigue. At the same time, because details in the images are difficult to clearly display, it may affect the doctor's accurate judgment of the surgical area, thereby increasing surgical risks.

[0003] To address these issues, Groge developed a high-resolution tracking system for eye movements. This system controls the movement of the laparoscope by tracking the surgeon's eye annotations, enabling more accurate and efficient surgical procedures. However, this technology is not stable; changes in lighting conditions or the surgeon's physical condition can interfere with the laparoscopic movement. King designed a computer vision-based robotic endoscopy technology that uses color markers to achieve autonomous control of the laparoscope without human intervention. However, these additional markers may affect the surgeon's perception, and the method is also susceptible to environmental influences, potentially failing to meet the surgeon's needs in some complex surgeries. There is an urgent need to propose a new solution to overcome the shortcomings of these existing technologies. Summary of the Invention

[0004] In response to the above-mentioned circumstances, this application proposes a method, system, and device for autonomous adjustment of laparoscopy based on visual saliency prediction, in order to overcome or at least partially overcome the shortcomings of the prior art.

[0005] This application discloses a laparoscopic self-adjustment method based on visual saliency prediction, the method comprising:

[0006] Collect eye movement data during the surgeon's operation and calculate the fixation point position of the eye movement data on the laparoscopic screen;

[0007] Based on the acquired gaze point locations, surgical region of interest (ROI) heatmap labels are created. A depth model is trained using these labels to achieve automatic identification of ROI heatmaps.

[0008] Surgical instruments are detected using an instance segmentation model to obtain the position of the instrument tip.

[0009] Based on the identified region of interest heatmap and the position of the surgical instrument tip, the position and size of the cutting area are calculated;

[0010] Further calculations and constraints are applied to the dimensions of the cropping area to complete the adjustment of the surgical view.

[0011] In one embodiment, the step of collecting eye movement data during the surgeon's operation and calculating the fixation point position of the eye movement data on the laparoscopic screen further includes:

[0012] Obtain the coordinates of the four corners of the laparoscopic screen in the eye-tracking video image;

[0013] Calculate the perspective transformation matrix for the coordinates of the four corners of the laparoscopic screen in the eye-tracking video image;

[0014] The position of the gaze point on the laparoscopic screen is calculated based on the obtained perspective transformation matrix.

[0015] In one embodiment, the step of creating surgical region of interest heatmap labels based on the acquired gaze point location further includes:

[0016] Use eye movement data within one second at the current moment to capture the fixation point location on the current laparoscopic screen to create a heatmap label for the region of interest.

[0017] In one embodiment, the step of taking the gaze point position of the eye movement data within one second at the current moment in the current laparoscopic screen to create a heatmap label for the region of interest further includes: initializing a heatmap matrix with the same size as the target image, and setting all values ​​to 0;

[0018] Iterate through all eye-tracking data within one second of the current moment and assign values ​​to the corresponding positions of the eye-tracking data in the heatmap matrix;

[0019] Gaussian blur is used to blur the heatmap matrix to obtain heatmap labels for regions of interest.

[0020] In one embodiment, the step of using an instance segmentation model to detect surgical instruments and obtain the position of the instrument tip includes: calculating the position of the surgical instrument tip by projecting the outline of the surgical instrument onto the minimum bounding rectangle.

[0021] In one embodiment, calculating the position and size of the cutting area based on the identified region of interest heatmap and the position of the surgical instrument tip further includes:

[0022] Calculate the center location of the heatmap of the region of interest;

[0023] The center position of the region of interest heatmap is tracked, and the center position of the region of interest heatmap after Kalman filtering is obtained by using a preset cropping region position filtering process.

[0024] The cutting dimensions are calculated using the position of the tip of the surgical instrument with the fastest movement speed and the center point of the smoothed region of interest heatmap.

[0025] The clipping size is tracked using a Kalman filter, and the filter is updated according to a preset clipping size filtering process to obtain the clipping size smoothed by the Kalman filter.

[0026] In one embodiment, further calculating and constraining the size of the cropping area to complete the surgical screen adjustment further includes:

[0027] First, convert the cropping area position and cropping size into the coordinates of the top left and bottom right corners of the rectangle, and ensure that the coordinates of the rectangle do not exceed the boundaries of the image;

[0028] The size of the cropping area is then adjusted, and the ratio of the width and height of the cropping area to the width and height of the frame is calculated.

[0029] If the expanded cropping area exceeds the image range, keep the cropping frame size unchanged and adjust the position of the cropping frame by the minimum distance to ensure that the cropping area is within the surgical scene.

[0030] Calculate the intersection-union ratio (IUU) of the current clipping region and the clipping region at the previous moment. Update the clipping parameters only when the IUU is less than a set threshold to achieve a more stable surgical field of view.

[0031] The cropped area is enlarged and replaced with the original laparoscopic image, thus completing the surgical view adjustment.

[0032] This application also discloses a laparoscopic autonomous adjustment system based on visual saliency prediction, comprising:

[0033] The fixation point acquisition module collects eye movement data during the surgeon's operation and calculates the fixation point position of the eye movement data on the laparoscopic screen.

[0034] The region of interest acquisition module generates surgical region of interest heatmap labels based on the acquired gaze point positions. It then trains a depth model by training the surgical region of interest heatmap labels to achieve automatic identification of the region of interest heatmap.

[0035] The instrument tip position acquisition module uses an instance segmentation model to detect surgical instruments and acquire the position of the surgical instrument tip.

[0036] The trimming module calculates the position and size of the trimming area based on the identified region of interest heatmap and the position of the surgical instrument tip.

[0037] The adjustment module further calculates and constrains the size of the cropping area to complete the adjustment of the surgical screen.

[0038] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the laparoscopic autonomous adjustment method described above.

[0039] This application also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the laparoscopic autonomous adjustment method as described above.

[0040] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0041] This application uses a wearable eye tracker to collect the surgeon's gaze points in real surgical scenarios and create a surgical region of interest dataset. The eye tracker data directly reflects the surgeon's real gaze behavior during the surgery, avoiding subjective judgment and individual differences that occur during manual annotation.

[0042] This application takes into account the influence of the location of the surgical interest area and the location of the surgical instruments on the surgical operation. The position and size of the cutting area adapt to the surgical operation, which is more in line with the doctor's operating intuition.

[0043] This application uses a Kalman filter to process the position and size of the cropped area, reducing image shake and resulting in a more stable laparoscopic image, thus reducing the surgeon's workload.

[0044] This application can effectively reduce the workload of the surgeon operating the laparoscopic surgery by automatically cropping and adjusting the laparoscopic surgical image, thereby improving surgical efficiency and safety. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart illustrating the eye-tracking-based surgical microscope control method provided in this application;

[0047] Figure 2 This is a schematic diagram illustrating the three-dimensional coordinate principle of the surgical instrument markings provided in this application.

[0048] Figure 3This is a schematic diagram illustrating the generation of heatmap labels for the surgical region of interest provided in this application.

[0049] Figure 4 This is a schematic diagram of obtaining the tip of the surgical instrument provided in this application.

[0050] Figure 5 This is a flowchart of the cropping region location filtering process provided in this application.

[0051] Figure 6 This is a flowchart of the cropping size filtering process provided in this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] This application aims to develop a patented technology that enables automatic image adjustment during laparoscopic surgery. This technology can identify the area of ​​focus for the surgeon in real time, extract it from the laparoscopic image, and automatically enlarge it to replace the original laparoscopic screen. This technology can effectively improve the efficiency and safety of laparoscopic surgery, reduce visual fatigue for the surgeon, lighten the burden on the endoscope assistant, and provide patients with higher-quality medical services.

[0054] like Figure 1 As shown, the first embodiment of this application discloses a laparoscopic self-adjustment method based on visual saliency prediction, the method comprising:

[0055] The S100 collects eye movement data during the surgeon's operation and calculates the fixation point position of the eye movement data on the laparoscopic screen.

[0056] Specifically: Eye trackers are used to collect eye movement data during surgery, and the location of the doctor's gaze point on the laparoscopic screen is obtained, providing a basis for the subsequent creation of region of interest heatmap labels.

[0057] To obtain accurate eye-tracking data, this application uses a wearable eye tracker to acquire the doctor's gaze point information. The perspective transformation matrix is ​​calculated by detecting the four corners of the laparoscopic screen. Then, the doctor's gaze position on the laparoscopic image is calculated by combining the transformation matrix with the gaze position from the eye tracker's video feed. The specific process is as follows: Figure 2 As shown.

[0058] S100 further includes:

[0059] S110, obtain the coordinates of the four corners of the laparoscopic screen in the eye-tracking video image.

[0060] First, the coordinates of the four corners of the laparoscopic screen in the eye-tracking video are obtained through the screen key point detection model.

[0061] S120 calculates the perspective transformation matrix for the coordinates of the four corners of the laparoscopic screen in the eye-tracking video image.

[0062] The coordinates of the four corners of the laparoscopic screen in the eye-tracking video image are denoted as the upper left (x1, y1), upper right (x2, y2), lower right (x3, y3), and lower left (x4, y4), respectively. These coordinates are defined as the fixation points in the eye-tracking video image. The laparoscopic screen is pre-sized at 1920×1080, so its perspective-transformed coordinates are (0,0), (1919,0), (1919,1079), and (0,1079), respectively.

[0063] The perspective transformation matrix can be solved using the following system of equations:

[0064]

[0065] Where (x′,v′,w′) are the homogeneous coordinates after perspective transformation, (x,y) are the coordinates of the four corners of the laparoscopic screen in the eye tracker video image, and T is the perspective transformation matrix to be solved, which is obtained by solving the coordinates of the four corners before and after the transformation.

[0066] S130, calculate the fixation point position of the eye movement data on the laparoscopic screen based on the acquired perspective transformation matrix.

[0067] The eye-tracking data refers to the doctor's fixation point position recorded in the eye-tracking video feed. Once T is obtained, the position of the fixation point (x, y) on the laparoscopic screen can be calculated. g ,y g )for:

[0068]

[0069]

[0070] Where (x) g ,y g (x) represents the fixation point position in the eye tracker video image. g ',y g ') represents the homogeneous coordinates during the intermediate transformation process, obtained by adjusting the scaling factor w. g Dividing the coordinates yields the gaze point position on the laparoscopic screen, which is the coordinate position of the four corners of the eye tracker video image on the laparoscopic screen.

[0071] S200, based on the acquired gaze point location, creates surgical region of interest (ROI) heatmap labels. A depth model is trained using these labels to achieve automatic identification of ROI heatmaps. For example... Figure 3 The diagram shows the process of generating heatmap labels for surgical regions of interest.

[0072] S200 further includes:

[0073] S210 takes the eye movement data within one second at the current moment and the fixation point position on the current laparoscopic screen to create a heat map label of the region of interest.

[0074] S210 further includes:

[0075] Initialize a heatmap matrix with the same size as the target image, and set all values ​​to 0;

[0076] Iterate through all eye-tracking data within one second of the current moment and assign values ​​to the corresponding positions of the eye-tracking data in the heatmap matrix;

[0077] Gaussian blur is used to blur the heatmap matrix, and the heatmap labels for the regions of interest are generated. Specifically, the variance of the Gaussian kernel here is the pixel distance corresponding to a 1.5-degree field of view in the eye-tracking data acquisition scene.

[0078] The S300 uses an instance segmentation model to detect surgical instruments and obtain the position of the instrument tip.

[0079] This application achieves automatic identification of surgical instrument tips by training a surgical instrument instance segmentation model. For example... Figure 4 As shown, the instance segmentation results of surgical instruments only contain contour information. This application uses the method of projecting the contour of the surgical instrument onto the smallest bounding rectangle to calculate the position of the tip of the surgical instrument.

[0080] Calculating the position of the surgical instrument tip by projecting the surgical instrument profile onto its smallest bounding rectangle further includes:

[0081] First, obtain the position P of the center point of the circumscribed rectangle of the surgical instrument outline. box The width and height W of the rectangle box H box And the rotation angle θ.

[0082] The direction of the major axis of the rectangle is determined by the numerical values ​​of its width and height, and the direction vector of the major axis is calculated by rotating the rectangle by its rotation angle.

[0083]

[0084] Obtain the offset of the surgical instrument contour (Contours) relative to the center point of the bounding rectangle.

[0085] offset = Contours - P box (1.5)

[0086] Calculate the projection length of each point on the contour onto the major axis, i.e., the inner product of the contour offset and the direction vector: innerProb.

[0087]

[0088] The point with the largest or smallest inner product value is the tip of the surgical instrument. These two points are first identified and marked as P. max and P min :

[0089] P max =Contours[argmax(innProb)]

[0090] P min =Contours[argmin(innProb)] (0.5)

[0091] Obtain the distance matrix D from non-edge pixels to the nearest edge pixel using the binary edge map of the surgical scene image. The values ​​in this matrix represent the Euclidean distance from the pixel at that location to the nearest edge pixel. Then query P. max and P min The minimum distance from two points to the edge pixel is used to determine the tip point P of the instrument, with the point having the larger distance being the tip point P. tip The calculation formula is:

[0092]

[0093] S400 calculates the position and size of the cutting area based on the identified region of interest heatmap and the position of the surgical instrument tip.

[0094] The S400 further includes:

[0095] S410 calculates the center location of the heatmap of the region of interest.

[0096] Specifically, this includes: calculating the center location C of the heatmap of the region of interest. roi The calculation formula is as follows:

[0097]

[0098]

[0099] Where S is the sum of the heatmap values ​​of the region of interest, W and H are the width and height of the current image, respectively, and heatmap is the prediction result of the surgical region of interest.

[0100] S420 tracks the center position of the heatmap of the region of interest and updates it when the center position of the heatmap of the region of interest is detected, thereby obtaining the center position of the heatmap of the region of interest after Kalman filtering smoothing.

[0101] Specifically, it includes:

[0102] like Figure 5 As shown, in order to improve the stability of the cropping region position, this invention tracks the center point of the region of interest using Kalman filtering and adopts a preset cropping region position filtering process to finally obtain the center position of the heatmap of the region of interest after Kalman filtering smoothing.

[0103] This invention tracks the center position of a region of interest (ROI) heatmap using Kalman filtering. It updates the heatmap when the center position is detected, predicts when the center position is briefly lost, and updates the heatmap using the original center position when the center position is missing for an extended period. Finally, it obtains the center position C of the Kalman-filtered ROI heatmap. smooth .

[0104] The cropping region location filtering process specifically includes:

[0105] How to determine if the region of interest has been lost?

[0106] If yes, further determine if the number of lost regions of interest exceeds a threshold. If not, calculate the cropped region size and update the filter state.

[0107] If the number of lost regions of interest exceeds the threshold, the Kalman filter is updated using the center position of the heatmap of the region of interest.

[0108] If the number of times the region of interest is lost is less than or equal to the threshold, a Kalman filter is used for prediction once.

[0109] The S430 calculates the cutting dimensions using the position of the surgical instrument tip at the fastest movement speed and the center position of the smoothed region of interest heatmap.

[0110] Specifically, it includes:

[0111] Then use the tip position p of the surgical instrument with the fastest movement speed tip The center position C of the smoothed region of interest heatmap smooth Calculate the cutting dimensions (w) c ,h c The calculation formula is:

[0112] (w c ,h c )=2·(|C smooth -p tip|+scale·(W,H)) (0.9)

[0113] In the formula, scale is the set minimum cropping size ratio.

[0114] The S440 uses a Kalman filter to track the clipping size, updates the filter according to a preset clipping size filtering process, and obtains the clipping size after Kalman filter smoothing.

[0115] To improve the stability of the cutting dimensions, a Kalman filter is still used here to track the cutting dimensions, such as... Figure 6 As shown, the filter is updated when surgical instruments are detected, prediction is performed when instruments are briefly lost, and the image center position is used for updating when instruments are missing for a long time. Finally, the cropping size (w) after Kalman filtering smoothing is obtained. smooth ,h smooth ).

[0116] like Figure 6 As shown, the preset cropping size filtering process specifically includes:

[0117] How to determine if surgical instruments are missing?

[0118] If yes, further determine if the number of surgical instrument loss counts exceeds a threshold. If not, calculate the trimming area size and update the filter state.

[0119] How do you determine if the number of lost surgical instruments exceeds a threshold?

[0120] If so, update the Kalman filter using the center position of the region of interest heatmap.

[0121] If not, use a Kalman filter to predict once.

[0122] S500 performs further calculations and constraints on the dimensions of the trimming area to complete the surgical view adjustment. After calculating the position and size of the trimming area, further calculations and constraints need to be applied to the parameters of the trimming area. Specifically, this includes:

[0123] S510, first convert the cropping area position and cropping size into the top left point p of the rectangle. l coordinates (x) l ,y l ) and the bottom right corner p r coordinates (x) r ,y r And ensure that the coordinates of the rectangle do not exceed the image boundary. The calculation formula is:

[0124] p l =(x l ,y l )=(max(x smooth-w c / 2,0),max(y smooth -h c / 2,0)) (0.10)

[0125] p r =(x r ,y r )=(min(x smooth +w c / 2,W),min(y smooth +h c / 2,H)) (0.11)

[0126] In the formula x smooth and y smooth C respectively smooth The horizontal and vertical coordinates.

[0127] S520, adjust the size of the cropping area again, and calculate the ratio of the width and height of the cropping area to the width and height of the frame.

[0128] To avoid insufficient clarity in the adjusted surgical image due to excessive magnification, the size of the cropped area needs to be adjusted again, calculating the ratio of the cropped area's width and height to the frame's width and height:

[0129]

[0130]

[0131] If the width ratio is less than the height ratio or a minimum ratio is set, then expand the cropping area width:

[0132]

[0133] If the height ratio is less than the width ratio or a minimum ratio is set, then expand the cropping area height:

[0134]

[0135]

[0136] S530: If the expanded cropping area exceeds the image range, the cropping frame size remains unchanged, and the position of the cropping frame is adjusted by the minimum distance to ensure that the cropping area is within the surgical scene.

[0137] S540: Calculate the intersection-union ratio (IUU) of the current clipping region and the clipping region at the previous moment. Update the clipping parameters when the IUU is less than a set threshold to achieve a more stable surgical field of view.

[0138] The S550 enlarges the cropped area and replaces the original laparoscopic image, thus completing the surgical view adjustment.

[0139] Specifically, the laparoscopic autonomous adjustment system based on visual saliency prediction described above and the laparoscopic autonomous adjustment system based on visual saliency prediction described below can be referred to in correspondence.

[0140] The second embodiment of this application also provides a laparoscopic autonomous adjustment system based on visual saliency prediction, including:

[0141] The fixation point acquisition module collects eye movement data during the surgeon's operation and calculates the fixation point position of the eye movement data on the laparoscopic screen.

[0142] The region of interest acquisition module generates surgical region of interest heatmap labels based on the acquired gaze point positions. It then trains a depth model by training the surgical region of interest heatmap labels to achieve automatic identification of the region of interest heatmap.

[0143] The instrument tip position acquisition module uses an instance segmentation model to detect surgical instruments and acquire the position of the surgical instrument tip.

[0144] The trimming module calculates the position and size of the trimming area based on the identified region of interest heatmap and the position of the surgical instrument tip.

[0145] The adjustment module further calculates and constrains the size of the cropping area to complete the adjustment of the surgical screen.

[0146] The laparoscopic autonomous adjustment system based on visual saliency prediction provided in this application embodiment can realize all the method steps implemented by the method embodiment with the execution subject being a computing terminal, and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0147] This application uses a wearable eye tracker to collect the surgeon's gaze points in real surgical scenarios and create a surgical region of interest dataset. The eye tracker data directly reflects the surgeon's real gaze behavior during the surgery, avoiding subjective judgment and individual differences that occur during manual annotation.

[0148] This application takes into account the influence of the location of the surgical interest area and the location of the surgical instruments on the surgical operation. The position and size of the cutting area adapt to the surgical operation, which is more in line with the doctor's operating intuition.

[0149] This application uses a Kalman filter to process the position and size of the cropped area, reducing image shake and resulting in a more stable laparoscopic image, thus reducing the surgeon's workload.

[0150] This application can effectively reduce the workload of the surgeon operating the laparoscopic surgery by automatically cropping and adjusting the laparoscopic surgical image, thereby improving surgical efficiency and safety.

[0151] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the laparoscopic autonomous adjustment method described above.

[0152] This application also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the laparoscopic autonomous adjustment method as described above.

[0153] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A laparoscopic autonomous adjustment method based on visual saliency prediction, characterized in that, The method comprises: Collecting eye movement data of a doctor during a surgery, and calculating a gaze point position of the eye movement data in the laparoscope screen; Based on the obtained gaze point position, a surgery interest region heat map label is made, a deep model is trained using a trained surgery interest region heat map label, and automatic recognition of the interest region heat map is realized; An instance segmentation model is used to detect a surgical instrument, and a surgical instrument tip position is obtained; Based on the recognized interest region heat map and the surgical instrument tip position, the position and size of a cropped region are calculated; The size of the cropped region is further calculated and constrained, and surgery picture adjustment is completed.

2. The laparoscopic self-regulating method of claim 1, wherein, The collecting of the eye movement data of the doctor during the surgery and the calculation of the gaze point position of the eye movement data in the laparoscope screen further comprises: Obtaining the coordinates of the four corners of the laparoscope screen in the eye tracker video picture; Calculating a perspective transformation matrix for the coordinates of the four corners of the laparoscope screen in the eye tracker video picture; According to the obtained perspective transformation matrix, the gaze point position of the eye movement data in the laparoscope screen is calculated.

3. The laparoscopic self-regulating method of claim 1, wherein, The making of the surgery interest region heat map label based on the obtained gaze point position further comprises: Taking the gaze point position of the eye movement data in the current laparoscope screen within one second at the current time to make the interest region heat map label.

4. The laparoscopic self-regulating method of claim 3, wherein, The taking of the gaze point position of the eye movement data in the current laparoscope screen within one second at the current time to make the interest region heat map label further comprises: initializing a heat map matrix with the same size as the target image, and setting all values to 0; Iterating through all the eye movement data within one second at the current time, and assigning the eye movement data to the corresponding position of the heat map matrix; Using a Gaussian blur operation to blur the heat map matrix, and obtaining the interest region heat map label.

5. The laparoscopic self-regulating method of claim 1, wherein, The use of the instance segmentation model to detect the surgical instrument and obtain the instrument tip position comprises: calculating the surgical instrument tip position by projecting the surgical instrument contour on the minimum circumscribed rectangle frame.

6. The laparoscopic self-regulating method of claim 1, wherein, The calculation of the position and size of the cropped region based on the recognized interest region heat map and the surgical instrument tip position further comprises: Calculating the center position of the interest region heat map; Tracking the center position of the interest region heat map, and using a preset cropped region position filtering process to obtain the center position of the interest region heat map smoothed by Kalman filtering; Using the surgical instrument tip position with the fastest motion speed and the smoothed center position of the interest region heat map to calculate the cropped size; Using a Kalman filter to track the cropped size, updating the filter according to a preset cropped size filtering process, and obtaining the smoothed cropped size by the Kalman filter.

7. The laparoscopic self-regulating method of claim 1, wherein, The further calculation and constraint of the size of the cropped region to complete the surgery picture adjustment further comprises: First, convert the cropped region position and the cropped size into the coordinates of the top-left point and the coordinates of the bottom-right corner of the rectangle frame, and ensure that the coordinates of the rectangle frame do not exceed the boundaries of the image; Adjust the size of the cropped region again, and calculate the proportions of the width and height of the cropped region to the width and height of the frame; If the expanded cropped region exceeds the image range, keep the size of the cropped frame unchanged, and adjust the position of the cropped frame by the minimum distance to ensure that the cropped region is in the surgery scene picture. The intersection and union ratio of the current clipping region and the clipping region at the last time is calculated, and when the intersection and union ratio is less than a set threshold, the clipping parameters are updated again to realize a more stable surgical field of view. The picture of the clipping region is enlarged to replace the original picture of the laparoscope, that is, the adjustment of the surgical picture is completed.

8. A laparoscopic autonomous adjustment system based on visual saliency prediction, characterized in that, The method comprises the following steps: The gaze point acquisition module collects the eye movement data of the doctor during the surgery and calculates the gaze point position of the eye movement data in the laparoscope screen; The interest region acquisition module, based on the acquired gaze point position, makes a surgical interest region heat map label, uses a trained surgical interest region heat map label to train a deep model, and realizes automatic recognition of the interest region heat map; The instrument tip position acquisition module uses an instance segmentation model to detect the surgical instrument and acquires the position of the tip of the surgical instrument; The clipping module calculates the position and size of the clipping region based on the recognized interest region heat map and the position of the tip of the surgical instrument; The adjustment module further calculates and constrains the size of the clipping region to complete the adjustment of the surgical picture.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the laparoscope autonomous adjustment method of any one of claims 1-7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the laparoscope autonomous adjustment method of any one of claims 1-7.

Images