A method to improve the accuracy of in-vivo device positioning using doctor's eye movement data

Abstract

This invention discloses a method for improving the accuracy of internal device positioning using a doctor's eye movement data. The method comprises the following steps: Step 1: Acquisition and preprocessing of eye movement data; Step 2: Generation of a mask matrix and gaze point heat map; Step 3: Input image preprocessing; Step 4: Feature extraction using a convolutional neural network; Step 5: Localization of internal devices using a Transformer; and Step 6: Optimization of positioning results based on a teacher-student model and eye movement information. By capturing the doctor's eye movement information and combining it with image processing techniques, this system achieves precise positioning of internal devices, improving the accuracy and efficiency of surgical procedures and reducing surgical risks.

Description

Technical Field

[0001] The present invention relates to the field of surgical navigation, and in particular to a method for improving the positioning accuracy of in-vivo instruments by using doctor's eye movement data. Background Art

[0002] In traditional surgeries, surgeons typically rely on the naked eye and simple microscopes to locate and manipulate instruments. However, the surgeon's clinical experience and the complexity of the surgical scenario can affect the accuracy of positioning. Inaccurate positioning can lead to surgical failure or complications. Existing in-vivo instrument localization technology analyzes real-time medical images to provide accurate position and angle information for in-vivo instruments, thereby reducing surgical risks.

[0003] Current in vivo device positioning technologies typically rely on sensors and image processing algorithms to achieve positioning. However, due to the complexity of human tissue and the variability of the surgical environment, sensors can be affected by factors such as local interference or rapid instrument movement, resulting in insufficient positioning accuracy. Tracking the surgeon's eye movements and integrating them into the positioning system provides an additional reference for positioning, thereby improving positioning accuracy and precision. This is particularly true in complex surgical scenarios involving intersecting nerves and blood vessels, significantly reducing the risk of iatrogenic injuries. Summary of the Invention

[0004] In response to the shortcomings of the existing technology, the present invention provides a method for improving the accuracy of internal device positioning through the doctor's eye movement data, which can improve the accuracy of existing internal device positioning technology and thus reduce surgical risks.

[0005] In order to achieve the purpose of the present invention, the technical solution adopted is:

[0006] A method for improving the accuracy of positioning an in-vivo device using a doctor's eye movement data comprises the following steps:

[0007] Step 1: Collection and preprocessing of eye movement data;

[0008] Step 2: Generate mask matrix and gaze point heat map;

[0009] Step 3: Preprocess the input image;

[0010] Step 4: Feature extraction based on convolutional neural network;

[0011] Step 5: Use Transformer to locate the device inside the body;

[0012] Step 6: Optimize the positioning effect based on the teacher-student model and eye movement information.

[0013] Preferably, in step 1, the collection and preprocessing of eye movement data includes the following steps:

[0014] Step 1.1: Use an eye tracker to collect eye movement data of the doctor while observing the surgical video;

[0015] Step 1.2: Differentiate the data points into fixation data and saccade data based on the displacement length of each collected data point, and treat the data points within the threshold range of the preset step length as fixation data points;

[0016] Step 1.3: Save the information of the gaze data point and record it as (x, y, t), where (x, y) is the position information of the gaze data point and t is the timestamp information of the image frame where the gaze data point is located.

[0017] Preferably, in step 2, generating the mask matrix and the gaze point heat map includes the following steps:

[0018] Step 2.1. Create an eye movement data matrix: Based on the information of the recorded gaze data points, create an eye movement data matrix E of size X*Y*T, where X*Y represents the resolution corresponding to the image frame, and T is the size of the image frame;

[0019] Step 2.2, generate mask matrix: First, set the value of the eye movement data matrix according to the information of the recorded gaze data point. The matrix value E(x, y, t) of the location of the gaze data point is set to 1, and the matrix values ​​of the other locations are set to 0. Then, calculate the visual attention radius Where D represents the distance from the doctor's eyes to the screen, and DPI is the resolution of the display. Finally, traverse all the locations in the eye movement matrix where the matrix value is 1, and set the matrix values ​​of all locations within the visual attention radius r of these locations to 1 to obtain the mask matrix E;

[0020] Step 2.3, generate gaze point heat map: First, create a matrix D with the same size as the image frame, and set the value D(x, y, t) of the gaze point position in the matrix to 255, and the rest of the positions to 0, so as to obtain the gaze data matrix D. Then use the Gaussian convolution kernel G with the size of the visual attention radius r, standard deviation σ=0.5, and mean μ=1 kernel Perform convolution operation with D. Finally, we get the gaze heat map In the formula is the symbol for the convolution operation.

[0021] Preferably, in step 3, the input image preprocessing includes the following steps:

[0022] Step 3.1: Perform random rotation and translation on ultrasound images containing internal devices to enhance data and improve the generalization and robustness of the model.

[0023] Step 3.2: Perform a dot product operation between the generated mask matrix and the ultrasound image containing the internal device to obtain new training set data and expand the data set;

[0024] Step 3.3: Divide the dataset into training set, validation set, and test set in a ratio of 7:2:1. The training set is used to build the model, the validation set is used to evaluate the performance of the model during training, and the test set is used to evaluate the performance of the trained model.

[0025] Preferably, in step 4, feature extraction based on convolutional neural network includes the following steps:

[0026] Step 4.1. Data preprocessing: Convert the input ultrasound image to the size required by the convolutional neural network.

[0027] Step 4.2, Load weights: Load pre-trained weights to accelerate model convergence and improve model performance;

[0028] Step 4.3, Convolutional layer feature extraction: The input image is convolved with the convolution kernel. Here, the input image passes through four convolution layers with different kernel sizes in sequence to extract features at different scales.

[0029] Step 4.4, Pooling: We use max pooling to downsample the feature map. First, specify the size of the pooling window. Then, slide the window across the feature map output by the convolutional layer and pool the values ​​within each window. This means taking the maximum value within the window as the pooled value. Repeat this process until all windows in the feature map have been max pooled, resulting in a new feature map.

[0030] Preferably, in step 5, using the Transformer to locate the internal device includes the following steps:

[0031] Step 5.1, Position Encoding: Calculate the encoding values ​​of all positions in the output feature map of the convolutional neural network, and then add the position encoding to the input features to embed the position information in the feature map to obtain the embedded feature map;

[0032] Step 5.2, Encoder-Decoder Operation: The embedded feature map is fed into a layered Transformer encoder. Each Transformer layer includes a multi-head self-attention mechanism and a feedforward neural network to generate a new feature matrix that effectively captures the semantic information and spatial structure in the feature map. The resulting feature matrix is ​​then passed through a series of decoder layers to convert the global semantic information into local information for a specific task.

[0033] Step 5.3, post-processing to obtain model output: The target detection frame output by the encoder may include multiple highly overlapping rotated target detection frames. Use the non-maximum suppression algorithm to sort the confidence scores from high to low, and then calculate the degree of overlap between the target detection frame output by the model and the true value frame. If the overlap value is higher than the preset threshold Thr=0.6, the result of the device positioning is output.

[0034] Preferably, in step 6, optimizing the positioning effect based on the teacher-student model and eye movement information includes the following steps:

[0035] Step 6.1, teacher model training: Use the ground truth data and gaze heatmap to train the teacher model to quantify high-quality uncertainty;

[0036] Step 6.2, Student Model Training: Use the ground truth data and gaze heatmap to train the student model to extract the prediction uncertainty of the teacher model;

[0037] Step 6.3. Construct the teacher-student model loss function: Use Kullback-Leibler divergence loss to transfer the uncertainty knowledge of the teacher model to the student model:

[0038]

[0039] Where, is the predicted value based on the perturbed input image, Refers to the average probability map of the softmax layer of the teacher model, p c (α) is the average probability map of the softmax layer of the student model, D represents each pixel in the image, and the calculated L KL The smaller the loss value, the better the matching effect between the student model and the teacher model;

[0040] Step 6.4: Calculation of uncertainty of the teacher model. The uncertainty of the teacher model can be calculated using the following formula:

[0041]

[0042] Where, is the student model's estimate of the prediction uncertainty, is the mean of any sample prediction, is the mean of all sample predictions;

[0043] Step 6.5: Construct the uncertainty loss function. Since we cannot match the distribution of two unbound uncertainties when regressing the position and angle information of the internal device, we construct the root mean square logarithmic loss:

[0044]

[0045] Where, is the prediction uncertainty of the teacher model, is the prediction uncertainty of the student model, N represents the total number of model predictions, and this loss function is introduced to pay special attention to pixels with higher uncertainty;

[0046] Step 6.6, Positioning Optimization: Calculate the loss function and uncertainty loss function of the joint teacher-student model based on eye movement information, and feed the loss value back to the Transformer-based model for in-vivo device positioning to further optimize the model parameters to achieve higher positioning accuracy.

[0047] Compared with the existing technology, the present invention has the following beneficial effects: by introducing the doctor's eye movement information, the model can more accurately understand the doctor's intentions and focus, and thus provide more accurate position and angle information of the internal instruments, thereby improving the precision and accuracy of the operation and providing reliable technical support for the successful implementation of clinical surgery. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] Example 1

[0051] like Figure 1 As shown, a method for improving the accuracy of positioning an in-vivo device using a doctor's eye movement data includes the following steps:

[0052] Step 1: Collection and preprocessing of eye movement data;

[0053] Step 2: Generate mask matrix and gaze point heat map;

[0054] Step 3: Preprocess the input image;

[0055] Step 4: Feature extraction based on convolutional neural network;

[0056] Step 5: Use Transformer to locate the device inside the body;

[0057] Step 6: Optimize the positioning effect based on the teacher-student model and eye movement information.

[0058] Preferably, in step 1, the collection and preprocessing of eye movement data includes the following steps:

[0059] Step 1.1: Use an eye tracker to collect eye movement data of the doctor while observing the surgical video;

[0060] Step 1.2: Differentiate the data points into fixation data and saccade data based on the displacement length of each collected data point, and treat the data points within the threshold range of the preset step length as fixation data points;

[0061] Step 1.3: Save the information of the gaze data point and record it as (x, y, t), where (x, y) is the position information of the gaze data point and t is the timestamp information of the image frame where the gaze data point is located.

[0062] Preferably, in step 2, generating the mask matrix and the gaze point heat map includes the following steps:

[0063] Step 2.1. Create an eye movement data matrix: Based on the information of the recorded gaze data points, create an eye movement data matrix E of size X*Y*T, where X*Y represents the resolution corresponding to the image frame, and T is the size of the image frame;

[0064] Step 2.2, generate mask matrix: First, set the value of the eye movement data matrix according to the information of the recorded gaze data point. The matrix value E(x, y, t) of the location of the gaze data point is set to 1, and the matrix values ​​of the other locations are set to 0. Then, calculate the visual attention radius Where D represents the distance from the doctor's eyes to the screen, and DPI is the resolution of the display. Finally, traverse all the locations in the eye movement matrix where the matrix value is 1, and set the matrix values ​​of all locations within the visual attention radius r of these locations to 1 to obtain the mask matrix E;

[0065] Step 2.3, generate gaze point heat map: First, create a matrix D with the same size as the image frame, and set the value D(x, y, t) of the gaze point position in the matrix to 255, and the rest of the positions to 0, so as to obtain the gaze data matrix D. Then use the Gaussian convolution kernel G with the size of the visual attention radius r, standard deviation σ=0.5, and mean μ=1 kernel Perform convolution operation with D. Finally, we get the gaze heat map In the formula is the symbol for the convolution operation.

[0066] Preferably, in step 3, the input image preprocessing includes the following steps:

[0067] Step 3.1: Perform random rotation and translation on ultrasound images containing internal devices to enhance data and improve the generalization and robustness of the model.

[0068] Step 3.2: Perform a dot product operation between the generated mask matrix and the ultrasound image containing the internal device to obtain new training set data and expand the data set;

[0069] Step 3.3: Divide the dataset into training set, validation set, and test set in a ratio of 7:2:1. The training set is used to build the model, the validation set is used to evaluate the performance of the model during training, and the test set is used to evaluate the performance of the trained model.

[0070] Preferably, in step 4, feature extraction based on convolutional neural network includes the following steps:

[0071] Step 4.1. Data preprocessing: Convert the input ultrasound image to the size required by the convolutional neural network.

[0072] Step 4.2, Load weights: Load pre-trained weights to accelerate model convergence and improve model performance;

[0073] Step 4.3, Convolutional layer feature extraction: The input image is convolved with the convolution kernel. Here, the input image passes through four convolution layers with different kernel sizes in sequence to extract features at different scales.

[0074] Step 4.4, Pooling: We use max pooling to downsample the feature map. First, specify the size of the pooling window. Then, slide the window across the feature map output by the convolutional layer and pool the values ​​within each window. This means taking the maximum value within the window as the pooled value. Repeat this process until all windows in the feature map have been max pooled, resulting in a new feature map.

[0075] Preferably, in step 5, using the Transformer to locate the internal device includes the following steps:

[0076] Step 5.1, Position Encoding: Calculate the encoding values ​​of all positions in the output feature map of the convolutional neural network, and then add the position encoding to the input features to embed the position information in the feature map to obtain the embedded feature map;

[0077] Step 5.2, Encoder-Decoder Operation: The embedded feature map is fed into a layered Transformer encoder. Each Transformer layer includes a multi-head self-attention mechanism and a feedforward neural network to generate a new feature matrix that effectively captures the semantic information and spatial structure in the feature map. The resulting feature matrix is ​​then passed through a series of decoder layers to convert the global semantic information into local information for a specific task.

[0078] Step 5.3, post-processing to obtain model output: The target detection frame output by the encoder may include multiple highly overlapping rotated target detection frames. Use the non-maximum suppression algorithm to sort the confidence scores from high to low, and then calculate the degree of overlap between the target detection frame output by the model and the true value frame. If the overlap value is higher than the preset threshold Thr=0.6, the result of the device positioning is output.

[0079] Preferably, in step 6, optimizing the positioning effect based on the teacher-student model and eye movement information includes the following steps:

[0080] Step 6.1, teacher model training: Use the ground truth data and gaze heatmap to train the teacher model to quantify high-quality uncertainty;

[0081] Step 6.2, Student Model Training: Use the ground truth data and gaze heatmap to train the student model to extract the prediction uncertainty of the teacher model;

[0082] Step 6.3. Construct the teacher-student model loss function: Use Kullback-Leibler divergence loss to transfer the uncertainty knowledge of the teacher model to the student model:

[0083]

[0084] Where, is the predicted value based on the perturbed input image, Refers to the average probability map of the softmax layer of the teacher model, p c (α) is the average probability map of the softmax layer of the student model, D represents each pixel in the image, and the calculated L KL The smaller the loss value, the better the matching effect between the student model and the teacher model;

[0085] Step 6.4: Calculation of uncertainty of the teacher model. The uncertainty of the teacher model can be calculated using the following formula:

[0086]

[0087] Where, is the student model's estimate of the prediction uncertainty, is the mean of any sample prediction, is the mean of all sample predictions;

[0088] Step 6.5: Construct the uncertainty loss function. Since we cannot match the distribution of two unbound uncertainties when regressing the position and angle information of the internal device, we construct the root mean square logarithmic loss:

[0089]

[0090] Where, is the prediction uncertainty of the teacher model, is the prediction uncertainty of the student model, N represents the total number of model predictions, and this loss function is introduced to pay special attention to pixels with higher uncertainty;

[0091] Step 6.6, Positioning Optimization: Calculate the loss function and uncertainty loss function of the joint teacher-student model based on eye movement information, and feed the loss value back to the Transformer-based model for in-vivo device positioning to further optimize the model parameters to achieve higher positioning accuracy.

Claims

1. A method for improving the accuracy of positioning of in-vivo devices using doctor's eye movement data, characterized in that: The steps include: Step 1: Collection and preprocessing of eye movement data; Step 2: Generate mask matrix and gaze point heat map; Step 3: Preprocess the input image; Step 4: Feature extraction based on convolutional neural network; Step 5: Use Transformer to locate the device inside the body; Step 6: Optimize the positioning effect based on the teacher-student model and eye movement information; The step 6 specifically includes the following steps: Step 6.1, teacher model training: Use the ground truth data and gaze heatmap to train the teacher model to quantify high-quality uncertainty; Step 6.2, Student Model Training: Use the ground truth data and gaze heatmap to train the student model to extract the prediction uncertainty of the teacher model; Step 6.

3. Construct the teacher-student model loss function and use the Kullback-Leibler divergence loss to transfer the uncertainty knowledge of the teacher model to the student model: ， Where, is the predicted value based on the perturbed input image, Refers to the average probability map of the softmax layer of the teacher model, is the average probability map of the softmax layer of the student model, Represents each pixel in the image; Calculated The smaller the loss value, the better the matching effect between the student model and the teacher model; Step 6.4: Calculation of uncertainty of the teacher model. The uncertainty of the teacher model can be calculated using the following formula: ， Where, is the student model's estimate of the prediction uncertainty, is the mean of any sample prediction, is the mean of all sample predictions; Step 6.5: Construct the uncertainty loss function. Since we cannot match the distribution of two unbound uncertainties when regressing the position and angle information of the internal device, we construct the root mean square logarithmic loss: ， Where, is the prediction uncertainty of the teacher model, is the prediction uncertainty of the student model, Represents the total number of model predictions. This loss function is introduced to pay special attention to pixels with higher uncertainty. Step 6.6, Positioning Optimization: Calculate the loss function and uncertainty loss function of the joint teacher-student model based on eye movement information, and feed the loss value back to the Transformer-based model for in-vivo device positioning to further optimize the model parameters to achieve higher positioning accuracy.

2. The method for improving the positioning accuracy of in-vivo devices using doctor's eye movement data according to claim 1, characterized in that: In step 1, the collection and preprocessing of eye movement data specifically includes the following steps: Step 1.1: Use an eye tracker to collect eye movement data of the doctor while observing the surgical video; Step 1.2: Differentiate the data points into fixation data and saccade data based on the displacement length of each collected data point, and treat the data points within the threshold range of the preset step length as fixation data points; Step 1.3: Save the information of the gaze data point and record it as ,in is the location information of the gaze data point, It is the timestamp information of the image frame where the gaze data point is located.

3. The method for improving the positioning accuracy of in-vivo devices using doctor's eye movement data according to claim 1, characterized in that: In step 2, generating the mask matrix and the gaze point heat map specifically includes the following steps: Step 2.

1. Create an eye movement data matrix: Create an eye movement data matrix of size Eye movement data matrix ,here Indicates the resolution corresponding to the image frame, is the size of the image frame; Step 2.2, generate mask matrix: first, set the value of eye movement data matrix according to the information of recorded gaze data point, the matrix value of the location of gaze data point , the matrix values ​​of the remaining positions are set to 0, and then the visual attention radius is calculated , where D represents the distance from the doctor's eyes to the screen, and DPI is the resolution of the display. Finally, all the positions where the matrix value is 1 in the eye movement matrix are traversed and the visual attention radius of these positions is calculated. The matrix values ​​of all positions within the range are set to 1 to obtain the mask matrix ; Step 2.3, generate gaze heat map: first create a matrix with the same size as the image frame , and the value of the gaze point in the matrix Set it to 255 and the rest of the positions to 0 to obtain the gaze data matrix , and then use the visual attention radius Size, standard deviation , mean Gaussian convolution kernel and Perform convolution operation, and finally, get the gaze heat map , where is the symbol for the convolution operation.

4. The method for improving the positioning accuracy of in-vivo devices using doctor's eye movement data according to claim 1, characterized in that: In step 3, the input image preprocessing specifically includes the following steps: Step 3.1: Perform random rotation and translation operations on ultrasound images containing internal devices to perform data augmentation to improve the generalization ability and robustness of the model; Step 3.2: Perform a dot product operation between the generated mask matrix and the ultrasound image containing the internal device to obtain new training set data and expand the data set; Step 3.3: Divide the dataset into training set, validation set, and test set in a ratio of 7:2:

1. The training set is used to build the model, the validation set is used to evaluate the performance of the model during training, and the test set is used to evaluate the performance of the trained model.

5. The method for improving the positioning accuracy of in-vivo devices using doctor's eye movement data according to claim 1, characterized in that: In step 4, feature extraction based on convolutional neural network specifically includes the following steps: Step 4.

1. Data preprocessing: Convert the input ultrasound image to the size required by the convolutional neural network. Step 4.2, Load weights: Load pre-trained weights to accelerate model convergence and improve model performance; Step 4.3, Convolutional layer feature extraction: The input image is convolved with the convolution kernel. Here, the input image passes through four convolution layers with different kernel sizes in sequence to extract features at different scales. Step 4.4, pooling operation: We use the maximum pooling operation to downsample the feature map. First, specify the size of the pooling window, then slide the window on the feature map output by the convolutional layer, and perform a pooling operation on the values ​​in each window, that is, take the maximum value in the window as the pooled value. Repeat the above operation until all windows in the feature map are subjected to the maximum pooling operation, and a new feature map can be obtained.

6. The method for improving the positioning accuracy of in-vivo devices using doctor's eye movement data according to claim 1, characterized in that: In step 5, using Transformer to locate the internal device specifically includes the following steps: Step 5.1, Position Encoding: Calculate the encoding values ​​of all positions in the output feature map of the convolutional neural network, and then add the position encoding to the input features to embed the position information in the feature map to obtain the embedded feature map; Step 5.2, Encoder-Decoder Operation: The embedded feature map is fed into a hierarchical Transformer encoder. Each Transformer layer includes a multi-head self-attention mechanism and a feedforward neural network to generate a new feature matrix that effectively captures the semantic information and spatial structure in the feature map. The resulting feature matrix is ​​then passed through a series of decoder layers to convert the global semantic information into local information for a specific task. Step 5.3, post-processing to obtain model output: The target detection frame output by the encoder may include multiple highly overlapping rotated target detection frames. Use the non-maximum suppression algorithm to sort the confidence scores from high to low, and then calculate the degree of overlap between the target detection frame output by the model and the true value frame. If the overlap value is higher than the preset threshold, , then the result of the device positioning is output.

Images