Method for improving in-vivo instrument positioning accuracy through doctor eye movement data
Through doctor eye movement data acquisition and deep learning model optimization, the problem of inaccurate positioning in traditional surgery is solved, the accuracy and accuracy of in vivo device positioning is improved, and the risk of surgical treatment is reduced, especially in complex surgical scenarios.
Patent Information
- Application Number
- CN202410803452.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-20
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2044-06-20
AI Technical Summary
In traditional surgery, doctors rely on naked eyes and microscopy to position the positioning is not accurate enough, resulting in high risk of surgery. Especially in complex surgical scenarios, the existing in vivo device positioning technology is affected by the complexity of human tissues and environmental changes, and the positioning accuracy is insufficient.
Through doctor eye movement data acquisition and preprocessing, mask matrix and gaze point heat map are generated, combined with convolutional neural network and Transformer model, the teacher-student model is used to optimize the positioning effect and integrate eye movement information to improve the accuracy of in vivo device positioning. .
It improves the accuracy and accuracy of in vivo device positioning, reduces the risk of surgery, especially in complex surgical scenarios, reduces the occurrence of iatrogenic injuries, and provides higher surgical precision and reliability.
Smart Images

Figure CN118887286A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of surgical navigation, and in particular to a method for improving the accuracy of positioning an in-vivo instrument through doctor's eye movement data. Background Art
[0002] In traditional surgery, surgeons usually rely on the naked eye and simple microscopes to locate and operate instruments. However, the doctor's clinical experience and the complexity of the surgical scene will affect the accuracy of positioning. Inaccurate positioning may lead to surgical failure or complications. Existing in-vivo instrument positioning technology can provide accurate position and angle information of in-vivo instruments by analyzing real-time medical images, thereby reducing surgical risks.
[0003] Current in vivo device positioning technology usually relies on sensors and image processing algorithms to achieve positioning. However, due to the complexity of human tissue and changes in the surgical environment, sensors may be affected by local interference or excessive movement of instruments, resulting in insufficient positioning accuracy. By tracking the doctor's eye movement information and integrating it into the positioning system, additional positioning references can be provided to improve the accuracy and precision of positioning, especially in complex surgical scenarios where nerves and blood vessels are intertwined, which can significantly reduce the occurrence of iatrogenic injuries. Summary of the invention
[0004] In view of the defects of the prior art, the present invention provides a method for improving the accuracy of positioning in vivo instruments through the doctor's eye movement data, which can improve the accuracy of the existing in vivo instrument positioning technology and thus reduce surgical risks.
[0005] To achieve the purpose of the present invention, the technical solution adopted is:
[0006] A method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data comprises the following steps:
[0007] Step 1: Collection and preprocessing of eye movement data;
[0008] Step 2: Generate mask matrix and gaze point heat map;
[0009] Step 3: Preprocess the input image;
[0010] Step 4: Feature extraction based on convolutional neural network;
[0011] Step 5: Use Transformer to locate the device in the body;
[0012] Step 6: Optimize the positioning effect based on the teacher-student model and eye movement information.
[0013] Preferably, in step 1, the collection and preprocessing of eye movement data includes the following steps:
[0014] Step 1.1, use an eye tracker to collect the eye movement data of the doctor when observing the surgical video;
[0015] Step 1.2, according to the displacement length of each collected data point, the data points are divided into fixation data and scanning data, and the data points within the threshold range of the preset step length are regarded as fixation data points;
[0016] Step 1.3: Save the information of the gaze data point and record it as (x, y, t), where (x, y) is the position information of the gaze data point, and t is the timestamp information of the image frame where the gaze data point is located.
[0017] Preferably, in step 2, generating a mask matrix and a gaze point heat map comprises the following steps:
[0018] Step 2.1, create an eye movement data matrix: according to the information of the recorded gaze data points, create an eye movement data matrix E of size X*Y*T, where X*Y represents the resolution corresponding to the image frame, and T is the size of the image frame;
[0019] Step 2.2, generate mask matrix: first, according to the information of the recorded fixation data points, set the value of the eye movement data matrix, the matrix value E(x, y, t) of the fixation data point is 1, and the matrix value of the other positions is set to 0. Then, calculate the visual attention radius Where D is the distance from the doctor's eyes to the screen, and DPI is the resolution of the display. Finally, traverse all the positions with a matrix value of 1 in the eye movement matrix, set the matrix values of all the positions within the visual attention radius r of these positions to 1, and obtain the mask matrix E;
[0020] Step 2.3, generate gaze point heat map: first create a matrix D with the same size as the image frame, and set the value D(x, y, t) of the gaze point position in the matrix to 255, and the rest of the positions to 0, so as to obtain the gaze data matrix D. Then use the Gaussian convolution kernel G with a visual attention radius r, standard deviation σ=0.5, and mean μ=1 kernel Convolve with D. Finally, we get the gaze heat map In the formula is the symbol for the convolution operation.
[0021] Preferably, in step 3, the input image preprocessing comprises the following steps:
[0022] Step 3.1, perform random rotation, random translation and other operations on the ultrasound images containing the internal instruments to perform data enhancement to improve the generalization ability and robustness of the model;
[0023] Step 3.2, using the generated mask matrix to perform a dot multiplication operation with the ultrasound image containing the in-vivo device to obtain new training set data and expand the data set;
[0024] Step 3.3, divide the dataset into training set, validation set and test set in a ratio of 7:2:1. The training set is used to build the model, the validation set is used to evaluate the performance of the model during the training process, and the test set is used to evaluate the performance of the trained model.
[0025] Preferably, in step 4, feature extraction based on convolutional neural network includes the following steps:
[0026] Step 4.1, data preprocessing: processing the size of the input ultrasound image into the size required by the convolutional neural network;
[0027] Step 4.2, load weights: load pre-trained weights to accelerate model convergence and improve model performance;
[0028] Step 4.3, convolutional layer feature extraction: The input image and the convolution kernel are convolved. Here, the input image passes through four convolutional layers with different convolution kernel sizes in sequence to extract features at different scales.
[0029] Step 4.4, Pooling operation: We use the maximum pooling operation to downsample the feature map. First, specify the size of the pooling window, then slide the window on the feature map output by the convolutional layer, and perform a pooling operation on the values in each window, that is, take the maximum value in the window as the pooled value. Repeat the above operation until all windows in the feature map are pooled, and a new feature map can be obtained.
[0030] Preferably, in step 5, using a Transformer to locate the in-vivo device comprises the following steps:
[0031] Step 5.1, position encoding: Calculate the encoding values of all positions in the output feature map of the convolutional neural network, and then add the position encoding to the input feature to embed the position information into the feature map to obtain an embedded feature map;
[0032] Step 5.2, encoder-decoder operation: The embedded feature map is fed into a hierarchical Transformer encoder, where each Transformer layer includes a multi-head self-attention mechanism and a feedforward neural network to obtain a new feature matrix that effectively captures the semantic information and spatial structure in the feature map. The obtained feature matrix is then passed through a series of decoder layers to convert the global semantic information into local information for a specific task;
[0033] Step 5.3, post-processing to obtain model output: The target detection frame output by the encoder may include multiple highly overlapping rotated target detection frames. Use the non-maximum suppression algorithm to sort the confidence scores from high to low, and then calculate the degree of overlap between the target detection frame output by the model and the true value frame. If the overlap value is higher than the preset threshold Thr=0.6, the result of the device positioning is output.
[0034] Preferably, in step 6, optimizing the positioning effect based on the teacher-student model and eye movement information includes the following steps:
[0035] Step 6.1, teacher model training: Use the true value data and gaze heat map to train the teacher model to quantify high-quality uncertainty;
[0036] Step 6.2: Student model training: Use the true value data and gaze heat map to train the student model to extract the prediction uncertainty of the teacher model;
[0037] Step 6.3, construct the teacher-student model loss function: Use Kullback-Leibler divergence loss to transfer the uncertainty knowledge of the teacher model to the student model:
[0038]
[0039] In the formula, is the predicted value based on the perturbed input image, Refers to the average probability map of the softmax layer of the teacher model, p c (α) is the average probability map of the softmax layer of the student model, D represents each pixel in the image, and the calculated L KL The smaller the loss value, the better the match between the student model and the teacher model;
[0040] Step 6.4: Calculation of uncertainty of the teacher model. The uncertainty of the teacher model can be calculated by the following formula:
[0041]
[0042] In the formula, is the prediction uncertainty estimate of the student model, is the mean of any sample prediction, is the mean of all sample predictions;
[0043] Step 6.5: Construct the uncertainty loss function. Since we cannot match the distribution of two unbound uncertainties when regressing the position and angle information of the in-vivo device, we construct the root mean square logarithmic loss:
[0044]
[0045] In the formula, is the prediction uncertainty of the teacher model, is the prediction uncertainty of the student model, N represents the total number of model predictions, and this loss function is introduced to pay special attention to pixels with higher uncertainty;
[0046] Step 6.6, optimization of positioning effect: Calculate the loss function and uncertainty loss function of the joint teacher-student model based on eye movement information, and feed the loss value back to the Transformer-based model for positioning the in vivo device to further optimize the model parameters to achieve higher positioning accuracy.
[0047] Compared with the prior art, the present invention has the following beneficial effects: by introducing the doctor's eye movement information, the model can more accurately understand the doctor's intentions and focus, and then provide more accurate information on the position and angle of the internal instruments, thereby improving the precision and accuracy of the operation, and providing reliable technical guarantee for the successful implementation of clinical surgery. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] Figure 1 It is a schematic diagram of the method flow of the present invention. DETAILED DESCRIPTION
[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0050] Example 1
[0051] like Figure 1 As shown, a method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data comprises the following steps:
[0052] Step 1: Collection and preprocessing of eye movement data;
[0053] Step 2: Generate mask matrix and gaze point heat map;
[0054] Step 3: Preprocess the input image;
[0055] Step 4: Feature extraction based on convolutional neural network;
[0056] Step 5: Use Transformer to locate the device in the body;
[0057] Step 6: Optimize the positioning effect based on the teacher-student model and eye movement information.
[0058] Preferably, in step 1, the collection and preprocessing of eye movement data includes the following steps:
[0059] Step 1.1, use an eye tracker to collect the eye movement data of the doctor when observing the surgical video;
[0060] Step 1.2, according to the displacement length of each collected data point, the data points are divided into fixation data and scanning data, and the data points within the threshold range of the preset step length are regarded as fixation data points;
[0061] Step 1.3: Save the information of the gaze data point and record it as (x, y, t), where (x, y) is the position information of the gaze data point, and t is the timestamp information of the image frame where the gaze data point is located.
[0062] Preferably, in step 2, generating a mask matrix and a gaze point heat map comprises the following steps:
[0063] Step 2.1, create an eye movement data matrix: according to the information of the recorded gaze data points, create an eye movement data matrix E of size X*Y*T, where X*Y represents the resolution corresponding to the image frame, and T is the size of the image frame;
[0064] Step 2.2, generate mask matrix: first, according to the information of the recorded fixation data points, set the value of the eye movement data matrix, the matrix value E(x, y, t) of the fixation data point is 1, and the matrix value of the other positions is set to 0. Then, calculate the visual attention radius Where D is the distance from the doctor's eyes to the screen, and DPI is the resolution of the display. Finally, traverse all the positions with a matrix value of 1 in the eye movement matrix, set the matrix values of all the positions within the visual attention radius r of these positions to 1, and obtain the mask matrix E;
[0065] Step 2.3, generate gaze point heat map: first create a matrix D with the same size as the image frame, and set the value D(x, y, t) of the gaze point position in the matrix to 255, and the rest of the positions to 0, so as to obtain the gaze data matrix D. Then use the Gaussian convolution kernel G with a visual attention radius r, standard deviation σ=0.5, and mean μ=1 kernel Convolve with D. Finally, we get the gaze heat map In the formula is the symbol for the convolution operation.
[0066] Preferably, in step 3, the input image preprocessing comprises the following steps:
[0067] Step 3.1, perform random rotation, random translation and other operations on the ultrasound images containing the internal instruments to perform data enhancement to improve the generalization ability and robustness of the model;
[0068] Step 3.2, using the generated mask matrix to perform a dot multiplication operation with the ultrasound image containing the in-vivo device to obtain new training set data and expand the data set;
[0069] Step 3.3, divide the dataset into training set, validation set and test set in a ratio of 7:2:1. The training set is used to build the model, the validation set is used to evaluate the performance of the model during the training process, and the test set is used to evaluate the performance of the trained model.
[0070] Preferably, in step 4, feature extraction based on convolutional neural network includes the following steps:
[0071] Step 4.1, data preprocessing: processing the size of the input ultrasound image into the size required by the convolutional neural network;
[0072] Step 4.2, load weights: load pre-trained weights to accelerate model convergence and improve model performance;
[0073] Step 4.3, convolutional layer feature extraction: The input image and the convolution kernel are convolved. Here, the input image passes through four convolutional layers with different convolution kernel sizes in sequence to extract features at different scales.
[0074] Step 4.4, Pooling operation: We use the maximum pooling operation to downsample the feature map. First, specify the size of the pooling window, then slide the window on the feature map output by the convolutional layer, and perform a pooling operation on the values in each window, that is, take the maximum value in the window as the pooled value. Repeat the above operation until all windows in the feature map are pooled, and a new feature map can be obtained.
[0075] Preferably, in step 5, using a Transformer to locate the in-vivo device comprises the following steps:
[0076] Step 5.1, position encoding: Calculate the encoding values of all positions in the output feature map of the convolutional neural network, and then add the position encoding to the input feature to embed the position information into the feature map to obtain an embedded feature map;
[0077] Step 5.2, encoder-decoder operation: The embedded feature map is fed into a hierarchical Transformer encoder, where each Transformer layer includes a multi-head self-attention mechanism and a feedforward neural network to obtain a new feature matrix that effectively captures the semantic information and spatial structure in the feature map. The obtained feature matrix is then passed through a series of decoder layers to convert the global semantic information into local information for a specific task;
[0078] Step 5.3, post-processing to obtain model output: The target detection frame output by the encoder may include multiple highly overlapping rotated target detection frames. Use the non-maximum suppression algorithm to sort the confidence scores from high to low, and then calculate the degree of overlap between the target detection frame output by the model and the true value frame. If the overlap value is higher than the preset threshold Thr=0.6, the result of the device positioning is output.
[0079] Preferably, in step 6, optimizing the positioning effect based on the teacher-student model and eye movement information includes the following steps:
[0080] Step 6.1, teacher model training: Use the true value data and gaze heat map to train the teacher model to quantify high-quality uncertainty;
[0081] Step 6.2: Student model training: Use the true value data and gaze heat map to train the student model to extract the prediction uncertainty of the teacher model;
[0082] Step 6.3, construct the teacher-student model loss function: Use Kullback-Leibler divergence loss to transfer the uncertainty knowledge of the teacher model to the student model:
[0083]
[0084] In the formula, is the predicted value based on the perturbed input image, Refers to the average probability map of the softmax layer of the teacher model, p c (α) is the average probability map of the softmax layer of the student model, D represents each pixel in the image, and the calculated L KL The smaller the loss value, the better the match between the student model and the teacher model;
[0085] Step 6.4: Calculation of uncertainty of the teacher model. The uncertainty of the teacher model can be calculated by the following formula:
[0086]
[0087] In the formula, is the prediction uncertainty estimate of the student model, is the mean of any sample prediction, is the mean of all sample predictions;
[0088] Step 6.5: Construct the uncertainty loss function. Since we cannot match the distribution of two unbound uncertainties when regressing the position and angle information of the in-vivo device, we construct the root mean square logarithmic loss:
[0089]
[0090] In the formula, is the prediction uncertainty of the teacher model, is the prediction uncertainty of the student model, N represents the total number of model predictions, and this loss function is introduced to pay special attention to pixels with higher uncertainty;
[0091] Step 6.6, optimization of positioning effect: Calculate the loss function and uncertainty loss function of the joint teacher-student model based on eye movement information, and feed the loss value back to the Transformer-based model for positioning the in vivo device to further optimize the model parameters to achieve higher positioning accuracy.
Claims
1. A method for improving the accuracy of positioning an in-vivo device by using a doctor's eye movement data, characterized in that: The steps include: Step 1: Collection and preprocessing of eye movement data; Step 2: Generate mask matrix and gaze point heat map; Step 3: Preprocess the input image; Step 4: Feature extraction based on convolutional neural network; Step 5: Use Transformer to locate the device in the body; Step 6: Optimize the positioning effect based on the teacher-student model and eye movement information.
2. A method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data as claimed in claim 1, characterized in that: In step 1, the collection and preprocessing of eye movement data specifically includes the following steps: Step 1.1, use an eye tracker to collect the eye movement data of the doctor when observing the surgical video; Step 1.2, according to the displacement length of each collected data point, the data point is divided into fixation data and scanning data, and the data point within the threshold range of the preset step length is regarded as the fixation data point; Step 1.3: Save the information of the gaze data point and record it as (x, y, t), where (x, y) is the position information of the gaze data point, and t is the timestamp information of the image frame where the gaze data point is located.
3. The method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data as claimed in claim 1, characterized in that: In step 2, generating the mask matrix and the gaze point heat map specifically includes the following steps: Step 2.1, create an eye movement data matrix: according to the information of the recorded gaze data points, create an eye movement data matrix E of size X*Y*T, where X*Y represents the resolution corresponding to the image frame, and T is the size of the image frame; Step 2.2, generate the mask matrix: first, according to the information of the recorded fixation data points, set the value of the eye movement data matrix, the matrix value E(x, y, t) = 1 at the location of the fixation data point, and the matrix values of the other locations are set to 0. Then, calculate the visual attention radius Where D represents the distance from the doctor's eyes to the screen, and DPI is the resolution of the display. Finally, all the positions with matrix values of 1 in the eye movement matrix are traversed, and the matrix values of all the positions within the visual attention radius r of these positions are set to 1 to obtain the mask matrix E; Step 2.3, generate gaze point heat map: first create a matrix D with the same size as the image frame, and set the value D(x, y, t) of the gaze point position in the matrix to 255, and the rest of the positions to 0, so as to obtain the gaze data matrix D, and then use the Gaussian convolution kernel G with a visual attention radius r, standard deviation σ=0.5, and mean μ=1 kernel Convolution operation is performed with D, and finally, the gaze heat map is obtained In the formula is the symbol for the convolution operation.
4. The method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data as claimed in claim 1, characterized in that: In step 3, the input image preprocessing specifically includes the following steps: Step 3.1, perform random rotation, random translation and other operations on the ultrasound images containing the internal instruments to perform data enhancement to improve the generalization ability and robustness of the model; Step 3.2, using the generated mask matrix to perform a dot multiplication operation with the ultrasound image containing the in-vivo device to obtain new training set data and expand the data set; Step 3.3, divide the dataset into training set, validation set and test set in a ratio of 7:2:
1. The training set is used to build the model, the validation set is used to evaluate the performance of the model during the training process, and the test set is used to evaluate the performance of the trained model.
5. The method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data as claimed in claim 1, characterized in that: In step 4, feature extraction based on convolutional neural network specifically includes the following steps: Step 4.1, data preprocessing: processing the size of the input ultrasound image into the size required by the convolutional neural network; Step 4.2, load weights: load pre-trained weights to accelerate model convergence and improve model performance; Step 4.3, convolutional layer feature extraction: The input image and the convolution kernel are convolved. Here, the input image passes through four convolutional layers with different convolution kernel sizes in sequence to extract features at different scales. Step 4.4, pooling operation: We use the maximum pooling operation to downsample the feature map. First, specify the size of the pooling window, then slide the window on the feature map output by the convolutional layer, and perform a pooling operation on the values in each window, that is, take the maximum value in the window as the pooled value. Repeat the above operation until all windows in the feature map are subjected to the maximum pooling operation, and then a new feature map can be obtained.
6. The method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data as claimed in claim 1, characterized in that: In step 5, using Transformer to locate the in-vivo device specifically includes the following steps: Step 5.1, position encoding: Calculate the encoding values of all positions in the output feature map of the convolutional neural network, and then add the position encoding to the input feature to embed the position information into the feature map to obtain an embedded feature map; Step 5.2, encoder-decoder operation: The embedded feature map is fed into a hierarchical Transformer encoder, where each Transformer layer includes a multi-head self-attention mechanism and a feedforward neural network to obtain a new feature matrix that effectively captures the semantic information and spatial structure in the feature map. The obtained feature matrix is then passed through a series of decoder layers to convert the global semantic information into local information for a specific task. Step 5.3, post-processing to obtain model output: The target detection frame output by the encoder may include multiple highly overlapping rotated target detection frames. Use the non-maximum suppression algorithm to sort the confidence scores from high to low, and then calculate the degree of overlap between the target detection frame output by the model and the true value frame. If the overlap value is higher than the preset threshold Thr=0.6, the result of the device positioning is output.
7. The method for improving the positioning accuracy of in-vivo devices by using doctor's eye movement data as claimed in claim 1, characterized in that: In step 6, the positioning effect is optimized based on the teacher-student model and eye movement information, which specifically includes the following steps: Step 6.1, teacher model training: Use the true value data and gaze heat map to train the teacher model to quantify high-quality uncertainty; Step 6.2: Student model training: Use the true value data and gaze heat map to train the student model to extract the prediction uncertainty of the teacher model; Step 6.3, construct the teacher-student model loss function, and use the Kullback-Leibler divergence loss to transfer the uncertainty knowledge of the teacher model to the student model: In the formula, is the predicted value based on the perturbed input image, Refers to the average probability map of the softmax layer of the teacher model, p c (α) is the average probability map of the softmax layer of the student model, and D represents each pixel in the image; The calculated L KL The smaller the loss value, the better the match between the student model and the teacher model; Step 6.4: Calculation of uncertainty of the teacher model. The uncertainty of the teacher model can be calculated by the following formula: In the formula, is the prediction uncertainty estimate of the student model, is the mean of any sample prediction, is the mean of all sample predictions; Step 6.5: Construct the uncertainty loss function. Since we cannot match the distribution of two unbound uncertainties when regressing the position and angle information of the in-vivo device, we construct the root mean square logarithmic loss: In the formula, is the prediction uncertainty of the teacher model, is the prediction uncertainty of the student model, N represents the total number of model predictions, and this loss function is introduced to pay special attention to pixels with higher uncertainty; Step 6.6, optimization of positioning effect: Calculate the loss function and uncertainty loss function of the joint teacher-student model based on eye movement information, and feed the loss value back to the Transformer-based model for positioning the in vivo device to further optimize the model parameters to achieve higher positioning accuracy.
Citation Information
Patent Citations
Image generation method based on eye gaze data and image data set
CN108230426A
Method of ultrasonic three-dimensional structure rapid acquisition system independent of external positioning equipment
CN114581599A
Method for improving accuracy of ultrasonic image recognition model through doctor eye movement data
CN115641496A
Chinese reading training system based on artificial intelligence and eye movement capture technology
CN116741020A
Spinal surgery robot navigation positioning system
CN117323004A
Cited By
Ophthalmologic operation real-time navigation system and method based on dynamic visual field tracking
CN119818291A
An Ophthalmic Surgery Real-Time Navigation System and Method Based on Dynamic Visual Field Tracking
CN119818291B
