Neural network-based operation field optimization method in laparoscopic cholecystectomy
Through neural network technology and collaborative annotation by multiple doctors, the surgical field in laparoscopic cholecystectomy is adjusted in real time, solving the balance between accuracy and real-time performance of the existing technology field optimization methods, and achieving efficient and concise The surgical field reconstruction and dynamic tracking improve the real-time and efficiency of the surgery.
Patent Information
- Application Number
- CN202411045259.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-15
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing laparoscopic cholecystectomy intraoperative field optimization methods are difficult to balance between three-dimensional reconstruction accuracy and real-time. The system is complex, the hardware requirements are strict, the adaptability is poor, and the user experience is unfriendly, making it difficult to be in the dynamically changing anatomical structure. Provide continuous and effective support.
Using a neural network-based method, through collaborative annotation and deep learning model training by multiple doctors, combining convolutional neural networks and long-term memory networks, real-time adjustment of camera angle, focal length and lighting conditions, dynamically tracking anatomical structure changes, and using most opinions to make decisions and feedback system optimization field.
Achieve high-precision field reconstruction without increasing hardware complexity, improves the real-time and accuracy of the operation, reduces the burden of manual adjustment by doctors, ensures the best visual status of the surgical field, and improves surgical efficiency and images. Clarity.
Smart Images

Figure CN118967624A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a method for optimizing the surgical field during laparoscopic cholecystectomy, in particular to a method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network. Background Art
[0002] The current methods for optimizing the surgical field in laparoscopic cholecystectomy have some significant drawbacks, especially in terms of 3D reconstruction and real-time tracking. The Chinese invention patent, patent number 2020110849528, provides a system and method for 3D reconstruction of the surgical field in microsurgery, which performs 3D reconstruction by combining visible light and infrared light. However, this method faces some challenges and limitations when applied to laparoscopic cholecystectomy. First, there are problems with the accuracy and real-time performance of 3D reconstruction. The method of patent 2020110849528 relies on the joint optimization of multiple viewpoints and a texture enhancement mechanism based on infrared speckle, which requires complex calculations and a lot of time to process data. In laparoscopic cholecystectomy, the dynamic changes of the surgical field and anatomical structures are very rapid, requiring real-time image processing and analysis. However, current 3D reconstruction methods often fail to find a balance between real-time and high precision, resulting in image lag and insufficient precision during surgery, which in turn affects the smooth progress of the operation. Secondly, the system is complex and has high hardware requirements. Patent 2020110849528 mentions that the structural design of the infrared photosensitive element and the speckle projector can accurately obtain the external structure of the surgical field. However, this complex hardware configuration increases the cost and maintenance difficulty of the system. In actual laparoscopic surgery, the surgical environment is complex and the equipment is numerous. Adding additional hardware not only increases the burden on the operating room, but also affects the flexibility and efficiency of the operation. In addition, the calibration and maintenance of infrared photosensitive elements and speckle projectors require professional technicians, which places higher demands on the hospital's resources and staffing.
[0003] Thirdly, the existing methods have poor adaptability to the surgical environment. In laparoscopic surgery, the surgical field is affected by many factors, including the movement of instruments, interference from blood and other body fluids, etc. Although the method of patent 2020110849528 enhances the texture through infrared speckle patterns, the effect of infrared speckle will be disturbed in complex surgical environments, resulting in a decrease in the accuracy of three-dimensional reconstruction. In addition, the light conditions in the surgical field vary greatly, and the stability and adaptability of the existing methods under different light conditions are insufficient, and it is impossible to guarantee high-quality three-dimensional reconstruction in various environments. In addition, the method of patent 2020110849528 has limitations in the use of prior knowledge of the surgical field. Although this method optimizes the surgical field model through three-dimensional reconstruction, it lacks full consideration of real-time dynamic changes during the operation. In laparoscopic cholecystectomy, the position and shape of the anatomical structure will change significantly as the operation progresses, and it is difficult to cope with such dynamic changes by relying solely on a priori three-dimensional reconstruction models. The lack of real-time update and adjustment capabilities makes it difficult for existing methods to provide continuous and effective support during surgery. Finally, the user experience of existing methods is not friendly enough. Laparoscopic surgery requires doctors to quickly obtain clear images of the surgical field during the operation and make quick decisions based on actual conditions. Although the method of patent 2020110849528 can theoretically provide high-precision three-dimensional reconstruction, it requires setup and debugging in actual operation, which has high technical requirements for doctors and increases the complexity and time cost of the operation. Summary of the invention
[0004] The purpose of the present invention is to provide a method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network, thereby solving some of the drawbacks and shortcomings pointed out in the background technology.
[0005] The present invention solves the above-mentioned technical problems by adopting the following technical solutions:
[0006] First, laparoscopic surgery video and image data were collected and preprocessed. The key structures of the gallbladder, bile duct, and liver in the videos and images were annotated by multiple doctors in accordance with the majority opinion principle. Image processing technology was used to reduce noise, enhance contrast, and detect edges.
[0007] The majority opinion compliance principle is a cloud-based collaborative platform that allows multiple doctors to log in remotely and annotate the same surgical video and image data in real time. The opinions of multiple doctors are integrated by implementing a majority opinion decision algorithm. Whenever a doctor completes a single annotation, the computer system counts the annotation results of all doctors and uses the majority opinion as the final structural positioning.
[0008] Wherein, the edge detection is used to identify and track the edges of anatomical structures during surgery;
[0009] Then, we designed and trained a deep learning model, and based on the improved convolutional neural network architecture, we added a specific attention mechanism to focus on feature learning in key structural areas, trained it using labeled data, and optimized the model parameters through cross-validation methods.
[0010] Finally, a surgical field optimization algorithm was developed. The trained neural network model was used to analyze the surgical video in real time, predict the position changes of key anatomical structures, and adjust the camera angle, focal length, and lighting conditions in real time based on the prediction results. At the same time, a feedback system was introduced to make dynamic adjustments based on the doctor's operational feedback during the operation.
[0011] Furthermore, the cloud-based collaboration platform construction process is as follows:
[0012] S1. Using a cloud-based collaborative platform, multiple doctors remotely logged in and annotated the gallbladder, bile duct, and liver anatomical structures in the surgical video in real time, using the decision formula:
[0013]
[0014] Where D i represents the labeled dataset of the i-th anatomical structure, x in is the annotation result of the nth doctor on the i-th anatomical structure, w n is the weight based on the historical accuracy of the doctor's annotations, μ i and σ i The datasets D are i The mean and standard deviation of are used to calculate the weighted average of the annotations;
[0015] S2, and train the neural network model based on the obtained labeled data, using the adjustment equation:
[0016]
[0017] Where S is the surgical scene data, V is the predetermined field of view quality parameter, α and β are dynamically adjusted coefficients, and H(S t ) is the visual field quality assessment function for the surgical scene S at time t.
[0018] Furthermore, the edge detection implementation method includes: first, using a generative adversarial network (GAN) and a deep convolutional neural network (DCNN) to train a model using the formula:
[0019] L GAN (E,θ)=∫ V log(1+e -(θ·E(x)) )dx
[0020] Where x represents the pixel in the image, E represents the edge feature vector extracted from the surgical video, θ is the model parameter, and V is all the visualization areas;
[0021] Next, a real-time edge tracking model based on the long short-term memory network LSTM is applied, using the formula:
[0022]
[0023] Calculate, where P(t) represents the edge position prediction at time t, and δ(t) is the time-dependent adjustment factor;
[0024] Finally, the data from CT and MRI imaging technologies are combined through a fusion algorithm:
[0025]
[0026] Where x represents the pixel in the image, C and M represent the image data of CT and MRI respectively, and λ is the fusion coefficient.
[0027] Furthermore, the deep learning model construction process is as follows:
[0028] First, the network architecture is used to adjust the special design and parameters of the convolution kernel:
[0029]
[0030] Where S represents the surgical image data, θ and μ k Represent the learning parameters and the center of the kth convolution kernel respectively, and K is the total number of convolution kernels;
[0031] Next, implement the loss function:
[0032]
[0033] Where Y is the real annotated surgical field optimization target area, P is the predicted area, and y n and p n are the values of Y and P at the nth pixel, respectively, and N is the total number of pixels in the image;
[0034] Finally, a cross-validation method is used:
[0035]
[0036] Where V m represents the actual output in the mth cross validation, is the model prediction output, and m is the number of verifications.
[0037] Furthermore, the surgical field optimization algorithm construction method includes:
[0038] S1. Develop a dynamic tracking algorithm based on deep learning, using recurrent neural network (RNN) and long short-term memory (LSTM) network to process surgical video sequence data, detect and track the dynamic changes of key anatomical structures in real time, integrate computer vision technology, including instance segmentation and object detection, identify and predict the position and shape of key structures;
[0039] S2. Then, an intelligent camera control system is designed to adjust the angle and focal length of the camera and the lighting conditions of the surgical area according to the key structure position data obtained from the neural network model;
[0040] S3. Build a real-time feedback system and develop a dynamic adjustment algorithm. When the doctor provides instant feedback on the camera adjustment through the input device during the operation, the neural network prediction and camera control system parameters are adjusted in real time according to the doctor's feedback. Use reinforcement learning to learn and adapt to the doctor's operating habits and preferences in real time.
[0041] Furthermore, the dynamic tracking algorithm is implemented as follows:
[0042] First, a composite model combining convolutional neural network (CNN) and long short-term memory (LSTM) is used to process the surgical video sequence data through the equation:
[0043]
[0044] Where S t represents the video frame at time t, H t-1 is the hidden state of LSTM at the previous moment, W k and B k is the weight and bias of the kth convolution kernel, K is the number of convolution kernels, and * represents the convolution operation, which detects and tracks the dynamic changes of key anatomical structures in real time;
[0045] It also integrates computer vision techniques, including instance segmentation and object detection, to identify and predict the location and shape of key structures through integral equations:
[0046]
[0047] Where R(x,y) represents the response intensity of the position (x,y), which represents the possibility that the pixel at this position belongs to the key structure, M is the mask operation matrix, and V ij (X) is the feature map of image X at position (i, j) after the convolution layer, c is the normalization constant, and N is the size of the mask.
[0048] Furthermore, the intelligent camera control system construction method includes:
[0049] S1. First, the intelligent camera control system uses the anatomical structure position data output by the neural network model through the formula:
[0050]
[0051] Adjust the camera angle θ and focal length φ, where S i (t) represents the position of the ith anatomical structure at time t, α(t), β i and γ(t) are the time-dependent adjustment coefficient and weight, respectively;
[0052] S2, the intelligent lighting adjustment system evaluates the impact of current lighting conditions on image quality based on a neural network, using the formula:
[0053]
[0054] Where L(x,y) represents the lighting intensity at position (x,y), κ is the data fed back by the light sensor, and η(x,y) is the brightness evaluation of the current pixel output by the neural network. The lighting conditions are optimized by adjusting the light intensity in the N region.
[0055] Furthermore, the real-time feedback system construction process is as follows:
[0056] First, a real-time feedback system was designed and implemented. Multimodal input devices were used to collect the doctor's immediate feedback on the camera's viewing angle and focal length during the operation, and the feedback was processed using the formula:
[0057]
[0058] Where θ(t) and φ(t) represent the angle and focal length of the camera at time t, respectively, and f(t, θ, φ) is the evaluation function;
[0059] Next, we developed a dynamic adjustment algorithm based on reinforcement learning to adjust the predictions of the neural network and the parameters of the camera control system, and update the parameters using the formula:
[0060]
[0061] Where Δp is the change in the parameter, γ is the learning rate, and r k is the reward obtained at step k, π k (a k |s k ; θ) is in state s k Next select action a k The strategy probability depends on the parameter θ.
[0062] Beneficial effects of the present invention:
[0063] 1. By combining the composite model of convolutional neural network (CNN) and long short-term memory network (LSTM), the system can process surgical video sequence data in real time, dynamically track and detect changes in key anatomical structures. This real-time performance greatly improves the accuracy of surgery, ensuring that doctors can clearly see and track important anatomical structures throughout the operation, thereby reducing surgical risks.
[0064] 2. Unlike traditional 3D reconstruction methods, the present invention uses a neural network model for efficient 3D reconstruction, which can achieve high-precision surgical field reconstruction without increasing hardware complexity. This method reduces the reliance on complex hardware and multi-viewpoint acquisition systems, making the system more concise and easy to maintain.
[0065] 3. The system uses the anatomical structure position data output by the neural network model to adjust the camera angle and focal length in real time to ensure the best state of the surgical field of view. The camera control system can automatically optimize the viewing angle and focal length based on real-time feedback to provide the best surgical field of view, reducing the burden of doctors manually adjusting the camera during surgery and improving surgical efficiency.
[0066] 4. The system evaluates the impact of current lighting conditions on image quality based on a neural network and optimizes the lighting conditions in the surgical area by adjusting the light intensity. This intelligent lighting adjustment ensures that the surgical area is always under optimal lighting conditions, reduces visual impairments caused by uneven light, and improves the clarity and recognizability of surgical images. BRIEF DESCRIPTION OF THE DRAWINGS
[0067] Figure 1 The present invention is a flow chart of the method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network.
[0068] Figure 2 A module diagram for building a cloud-based collaboration platform of the present invention.
[0069] Figure 3 This is a module diagram of the edge detection implementation method of the present invention.
[0070] Figure 4 A flow chart is constructed for the surgical field optimization algorithm of the present invention. DETAILED DESCRIPTION
[0071] The specific implementation modes of the present invention will be described in detail below in conjunction with the accompanying drawings.
[0072] Combined with Figure 1The neural network-based laparoscopic cholecystectomy surgical field optimization method includes collecting and preprocessing surgical video and image data, involving multiple doctors to jointly annotate key structures such as the gallbladder, bile duct, and liver in the video and images in real time through a cloud-based collaborative platform. In this collaborative process, each doctor submits his or her own annotation results, and the system integrates these annotation results through the principle of majority opinion, that is, for each annotation point, the position supported by the majority of doctors is selected as the final structure location. In addition, the image data undergoes a series of image processing techniques, including noise reduction, contrast enhancement, and edge detection, in order to more accurately identify and track the edges of anatomical structures during surgery.
[0073] The next step is to design and train a deep learning model. This model is based on an improved convolutional neural network architecture, in which an attention mechanism is added to enable the model to more focusedly learn the features of key anatomical areas in surgical videos. The model is trained using data accurately annotated by doctors, and a cross-validation method is used to optimize the model parameters to improve the model's adaptability to new surgical scenarios and overall prediction accuracy.
[0074] Finally, we developed an algorithm for optimizing the surgical field, using the trained neural network model to analyze the surgical video in real time and predict the position changes of the anatomical structure during the operation. We adjusted the angle and focal length of the camera and the lighting conditions of the surgical area according to the prediction results of the model to ensure the clearest surgical field of view. At the same time, through the feedback mechanism, doctors can provide operational feedback through input devices including touch screens, foot pedals or voice commands during the operation. The system adjusts the output of the neural network and the camera control parameters in real time based on these feedbacks, so that the system can better adapt to the doctor's operating habits and the specific needs of the operation.
[0075] Embodiment 1:
[0076] Combined with Figure 2 In the embodiment, multiple doctors remotely log in through the cloud collaboration platform to annotate the gallbladder, bile duct, and liver anatomical structures in the surgical video. The following decision formula is used to integrate the doctors' annotation results:
[0077]
[0078] Among them, D i represents the labeled dataset of the i-th anatomical structure, x in is the annotation result of the nth doctor on the i-th anatomical structure, w n is the weight based on the historical accuracy of the doctor's annotations, μ i and σ i The datasets D are i The mean and standard deviation of are used to calculate the weighted average of the annotations.
[0079] Suppose that three doctors give the labeling results of gallbladder (the first anatomical structure) as x 1,1 =5, x 2,1 =5.5, x 3,1 =6, the doctors’ weights are w1=0.9, w2=0.8, w3=0.85, with mean μ1=5.5 and standard deviation σ1=0.25.
[0080] Substitute the above data into the formula to calculate the weighted average annotation result:
[0081]
[0082] Calculation results:
[0083]
[0084]
[0085]
[0086] Therefore, combining the labeling results of multiple doctors, the system finally obtained the labeling position of the gallbladder as 1.5253.
[0087] The embodiment further trains a neural network model based on the obtained annotated data and uses an adjustment equation to optimize the quality of the surgical field of view. First, the annotated data of the gallbladder, bile duct, and liver will be used to train a deep convolutional neural network model. The following adjustment equation is used to optimize the quality of the surgical field of view:
[0088]
[0089] in:
[0090] S is the surgical scene data;
[0091] V is the predetermined visual field quality parameter;
[0092] α and β are the coefficients of dynamic adjustment;
[0093] H(S t ) is the visual field quality assessment function for the surgical scene S at time t.
[0094] Annotated dataset:
[0095] Gallbladder position data: The marked gallbladder position is [50,60], with a standard deviation of 5;
[0096] Biliary tract position data: The marked bile duct position is [55,65], with a standard deviation of 4;
[0097] Liver position data: The marked liver position is [70,80] with a standard deviation of 6.
[0098] The above labeled data were input into the deep convolutional neural network ResNet50 for training. The model architecture includes convolutional layers, pooling layers, fully connected layers, and integrates a specific attention mechanism CBAM. During the training process, an adjustment equation is used to optimize the quality of the surgical field of view. The field of view quality parameter V is set to 0.9, indicating the expected field of view clarity. The dynamic adjustment coefficient is set to the range of α∈[0.01,0.1] and β∈[0.5,2.0]. The specific values are α=0.05 and β=1.5.
[0099] During the operation, the system evaluates the visual field quality in real time and sets the visual field quality evaluation function H(S t ) outputs a value of 0.7. Substituting these values into the adjustment equation:
[0100]
[0101] By calculating:
[0102]
[0103]
[0104] Set T = 10 seconds:
[0105]
[0106] therefore:
[0107] g(S,0.9)=0.3×12.974=3.8922
[0108] The results indicate that under the current surgical field of view, the system needs to be adjusted to improve field of view clarity.
[0109] Embodiment 2:
[0110] Combined with Figure 3 In the laparoscopic cholecystectomy, the edge detection of the surgical field is optimized by using the generative adversarial network (GAN) and the deep convolutional neural network (DCNN) technology, and the model is trained using the following formula:
[0111] L GAN (E,θ)=∫ V log(1+e -(θ·E(x)) )dx
[0112] in:
[0113] x represents the pixel in the image;
[0114] E represents the edge feature vector extracted from the surgical video;
[0115] θ is the model parameter;
[0116] V is the total visualization area.
[0117] It is assumed that during the operation, the system extracts multiple frames of images from the surgical video, each of which is 256×256 pixels in size. Through the preprocessing step, the edge feature vector E is extracted, and these feature vectors are used to train the GAN and DCNN models. The edge detection model is optimized by training the generator and discriminator in GAN against each other. The initial value range of the model parameter θ is set to [-0.1, 0.1], and it is gradually optimized during the training process. The pixel point x of the input image generates the feature vector E(x) through the generator. The loss function of the generative adversarial network is calculated using the following formula:
[0118] L GAN (E,θ)=∫ V log(1+e -(θ·E(x))) )dx
[0119] Set the value range V to an image area of 256×256, the model parameter θ=0.05, and the edge feature vector F(x) of the pixel point to 0.8. The loss function at this pixel point is calculated as:
[0120]
[0121] The calculation results are:
[0122] L GAN (0.8,0.05)=256×256×log(1+e -0.04 )
[0123] log(1+e -0.04 )≈log(1+0.9608)=log1.9608≈0.674
[0124] L GAN (0.8,0.05)=256×256×0.674≈44236.16
[0125] During the training process, the loss function L is minimized by adjusting the model parameters θ and the feature vector E(x). GAN , gradually improving the edge detection accuracy of the model.
[0126] Example In a laparoscopic cholecystectomy, after the surgical team achieved high-precision edge detection through GAN and DCNN technologies, they continued to apply a real-time edge tracking model based on a long short-term memory network (LSTM) to further optimize the tracking and adjustment capabilities of the surgical field of view.
[0127] Apply the LSTM-based real-time edge tracking model and use the following formula for calculation:
[0128]
[0129] in:
[0130] P(t) represents the edge position prediction at time t
[0131] δ(t) is the time-dependent adjustment factor
[0132] The edge position data are set as follows: P(1)=0.7; P(2)=0.75; P(3)=0.72; P(4)=0.78.
[0133] The time-dependent adjustment factor δ(t) ranges from [0.8, 1.2], and the specific values are δ(1) = 1.0, δ(2) = 0.9, δ(3) = 1.1, and δ(4) = 1.0.
[0134] Substitute the above data into the LSTM model formula to calculate the real-time edge tracking results:
[0135]
[0136] The calculation process is as follows:
[0137] When t=1, P(t-1) does not exist, so this term is ignored.
[0138] When t = 2:
[0139]
[0140] When t=3:
[0141]
[0142] When t=4:
[0143]
[0144] Add all the results together:
[0145] T LSTM (P, δ)=0.0466-0.0326+0.0624=0.0764
[0146] Next, the embodiment applies data from CT and MRI imaging technologies and performs data fusion through the following fusion algorithm:
[0147]
[0148] in:
[0149] x represents the pixel in the image;
[0150] C and M represent the image data of CT and MRI, respectively;
[0151] λ is the fusion coefficient, which ranges from 0 to 1.
[0152] Assume that during surgery, the system obtains high-resolution images of the anatomical structure from the patient's CT and MRI scans. Both the CT image C and the MRI image M are grayscale images of 512×512 pixels. Take λ=0.6, which means that the CT data is more preferred while retaining the advantages of some MRI data. The CT and MRI image data are input into the fusion algorithm for processing.
[0153] Set the gradient of the CT image at pixel x Gradients of MRI images Substituting into the formula:
[0154] F(C,M,0.6)=0.6·0.8+0.4·0.5
[0155] F(C,M,0.6)=0.48+0.2=0.68
[0156] At this pixel, the gradient of the fused image is 0.68.
[0157] Embodiment 3:
[0158] In the laparoscopic cholecystectomy, the network architecture is first used to adjust the special design and parameters of the convolution kernel, and the convolution operation is performed using the following formula:
[0159]
[0160] in:
[0161] S represents surgical image data;
[0162] θ and μ k Represent the learning parameters and the center of the kth convolution kernel respectively;
[0163] K is the total number of convolution kernels.
[0164] The size of each frame of the surgical video is set to 256×256 pixels. The system extracts multiple frames of surgical image data for processing. The total number of convolution kernels is set to K=5, and the initial learning parameter of the convolution kernel is θ k and the center μ k The ranges are (0.01,0.1) and (0,256) respectively.
[0165] The surgical image data S is input into the deep learning model and convolution operation is performed. The convolution kernel center μ is set k =128, learning parameter θ k=0.05, the pixel value of the surgical image data S is 150. Substitute these data into the formula for calculation:
[0166]
[0167] The calculation results are:
[0168]
[0169] A(S,0.05)=0.05·3.33×10 -11 =1.665×10 -12
[0170] This result represents the convolution result under the action of a specific convolution kernel.
[0171] The embodiment further uses the following loss function to measure the difference between the predicted area and the true labeled area, and performs optimization:
[0172]
[0173] in:
[0174] Y is the truly marked target area for surgical field optimization;
[0175] P is the prediction area;
[0176] y n and p n are the values of Y and P at the nth pixel respectively;
[0177] N is the total number of pixels in the image.
[0178] Assume that during the operation, the system extracts a 256×256 pixel image from the surgical video, and the pixel values corresponding to the real labeled area Y and the model predicted area P are stored in the matrix respectively.
[0179] Some pixel values of the real marked area Y and the predicted area P are as follows:
[0180] y1=0.8, y2=0.7, y3=0.9,…,y 65536 =0.5
[0181] p1=0.75, p2=0.65, p3=0.85,…,p 65536 =0.55
[0182] Substitute the above data into the loss function formula for calculation:
[0183]
[0184] calculate
[0185]
[0186] calculate and
[0187]
[0188]
[0189] Calculate the loss function value:
[0190]
[0191]
[0192] L(Y,P)=1-0.7846
[0193] L(Y,P)=0.2154
[0194] The embodiment uses a cross-validation method to verify and further optimize the model, using the following formula:
[0195]
[0196] in:
[0197] V m Represents the actual output in the mth cross validation;
[0198] Predict output for the model;
[0199] M is the number of verifications.
[0200] The surgical video dataset is divided into 5 parts for cross-validation (ie, M = 5). In each cross-validation, the model is trained on 4 parts of the data and validated on the remaining 1 part of the data.
[0201] Take the first cross validation as an example:
[0202] The actual output V1 is 0.85, and the model predicts the output is 0.80;
[0203] The actual output V2 is 0.90, and the model predicts the output is 0.88;
[0204] The actual output V3 is 0.75, and the model predicts the output is 0.78;
[0205] The actual output V4 is 0.82, and the model predicts the output is 0.80;
[0206] The actual output V5 is 0.88, and the model predicts the output is 0.85.
[0207] Substitute the above data into the formula to calculate:
[0208]
[0209] The specific calculation is as follows:
[0210] When m=1:
[0211]
[0212] When m=2:
[0213]
[0214] When m=3:
[0215]
[0216] When m=4:
[0217]
[0218] When m=5:
[0219]
[0220] Calculate the sum:
[0221]
[0222]
[0223] In the 5 cross-validations, the calculation results of the actual output and the model predicted output are close to 1, indicating that the model's prediction is very close to the actual value.
[0224] Embodiment 4:
[0225] In the laparoscopic cholecystectomy, a composite model combining a convolutional neural network (CNN) and a long short-term memory (LSTM) network is used to process the surgical video sequence data through the following equation:
[0226]
[0227] in:
[0228] S t represents the video frame at time t;
[0229] H t-1 is the hidden state of LSTM at the previous moment;
[0230] W k and B k is the weight and bias of the kth convolution kernel;
[0231] K is the number of convolution kernels;
[0232] * indicates a convolution operation.
[0233] During the operation, the system extracts multiple frames of image data from the operation video, and the size of each frame is 256×256 pixels. t Input the composite model and perform convolution and LSTM processing. Set the total number of convolution kernels K = 3, and the weight of each convolution kernel W k and bias B k The initial ranges are 0.01,0.1 and -0.1,0.1 respectively.
[0234] Assume that at a certain time t, video frame S t Some pixel values of are 0.5, 0.6, 0.4, and the hidden state H at the previous moment t-1 0.2, 0.3, 0.1. Substitute these data into the equation for calculation:
[0235]
[0236] Set the specific value of the convolution kernel to:
[0237] W1=0.05,B1=0.02
[0238] W2=0.07,B2=-0.01
[0239] W3=0.1,B3=0.03
[0240] Perform convolution operation and activation function ReLU calculation:
[0241] CNN1(S t )=0.05*0.5+0.02=0.025+0.02=0.045
[0242] CNN2(S t )=0.07*0.6-0.01=0.042-0.01=0.032
[0243] CNN3(S t )=0.1*0.4+0.03=0.04+0.03=0.07
[0244] These convolution results are passed through the ReLU activation function:
[0245] Relu(0.045)=0.045
[0246] Relu(0.032)=0.032
[0247] Relu(0.07)=0.07
[0248] The result after ReLU activation is input into LSTM for calculation:
[0249] F(S t )=LSTM([0.045,0.032,0.07],[0.2,0.3,0.1])
[0250] The calculation results of LSTM are set to: 0.15, 0.25, 0.12, which means that at time t, the video frame S t The results after convolution and LSTM processing are used to detect and track the dynamic changes of key anatomical structures in real time.
[0251] Embodiments further integrate computer vision techniques, including instance segmentation and object detection, by identifying and predicting the location and shape of key structures via the following integral equations:
[0252]
[0253] Where R(x,y) represents the response intensity of the position (x,y), which represents the possibility that the pixel at this position belongs to the key structure; M is the mask operation matrix; V ij (X) is the feature map of image X at position (i, j) after the convolution layer; c is the normalization constant; N is the size of the mask.
[0254] During the operation, the system extracts a 256×256 pixel image from the surgical video and obtains the feature map V after processing by the convolution layer. ij (X). The mask operation matrix M is used to identify specific regions. Set N = 5 and the normalization constant c = 1. ij (X) and the mask operation matrix M are substituted into the integral equation for calculation. Taking a certain position (x, y) in the image as an example, set V ij Some values of (X) are as follows:
[0255] V 00 (X) = 0.5
[0256] V 01 (X) = 0.7
[0257] V 10 (X) = 0.6
[0258] V 11 (X) = 0.8
[0259] Set the value of the mask operation matrix M to 1:
[0260]
[0261] First calculate the inner integral:
[0262] When i=0,j=0:
[0263] log(1+e 0.5-1 )=log(1+e -0.5 )≈log 1.6065≈0.475
[0264] When i=0,j=1:
[0265] log(1+e 0.7-1 )=log(1+e -0.3 )≈log 1.3499≈0.300
[0266] When i=1,j=0:
[0267] log(1+e 0.6-1 )=log(1+e -0.4 )≈log 1.4918≈0.400
[0268] When i=1,j=1:
[0269] log(1+e 0.8-1 )=log(1+e -0.2 )≈log 1.2214≈0.200
[0270] Substituting these results into the integral equation and calculating the sum:
[0271]
[0272]
[0273] Calculate the integral result:
[0274] R(x,y)=1.375×(2.5-(-2.5))×(2.5-(-2.5))=1.375×5×5=34.375
[0275] The system extracts a frame of image from the surgical video and obtains the feature map V after processing through the convolution layer. ij (X). Then, the mask operation matrix M and the integral equation are used to calculate the response intensity R(x,y)=34.375 at the position (x,y). This result indicates that the pixel at this position is likely to belong to the key structure.
[0276] Embodiment 5:
[0277] Embodiment An intelligent camera control system is designed in laparoscopic cholecystectomy surgery. According to the key structure position data output by the neural network model, the angle and focal length of the camera are adjusted in real time, and the lighting conditions of the surgical area are adjusted. The angle θ and focal length φ of the camera are adjusted by the following formula:
[0278]
[0279] in:
[0280] S i (t) represents the position of the i-th anatomical structure at time t;
[0281] α(t), β i and γ(t) are the time-dependent adjustment coefficient and weight, respectively;
[0282] θ and φ are the angle and focal length of the camera.
[0283] During the operation, the system obtains real-time updated key anatomical structure position data S from the neural network model. i (t). The total time of camera adjustment is set to T = 10 seconds, and the time-related adjustment coefficients α(t) and γ(t) range from 0.1, 1.0 to -0.5, 0.5, respectively. The weight β i The range is 0.1, 1.0.
[0284] Assume that the anatomical structure position data S at a certain time t i (t) are: S1(t)=0.5; S2(t)=0.7; S3(t)=0.6.
[0285] The specific values of the time-related adjustment coefficients and weights are set as: α(t)=0.5; β1=0.4; β2=0.3; β3=0.3; γ(t)=0.2.
[0286] Substitute these data into the formula to calculate:
[0287]
[0288] First calculate the inner sum:
[0289]
[0290] Compute the gradient:
[0291]
[0292] Substituting into the formula:
[0293]
[0294] Calculate the integral:
[0295] C(θ,φ)=0.5·0.79·10=3.95
[0296] The embodiment designs an intelligent lighting adjustment system, which uses a neural network to evaluate the impact of current lighting conditions on image quality, and optimizes the lighting conditions of the surgical area by adjusting the light intensity. The lighting intensity at position (x, y) is calculated by the following formula:
[0297]
[0298] in:
[0299] L(x,y) represents the lighting intensity at position (x,y);
[0300] κ is the data fed back by the light sensor;
[0301] η(x,y) is the brightness evaluation of the current pixel output by the neural network;
[0302] N is the region size, and its value range is (3,7).
[0303] During the operation, the system obtains the data of the current lighting conditions from the light sensor and evaluates the brightness of the current pixel through the neural network. Set N = 5, that is, the range of x and y coordinates is -2 to 2.
[0304] The specific values of the light sensor feedback data κ and the brightness evaluation η at a certain position (x, y) are set as follows:
[0305] κ(x-2,y-2)=0.8
[0306] κ(x-1,y-1)=0.7
[0307] κ(x,y)=0.9
[0308] η(x,y)=0.6
[0309] Substitute these data into the formula to calculate:
[0310]
[0311] When (i,j)=(-2,-2):
[0312] κ(x-2,y-2)·log(1+e η(x,y) )=0.8·log(1+e 0.6 )=0.8·log1.822≈0.8·0.6=0.48
[0313] When (i,j)=(-1,-1):
[0314] κ(x-1,y-1)·log(1+e η(x,y) )=0.7·log(1+e 0.6 )=0.7·log1.822≈0.7·0.6=0.42
[0315] When (i,j)=(0,0):
[0316] κ(x,y)·log(1+e η(x,y) )=0.9·log(1+e 0.6 )=0.9·log1.822≈0.9·0.6=0.54
[0317] Substituting these results into the integral equation and calculating the sum:
[0318]
[0319] Calculate the integral result:
[0320] L(x,y)=(0.48+0.42+0.54)×(2-(-2))×(2-(-2))=1.44×4×4=23.04
[0321] At position (x, y), the system calculated the lighting intensity L(x, y) = 23.04. This result is used to adjust the light intensity in the surgical area to ensure the optimal surgical field of view.
[0322] Embodiment 6:
[0323] Combined with Figure 4 In the embodiment, a method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network is used to design and implement a real-time feedback system, which uses a multimodal input device to collect the doctor's instant feedback on the camera viewing angle and focal length adjustment during the operation, and processes the feedback through the following formula:
[0324]
[0325] in:
[0326] θ(t) and φ(t) represent the angle and focal length of the camera at time t, respectively;
[0327] f(t,θ,φ) is an evaluation function, which represents the evaluation of the effect of adjusting the camera viewing angle and focal length at time t.
[0328] During the operation, the doctor provides instant feedback on the camera angle and focal length through the touch screen and voice command input device. The system records the camera angle and focal length adjustment data at multiple time points during the operation. The total adjustment time is set to T = 10 seconds.
[0329] Set the camera angle and focal length data as follows:
[0330] θ(1)=30°,φ(1)=50mm;
[0331] θ(5)=45°,φ(5)=60mm;
[0332] θ(10)=40°,φ(10)=55mm.
[0333] Substitute these data into the formula for calculation and set the evaluation function:
[0334] f(t,θ,φ)=α(t)·(θ(t)-θ desired )+β(t)·(φ(t)-φ desired )
[0335] where θ desired and φ desired are the desired angle and focal length, α(t) and β(t) are time-related adjustment coefficients, and their value range is 0.1, 1.0.
[0336] Set the desired camera angle and focal length to θ respectively desired =35° and φ desired =55mm, time-related adjustment coefficients α(t)=0.5 and β(t)=0.3.
[0337] The specific calculations at time points t=1, t=5 and t=10 are as follows:
[0338] When t=1:
[0339] f(1,θ(1),φ(1))=0.5·(30-35)+0.3·(50-55)=0.5·(-5)+0.3·(-5)=-2.5-1.5
[0340] =-4
[0341] When t=5:
[0342] f(5,θ(5),φ(5))=0.5·(45-35)+0.3·(60-55)=0.5·10+0.3·5=5+1.5=6.5
[0343] When t=10:
[0344] f(10,θ(10),φ(10))=0.5·(40-35)+0.3·(55-55)=0.5·5+0.3·0=2.5+0=2.5
[0345] Substituting these results into the integral equation and calculating the sum:
[0346]
[0347] Then, the embodiment develops a dynamic adjustment algorithm based on reinforcement learning, adjusts the model parameters in real time according to the feedback and reward information during the operation, and updates the parameters through the following formula:
[0348]
[0349] in:
[0350] Δp is the change in the parameter;
[0351] γ is the learning rate, ranging from 0.01 to 0.1;
[0352] r k is the reward obtained at step k;
[0353] π k (a k |s k ; θ) is in state s k Next select action a k The strategy probability depends on the parameter θ.
[0354] During the operation, the system records the doctor's feedback and the effect of camera adjustment at each time step. The system uses this data as reward information for the reinforcement learning algorithm. The learning rate γ is set to 0.05.
[0355] Set at a certain point in time, the system records the reward r k and status k The strategy probability π under k (a k |s k ; θ) is as follows:
[0356] r1=1.0,π1(a1|s1;θ)=0.8
[0357] r2=0.5,π2(a2|s2;θ)=0.7
[0358] r3=0.8,π3(a3|s3;θ)=0.9
[0359] Substitute these data into the formula to calculate:
[0360]
[0361] Compute the gradient at each step:
[0362] When k=1:
[0363]
[0364] When k=2:
[0365]
[0366] When k=3:
[0367]
[0368] Substitute these results into the parameter update formula and calculate the sum:
[0369] Δp=0.05·(1.0·1.25+0.5·1.43+0.8·1.11)
[0370] Δp=0.05·(1.25+0.715+0.888)
[0371] Δp=0.05·2.853=0.14265
[0372] By calculating the parameter change Δp=0.14265, the system can adjust the parameters of the neural network prediction and camera control system in real time to ensure the optimal state of the surgical field of view.
[0373] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.
Claims
1. A neural network-based method for optimizing the surgical field during laparoscopic cholecystectomy, characterized by The following steps are involved: First, laparoscopic surgery video and image data were collected and preprocessed. Multiple doctors participated in the majority opinion analysis to annotate key structures of the gallbladder, bile duct, and liver in the videos and images. Image processing techniques were then used to perform noise reduction, contrast enhancement, and edge detection. The majority opinion principle is a cloud-based collaborative platform that allows multiple doctors to remotely log in and annotate the same surgical video and image data in real time. The opinions of multiple doctors are integrated by implementing a majority opinion decision-making algorithm. Each time a doctor completes a single annotation, the computer system counts the annotation results of all doctors and uses the majority opinion as the final structural positioning. Wherein, the edge detection is used to identify and track the edges of anatomical structures during surgery; Then, a deep learning model is designed and trained. Based on an improved convolutional neural network architecture, a specific attention mechanism is added to focus on feature learning in key structural areas. The model parameters are optimized using labeled data for training and cross-validation. Finally, a surgical field optimization algorithm was developed, which used a trained neural network model to analyze surgical videos in real time, predict the position changes of key anatomical structures, and adjust the camera angle, focal length, and lighting conditions in real time based on the prediction results. At the same time, a feedback system was introduced to make dynamic adjustments based on the doctor's operational feedback during the operation.
2. The method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network according to claim 1, characterized in that The process of building the cloud-based collaboration platform is as follows: S1. Using a cloud-based collaborative platform, multiple doctors remotely logged in and annotated the gallbladder, bile duct, and liver anatomical structures in the surgical video in real time, using the decision formula: Among them D i represents the labeled dataset of the i-th anatomical structure, x in is the annotation result of the nth doctor on the i-th anatomical structure, w n is the weight based on the historical accuracy of doctors’ annotations, μ i and σ i The dataset D i The mean and standard deviation of are used to calculate the weighted average of the annotations; S2. And train the neural network model based on the obtained labeled data, using the adjustment equation: Where S is the surgical scene data, V is the predetermined field of view quality parameter, α and β are dynamically adjusted coefficients, and H(S t ) is the visual field quality evaluation function for the surgical scene S at time t.
3. The method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network according to claim 1, characterized in that The edge detection implementation method includes: first, using a generative adversarial network (GAN) and a deep convolutional neural network (DCNN) to train a model using the formula: L GAN (E,θ)6∫ V log(1+e -(θ·E(x)) )dx Where x represents the pixel in the image, E represents the edge feature vector extracted from the surgical video, θ is the model parameter, and V is the total visualization area; Next, a real-time edge tracking model based on the long short-term memory network (LSTM) is applied, using the formula: Calculate where P(t) represents the edge position prediction at time t and δ(t) is the time-dependent adjustment factor; Finally, combining data from CT and MRI imaging technologies through fusion algorithms: Where x represents the pixel in the image, C and M represent the image data of CT and MRI respectively, and λ is the fusion coefficient.
4. The method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network according to claim 1, characterized in that The deep learning model construction process is as follows: First, the network architecture is used to adjust the special design and parameters of the convolution kernel: Where S represents the surgical image data, θ and μk represent the learning parameters and the center of the kth convolution kernel, respectively, and K is the total number of convolution kernels; Next, implement the loss function: Where Y is the actual annotated target area for surgical field optimization, P is the predicted area, yn and pn are the values of Y and P at the nth pixel respectively, and N is the total number of pixels in the image; Finally, a cross-validation method is used: Where V m represents the actual output in the mth cross validation, is the model prediction output, and M is the number of verifications.
5. The method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network according to claim 1, characterized in that The surgical field optimization algorithm construction method includes: S1. Develop a deep learning-based dynamic tracking algorithm that uses recurrent neural networks (RNNs) and long short-term memory (LSTMs) to process surgical video sequence data, detect and track dynamic changes in key anatomical structures in real time, and integrate computer vision techniques, including instance segmentation and object detection, to identify and predict the location and shape of key structures. S2. Then, an intelligent camera control system is designed to adjust the angle and focal length of the camera and the lighting conditions of the surgical area according to the key structure position data obtained from the neural network model; S3. Build a real-time feedback system and develop a dynamic adjustment algorithm. When the doctor provides immediate feedback on the camera adjustment through the input device during the operation, the neural network prediction and camera control system parameters are adjusted in real time based on the doctor's feedback. Reinforcement learning is used to learn and adapt to the doctor's operating habits and preferences in real time.
6. The method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network according to claim 1, characterized in that The dynamic tracking algorithm implementation process is as follows: First, we use a composite model combining convolutional neural networks (CNN) and long short-term memory (LSTM) networks to process surgical video sequence data using the equation: Among them S t represents the video frame at time t, H t-1 is the hidden state of LSTM at the previous moment, W k and B k is the weight and bias of the kth convolution kernel, K is the number of convolution kernels, * represents the convolution operation, and detects and tracks the dynamic changes of key anatomical structures in real time; It also integrates computer vision techniques, including instance segmentation and target detection, to identify and predict the location and shape of key structures through integral equations: Where R(x, y) represents the response intensity of the position (x, y), which characterizes the possibility that the pixel at this position belongs to the key structure, M is the mask operation matrix, and V ij (X) is the feature map of image X at position (i, j) after the convolution layer, c is the normalization constant, and N is the size of the mask.
7. The method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network according to claim 1, characterized in that The intelligent camera control system construction method includes: S1. First, the intelligent camera control system uses the anatomical structure position data output by the neural network model through the formula: Adjust the camera angle θ and focal length φ, where S i (t) represents the position of the anatomical structure i at time t, α(t), β i and γ(t) are the time-related adjustment coefficient and weight, respectively; S2. The intelligent lighting adjustment system evaluates the impact of current lighting conditions on image quality based on a neural network, using the formula: Where L(x, y) represents the lighting intensity at position (x, y), k is the data fed back by the light sensor, and η(x, y) is the brightness evaluation of the current pixel output by the neural network. The lighting conditions are optimized by adjusting the light intensity in the N area.
8. The method for optimizing the surgical field during laparoscopic cholecystectomy based on a neural network according to claim 1, characterized in that The real-time feedback system construction process is as follows: First, we designed and implemented a real-time feedback system. This system uses multimodal input devices to collect the doctor's immediate feedback on the camera's viewing angle and focal length during surgery. The feedback is then processed using the following formula: Where θ(t) and φ(t) represent the angle and focal length of the camera at time t, respectively, and f(t, θ, φ) is the evaluation function; Next, we developed a dynamic adjustment algorithm based on reinforcement learning to adjust the neural network predictions and the parameters of the camera control system, and used the formula to update the parameters: Where Δp is the amount of change in the parameter, γ is the learning rate, and r k is the reward obtained at step k, π k (a k |s k ;θ) is in state s k Next, select action a k The policy probability depends on the parameter θ.
Citation Information
Patent Citations
Surgery data processing method
CN109949880A
Focus tracking method under digestive endoscope based on sequential feature learning
CN111915573A
Deep learning-based spine endoscopic surgery real-time auxiliary method and system
CN116687561A
Breast cancer early screening diagnosis method based on attention mechanism enhanced CNN (Convolutional Neural Network)
CN117898767A
Medical image recognition method and system based on enhanced annotation and deep learning
CN118096721A
Cited By
Continuous grading and scoring method based on safety-critical surgical field
CN119360283A
Operation field space reconstruction method and system based on big data analysis
CN121353599A
