A surgical instrument needle tip trajectory prediction method

Abstract

The application belongs to the technical field of deep learning, and specifically discloses a surgical instrument needle tip trajectory prediction method, which comprises the following steps: constructing a surgical instrument motion trajectory dataset by using a key motion image sequence of a surgical instrument; dividing the key motion image sequence of the surgical instrument into a first image sequence and a second image sequence with the same frame number according to time sequence; constructing a generative adversarial network model by taking a 3D convolutional neural network as a generator and a multi-scale discriminator as a discriminator; and fixing the parameters of the generator and the discriminator after training the generative adversarial network model based on the mapping relationship between the first image sequence and the second image sequence. The real-time motion trajectory image sequence of the surgical instrument is obtained, the real-time motion trajectory image sequence is input into the generative adversarial network model to generate a predicted image sequence, and the prediction result of the needle tip trajectory of the surgical instrument is obtained based on the predicted image sequence. The application realizes high-precision prediction.

Description

Technical Field

[0001] This invention belongs to the field of deep learning technology and relates to a method for predicting the trajectory of a surgical instrument needle tip. Background Technology

[0002] Currently, the adoption rate of surgical navigation systems is gradually increasing globally, indicating a very optimistic market prospect. During high-precision surgical procedures, surgical navigation systems need to track the position of surgical instrument tips in real time to prevent misoperation. As the complexity of surgical procedures continues to increase, the precise control and positioning of surgical instrument tips has become one of the key factors in improving surgical success rates.

[0003] Current methods for predicting the trajectory of surgical instrument needle tips mainly include geometric modeling, physics-based simulation, and adaptive control strategies. Geometric modeling predicts the trajectory by establishing a geometric model of the interaction between the needle tip and tissue. Physics-based simulation uses physical laws to simulate the behavior of the needle tip in tissue. Adaptive control strategies dynamically adjust the needle tip's movement path through feedback mechanisms. These traditional methods have promoted the development of needle tip trajectory prediction to some extent. However, they are poor at processing three-dimensional space, causing blurring, artifacts, and even pixel loss in the predicted frames. As a result, there are still significant limitations in prediction efficiency and accuracy, which require further improvement.

[0004] In summary, how to predict the trajectory of surgical instrument needle tips efficiently and with high accuracy remains a challenge. Summary of the Invention

[0005] The purpose of this invention is to provide a method for predicting the trajectory of surgical instrument needle tips, which improves the accuracy and detail of the prediction, and achieves efficient and high-precision prediction.

[0006] To achieve the above objectives, the specific technical solution provided by the present invention is as follows:

[0007] A method for predicting the trajectory of a surgical instrument needle tip includes the following steps:

[0008] The historical motion trajectory of the surgical instrument is obtained, which includes all motion forms of the surgical instrument. Key motion image sequences of the surgical instrument are extracted from each motion trajectory to construct a surgical instrument motion trajectory dataset.

[0009] The key motion image sequences of surgical instruments in the surgical instrument motion trajectory dataset are preprocessed, and the preprocessed key motion image sequences of surgical instruments are divided into a first image sequence and a second image sequence with the same number of frames according to the time sequence.

[0010] A generative adversarial network model is constructed using a 3D convolutional neural network as the generator and a multi-scale discriminator as the discriminator. After training the generative adversarial network model based on the mapping relationship between the first image sequence and the second image sequence, the prediction accuracy of the generative adversarial network model is evaluated. When the prediction accuracy of the generative adversarial network model reaches a preset accuracy, the parameters of the generator and the discriminator are fixed.

[0011] A real-time motion trajectory image sequence of the surgical instrument is acquired, and the real-time motion trajectory image sequence is input into the generator to generate a prediction image sequence. Based on the prediction image sequence, the prediction result of the surgical instrument needle tip trajectory is obtained.

[0012] Preferably, the preprocessing of key motion image sequences of surgical instruments in the surgical instrument motion trajectory dataset includes the following steps:

[0013] Remove noise from each frame of a sequence of key motion images of surgical instruments;

[0014] The pixel values ​​of each frame in the key motion image sequence of surgical instruments are normalized, and the pixel values ​​are mapped from [0, 255] to [0, 1].

[0015] Preferably, training the generative adversarial network model based on the mapping relationship between the first image sequence and the second image sequence includes the following steps:

[0016] The surgical instrument motion trajectory dataset was divided into a training set and a test set.

[0017] A complete multi-frame image sequence consisting of the first image sequence and the second image sequence is selected from the training set as the input of the discriminator.

[0018] The first image sequence in the training set is fed into the generator to generate the predicted image sequence. The generator loss function is calculated and the generator is updated.

[0019] The generated predicted image sequence is input into the discriminator. The discriminator checks the temporal consistency between the generated predicted image sequence and the first image sequence, calculates the discriminator loss function, and updates the discriminator until the discriminator loss function is minimized.

[0020] Preferably, evaluating the prediction accuracy of the generative adversarial network model includes the following steps:

[0021] The first image sequence in the test set is taken as the input to the generator, which then generates a predicted image sequence.

[0022] A complete multi-frame image sequence consisting of the first and second image sequences in the test set is obtained as input to the discriminator to evaluate the difference between the predicted image and the real image, and to verify the error between the three-dimensional coordinates of the surgical instrument needle tip in the predicted image and the three-dimensional coordinates of the surgical instrument needle tip in the real image. The prediction accuracy of the generative adversarial network model is evaluated by comprehensively considering the above differences and errors.

[0023] Preferably, the following formula is used to evaluate the difference between the predicted image and the real image.

[0024] ,

[0025] in, This indicates the frame number of the image in the predicted image sequence. For the first The real images corresponding to each input image sequence For the first The predicted image corresponds to each input image sequence, and MSE (mean squared error) is used to evaluate the difference between the predicted image and the real image. It is a natural number. The value of is [1, ];

[0026] The lower the mean square error (MSE) value, the higher the prediction accuracy.

[0027] Preferably, verifying the error between the three-dimensional coordinates of the surgical instrument tip in the predicted image and the three-dimensional coordinates of the surgical instrument tip in the real image includes the following steps:

[0028] The three-dimensional coordinates of the surgical instrument needle tip in the predicted image and the three-dimensional coordinates of the surgical instrument needle tip in the real image are extracted respectively.

[0029] The error between the three-dimensional coordinates of the surgical instrument needle tip in the predicted image and the three-dimensional coordinates of the surgical instrument needle tip in the real image is calculated using the following formula.

[0030] ,

[0031] in, This indicates the frame number of the image in the predicted image sequence. The three-dimensional coordinates of the needle tip in the real image. To predict the 3D coordinates of the needle tip in the image, It's an error. It is a natural number. The value of is [1, ];

[0032] When error A value less than 1 indicates that the surgical instrument needle tip trajectory has reached the predicted accuracy.

[0033] Preferably, obtaining the predicted trajectory of the surgical instrument needle tip based on the predicted image sequence includes the following steps:

[0034] Based on the predicted image sequence, the three-dimensional coordinates of three markers on the surgical instrument were determined;

[0035] The three-dimensional coordinates of the surgical instrument needle tip are determined based on the three-dimensional coordinates of three markers on the surgical instrument, and the predicted trajectory of the surgical instrument needle tip is obtained.

[0036] Preferably, determining the three-dimensional coordinates of three markers on the surgical instrument based on the predicted image sequence includes the following steps:

[0037] Based on the predicted image sequence generated by the generative adversarial network model, the edge contours of three markers on the surgical instrument are obtained by the Canny edge detection algorithm;

[0038] Randomly select three pixels on the edge contour of the marker, and determine the coordinates of the marker's two-dimensional center point according to the following formula. ,

[0039] ,

[0040] in, , and These are the coordinates of three non-collinear points on the edge contour of the marker. These are the two-dimensional coordinates of the center of the marker.

[0041] Using a region similarity matching algorithm, markers in images acquired by the two cameras of a binocular vision measurement device are matched.

[0042] The three-dimensional coordinates of the marker are determined by the following formula:

[0043] ,

[0044] in, , It is the center distance between the two cameras of a binocular vision measurement device. It is the focal length of a binocular vision measurement device. , These are the two-dimensional coordinates of the marker in the images captured by the two cameras. These are the three-dimensional coordinates of the marker.

[0045] Preferably, the three-dimensional coordinates of the surgical instrument needle tip are determined based on the following formula:

[0046] ,

[0047] in, ,

[0048] ,

[0049] ,

[0050] These are the coordinates of the surgical instrument needle tip in the surgical instrument coordinate system. These are the three-dimensional coordinates of the surgical instrument needle tip. , and These are the coordinates of marker one, marker two, and marker three in three-dimensional space. It is a translation vector. It is a rotation vector. These are the unit vectors in the x, y, and z directions of the surgical instrument coordinate system.

[0051] Compared with existing technologies, the present invention provides a method for predicting the trajectory of a surgical instrument tip. This method acquires the historical motion trajectory of the surgical instrument, extracts key motion image sequences from each trajectory to construct a surgical instrument motion trajectory dataset, preprocesses the key motion image sequences, and divides them into a first image sequence and a second image sequence with the same number of frames according to time sequence. A generative adversarial network (GAN) model is constructed using a 3D convolutional neural network as the generator and a multi-scale discriminator as the discriminator. The GAN model is trained based on the mapping relationship between the first and second image sequences, and its prediction accuracy is evaluated. When the prediction accuracy reaches a preset accuracy, the parameters of the generator and discriminator are fixed. The method also acquires real-time motion trajectory image sequences of the surgical instrument and inputs them into the generator to generate prediction image sequences. The prediction result of the surgical instrument tip trajectory is obtained based on these prediction image sequences. This invention innovatively applies a generative adversarial network (GAN) model to the prediction of surgical instrument needle tip trajectories. In this GAN model, the generator is a 3D convolutional neural network, and the discriminator is a multi-scale discriminator. The generator attempts to generate realistic predicted images to deceive the discriminator, while the discriminator tries to distinguish between the real and predicted images. This adversarial process prompts the generator to continuously improve the realism of the generated data, further enhancing the accuracy and detail of surgical instrument needle tip trajectory prediction. This achieves efficient and high-precision prediction, demonstrating strong practicality and making it worthy of widespread application. Attached Figure Description

[0052] Figure 1 This is a flowchart of the present invention.

[0053] Figure 2 This is a simulation scenario diagram of the present invention.

[0054] Figure 3 This is a partial view of the present invention.

[0055] Figure 4 This is the generative adversarial network model used in this invention.

[0056] Figure label:

[0057] 1. Three-axis sliding guide test stand; 2. Surgical instruments; 3. Binocular vision measurement equipment; 4. Computer; 5. Marker 1; 6. Marker 2; 7. Marker 3; 8. Surgical instrument needle tip. Detailed Implementation

[0058] To accurately predict the trajectory of surgical instrument needle tips and help surgical robots or doctors perform surgical operations more precisely, this invention addresses the technical problem that existing prediction methods have poor performance in three-dimensional space processing, resulting in blurred prediction frames, artifacts, and even pixel loss, thus significantly limiting prediction efficiency and accuracy.

[0059] To enable those skilled in the art to better understand and implement the technical solutions of this invention, the following will be described in conjunction with the appendix. Figure 1 To the attached Figure 4 The technical solutions in this invention will be described clearly and in detail.

[0060] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0061] Furthermore, it should be further explained that in the description of the embodiments of the present invention, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present invention, "multiple" means two or more.

[0062] The terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of that feature, and in the description of this invention, unless otherwise stated, "multiple" means two or more.

[0063] Example 1

[0064] A flowchart for predicting the trajectory of surgical instrument needle tips is shown below. Figure 1 As shown, the specific steps include:

[0065] The historical motion trajectory of surgical instrument 2 is obtained, which includes all motion forms of surgical instrument 2. Key motion image sequences of surgical instrument are extracted from each motion trajectory to construct a surgical instrument motion trajectory dataset.

[0066] The key motion image sequences of surgical instruments in the surgical instrument motion trajectory dataset are preprocessed, and the preprocessed key motion image sequences of surgical instruments are divided into a first image sequence and a second image sequence with the same number of frames according to the time sequence.

[0067] A generative adversarial network (GAN) model is constructed using a 3D convolutional neural network as the generator and a multi-scale discriminator as the discriminator. After training the GAN model based on the mapping relationship between the first image sequence and the second image sequence, the prediction accuracy of the GAN model is evaluated. When the prediction accuracy of the GAN model reaches a preset accuracy, the parameters of the generator and the discriminator are fixed.

[0068] The real-time motion trajectory image sequence of surgical instrument 2 is acquired, and the real-time motion trajectory image sequence is input into the generator to generate a prediction image sequence. Based on the prediction image sequence, the prediction result of the needle tip trajectory of the surgical instrument is obtained.

[0069] The detailed implementation plan for the main steps is as follows:

[0070] 1. Construct a dataset of surgical instrument motion trajectories

[0071] Figure 2 It is a simulated scene diagram. The required devices in the scene diagram include surgical instruments 2, binocular vision measurement equipment 3, three-axis sliding guide test platform 1, and computer 4. Among them, surgical instruments 2, binocular vision measurement equipment 3, and three-axis sliding guide test platform 1 are all electrically connected to computer 4.

[0072] To obtain the historical motion trajectory of surgical instrument 2, surgical instrument 2 is first fixed on the three-axis sliding guide test table 1, so that it faces the binocular vision measurement device 3, as shown below. Figure 2 As shown. Then, the surgical instrument 2 is slowly moved by controlling the three-axis sliding guide test stage 1, while simultaneously using a binocular vision measurement device 3 to acquire video of the movement of the surgical instrument 2. These videos must cover the movement of the surgical instrument 2 along the three-axis sliding guide test stage 1. , , All motion forms in three directions. Special attention must be paid to lighting conditions during video acquisition to ensure clear and distortion-free images. Ultimately, 1000 video segments were acquired, and a sequence of 14 consecutive frames of key surgical instrument motion images was extracted from each segment to construct a surgical instrument motion trajectory dataset.

[0073] 2. Preprocessing of key motion image sequences of surgical instruments

[0074] Since the video sequence of the historical motion trajectory of surgical instrument 2 is available, and noise is inevitably present in the video, in order to prevent the noise from affecting the generative adversarial network model, it is necessary to preprocess the key motion image sequence of the surgical instrument to improve the image quality. Specifically, Kalman filtering technology is used to denoise the image, thereby more accurately estimating the true state of each frame and reducing noise interference.

[0075] Next, the image pixel values ​​are normalized, mapping the pixel values ​​from [0, 255] to [0, 1], in order to improve the stability and convergence speed of the network model training.

[0076] 3. Construct a generative adversarial network model

[0077] Specifically, in practical applications, this invention predicts the trajectory of the surgical instrument 2 in the next 7 frames by analyzing the historical motion data of the surgical instrument 2 during the surgical process, especially the trajectory of the surgical instrument in the first 7 frames. The core idea of ​​this method is to utilize the generative capability and adversarial learning mechanism of the generative adversarial network model, combined with the binocular vision measurement device 3, to accurately predict the future motion path of the surgical instrument needle tip 8.

[0078] Generative Adversarial Networks (GANs) are deep learning models that generate data through adversarial training. A GAN consists of two main parts: a generator and a discriminator. The generator attempts to produce realistic predicted images to fool the discriminator, while the discriminator tries to distinguish between real and predicted images. This adversarial process drives the generator to continuously improve the accuracy of its predicted images. In the prediction of the motion trajectory of surgical instrument 2, GANs can be used to generate realistic motion trajectories of surgical instrument 2, which can then be used for simulation, training, or decision support during surgery. Specifically, the design and training of the generator and discriminator are as follows:

[0079] The generator employs a 3D Convolutional Neural Network (3D-CNN), whose advantage lies in its ability to capture the spatial and temporal motion patterns of surgical instrument 2. 3D-CNN, through convolutional operations in both spatial and temporal dimensions, effectively processes and generates complex spatiotemporal data. The generator's main structure includes an input layer, 3D convolutional layers, an upsampling layer, and an output layer. The input layer receives the motion trajectory of surgical instrument 2 from the previous time step. The 3D convolutional layers extract spatiotemporal features; by using multiple 3D convolutional layers, the complex dynamics of surgical instrument 2's motion can be captured. The upsampling layer upsamples the low-dimensional feature map to the target dimension using methods such as deconvolution or interpolation, generating the motion trajectory of surgical instrument 2. The output layer outputs the generated motion trajectory data of surgical instrument 2.

[0080] The discriminator employs a multi-scale discriminator design. This design enables the evaluation of generated data at different scales, thereby improving the discriminator's sensitivity to details and its grasp of the overall structure. The main structure of the multi-scale discriminator includes an input layer, multi-scale convolutional layers, and fully connected layers. The input layer receives real or generated surgical instrument motion trajectory data. The multi-scale convolutional layers consist of multiple convolutional layers at different scales, such as different kernel sizes and stride settings, to extract features at different scales. Each scale convolutional layer is responsible for capturing different levels of motion features. The fully connected layers fuse and discriminate the features extracted by the convolutional layers, determining whether the output data is real or generated.

[0081] When constructing the generative adversarial network model, each frame in the preprocessed 14-frame sequence of key motion images of surgical instruments is divided into two parts with the same number of frames according to the time sequence. The first 7 frames are used as the first image sequence for input, and the last 7 frames are used as the second image sequence for target output.

[0082] The idea behind generative adversarial networks (GANs) originates from game theory and consists of a generator and a discriminator. The model continuously engages in adversarial games until it reaches a Nash equilibrium, thereby generating high-quality samples. The generator simulates the distribution of a second image sequence to produce realistic image samples, while the discriminator distinguishes between real and generated trajectories. The generator includes an input layer, a 3D convolutional layer, an upsampling layer, and an output layer. The input layer receives the input image sequence from the previous time step. The 3D convolutional layer extracts low-dimensional spatiotemporal features from the input image sequence. The upsampling layer upsamples these low-dimensional features to the target dimension, resulting in the output image sequence. The output layer outputs the generated image sequence. The discriminator employs a multi-scale discriminator, judging the generator's output image sequence at different resolutions to obtain the probability of determining whether it is a true or false image.

[0083] Generative adversarial network models such as Figure 4 As shown, the generator's main function is to simulate the distribution of the second image sequence to generate realistic image samples. The generator uses random noise... As input, the output image is generated through a neural network. The generator learns from real image data continuously. The system attempts to generate samples that are highly similar to real image data by analyzing the characteristics and distribution of these images, thereby deceiving the discriminator and causing it to misclassify the generated images as real images. It is a binary classifier used to determine whether an input image is a real image or a predicted image. When the input is a real image... At that time, the output of the discriminator It should be close to 1; while when the input is the predicted image hour, The output value should be close to 0. The two processors feed back to each other through the loss function, continuously adjusting their parameters. In adversarial learning, the generator and discriminator mutually boost each other, eventually approaching a Nash equilibrium. At this point, the predicted image generated by the generator is almost indistinguishable from the real image, and the discriminator also struggles to differentiate between them.

[0084] Since each frame in the key motion image sequence of surgical instruments contains temporal information, a 3D convolutional neural network is used in the generator to effectively extract the spatiotemporal correlation features between image frames. 3D convolution captures the dynamic changes and spatiotemporal features of the image sequence by simultaneously performing convolution operations on the width and height of the spatial image and the relationship between temporal frames. When processing each frame in the first 7 frames of the key motion image sequence of surgical instruments, the dimensions of the input tensor are [N, C, T=7, H, W], where N represents the batch size, i.e., the number of samples input into the network at one time; C represents the number of channels; T represents the number of frames, with T=7 indicating a fixed number of 7 frames; and H and W represent the height and width of the image. First, the network uses a 3×3×3 3D convolutional kernel to extract low-level spatiotemporal features of the image sequence. This convolutional kernel can establish local correlations between temporal frames and spatial locations. Subsequently, by stacking multiple layers of 3D convolutional layers, higher-level spatiotemporal features are extracted progressively. To enable the network to capture more complex features while reducing computational overhead and mitigating overfitting, a ReLU activation function is added after each convolutional layer. Furthermore, to alleviate potential degradation and vanishing gradient problems in deep network structures, a residual module is added every three convolutional layers to improve training efficiency and performance. Finally, to progressively restore the extracted spatiotemporal features to the target output resolution, the network upsamples the features using 3D transposed convolutions while gradually reducing the number of channels in the feature maps. Through this process, the network's final output tensor has dimensions consistent with the target, i.e., [N, C, T=7, H, W], thus generating a satisfactory image sequence.

[0085] To better perceive local and global information in image sequences, the discriminator employs a multi-scale discriminator. By discriminating images at different resolutions, the multi-scale discriminator effectively captures spatiotemporal features at different scales. The probability of determining whether an image is true or false is obtained by discriminating the generator's output image sequence at different resolutions, including the following steps:

[0086] The generator's output image sequence is downsampled at multiple resolutions to generate image sequences at different resolutions, including the original resolution, 1 / 2 resolution, 1 / 4 resolution, and 1 / 8 resolution.

[0087] Subsequently, for each image sequence at each resolution, a 3D convolutional neural network is used to extract its spatiotemporal features, and a true / false distinction is made independently at each resolution, outputting the corresponding probability.

[0088] Finally, by averaging the output probabilities of all resolutions, an overall true / false probability is obtained. If the overall true / false probability of the output is close to 1, it is determined to be a real image; if the overall true / false probability of the output is close to 0, it is determined to be a predicted image.

[0089] In this way, the multi-scale discriminator can more comprehensively analyze the features of image sequences at different scales, thereby improving the robustness and discrimination ability of the discriminator.

[0090] One of the key aspects of generative adversarial network (GAN) models lies in the rational design of the loss function. In this invention, both the generator and discriminator employ binary cross-entropy loss as the optimization objective. The generator loss function is defined as:

[0091] ,

[0092] in, Indicates the discriminator, Represents a generator. For the input image, The image generated by the generator. This represents the probability that the discriminator's predicted image output is a real image. The generator continuously optimizes the quality of the generated images by minimizing the loss function, gradually making them approximate the real image.

[0093] Ultimately, the goal of the generator is to make The value approaches 1, making it difficult for the discriminator to distinguish between the predicted image and the real image, thus increasing the difficulty of the discriminator's judgment.

[0094] The discriminator loss function is defined as:

[0095] ,

[0096] The discriminator loss function is divided into two parts, the first part being... The second image sequence is expected in this part. It was judged to be true, that is The first part is 1; the second part is The image to be predicted in this section It was judged to be counterfeit, that is It is 0.

[0097] Therefore, the objective of the discriminator loss function is to maximize the predicted second image sequence output as close to 1 as possible, and minimize the predicted image output as close to 0 as possible. This is achieved by minimizing the discriminator loss function. The discriminator can learn better how to distinguish between real and predicted images.

[0098] After the generative adversarial network model is built, training of the model begins.

[0099] First, the dataset was divided into training and testing sets in an 8:2 ratio, with 80% used for training and 20% for testing. Meanwhile, to accelerate the training process and improve model stability, the Adam optimizer was employed. This optimizer adaptively adjusts the learning rate, effectively mitigating potential gradient vanishing or exploding problems during training.

[0100] Furthermore, the use of the Adam optimizer can improve the training efficiency and prediction performance of the model, enabling it to converge faster and achieve better results. In generative adversarial network models, the generator and discriminator are trained alternately, and the specific training process is as follows:

[0101] First, 14 real images are input into the discriminator, and the discriminator is expected to output a probability close to 1 in order to accurately identify the real images.

[0102] Next, the first 7 frames of image sequence are fed into the generator, which generates the next 7 frames of image sequence. Then, the generator loss function is calculated and the generator is updated.

[0103] Subsequently, the generator-generated sequence of the last 7 frames is input into the discriminator. The discriminator evaluates the quality of the generated last 7 frames based on the complete 14 real frames, and checks the temporal consistency between the generated last 7 frames and the first 7 frames. The discriminator loss function is calculated, and the discriminator is updated.

[0104] By continuously training the discriminator and generator alternately, the generator gradually generates more realistic image sequences, while the discriminator gradually enhances its ability to distinguish the authenticity of images and the rationality of their temporal sequence.

[0105] When the discriminator output probability approaches 0.5 during training, it indicates that the discriminator can no longer distinguish between real and predicted images, and the training process ends.

[0106] The trained generative adversarial network (GAN) model is tested using a test set to evaluate its prediction accuracy. Training ends when the prediction accuracy of the GAN model meets the requirements.

[0107] Specifically, after the generative adversarial network model is trained, a sequence of 14 consecutive frames of key motion images of surgical instruments is randomly selected from the test set.

[0108] First, the first 7 frames of image sequence are input into the generator, which then generates the next 7 frames of predicted image sequence.

[0109] Simultaneously, a complete sequence of 14 frames of key motion images of surgical instruments is input into the discriminator to evaluate the differences between the generated last 7 frames and the real images.

[0110] After the generator completes the prediction of 7 frames of image sequence, the model's prediction accuracy is verified in two ways.

[0111] On the one hand, to fully evaluate the quality of the predicted image, the mean squared error (MSE) is used as an evaluation metric for prediction accuracy. MSE is a commonly used metric to measure the degree of difference between the predicted and actual values. The MSE is calculated using the following formula:

[0112] ,

[0113] in, Indicates the number of predicted image sequences. For the first The real images corresponding to each input image sequence For the first The predicted image corresponds to each input image sequence. MSE is the mean squared error, used to evaluate the difference between the predicted image and the real image.

[0114] when When the value is 7, the mean square error (MSE) is calculated as follows:

[0115] ,

[0116] Mean Squared Error (MSE) is used to evaluate the difference between the predicted image and the real image, and its value reflects the accuracy of the prediction result. The smaller the loss of pixel values, that is, the lower the MSE value, the closer the predicted image is to the real image, and the higher the prediction accuracy.

[0117] In this invention, when the mean square error (MSE) is less than 50, it indicates that the quality of the generated image has met the expected requirements.

[0118] On the other hand, in order to obtain the trajectory of the surgical instrument tip 8 in three-dimensional space and verify the prediction accuracy, the computer 4 extracts the three-dimensional coordinates of the surgical instrument tip 8 in the generated last 7 frame image sequence and the three-dimensional coordinates of the surgical instrument tip 8 in the real last 7 frame image sequence. The prediction accuracy of the model is evaluated by calculating the error between the two sets of three-dimensional coordinates.

[0119] To accurately determine the three-dimensional coordinates of the surgical instrument tip 8, three markers are set on the surgical instrument 2. The markers are optical positioning marker balls that can reflect infrared light. The relative positions of the three markers on the surgical instrument 2 and the surgical instrument tip 8 are determined. Therefore, based on the predicted image sequence and the binocular vision measurement device 3, the three-dimensional coordinates of the three markers on the surgical instrument 2 can be determined. Based on the three-dimensional coordinates of the three markers on the surgical instrument 2, the three-dimensional coordinates of the surgical instrument tip 8 can be determined, and the predicted result of the surgical instrument tip trajectory can be obtained.

[0120] Specifically, the calculation process for the three-dimensional coordinates of the surgical instrument needle tip 8 is as follows:

[0121] (1) Based on the predicted image sequence generated by the generative adversarial network model, the edge contours of the three markers on the surgical instrument 2 are obtained by the Canny edge detection algorithm.

[0122] Specifically, the markers are marker 5, marker 6, and marker 7, all of which are placed on surgical instrument 2. Marker 5, marker 6, and marker 7 appear as a circle in the image, and the Canny algorithm can detect the circular outline of the markers.

[0123] (2) Calculate the two-dimensional center coordinates of the marker based on the edge pixels on the edge contour.

[0124] The specific method is as follows: Randomly select three pixels on the edge contour of the marker, and let the coordinates of the three non-collinear points on the edge contour of the marker be... , and The two-dimensional center coordinates of the marker are determined according to the following formula. ,

[0125] ,

[0126] in, , and These are the coordinates of three non-collinear points on the edge contour of the marker. These are the two-dimensional coordinates of the center of the marker.

[0127] (3) By using the region similarity matching algorithm, the markers in the images obtained by the two cameras of the binocular vision measurement device 3 are accurately matched.

[0128] (4) Calculate the three-dimensional coordinates of the marker.

[0129] Let the two-dimensional coordinates of the marker in the images acquired by the two cameras be respectively , Since camera one and camera two are parallel in the binocular vision measurement device 3, then The three-dimensional coordinates of the marker are The three-dimensional coordinates of the marker are determined by the following formula:

[0130] ,

[0131] in, It is the center distance between the two cameras of the binocular vision measurement device 3. It is the focal length of the binocular vision measurement device 3. , These are the two-dimensional coordinates of the marker in the images captured by the two cameras. These are the three-dimensional coordinates of the marker.

[0132] (5) such as Figure 3 As shown, let the coordinates of marker 5, marker 6, and marker 7 in three-dimensional space be respectively... , and Then the translation vector of surgical instrument 2 relative to binocular vision measurement device 3 is:

[0133] ,

[0134] The rotation matrix of surgical instrument 2 relative to binocular vision measurement device 3 is: ,

[0135] in, , , Surgical instrument coordinate system The unit vector of the axis, the rules for establishing the coordinate system of surgical instruments are as follows: Figure 4 As shown,

[0136] ,

[0137] Let the coordinates of the surgical instrument needle tip 8 in the surgical instrument coordinate system obtained after calibration by surgical instrument 2 be... Then the three-dimensional coordinates of the surgical instrument needle tip 8 are:

[0138] ,

[0139] The error is calculated by extracting the three-dimensional coordinates of the surgical instrument tip 8 from 7 real images and 7 generated predicted images. The error calculation formula is as follows:

[0140] ,

[0141] in, This indicates the number of images in the predicted image sequence. The three-dimensional coordinates of the surgical instrument tip 8 in the real image. To predict the three-dimensional coordinates of the surgical instrument tip 8 in the image, It's an error. It is a natural number. The value of is [1, When the error A value less than 1 indicates that the surgical instrument needle tip trajectory has reached the predicted accuracy.

[0142] To predict the number of images in an image sequence Taking 7 as an example, then

[0143] .

[0144] In this invention, when the error When the value is less than 1, it indicates that the surgical instrument needle tip trajectory has reached the prediction accuracy, and the training of the generative adversarial network model is complete.

[0145] During the training process of a generative adversarial network (GAN) model, the parameters are dynamically adjusted. Once training is complete, the final parameters are selected as the final parameters of the GAN model. At this point, the parameters are fixed, and the generator can be used directly for prediction in the later stages.

[0146] In use, the real-time motion trajectory image sequence of surgical instrument 2 is directly acquired, and the real-time motion trajectory image sequence is input into the generator to generate a prediction image sequence. Based on the prediction image sequence, the prediction result of the surgical instrument needle tip trajectory is obtained. At this time, the surgical instrument needle tip trajectory is drawn, and the predicted surgical instrument needle tip trajectory is fed back to the doctor or surgical robot to help guide the surgical verification operation.

[0147] In summary, the surgical instrument needle tip trajectory prediction method provided by this invention has the following advantages:

[0148] 1. This invention uses a generative adversarial network (GAN) model to predict the motion trajectory of the surgical instrument needle tip 8. The GAN model consists of a generator and a discriminator. Through adversarial training between the two, the generator can continuously optimize its output, thereby generating high-precision prediction results. This adversarial learning mechanism can effectively capture complex motion patterns of the surgical instrument 2 and improve the accuracy of surgical instrument needle tip trajectory prediction.

[0149] 2. In the generative adversarial network model used in this invention, the 3D convolutional neural network used by the generator can perform convolution operations simultaneously in the spatial and temporal dimensions, effectively capturing the complex motion trajectory of the surgical instrument 2 in three-dimensional space, and accurately predicting the dynamic changes of the surgical instrument needle tip 8 during the operation.

[0150] 3. In the generative adversarial network model used in this invention, the discriminator employs a multi-scale discriminator. This discriminator can evaluate the generated motion trajectory at different scales, capturing features at different levels, thereby improving the ability to judge the authenticity of the generated trajectory. This multi-scale feature extraction capability can effectively distinguish between real and fake trajectories, improving the prediction accuracy of the overall system.

[0151] 4. The binocular vision measurement device 3 used in this invention is a near-infrared binocular vision measurement device 3. It tracks and positions the device by reflecting a marker ball, and further verifies the prediction accuracy of the generative adversarial network model by calculating the three-dimensional coordinates of the surgical instrument needle tip 8, thereby improving the accuracy and detail of the surgical instrument needle tip trajectory prediction and ensuring high accuracy of the prediction.

[0152] This invention, when applied to surgical navigation systems, helps surgeons make more accurate decisions during surgery by predicting the trajectory of surgical instrument tips, reducing the risk of errors and increasing the success rate of the operation. In robot-assisted surgery, predicting the trajectory of surgical instrument tips helps improve the accuracy and reliability of the surgical robot's operation, making the surgical process smoother.

[0153] Furthermore, by recording and analyzing the trajectory of surgical instrument needles, a large amount of surgical data can be accumulated, which helps to improve surgical methods and instrument design.

[0154] It is understood that the present invention has been described through some embodiments, and those skilled in the art will know that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention.

[0155] Furthermore, based on the teachings of this invention, modifications can be made to these features and embodiments to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, this invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims are protected by this invention.

Claims

1. A surgical instrument needle tip trajectory prediction method, characterized by, The method comprises the following steps: acquiring a historical motion trajectory of a surgical instrument (2), the historical motion trajectory comprising all motion forms of the surgical instrument (2), extracting a key motion image sequence of the surgical instrument in each motion trajectory for constructing a surgical instrument motion trajectory dataset; preprocessing the key motion image sequence of the surgical instrument in the surgical instrument motion trajectory dataset, and dividing the preprocessed key motion image sequence of the surgical instrument into a first image sequence and a second image sequence with the same number of frames according to time sequence; constructing a generative adversarial network model by taking a 3D convolutional neural network as a generator and a multi-scale discriminator as a discriminator, training the generative adversarial network model based on a mapping relationship between the first image sequence and the second image sequence, and evaluating the prediction accuracy of the generative adversarial network model, wherein the parameters of the generator and the discriminator are fixed when the prediction accuracy of the generative adversarial network model reaches a preset accuracy; acquiring a real-time motion trajectory image sequence of the surgical instrument (2), inputting the real-time motion trajectory image sequence into the generator to generate a predicted image sequence, and obtaining a prediction result of a needle tip trajectory of the surgical instrument based on the predicted image sequence; training the generative adversarial network model based on the mapping relationship between the first image sequence and the second image sequence, comprising the following steps: dividing the surgical instrument motion trajectory dataset into a training set and a test set; selecting a complete multi-frame image sequence composed of the first image sequence and the second image sequence from the training set as input of the discriminator; inputting the first image sequence in the training set into the generator as input to generate a predicted image sequence, calculating a generator loss function, and updating the generator; inputting the predicted image sequence generated by the generator into the discriminator, checking the timing consistency between the generated predicted image sequence and the first image sequence by the discriminator, calculating a discriminator loss function, updating the discriminator, and minimizing the discriminator loss function until the discriminator loss function is minimized; obtaining a prediction result of a needle tip trajectory of the surgical instrument based on the predicted image sequence, comprising the following steps: determining three-dimensional coordinates of three markers on the surgical instrument (2) based on the predicted image sequence; determining three-dimensional coordinates of a needle tip (8) of the surgical instrument based on the three-dimensional coordinates of the three markers on the surgical instrument (2), and obtaining a prediction result of a needle tip trajectory of the surgical instrument.

2. The surgical instrument needle tip trajectory prediction method of claim 1, wherein, The preprocessing of the key motion image sequence of the surgical instrument in the surgical instrument motion trajectory dataset comprises the following steps: removing noise in each image in the key motion image sequence of the surgical instrument; normalizing pixel values of each image in the key motion image sequence of the surgical instrument, and mapping the pixel values from [0, 255] to [0, 1].

3. The surgical instrument needle tip trajectory prediction method of claim 1, wherein, The evaluation of the prediction accuracy of the generative adversarial network model comprises the following steps: acquiring the first image sequence in the test set as input of the generator, and generating a predicted image sequence by the generator; The complete multi-frame image sequence composed of the first image sequence and the second image sequence in the test set is taken as an input of the discriminator to evaluate the difference between the predicted image and the real image and verify the error between the three-dimensional coordinates of the surgical instrument needle tip (8) in the predicted image and the three-dimensional coordinates of the surgical instrument needle tip (8) in the real image, and the prediction accuracy of the generative adversarial network model is comprehensively evaluated by using the difference and the error.

4. The surgical instrument needle tip trajectory prediction method of claim 3, wherein, The difference between the predicted image and the real image is evaluated by using the following formula, ， wherein, denotes the frame number of an image in the sequence of predicted images, is the real image corresponding to the input image sequence, is the predicted image corresponding to the input image sequence, MSE is the mean square error for evaluating the difference between the predicted image and the real image, is a natural number, the value of is [1, ]. The lower the mean square error (MSE) value, the higher the prediction accuracy.

5. The surgical instrument needle tip trajectory prediction method of claim 4, wherein, The error between the three-dimensional coordinates of the surgical instrument needle tip (8) in the predicted image and the three-dimensional coordinates of the surgical instrument needle tip (8) in the real image is verified, including the following steps: The three-dimensional coordinates of the surgical instrument needle tip (8) in the predicted image and the three-dimensional coordinates of the surgical instrument needle tip (8) in the real image are extracted respectively; The error between the three-dimensional coordinates of the surgical instrument needle tip (8) in the predicted image and the three-dimensional coordinates of the surgical instrument needle tip (8) in the real image is calculated by using the following formula, ， wherein, denotes the frame number of an image in the sequence of predicted images, is the three-dimensional coordinate of the needle tip in the real image, is the three-dimensional coordinate of the needle tip in the predicted image, is the error, is a natural number, the value of is [1, ]. When the error is less than 1, it indicates that the surgical instrument needle tip trajectory has reached the prediction accuracy.

6. The surgical instrument needle tip trajectory prediction method of claim 1, wherein, Based on the predicted image sequence, the three-dimensional coordinates of the three markers on the surgical instrument are determined, including the following steps: Based on the predicted image sequence generated by the generative adversarial network model, the edge contours of the three markers on the surgical instrument (2) are obtained by using the Canny edge detection algorithm; Three pixel points on the edge contour of the marker are randomly selected, and the two-dimensional center coordinates of the marker are determined according to the following formula , ， wherein, , and are the coordinates of three points on the edge profile of the marker that are not collinear, is the two-dimensional center of the marker coordinate; The markers in the images obtained by the two cameras of the binocular vision measurement device (3) are matched by using a region similarity matching algorithm; The three-dimensional coordinates of the markers are determined by using the following formula, ， wherein, , is the center distance between the two cameras of the binocular vision measurement device (3), is the focal length of the binocular vision measurement device (3), , are the two-dimensional coordinates of the marker in the images acquired by the two cameras, respectively, is the three-dimensional coordinate of the marker.

7. The surgical instrument needle tip trajectory prediction method of claim 1, wherein, The three-dimensional coordinates of the surgical instrument needle tip (8) are determined based on the following formula, ， wherein , ， ， is the coordinate of the tip of the surgical instrument (8) in the coordinate system of the surgical instrument, is the three-dimensional coordinate of the tip of the surgical instrument (8), , and are the coordinates of the marker one (5), the marker two (6) and the marker three (7) in the three-dimensional space, respectively, is the translation vector, is the rotation vector, are the unit vectors of the x, y, z directions on the coordinate system of the surgical instrument, respectively.

Images