Semantic segmentation and pose estimation method and system of surgical robot instrument under endoscope image

Abstract

The invention discloses a semantic segmentation and pose estimation method and system for a surgical robot instrument under an endoscope image, and relates to the field of computer vision. The objective of the invention is to solve the problems that an existing semantic segmentation and pose estimation method of a surgical robot instrument is large in error rate and poor in real-time performance and generalization performance, the constraint effect of time sequence information is rarely considered, and the requirement of accurate surgical operation is difficult to meet. According to the technical key points, a backbone network is used for extracting multi-scale information of an original image of an endoscope, and a feature encoding module aggregates encoded features from features extracted from the backbone network and comprises image local feature information of the image at different abstraction levels; the semantic segmentation decoder is used for generating segmented images from the image feature map, and the segmented images provide implicit geometric constraints for the pose estimation decoder; based on geometric constraints provided by a semantic segmentation result, the position and posture estimation decoder regresses the positions and posture parameters of the wrist and the rod of the surgical instrument relative to the camera from the feature map, and the opening and closing angle of the tail end of the instrument; and a time sequence module is used to construct a time sequence data buffer queue for continuous frame analysis, inter-frame jitter is reduced, and the precision of pose estimation and end opening and closing angle estimation algorithms is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method and system for semantic segmentation and posture estimation of surgical robot instruments under a laparoscope field of view. Background Art

[0002] Laparoscopic minimally invasive surgery has become one of the important directions in the field of modern surgery. Compared with traditional open surgery, laparoscopic minimally invasive surgery has the advantages of less trauma, faster patient recovery, and fewer postoperative complications. In laparoscopic minimally invasive surgery, the precise operation of surgical instruments is crucial to the surgical effect. Accurately segmenting the surgical instrument mask, estimating and tracking the spatial position and posture of the wrist and rod of the surgical instrument, and the opening and closing angle of the end clamp during the operation can provide strong support for doctors, improve the accuracy, safety and efficiency of the operation, and also provide strong support for posture correction and auxiliary decision-making during the operation.

[0003] Semantic segmentation is an image processing technology that aims to classify each pixel in an image into different semantic categories, identify different types of instruments, and biological tissues. Compared with traditional image segmentation technology, its advantage is that it can better enable computers to recognize and understand the content in the image, the segmentation results are more accurate, and the algorithm runs faster. At present, the semantic segmentation algorithm of laparoscopic surgical robot instruments mainly uses convolutional neural networks or visual Transformer networks as the main structure, and the accuracy is acceptable.

[0004] At present, there are mainly two types of methods for estimating the posture of surgical instruments: the method based on feature points estimates the posture by extracting and matching key points in the image, but in the case of complex surgical scenes, the extraction and matching of feature points are prone to errors; the method based on template matching requires the establishment of a three-dimensional model library of the instrument in advance, and the real-time and generalization are poor. However, the existing methods have a large error rate, or poor real-time and generalization, and rarely consider the constraint effect of timing information. The existing patent document CN118236166A discloses an automatic tracking system and method for surgical instruments, which includes: a mechanical arm; a visual sensing unit, which is arranged at the end of the mechanical arm and is used to obtain image data; a host computer, which is connected to the visual sensing unit, and is used to analyze the image data based on a neural network model to obtain the position information of the surgical instrument and the position information of the biological cavity tissue, and obtain the movement information of the mechanical arm according to the position information of the surgical instrument and the position information of the biological cavity tissue; a control unit, which is connected to the host computer and the mechanical arm, and is used to control the movement direction of the mechanical arm according to the movement information of the mechanical arm. This prior art does not combine semantic segmentation for pose estimation, and does not mention the opening and closing angle of the end clamp.

[0005] In addition, in practical applications, the posture feedback system of surgical instruments usually relies on various sensors in the mechanical structure, such as joint encoders, force sensors, etc. However, due to the hysteresis characteristics, plastic deformation, return difference and other problems of the mechanical transmission structure, and the problem that the elastic deformation caused by insufficient stiffness of the instrument cannot be fully and accurately fed back by the sensor, posture feedback relying solely on sensors often has delays and errors, which is difficult to meet the needs of precise surgical operations.

[0006] Therefore, it is urgent to improve the control accuracy and safety of laparoscopic robotic surgery. Summary of the invention

[0007] The technical problems to be solved by the present invention are:

[0008] In order to solve the problems of existing semantic segmentation and pose estimation methods of surgical robot instruments, such as large error rate, poor real-time and generalization, less consideration of the constraints of timing information, and difficulty in meeting the needs of precise surgical operations, the present invention provides a semantic segmentation and pose estimation method and system for surgical robot instruments under laparoscopic images.

[0009] The technical solution adopted by the present invention to solve the above technical problems is:

[0010] The semantic segmentation and pose estimation method of surgical robot instruments under laparoscopic images, the implementation process of the method is as follows:

[0011] Based on the encoder-decoder architecture, a single encoder-dual decoder surgical robot instrument pose estimation model is established, including a backbone network, a feature encoder (feature encoding module), a semantic segmentation decoder, a pose estimation decoder and its data cache queue and time series module;

[0012] The backbone network is used to extract multi-scale information of the original endoscope image. The feature encoder aggregates the encoded features from the features extracted by the backbone network, which contains the local feature information of the image at different abstraction levels.

[0013] The semantic segmentation decoder is used to generate a segmentation image from the image feature map. The segmentation image provides implicit geometric constraints for the pose estimation decoder.

[0014] Based on the geometric constraints provided by the semantic segmentation results, the pose estimation decoder regresses the position and pose parameters of the wrist and rod of the surgical instrument relative to the camera, as well as the opening and closing angle of the instrument end from the feature map;

[0015] Use the time series module to build a time series data cache queue for continuous frame analysis, reduce inter-frame jitter, and improve the accuracy of the pose estimation and terminal opening and closing angle estimation algorithms.

[0016] Furthermore, the backbone network extracts features by connecting MiST blocks and patch embedding in series. The backbone network uses a network based on MiST blocks (Mixed Swin-Transformer) to extract multi-scale features of the target. The feature map sizes are H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32 respectively; each extraction stage contains Patch embedding and MiSTBlock for feature extraction and transformation, specifically:

[0017] (1) Among them, the MiST block is mainly based on Swin-Transformer, normalization is used before the Swin-Transformer-T module, and the input is input to Swin-Transformer-T for attention operation. Its output is concatenated with the features of the input MiST block, and the concatenation is used as the output of the MiST block;

[0018] (2) For the PatchEmbedding patch embedding, the first embedded patch uses a 5x5 convolution kernel and a step size of 4 to extract 1 / 4 size features, and the subsequent embedded patches use a 3x3 convolution kernel and a step size of 2 to extract 1 / 8, 1 / 16, and 1 / 32 size features respectively;

[0019] (3) In the backbone network, the patch embedding module is connected in series with Ni (i=1~4) MiST blocks. The input image undergoes patch embedding and N1 MiST blocks to obtain the matrix of feature 1, whose dimension is H / 4x W / 4x C1. Feature 2 undergoes patch embedding and N2 MiST blocks to obtain the matrix of feature 2, and so on. The value of Ci (i=1~4) represents the richness of the feature. C1~C4 takes 32, 64, 128, 256, and N1~N4 takes 2, 2, 2, 2. Feature 1~Feature 4 are the low-level to high-level feature information contained in the image. The low-level features include information such as texture and edge, and the high-level features include information such as abstracted semantic level information.

[0020] Furthermore, the semantic segmentation decoder is: the features aggregated in the encoder are mapped through an MLP, and the output is the result of semantic segmentation, which provides constraints for the pose estimation decoder; the pose estimation decoder is: to obtain the pose of the surgical instrument, the pose estimation decoder uses three series-connected MLP blocks, accepts the input of the aggregated features and the semantic segmentation results, obtains the estimated cache of the instrument pose in the current frame, and writes it into the data cache queue of the pose estimation decoder; the time series module uses the xLSTM network, taking into account both long-term and short-term accuracy; the time series module obtains the timing cache from the pose estimation decoder, and outputs the pose parameters of the instrument rod and wrist and the opening and closing angle of the end clamp; the instrument rod and wrist have 6 degrees of freedom respectively, represented by x, y, z, pitch, yaw, roll, where: x, y, z represent the displacement of the coordinate system of the rod or wrist relative to the camera coordinate system; pitch, yaw, roll represent the rotation angle of the coordinate system of the rod or wrist relative to the camera coordinate system.

[0021] Furthermore, for the single encoder-dual decoder surgical robot instrument pose estimation model, its training and environment settings are as follows:

[0022] (1) Optimizer and learning rate scheduling: The AdamW optimizer is used, and the default learning rate is set to 2e-4. The learning rate scheduling is as follows: in the first quarter of training, the encoder learning rate is gradually increased from the default value of 1 / 8 to 1 / 4, and in the last three quarters of training, the encoder learning rate is gradually reduced from the default value of 1 / 4 to 1 / 16; the decoder learning rate is maintained at 1 times the default learning rate until the last quarter of training, when the learning rates of the two decoders are reduced to 1 / 3 of the default learning rate;

[0023] (2) Dataset data enhancement: Data enhancement is performed based on the mmseg library, and the following are performed in series: random size cropping and random position cropping with a probability of 0.5, horizontal and vertical random flipping with a probability of 0.5, and random distortion of brightness, contrast, saturation, and hue with a probability of 0.5 and a range of ±25%. The original data is enhanced with different data enhancements using a data wrapper, and the data is repeated 10 times to increase the generalization of the enhanced data;

[0024] (3) Training environment: The important environment configuration components are as follows: numpy = 1.19.5, CUDA = 11.3, ptorch = 1.11.0, torchvision = 0.12.0, mmcv = 1.6.0, mmseg = 0.29.1; two RTX3090 graphics cards are used for graphics card training, and 20,000 iter are trained;

[0025] (4) Forward and backward propagation of training: During training, the pre-trained weights of Swin-T are loaded, and the weights of the remaining network modules are initialized using the kaiming initialization method.

[0026] Furthermore, the semantic segmentation prediction image output by the semantic segmentation decoder is compared with the label image of the original image through a designed loss function. The value of the loss function is used to measure the difference between the two, and the optimizer uses the derivative of the loss function to update the network parameters.

[0027] Furthermore, the loss function mixes the following functions according to the corresponding weights, specifically: (1) Edge loss function 1Boundary_loss: The gradient edge mask map of the true value image is extracted by the laplacian operator. The mask map is obtained by using the laplacian operator of size 3*3 to extract the edge of the true value image; dice_loss and ce_loss are calculated under the corresponding gradient edge mask map and added to the total loss;

[0028] (2) Edge loss function 2InverseTransformLoss: A boundary-aware segmentation loss function that focuses on the alignment between the target and the predicted boundary. It uses a distance metric based on the transformation parameters to measure the difference between the predicted result and the true boundary, and obtains the distance between the two based on the transformation relationship. Specifically, a small neural network is used to predict the affine transformation matrix between the two boundaries, and outputs a matrix with 6 parameters, which represents the affine transformation between the predicted result and the true boundary. The InverseTransformLoss is calculated from the above 6 parameters and added to the total loss.

[0029] (3) diceloss: as shown in formula (1);

[0030] The loss function diceloss based on the Dice coefficient is often used to deal with class imbalance problems

[0031]

[0032] Where: p i is the probability predicted by the model, g i is the true label (ground truth), ∈ is a smoothing term to prevent the denominator from being 0;

[0033] (4) Focal Loss: As shown in formula (2), focal loss is an improved version of cross entropy loss.

[0034] FL(p t )=-α t (1-p t ) γlog(p t ) (2)

[0035] in:

[0036] p t is the predicted probability of class t;

[0037] α t is the category weight factor (used to balance positive and negative samples), which is 0.5;

[0038] γ is the focusing parameter, which is used to reduce the weight of easy-to-separate samples and is set to 3;

[0039] y is the value of the true image pixel label. When y=1, p t =p; when y = 0, p t =1-p;

[0040] (5) BCELoss: as described in formula (3);

[0041]

[0042] Where N is the number of pixels involved in the loss calculation; y i is the true label of pixel i; is the predicted probability of pixel i.

[0043] A system for semantic segmentation and posture estimation of surgical robot instruments under laparoscopic images, the system having program modules corresponding to the steps of the above technical solution, and executing the steps in the method for semantic segmentation and posture estimation of surgical robot instruments under laparoscopic images during operation.

[0044] A computer-readable storage medium stores a computer program, wherein the computer program is configured to implement the steps of the method for semantic segmentation and posture estimation of surgical robot instruments under laparoscopic images when called by a processor.

[0045] The present invention has the following beneficial technical effects:

[0046] The present invention unifies semantic segmentation, instrument pose estimation and timing constraints into a multi-task learning framework; semantic segmentation results are added as constraints to the instrument pose estimation link to improve its calculation accuracy; and timing modeling improves the stability of pose estimation. The present invention introduces a mixed loss function and adds semantic segmentation edge information to the loss calculation. The pose estimation method based on visual neural network proposed in the present invention can effectively solve the above technical problems by combining semantic segmentation results with timing information. The method can provide intelligent tools for laparoscopic robotic surgery and improve the control accuracy and safety of laparoscopic robotic surgery. The present invention can calculate the position and posture parameters of the wrist and rod of the surgical instrument, as well as the opening and closing angle of the end clamp. The semantic segmentation of surgical instruments is realized, the binary segmentation mIoU index is not less than 92%, the part segmentation mIoU index is not less than 80%, and the frame rate running on the RTX3090 graphics card is not less than 20 frames per second. The target features are learned using semantic segmentation tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flowchart of the method of the present invention. Figure 2 It is a network structure block diagram of the surgical robot instrument posture estimation model in the present invention. Figure 3 : Schematic diagram of loss function and network parameter update in the training process (training process); The predicted image input by the semantic segmentation decoder is compared with the label image corresponding to the original image, and the loss value (loss) is output through the total loss function. The loss function is derived, the network is back-propagated, and the parameters of the network model are updated. Figure 4 : From left to right: original image, label image, and semantic segmentation prediction image output by the semantic segmentation decoder. Figure 5 This is a screenshot of the log output during the code training process. Figure 6 Figure 3 is a diagram showing the coordinate system changes of the six parameters x, y, z, pitch, yaw, and roll of the rod and wrist. The origin of the camera coordinate system is O, the origin of the rod / wrist coordinate system is O', (x, y, z) is the position of O' in the O coordinate system, and (pitch, yaw, roll) is the rotation angle of the rod / wrist coordinate system relative to the camera coordinate system, and the rotation directions are along the x / y / z axes of the camera coordinate system, respectively. Specifically, use one coordinate system, rotate the pitch angle along Xc, the yaw angle along Yc, the roll angle along Zc, and translate the origin to the (x, y, z) position of the camera coordinate system to obtain the rod / wrist coordinate system. DETAILED DESCRIPTION

[0048] Combination Figure 1-6The implementation of the semantic segmentation and pose estimation method of surgical robot instruments under the real laparoscope image of the present invention is described as follows:

[0049] like Figure 1 As shown in the figure, based on the encoder-decoder architecture, a single encoder-dual decoder surgical robot instrument pose estimation model is established, including a backbone network, an encoder, a semantic segmentation decoder, a pose estimation decoder, and its data cache queue and time series module. The backbone network is used to extract multi-scale information of the original endoscope image. The feature encoding module aggregates the encoded features from the features extracted from the backbone network, including the local feature information of the image at different abstract levels; the semantic segmentation decoder is used to generate a segmented image from the image feature map, and the segmented image provides implicit geometric constraints for the pose estimation decoder; the pose estimation decoder regresses the position and posture parameters of the wrist and rod of the surgical instrument relative to the camera, and the opening and closing angle of the end of the instrument from the feature map based on the geometric constraints provided by the semantic segmentation results; the time series module is used to build a time series data cache queue for continuous frame analysis, reduce inter-frame jitter, and improve the accuracy of the pose estimation and end opening and closing angle estimation algorithms.

[0050] Specifically, the network structure is as follows Figure 2 As shown, through Figure 2 The backbone network shown in Figure 1 extracts features by connecting MiST blocks and patch embedding in series. The backbone network uses a network based on MiST blocks (Mixed Swin-Transformer) to extract multi-scale features of the target. The feature map sizes are H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32. Each extraction stage contains Patch embedding and MiSTBlock for feature extraction and transformation.

[0051] (1) Among them, the MiST block is mainly based on Swin-Transformer, which uses normalization before the Swin-Transformer-T module and is input into Swin-Transformer-T for attention operation. Its output is concatenated with the features of the input MiST block and the concatenated features are used as the output of the MiST block.

[0052] (2) For the PatchEmbedding patch embedding, the first embedded patch uses a 5x5 convolution kernel and a step size of 4 to extract 1 / 4 size features, and the subsequent embedded patches use a 3x3 convolution kernel and a step size of 2 to extract features of 1 / 8, 1 / 16, and 1 / 32 sizes, respectively.

[0053] (3) Figure 2In the above design, C1~C4 can be 32, 64, 128, 256, and N1~N4 can be 2, 2, 2, 2. This ensures that high-level abstract features have enough channels to store a variety of abstract feature representations. Through the above design, image features can be captured at different scales, and hierarchical feature representations can be constructed to provide rich feature information for the downstream tasks of the subsequent decoder.

[0054] Features 1 to 4 are low-level to high-level feature information contained in the image. Low-level features include texture and edge information, and high-level features include abstract semantic level information. The matrix obtained by feature 1 from the MiST block contains the feature information in the image. The dimension of feature 1 is H / 4x W / 4x C1. Features 2 to 4 are similar. The value of Cx represents the richness of the feature. C1 to C4 is [32, 64, 128, 128]. N1 to N4 is the number of stacking times of MiST blocks, which is [2, 2, 2, 2]. In the backbone network, the patch embedding module is connected in series with Ni MiST blocks. The input image is subjected to patch embedding and N1 MiST blocks to obtain the matrix of feature 1, whose dimension is H / 4x W / 4x C1. Feature 2 is subjected to patch embedding and N2 MiST blocks to obtain the matrix of feature 2, and so on.

[0055] The aggregated features are mapped through an MLP and output as the result of semantic segmentation, which provides constraints for the pose estimation decoder. The pose estimation decoder uses three series-connected MLP blocks to accept the input of aggregated features and semantic segmentation results, obtain the estimated cache of the instrument pose in the current frame, and write it to the data cache queue. The time series module uses the xLSTM network to take into account both long-term and short-term accuracy. The time series module obtains the timing cache from the pose estimation decoder and outputs the pose parameters (6 degrees of freedom, x, y, z, pitch, yaw, roll) of the instrument rod and wrist, as well as the opening and closing angle of the end fixture.

[0056] Dataset

[0057] (1) Endovis2017 segmentation dataset: The robotics sub-dataset from the instrument segmentation and tracking sub-challenge of the Endoscopic Vision Challenge is a collection of 10 abdominal surgery sequences recorded by the da Vinci surgical robot system. It contains 3,000 images, each with an effective size of 1280*1024 pixels and containing 1 to 2 instruments.

[0058] (2) CholecInstanceSeg segmentation dataset: released in June 2024, including 41,900 annotated frames and 64,400 tool instances, each with a semantic mask and instance ID. Each image contains a specific difficult case, including motion blur, smoke, reflection, soft tissue occlusion or attachment, strong or weak light, etc.

[0059] (3) Surgical video and corresponding instrument pose dataset: This is a private dataset with a total video length of 6 hours, 30 frames per second for video data, and 2 frames per second for instrument pose data. For each frame with instrument pose data, the frame has a corresponding instrument semantic segmentation annotation.

[0060] Training and environment setup

[0061] (1) Optimizer and learning rate scheduling: The AdamW optimizer is used, and the default learning rate is set to 2e-4. The learning rate scheduling is as follows: In the first quarter of training, the learning rate of the encoder is gradually increased from the default value of 1 / 8 to 1 / 4, and in the last three quarters of training, the learning rate of the encoder is gradually reduced from the default value of 1 / 4 to 1 / 16. In this way, the learning speed of the model in the early stage and the fineness of adjustment in the later stage are taken into account, helping the model to achieve a better range optimal solution in the later stage. The decoder learning rate is maintained at 1 times the default learning rate until the last quarter of training, when the learning rate of the two decoders is reduced to 1 / 3 of the default learning rate.

[0062] (2) Dataset data enhancement: Data enhancement is performed based on the mmseg library, and the following series is performed: random size cropping and random position cropping with a probability of 0.5, horizontal and vertical random flipping with a probability of 0.5, and random distortion of brightness, contrast, saturation, and hue with a probability of 0.5 and a range of ±25%. The original data is subjected to different data enhancements using a data wrapper, and the data is repeated 10 times to increase the generalization of the enhanced data. In addition, before the final deployment, in order to make full use of the existing data, the K-Fold Cross-Validation method is used to divide the validation set. Specifically, K=5 is taken, and the dataset is divided into 5 parts. One of the parts is taken as the validation set each time, and the rest is used as the training set. This is repeated 5 times, and the model evaluation index is taken as the average of the 5 validation results. This method can balance the random errors under different data partitions, provide a more stable and reliable model performance evaluation, reduce the network's dependence on a specific part of the data during the final training process, and ensure that overfitting does not occur.

[0063] (3) Training environment: The important environment configuration components are as follows: numpy = 1.19.5, CUDA = 11.3, ptorch = 1.11.0, torchvision = 0.12.0, mmcv = 1.6.0, mmseg = 0.29.1. Two RTX3090 graphics cards are used for training 20,000 iter.

[0064] (4) Forward and backward propagation of training: During training, the pre-trained weights of Swin-T are loaded, and the weights of the remaining network modules are initialized through the kaiming initialization method. The pre-processed image is used as the network input, and the network is forward propagated. The semantic segmentation result is obtained from the semantic segmentation decoder and compared with the true value image. The loss of the semantic segmentation task is obtained through a specially designed loss function, and the gradient is then reversely propagated to update the network parameters; the pose estimation parameters obtained from the pose estimation decoder are compared with the true value, and then back-propagated; the optimized pose estimation parameters obtained through the data cache queue and time series module are compared with the true value, and then back-propagated.

[0065] Loss function design

[0066] Use a mixed loss function and introduce the edge loss functions Boundary and InverseTransformLoss. Mix the following loss functions according to the corresponding weights. The weight of the edge loss function can be appropriately increased in the later stage of training.

[0067] (1) Edge loss function 1Boundary_loss: The gradient edge mask map of the true value image is extracted by the laplacian operator. The mask map is obtained by using the laplacian operator of size 3*3 to extract the edge of the true value image. The dice_loss and ce_loss are calculated under the corresponding gradient edge mask map and added to the total loss.

[0068] (2) Edge loss function 2InverseTransformLoss: A boundary-aware segmentation loss function that focuses on the alignment between the target and the predicted boundary. A distance metric based on the transformation parameters is used to measure the difference between the predicted result and the true boundary, and the distance between the two is obtained based on the transformation relationship. Specifically, a small neural network is used to predict the affine transformation matrix between the two boundaries, and a matrix with 6 parameters is output, representing the affine transformation between the predicted result and the true boundary. The InverseTransformLoss is calculated from the above 6 parameters and added to the total loss.

[0069] (3) diceloss: as shown in formula (1);

[0070] The loss function diceloss based on the Dice coefficient is often used to deal with class imbalance problems

[0071]

[0072] Where: p i is the probability predicted by the model, g i is the true label (ground truth), ∈ is a smoothing term to prevent the denominator from being zero.

[0073] (4) focal loss: as shown in formula (2);

[0074] Focalloss is an improved version of cross entropy loss.

[0075] FL(p t )=-α t (1-p t ) γ log(p t ) (2)

[0076] in:

[0077] p t is the predicted probability of class t;

[0078] α t is the category weight factor (used to balance positive and negative samples), which is 0.5;

[0079] γ is the focusing parameter, which is used to reduce the weight of easy-to-separate samples and is set to 3;

[0080] y is the value of the true image pixel label. When y=1, p t =p t ; When y = 0, p t =1-p t .

[0081] (5) BCELoss: as described in formula (3).

[0082]

[0083] Where N is the number of pixels involved in the loss calculation; y i is the true label of pixel i; is the predicted probability of pixel i.

[0084] The weighted weights of the above five loss functions are obtained through debugging, and the weight ratio used in the actual method is 1.5:1:0.2:1:1.

[0085] Evaluation indicators

[0086] Including but not limited to mIoU, as described in formula (1-4). mIoU is used to measure the accuracy between the predicted P and the true value image GT.

[0087]

[0088] in:

[0089] C is the total number of categories,

[0090] P i represents the prediction area of ​​the i-th category,

[0091] GT i represents the region of the label of the i-th category,

[0092] |P i ∩GT i | and |P i ∪GT i | represents the area of ​​the intersection and union of two areas.

[0093] The present invention can calculate the position and posture parameters of the wrist and rod of the surgical instrument and the opening and closing angle of the end clamp. The semantic segmentation of the surgical instrument is realized, the binary segmentation mIoU index is not less than 92%, the part segmentation mIoU index is not less than 80%, and the frame rate running on the RTX3090 graphics card is not less than 20 frames per second. The semantic segmentation task can be used to learn the target features.

[0094] In summary, the method proposed in the present invention, based on the pose estimation of visual neural network, can effectively solve the technical problems proposed in the present invention by combining semantic segmentation results with time series information. The method of the present invention can provide intelligent tools for laparoscopic robotic surgery and improve the control accuracy and safety of laparoscopic robotic surgery. The method of the present invention has been verified by simulation experiments and practical applications to have the technical effects and practicality claimed by the present invention. Figure 5 As shown, it is a screenshot of the log output during the code training process. The code based on the present invention has been developed and put into practical use.

[0095] The method of the present invention has been verified through simulation experiments and practical applications, and the technical effects claimed by the present invention have been verified.

[0096] The algorithm (method) proposed in the present invention is the underlying technical core of the present invention, and various products can be derived based on the algorithm.

[0097] Based on the algorithm (method) proposed in the present invention, a semantic segmentation and posture estimation system for surgical robot instruments under laparoscopic images is developed using a programming language. The system has program modules corresponding to the steps of the above-mentioned technical solution, and executes the steps in the semantic segmentation and posture estimation method for surgical robot instruments under laparoscopic images during operation.

[0098] The computer program of the developed system (software) is stored on a computer-readable storage medium, and the computer program is configured to implement the steps of the semantic segmentation and pose estimation method of the surgical robot instrument under the laparoscope image when called by the processor. That is, the present invention is materialized on a carrier to become a computer program product.

[0099] The software developed based on the present invention is configured on a surgical robot instrument to form a terminal product.

[0100] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] The computer programs (also referred to as programs, software, software applications, or codes) of the present invention include machine instructions for programmable processors, and these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used in the present invention, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or device (e.g., disk, optical disk, memory, programmable logic device PLD) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0102] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, they are all within the scope of protection of the present invention.

Claims

1. A method for semantic segmentation and pose estimation of surgical robot instruments in laparoscopic images, characterized in that: The implementation process of the method is: Based on the encoder-decoder architecture, a single encoder-dual decoder surgical robot instrument pose estimation model is established, including a backbone network, feature encoder, semantic segmentation decoder, pose estimation decoder and its data cache queue and time series module; The backbone network is used to extract multi-scale information of the original endoscope image. The feature encoder aggregates the encoded features from the features extracted by the backbone network, which contains the local feature information of the image at different abstraction levels. The semantic segmentation decoder is used to generate a segmentation image from the image feature map. The segmentation image provides implicit geometric constraints for the pose estimation decoder. Based on the geometric constraints provided by the semantic segmentation results, the pose estimation decoder regresses the position and pose parameters of the wrist and rod of the surgical instrument relative to the camera, as well as the opening and closing angle of the instrument end from the feature map; Use the time series module to build a time series data cache queue for continuous frame analysis, reduce inter-frame jitter, and improve the accuracy of the pose estimation and terminal opening and closing angle estimation algorithms.

2. The method for semantic segmentation and pose estimation of surgical robot instruments under laparoscopic images according to claim 1, characterized in that: The backbone network extracts features by connecting MiST blocks and patch embedding in series. The backbone network uses a network based on MiST blocks (Mixed Swin-Transformer) to extract multi-scale features of the target. The feature map sizes are H / 4×W / 4, H / 8×W / 8, H / 16×W / 16, and H / 32×W / 32 respectively; each extraction stage contains Patch embedding and MiSTBlock for feature extraction and transformation, specifically: (1) Among them, The MiST block is mainly based on Swin-Transformer. Normalization is used before the Swin-Transformer-T module and input into Swin-Transformer-T for attention operation. Its output is concatenated with the features of the input MiST block and the concatenated output is used as the output of the MiST block. (2) For the PatchEmbedding patch embedding, the first embedded patch uses a 5x5 convolution kernel and a step size of 4 to extract 1 / 4 size features, and the subsequent embedded patches use a 3x3 convolution kernel and a step size of 2 to extract 1 / 8, 1 / 16, and 1 / 32 size features respectively; (3) In the backbone network, the patch embedding module is connected in series with Ni (i=1~4) MiST blocks. The input image undergoes patch embedding and N1 MiST blocks to obtain the matrix of feature 1, whose dimension is H / 4x W / 4x C1. Feature 2 undergoes patch embedding and N2 MiST blocks to obtain the matrix of feature 2, and so on. The value of Ci (i=1~4) represents the richness of the feature. C1~C4 takes 32, 64, 128, 256, and N1~N4 takes 2, 2, 2, 2. Feature 1~Feature 4 are the low-level to high-level feature information contained in the image. The low-level features include information such as texture and edge, and the high-level features include information such as abstracted semantic level information.

3. The method for semantic segmentation and pose estimation of surgical robot instruments under laparoscopic images according to claim 1 or 2, characterized in that: The semantic segmentation decoder is as follows: the features aggregated in the encoder are mapped through an MLP and the output is the result of semantic segmentation. The semantic segmentation result provides constraints for the pose estimation decoder. The pose estimation decoder is as follows: To obtain the pose of the surgical instrument, the pose estimation decoder uses three series-connected MLP blocks, accepts the input of aggregated features and semantic segmentation results, obtains the estimated cache of the instrument pose in the current frame, and writes it into the data cache queue of the pose estimation decoder; the time series module uses the xLSTM network to take into account both long-term and short-term accuracy; The time series module obtains the timing buffer from the pose estimation decoder and outputs the pose parameters of the instrument rod and wrist and the opening and closing angle of the end fixture; the instrument rod and wrist have 6 degrees of freedom respectively, represented by x, y, z, pitch, yaw, and roll, where: x, y, and z represent the displacement of the coordinate system of the rod or wrist relative to the camera coordinate system; pitch, yaw, and roll represent the rotation angle of the coordinate system of the rod or wrist relative to the camera coordinate system.

4. The method for semantic segmentation and pose estimation of surgical robot instruments under laparoscopic images according to claim 3, characterized in that: For the single encoder-dual decoder surgical robot instrument pose estimation model, the training and environment settings are as follows: (1) Optimizer and learning rate scheduling: The AdamW optimizer is used, and the default learning rate is set to 2e-4. The learning rate scheduling is as follows: in the first quarter of the training, the encoder's learning rate is gradually increased from the default value of 1 / 8 to 1 / 4, and in the last three quarters of the training, the encoder's learning rate is gradually reduced from the default value of 1 / 4 to 1 / 16. The decoder learning rate is kept at 1 times the default learning rate until the last quarter of training, when the learning rates of both decoders are reduced to 1 / 3 of the default learning rate; (2) Dataset data enhancement: Data enhancement is performed based on the mmseg library, and the following are performed in series: random size cropping and random position cropping with a probability of 0.5, horizontal and vertical random flipping with a probability of 0.5, and random distortion of brightness, contrast, saturation, and hue with a probability of 0.5 and a range of ±25%. The original data is enhanced with different data enhancements using a data wrapper, and the data is repeated 10 times to increase the generalization of the enhanced data; (3) Training environment: The important environment configuration components are as follows: numpy = 1.19.5, CUDA = 11.3, ptorch = 1.11.0, torchvision = 0.12.0, mmcv = 1.6.0, mmseg = 0.29.1; two RTX3090 graphics cards are used for graphics card training, and 20,000 iter are trained; (4) Forward and backward propagation of training: During training, the pre-trained weights of Swin-T are loaded, and the weights of the remaining network modules are initialized using the kaiming initialization method.

5. The method for semantic segmentation and pose estimation of surgical robot instruments under laparoscopic images according to claim 4, characterized in that: The semantic segmentation prediction image output by the semantic segmentation decoder is compared with the label image of the original image through the designed loss function, and the value of the loss function is used to measure the difference between the two.

6. The method for semantic segmentation and pose estimation of surgical robot instruments under laparoscopic images according to claim 5, characterized in that: The loss function mixes the following functions according to the corresponding weights, specifically: (1) Edge loss function 1Boundary_loss: The gradient edge mask map of the true value image is extracted by the laplacian operator. The mask map is obtained by using the laplacian operator of size 3*3 to extract the edge of the true value image; dice_loss and ce_loss are calculated under the corresponding gradient edge mask map and added to the total loss; (2) Edge loss function 2InverseTransformLoss: A boundary-aware segmentation loss function that focuses on the alignment between the target and the predicted boundary. It uses a distance metric based on the transformation parameters to measure the difference between the predicted result and the true boundary, and obtains the distance between the two based on the transformation relationship. Specifically, a small neural network is used to predict the affine transformation matrix between the two boundaries, and outputs a matrix with 6 parameters, which represents the affine transformation between the predicted result and the true boundary. The InverseTransformLoss is calculated from the above 6 parameters and added to the total loss. (3) diceloss: as shown in formula (1); The loss function diceloss based on the Dice coefficient is often used to deal with class imbalance problems Where: p i is the probability predicted by the model, g i is the true label (ground truth), ∈ is a smoothing term to prevent the denominator from being 0; (4) focalloss: as shown in formula (2); FL(p t )=-a t (1-p t ) γ log(p t ) (2) in: p t is the predicted probability of class t; α t is the category weight factor, which is 0.5; γ is the focusing parameter, which is used to reduce the weight of easy-to-separate samples and is set to 3; y is the value of the true image pixel label. When y=1, p t =p; when y = 0, p t =1-p; (5) BCELoss: as described in formula (3); Where N is the number of pixels involved in the loss calculation; y i is the true label of pixel i; is the predicted probability of pixel i.

7. A semantic segmentation and pose estimation system for surgical robot instruments under laparoscopic images, characterized by: The system has a program module corresponding to the steps of any one of claims 1 to 6, and executes the steps in the method for semantic segmentation and posture estimation of surgical robot instruments under laparoscopic images when running.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is configured to implement the steps of the method for semantic segmentation and pose estimation of a surgical robot instrument under a laparoscopic image according to any one of claims 1 to 6 when called by a processor.

Images