Semi-supervised surgical instrument segmentation method and system based on teacher-student model

Abstract

The invention relates to the technical field of medical image processing, in particular to a semi-supervised surgical instrument segmentation method and system based on a teacher-student model, and the method comprises the following steps: obtaining a three-dimensional ultrasonic image for surgical instrument segmentation; constructing and training a network structure formed by combining a single teacher network and three student networks to obtain a semi-supervised surgical instrument segmentation model for segmenting surgical instruments from the three-dimensional ultrasonic image; and outputting a surgical instrument segmentation result on the three-dimensional ultrasonic image by using the semi-supervised surgical instrument segmentation model. According to the method, the semi-supervised surgical instrument segmentation model is constructed through knowledge distillation of the single teacher network and the three student networks, so that the three student networks in the semi-supervised surgical instrument segmentation model are matched with three dimensions of three-dimensional ultrasound respectively, parallel processing of extraction and segmentation of three-dimensional image features is achieved, and the accuracy of three-dimensional image segmentation is improved. The information processing amount is small, the structural complexity is low, the efficiency is improved, and the timeliness requirement in the operation process can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a semi-supervised surgical instrument segmentation method and system based on a teacher-student model. Background Art

[0002] Surgical instrument segmentation algorithms automatically segment surgical instruments in medical images and are widely used in assisted surgery. They help doctors better control surgical procedures and reduce surgical risks. These algorithms primarily use computer vision and machine learning algorithms to extract and accurately segment surgical instruments from medical images.

[0003] In the existing technology, during the segmentation of surgical instruments, a segmentation model is usually established with three-dimensional ultrasound images as the overall processing unit, so that the segmentation model needs to process three-dimensional image information at the same time. The amount of information processing is large, and the difficulty of feature processing increases, which naturally increases the structural complexity and computational complexity of the segmentation model, resulting in a long segmentation time. It is difficult to apply to the three-dimensional ultrasound images fed back during the operation, that is, it is difficult to quickly locate the surgical instruments in the three-dimensional ultrasound images fed back during the operation, and cannot meet the real-time requirements. Summary of the Invention

[0004] The purpose of the present invention is to provide a semi-supervised surgical instrument segmentation method based on a teacher-student model to solve the technical problems in the existing technology that the segmentation model with three-dimensional ultrasound images as the overall processing is complex in structure and operation, the segmentation takes a long time, and cannot meet real-time requirements.

[0005] In order to solve the above technical problems, the present invention specifically provides the following technical solutions: A semi-supervised surgical instrument segmentation method based on a teacher-student model includes the following steps: acquiring a three-dimensional ultrasound image for surgical instrument segmentation; A network structure consisting of a single teacher network and three student networks was constructed and trained to obtain a semi-supervised surgical instrument segmentation model for segmenting surgical instruments on 3D ultrasound images. The semi-supervised surgical instrument segmentation model is used to output the surgical instrument segmentation results on the three-dimensional ultrasound image.

[0006] As a preferred solution of the present invention, the method for constructing the teacher network in the network structure includes: The HybridFormer model composed of a CNN network and a Transformer model is used as the feature extraction module in the teacher network, and Mask R-CNN is used as the mask prediction module in the teacher network; The 3D ultrasound image is used as the input of the teacher network, and the surgical instrument segmentation mask in the 3D ultrasound image is used as the output of the teacher network.

[0007] As a preferred solution of the present invention, the three student networks in the network structure include: The HybridFormer model composed of a CNN network and a Transformer model is used as the feature extraction module in each student network, and the Mask R-CNN is used as the mask prediction module in each student network; Using the X-axis component image of the three-dimensional ultrasound image as an input item of the first student network, and using the X-axis component of the surgical instrument segmentation mask in the X-axis component image as an output item of the first student network; The Y-axis component image of the three-dimensional ultrasound image is used as an input item of the second student network, and the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image is used as an output item of the second student network; The Z-axis component image of the three-dimensional ultrasound image is used as the input item of the third student network, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image is used as the output item of the third student network.

[0008] As a preferred solution of the present invention, the training method of the network structure includes: Construct a dataset containing mask-annotated data and mask-unannotated data of three-dimensional ultrasound images; The teacher network is trained using the masked data of the 3D ultrasound images in the dataset, and the surgical instrument segmentation masks of the masked unlabeled data of the 3D ultrasound images output by the teacher network after training are used as pseudo labels for training the student network. Separating the X-axis, Y-axis, and Z-axis components of the mask annotation data of the three-dimensional ultrasound image and the pseudo-label Persudo Labels data of the three-dimensional ultrasound image to obtain the X-axis component, Y-axis component, and Z-axis component of the mask annotation data of the three-dimensional ultrasound image, and the X-axis component, Y-axis component, and Z-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image; Construct loss functions for training three student networks; Based on the loss function, the first student network is trained using the mixed data consisting of the X-axis component of the masked annotation data of the three-dimensional ultrasound image and the X-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, the second student network is trained using the mixed data consisting of the Y-axis component of the masked annotation data of the three-dimensional ultrasound image and the Y-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, and the third student network is trained using the mixed data consisting of the Z-axis component of the masked annotation data of the three-dimensional ultrasound image and the Z-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, so as to complete the training of the three student networks.

[0009] As a preferred solution of the present invention, the loss function is: ; in, , ; , ; ; Where Ls is the total loss function of the three student networks, is the total distillation loss of the three student networks, is the total prediction loss of the three student networks, is the total reconstruction loss of the three student networks, are x, y, z identifiers, for The axis corresponds to the distillation loss of the student network, for The axis corresponds to the prediction loss of the student network, for The axis corresponds to the output of the student network, Persudo Labels are the pseudo labels output by the teacher network. The component on the axis, The ground truth of surgical instrument segmentation mask for mask annotation data of 3D ultrasound images is The component on the axis, for The axes correspond to the balanced hyperparameters of the student network, The ground truth value of surgical instrument segmentation mask for mask annotation data of 3D ultrasound images, Persudo Labels are the pseudo labels output by the teacher network.

[0010] As a preferred solution of the present invention, the method for determining the balance hyperparameters includes: Get The distillation loss of the student network corresponding to the axis Hewei The prediction loss of the student network corresponding to the axis ; use and Build The axis corresponds to the balanced hyperparameter of the student network ,in, ; Where, are x, y, z identifiers, for The axis corresponds to the distillation loss of the student network, for The axes correspond to the prediction loss of the student network.

[0011] As a preferred solution of the present invention, the parameters of the feature extraction module and the mask prediction module in the teacher network and the parameters of the feature extraction modules and the mask prediction modules in the three student networks are iteratively transferred and updated through the exponential moving average EMA method.

[0012] As a preferred solution of the present invention, the semi-supervised surgical instrument segmentation model is composed of three trained student networks.

[0013] As a preferred embodiment of the present invention, a method for outputting surgical instrument segmentation results on a three-dimensional ultrasound image using a semi-supervised surgical instrument segmentation model includes: The three student networks in the semi-supervised surgical instrument segmentation model are used to perform feature extraction and mask prediction on the X-axis component image, Y-axis component image, and Z-axis component image of the 3D ultrasound image, thereby obtaining the X-axis component of the surgical instrument segmentation mask in the X-axis component image, the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image. The X-axis component of the surgical instrument segmentation mask in the X-axis component image, the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image are superimposed to obtain a fused mask as the surgical instrument segmentation result.

[0014] As a preferred embodiment of the present invention, the present invention provides a semi-supervised surgical instrument segmentation system based on a teacher-student model, which is applied to a semi-supervised surgical instrument segmentation method based on a teacher-student model. The system includes: A data acquisition unit for acquiring a three-dimensional ultrasonic image for surgical instrument segmentation; A model building unit, which constructs and trains a network structure composed of a single teacher network and three student networks to obtain a semi-supervised surgical instrument segmentation model for segmenting surgical instruments on three-dimensional ultrasound images; The segmentation output unit is used to output the surgical instrument segmentation result on the three-dimensional ultrasound image using the semi-supervised surgical instrument segmentation model.

[0015] Compared with the prior art, the present invention has the following beneficial effects: The present invention constructs a semi-supervised surgical instrument segmentation model through knowledge distillation of a single teacher network and three student networks, so that the three student networks in the semi-supervised surgical instrument segmentation model respectively match the three dimensions of three-dimensional ultrasound, achieving parallel processing of three-dimensional image feature extraction and segmentation. Each student network only needs to process the image features of one dimension, with less information processing and low structural complexity, which improves efficiency and can meet the timeliness requirements during the operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0017] Figure 1 A flowchart of a semi-supervised surgical instrument segmentation method based on a teacher-student model provided in an embodiment of the present invention; Figure 2 A block diagram of a semi-supervised surgical instrument segmentation system based on a teacher-student model provided by an embodiment of the present invention; Figure 3 Schematic diagram of training a semi-supervised surgical instrument segmentation model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0019] like Figure 1 As shown, the present invention provides a semi-supervised surgical instrument segmentation method based on a teacher-student model, comprising the following steps: acquiring a three-dimensional ultrasound image for surgical instrument segmentation; A network structure consisting of a single teacher network and three student networks was constructed and trained to obtain a semi-supervised surgical instrument segmentation model for segmenting surgical instruments on 3D ultrasound images. The semi-supervised surgical instrument segmentation model is used to output the surgical instrument segmentation results on the three-dimensional ultrasound image.

[0020] In order to improve the efficiency of surgical instrument segmentation, the present invention utilizes the knowledge distillation framework of the teacher-student model and uses a teacher network to obtain three student networks that perform surgical instrument segmentation in the X-axis, Y-axis and Z-axis directions of the three-dimensional ultrasound image. Each student network corresponds to the segmentation of surgical instruments in one axis, thereby converting the surgical instrument segmentation based on the three-dimensional ultrasound image as a whole into the surgical instrument segmentation based on the three axes of the three-dimensional ultrasound image independently. This reduces the image feature information processed by each student network in the surgical instrument segmentation compared to the teacher network, and the corresponding network structure is more lightweight.

[0021] Therefore, the present invention utilizes three lightweight student networks as a semi-supervised surgical instrument segmentation model for segmenting surgical instruments from three-dimensional ultrasound images. This gives the semi-supervised surgical instrument segmentation model the advantages of a lightweight structure, including but not limited to: Improved computational efficiency: By reducing model parameters and computational complexity, the lightweight model can run quickly with limited hardware resources, significantly improving computational efficiency. Storage space savings: The lightweight model reduces data volume, thereby saving storage space and lowering storage costs. Faster rendering speed: Lightweight processing can reduce model complexity and size, thereby speeding up rendering and reducing loading and rendering time. Optimized user experience: The lightweight model improves processing speed and performance, making user experience smoother and optimizing the overall user experience. Adaptability to multiple devices: The lightweight model has a small memory footprint and can be easily deployed on mobile devices or embedded systems, making it suitable for a variety of devices and application scenarios. Improved response speed: The lightweight model excels in real-time data processing, enabling rapid data response, making it suitable for scenarios requiring real-time processing. Reduced deployment costs: Lightweight models have lower publishing and deployment costs and can be more easily deployed on various platforms.

[0022] Based on the advantages of the above-mentioned lightweight model, the three lightweight student models obtained in the present invention can respectively match the three axial dimensions of three-dimensional ultrasound, achieve parallel processing of extraction and segmentation of image features in the three axial dimensions, and adapt to the real-time, easy deployment, and fast response speed requirements in the surgical process.

[0023] While realizing the lightweight semi-supervised surgical instrument segmentation model, the present invention adds a reconstruction loss ( ), ensuring that the segmentation results in the X-axis, Y-axis, and Z-axis directions output by the three student networks have minimal loss after fusion and reconstruction. After fusion and reconstruction, the segmentation result can achieve the effect closest to that obtained by performing surgical instrument segmentation on the three-dimensional ultrasound image as a whole, that is, ensuring the overall accuracy of the fusion of the three student networks outputs.

[0024] Among them, the common losses of the teacher-student model are the losses between the student network output and the true value ( ), and the loss between the output of the student network and the output of the teacher network ( ), to ensure that the segmentation results of the three student networks in the X-axis, Y-axis and Z-axis directions are maximized, that is, to ensure the independent accuracy of the three student networks output.

[0025] Therefore, in the present invention, 、 and The loss function for training the student network is constructed to achieve lightweight semi-supervised surgical instrument segmentation model while obtaining the most accurate segmentation accuracy.

[0026] Furthermore, in order to maintain the task synchronization of the three student networks and avoid the deviation of the segmentation results after fusion reconstruction caused by asynchrony, the present invention 、 Hyperparameters for balancing the three student networks are added. The hyperparameters are quantitatively measured based on the loss between the output of each student network and the true value (prediction loss) and the loss between the output of each network and the output of the teacher network (distillation loss). The smaller the prediction loss and distillation loss in the three student networks, the faster the student network will complete the corresponding axial surgical instrument segmentation task. The student network with large prediction loss and distillation loss will be out of sync, or in other words, the student network with small prediction loss and distillation loss outputs high-precision axial segmentation results, while the student network with large prediction loss and distillation loss outputs low-precision axial segmentation results. Reconstructing and fusing the segmentation results output by each of the three student networks will inevitably produce accuracy deviations.

[0027] After adding hyperparameters to balance the three student networks, during the synchronous training of the three student networks, low loss weights are added to the student network with small prediction loss and distillation loss, and high loss weights are added to the student network with high prediction loss and distillation loss. This allows the training attention to be focused on high-weight items, that is, focusing on the training of the student network with high prediction loss and distillation loss, and pulling the training progress of the student network with high prediction loss and distillation loss towards the student network with low prediction loss and distillation loss, so as to achieve the purpose of balancing the progress of the two. Therefore, after adding hyperparameters to balance the three student networks, the present invention allows the three student networks to start training synchronously from low segmentation accuracy to high segmentation accuracy, and to achieve the maximum possible synchronization to end training at segmentation accuracy, so that the three student networks are trained synchronously, and there is no need to subsequently discover the insufficient segmentation accuracy of a student network and retrain it.

[0028] To improve the efficiency of surgical instrument segmentation, the present invention utilizes the knowledge distillation framework of the teacher-student model. A teacher network is used to obtain three student networks for surgical instrument segmentation in the X-axis, Y-axis, and Z-axis directions of a 3D ultrasound image. Each student network performs surgical instrument segmentation in a corresponding axis, thereby converting surgical instrument segmentation based on the 3D ultrasound image as a whole into surgical instrument segmentation based on the three axes of the 3D ultrasound image independently. This results in each student network processing less image feature information than the teacher network during surgical instrument segmentation, and the corresponding network structure is more lightweight. The specific network structure is as follows: The construction method of the teacher network in the network structure includes: The HybridFormer model composed of a CNN network and a Transformer model is used as the feature extraction module in the teacher network, and Mask R-CNN is used as the mask prediction module in the teacher network; The 3D ultrasound image is used as the input of the teacher network, and the surgical instrument segmentation mask in the 3D ultrasound image is used as the output of the teacher network.

[0029] The three student networks in the network structure include: The HybridFormer model composed of a CNN network and a Transformer model is used as the feature extraction module in each student network, and the Mask R-CNN is used as the mask prediction module in each student network; Using the X-axis component image of the three-dimensional ultrasound image as an input item of the first student network, and using the X-axis component of the surgical instrument segmentation mask in the X-axis component image as an output item of the first student network; The Y-axis component image of the three-dimensional ultrasound image is used as an input item of the second student network, and the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image is used as an output item of the second student network; The Z-axis component image of the three-dimensional ultrasound image is used as the input item of the third student network, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image is used as the output item of the third student network.

[0030] The three lightweight student models obtained by the present invention can respectively match the three axial dimensions of three-dimensional ultrasound, achieve parallel processing of extraction and segmentation of image features in the three axial dimensions, and adapt to the real-time, easy deployment, and fast response requirements in the surgical process.

[0031] The training methods of the network structure include: Construct a dataset containing mask-annotated data and mask-unannotated data of three-dimensional ultrasound images; The teacher network is trained using the masked data of the 3D ultrasound images in the dataset, and the surgical instrument segmentation masks of the masked unlabeled data of the 3D ultrasound images output by the teacher network after training are used as pseudo labels for training the student network. Separating the X-axis, Y-axis, and Z-axis components of the mask annotation data of the three-dimensional ultrasound image and the pseudo-label Persudo Labels data of the three-dimensional ultrasound image to obtain the X-axis component, Y-axis component, and Z-axis component of the mask annotation data of the three-dimensional ultrasound image, and the X-axis component, Y-axis component, and Z-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image; Construct loss functions for training three student networks; Based on the loss function, the first student network is trained using the mixed data consisting of the X-axis component of the masked annotation data of the three-dimensional ultrasound image and the X-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, the second student network is trained using the mixed data consisting of the Y-axis component of the masked annotation data of the three-dimensional ultrasound image and the Y-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, and the third student network is trained using the mixed data consisting of the Z-axis component of the masked annotation data of the three-dimensional ultrasound image and the Z-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, so as to complete the training of the three student networks.

[0032] In the present invention, 、 and The loss function for training the student network can achieve the most accurate segmentation accuracy while making the semi-supervised surgical instrument segmentation model lightweight. The details are as follows: The loss function is: ; in, , ; , ; ; Where Ls is the total loss function of the three student networks, is the total distillation loss of the three student networks, is the total prediction loss of the three student networks, is the total reconstruction loss of the three student networks, are x, y, z identifiers, for The axis corresponds to the distillation loss of the student network, for The axis corresponds to the prediction loss of the student network, for The axis corresponds to the output of the student network, Persudo Labels are the pseudo labels output by the teacher network. The component on the axis, The ground truth of surgical instrument segmentation mask for mask annotation data of 3D ultrasound images is The component on the axis, for The axes correspond to the balanced hyperparameters of the student network, The ground truth value of surgical instrument segmentation mask for mask annotation data of 3D ultrasound images, Persudo Labels are the pseudo labels output by the teacher network.

[0033] The present invention adds reconstruction loss to the loss function of the three student networks based on the universal loss of the teacher-student model ( ), ensuring that the segmentation results in the X-axis, Y-axis, and Z-axis directions output by the three student networks have minimal loss after fusion and reconstruction. After fusion and reconstruction, the segmentation result can achieve the effect closest to that obtained by performing surgical instrument segmentation on the three-dimensional ultrasound image as a whole, that is, ensuring the overall accuracy of the fusion of the three student networks outputs.

[0034] Among them, the common losses of the teacher-student model are the losses between the student network output and the true value ( ), and the loss between the output of the student network and the output of the teacher network ( ), to ensure that the segmentation results of the three student networks in the X-axis, Y-axis and Z-axis directions are maximized, that is, to ensure the independent accuracy of the three student networks output.

[0035] Among them, the student network corresponding to the X-axis is the first student network, the student network corresponding to the Y-axis is the second student network, and the student network corresponding to the Z-axis is the third student network.

[0036] In order to maintain the task synchronization of the three student networks and avoid the deviation of the segmentation results after fusion and reconstruction caused by asynchrony, the present invention 、 The hyperparameters for balancing the three student networks are added. The hyperparameters are quantified based on the loss between the output of each of the three student networks and the true value (prediction loss) and the loss between the output of each of the three networks and the output of the teacher network (distillation loss). They are as follows: Methods for determining balanced hyperparameters include: Get The distillation loss of the student network corresponding to the axis Hewei The prediction loss of the student network corresponding to the axis ; use and Build The axis corresponds to the balanced hyperparameter of the student network ,in, ; Where, are x, y, z identifiers, for The axis corresponds to the distillation loss of the student network, for The axis corresponds to the prediction loss of the student network.

[0037] After the present invention adds hyperparameters to balance the three student networks, during the synchronous training of the three student networks, low loss weights are added to the student network with small prediction loss and distillation loss, and high loss weights are added to the student network with high prediction loss and distillation loss. This allows the training attention to be focused on high-weight items, that is, the training of the student network with high prediction loss and distillation loss is focused on, and the training progress of the student network with high prediction loss and distillation loss is pulled towards the student network with low prediction loss and distillation loss, so as to achieve the purpose of balancing the progress of the two. Therefore, after the present invention adds hyperparameters to balance the three student networks, the three student networks start training synchronously from low segmentation accuracy to high segmentation accuracy, and as much as possible, end training synchronously with segmentation accuracy, so that the three student networks are trained synchronously, and there is no need to subsequently discover the insufficient segmentation accuracy of a student network and retrain it.

[0038] The parameters of the feature extraction module and mask prediction module in the teacher network and the parameters of the feature extraction module and mask prediction module in the three student networks are iteratively updated using the exponential moving average (EMA) method.

[0039] The semi-supervised surgical instrument segmentation model consists of three trained student networks.

[0040] The method of outputting surgical instrument segmentation results on a three-dimensional ultrasound image using a semi-supervised surgical instrument segmentation model includes: The three student networks in the semi-supervised surgical instrument segmentation model are used to perform feature extraction and mask prediction on the X-axis component image, Y-axis component image, and Z-axis component image of the 3D ultrasound image, thereby obtaining the X-axis component of the surgical instrument segmentation mask in the X-axis component image, the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image. The X-axis component of the surgical instrument segmentation mask in the X-axis component image, the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image are superimposed, and the obtained fusion mask is used as the surgical instrument segmentation result.

[0041] like Figure 2 As shown, the present invention provides a semi-supervised surgical instrument segmentation system based on a teacher-student model, which is applied to a semi-supervised surgical instrument segmentation method based on a teacher-student model. The system includes: A data acquisition unit for acquiring a three-dimensional ultrasonic image for surgical instrument segmentation; A model building unit, which constructs and trains a network structure composed of a single teacher network and three student networks to obtain a semi-supervised surgical instrument segmentation model for segmenting surgical instruments on three-dimensional ultrasound images; The segmentation output unit is used to output the surgical instrument segmentation result on the three-dimensional ultrasound image using the semi-supervised surgical instrument segmentation model.

[0042] The present invention constructs a semi-supervised surgical instrument segmentation model through knowledge distillation of a single teacher network and three student networks, so that the three student networks in the semi-supervised surgical instrument segmentation model respectively match the three dimensions of three-dimensional ultrasound, achieving parallel processing of three-dimensional image feature extraction and segmentation. Each student network only needs to process the image features of one dimension, with less information processing and low structural complexity, which improves efficiency and can meet the timeliness requirements during the operation.

[0043] The above embodiments are merely exemplary embodiments of the present application and are not intended to limit the scope of the present application. The scope of protection of the present application is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present application within the essence and scope of protection of the present application, and such modifications or equivalent substitutions shall also be deemed to fall within the scope of protection of the present application.

Claims

1. A semi-supervised surgical instrument segmentation method based on a teacher-student model, characterized in that: The following steps are involved: acquiring a three-dimensional ultrasound image for surgical instrument segmentation; A network structure consisting of a single teacher network and three student networks was constructed and trained to obtain a semi-supervised surgical instrument segmentation model for segmenting surgical instruments on 3D ultrasound images. The semi-supervised surgical instrument segmentation model is used to output the surgical instrument segmentation results on the three-dimensional ultrasound image.

2. A semi-supervised surgical instrument segmentation method based on a teacher-student model according to claim 1, characterized in that: The method for constructing the teacher network in the network structure includes: The HybridFormer model composed of a CNN network and a Transformer model is used as the feature extraction module in the teacher network, and Mask R-CNN is used as the mask prediction module in the teacher network; The 3D ultrasound image is used as the input of the teacher network, and the surgical instrument segmentation mask in the 3D ultrasound image is used as the output of the teacher network.

3. The semi-supervised surgical instrument segmentation method based on the teacher-student model according to claim 2, characterized in that: The three student networks in the network structure include: The HybridFormer model composed of a CNN network and a Transformer model is used as the feature extraction module in each student network, and the Mask R-CNN is used as the mask prediction module in each student network; Using the X-axis component image of the three-dimensional ultrasound image as an input item of the first student network, and using the X-axis component of the surgical instrument segmentation mask in the X-axis component image as an output item of the first student network; The Y-axis component image of the three-dimensional ultrasound image is used as an input item of the second student network, and the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image is used as an output item of the second student network; The Z-axis component image of the three-dimensional ultrasound image is used as the input item of the third student network, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image is used as the output item of the third student network.

4. The semi-supervised surgical instrument segmentation method based on the teacher-student model according to claim 3, characterized in that: The training method of the network structure includes: Construct a dataset containing mask-annotated data and mask-unannotated data of three-dimensional ultrasound images; The teacher network is trained using the masked data of the 3D ultrasound images in the dataset, and the surgical instrument segmentation masks of the masked unlabeled data of the 3D ultrasound images output by the teacher network after training are used as pseudo labels for training the student network. Separating the X-axis, Y-axis, and Z-axis components of the mask annotation data of the three-dimensional ultrasound image and the pseudo-label Persudo Labels data of the three-dimensional ultrasound image to obtain the X-axis component, Y-axis component, and Z-axis component of the mask annotation data of the three-dimensional ultrasound image, and the X-axis component, Y-axis component, and Z-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image; Construct loss functions for training three student networks; Based on the loss function, the first student network is trained using the mixed data consisting of the X-axis component of the masked annotation data of the three-dimensional ultrasound image and the X-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, the second student network is trained using the mixed data consisting of the Y-axis component of the masked annotation data of the three-dimensional ultrasound image and the Y-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, and the third student network is trained using the mixed data consisting of the Z-axis component of the masked annotation data of the three-dimensional ultrasound image and the Z-axis component of the pseudo-label Persudo Labels data of the three-dimensional ultrasound image, so as to complete the training of the three student networks.

5. The semi-supervised surgical instrument segmentation method based on the teacher-student model according to claim 4, characterized in that: The loss function is: ； in, , ; ， ； ； Where Ls is the total loss function of the three student networks, is the total distillation loss of the three student networks, is the total prediction loss of the three student networks, is the total reconstruction loss of the three student networks, are x, y, z identifiers, for The axis corresponds to the distillation loss of the student network, for The axis corresponds to the prediction loss of the student network, for The axis corresponds to the output of the student network, Persudo Labels are the pseudo labels output by the teacher network. The component on the axis, The ground truth of surgical instrument segmentation mask for mask annotation data of 3D ultrasound images is The component on the axis, for The axes correspond to the balanced hyperparameters of the student network, The ground truth value of surgical instrument segmentation mask for mask annotation data of 3D ultrasound images, Persudo Labels are the pseudo labels output by the teacher network.

6. The semi-supervised surgical instrument segmentation method based on the teacher-student model according to claim 5, characterized in that: The method for determining the balance hyperparameters includes: Get The distillation loss of the student network corresponding to the axis Hewei The prediction loss of the student network corresponding to the axis ; use and Build The axis corresponds to the balanced hyperparameter of the student network ,in, ； Where, are x, y, z identifiers, for The axis corresponds to the distillation loss of the student network, for The axis corresponds to the prediction loss of the student network.

7. The semi-supervised surgical instrument segmentation method based on the teacher-student model according to claim 6, characterized in that: The parameters of the feature extraction module and the mask prediction module in the teacher network and the parameters of the feature extraction modules and the mask prediction modules in the three student networks are iteratively transferred and updated through the exponential moving average (EMA) method.

8. The semi-supervised surgical instrument segmentation method based on the teacher-student model according to claim 7, characterized in that: The semi-supervised surgical instrument segmentation model consists of three trained student networks.

9. The semi-supervised surgical instrument segmentation method based on the teacher-student model according to claim 8, characterized in that: The method of outputting surgical instrument segmentation results on a three-dimensional ultrasound image using a semi-supervised surgical instrument segmentation model includes: The three student networks in the semi-supervised surgical instrument segmentation model are used to perform feature extraction and mask prediction on the X-axis component image, Y-axis component image, and Z-axis component image of the 3D ultrasound image, thereby obtaining the X-axis component of the surgical instrument segmentation mask in the X-axis component image, the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image. The X-axis component of the surgical instrument segmentation mask in the X-axis component image, the Y-axis component of the surgical instrument segmentation mask in the Y-axis component image, and the Z-axis component of the surgical instrument segmentation mask in the Z-axis component image are superimposed to obtain a fused mask as the surgical instrument segmentation result.

10. A semi-supervised surgical instrument segmentation system based on a teacher-student model, characterized in that: A semi-supervised surgical instrument segmentation method based on a teacher-student model as described in any one of claims 1 to 9, the system comprising: A data acquisition unit for acquiring a three-dimensional ultrasonic image for surgical instrument segmentation; A model building unit, which constructs and trains a network structure composed of a single teacher network and three student networks to obtain a semi-supervised surgical instrument segmentation model for segmenting surgical instruments on three-dimensional ultrasound images; The segmentation output unit is used to output the surgical instrument segmentation result on the three-dimensional ultrasound image using the semi-supervised surgical instrument segmentation model.

Images