Target segmentation model training method and target segmentation method

Abstract

The application provides a target segmentation model training method and a target segmentation method, and relates to the technical field of image processing. The method comprises the following steps: obtaining a labeled training data set and an unlabeled training data set; determining a mixed training data set based on the labeled training data set and the unlabeled training data set; training a pre-constructed initial student network model based on the labeled training data set, the mixed training data set, the unlabeled training data set and a pre-constructed initial teacher network model, updating the parameters of the initial teacher network model based on the parameters of a first student network model obtained, and taking the updated teacher network model as a new initial teacher network model; and determining a target segmentation model based on the parameters of the finally updated first student network model. When the student network model is trained, the speed of training the target segmentation model can be accelerated, and the accuracy of the target segmentation model in segmenting surgical instruments can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a target segmentation model training method and a target segmentation method. Background Technology

[0002] Minimally invasive surgery, also known as interventional surgery, involves inserting surgical instruments into the body through external incisions using images from a visual display system for treatment or diagnosis. Therefore, endoscopic image-based instrument segmentation is crucial for robot-assisted minimally invasive surgery.

[0003] Currently, most surgical instrument segmentation methods employ convolutional neural networks (CNNs), which require labeled datasets for training. However, CNNs fail to utilize contextual information, and real-world surgical scenarios are complex, with few labeled data available in clinical datasets, resulting in low accuracy in surgical instrument segmentation. Summary of the Invention

[0004] This invention provides a target segmentation model training method and a target segmentation method to solve the problem of low accuracy in the segmentation of surgical instruments in the prior art.

[0005] This invention provides a method for training a target segmentation model, comprising:

[0006] Obtain a labeled training dataset and an unlabeled training dataset; the labeled training dataset includes multiple first sample images and first label data for each first sample image; the unlabeled dataset includes multiple second sample images;

[0007] Based on the labeled training dataset and the unlabeled training dataset, a hybrid training dataset is determined; the hybrid training dataset includes multiple third sample images and second label data for each of the third sample images;

[0008] Based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and the pre-built initial teacher network model, the pre-built initial student network model is trained to obtain the first student network model.

[0009] Based on the parameters of the first student network model, the parameters of the initial teacher network model are updated, and the updated teacher network model is used as the new initial teacher network model.

[0010] The target segmentation model is determined based on the parameters of the final updated first student network model.

[0011] According to a target segmentation model training method provided by the present invention, determining a hybrid training dataset based on the labeled training dataset and the unlabeled training dataset includes:

[0012] Based on the labeled training dataset and the unlabeled training dataset, at least one set of sample images is determined; each set of sample images includes the first sample image and the second sample image.

[0013] A mixed training dataset is determined based on at least one set of sample images, the initial student network model, and the initial teacher network model; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images.

[0014] According to a target segmentation model training method provided by the present invention, determining a mixed training dataset based on at least one set of sample images, the initial student network model, and the initial teacher network model includes:

[0015] The first sample image in each group of sample images is input into the initial student network model to obtain the first segmented image output by the initial student network model;

[0016] The second sample image from each group of sample images is input into the initial teacher network model to obtain the second segmentation image output by the initial teacher network model;

[0017] Randomly select the first label data corresponding to the target number of pixels in the first segmented image;

[0018] A binary mask is generated based on the first label data corresponding to the target number of pixels;

[0019] Based on the binary mask, the first sample image and the second sample image, as well as the first segmented image and the second segmented image, are mixed to obtain the third sample image and the second label data of the third sample image;

[0020] Based on the third sample image and the second label data, a mixed training dataset is determined.

[0021] According to a target segmentation model training method provided by the present invention, the first student network model is obtained by training an initial student network model based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and the initial teacher network model, comprising:

[0022] Based on the first segmented image and the second label data, calculate the first classification cross-entropy loss function;

[0023] The third sample image from the mixed training dataset is input into the initial student network model to obtain the third segmentation image output by the initial student network model;

[0024] Based on the pixel values ​​of at least one pixel block corresponding to the third segmented image and the binary mask corresponding to the third sample image, the structural similarity loss function is calculated;

[0025] The first student network model is determined based on the first classification cross-entropy loss function, the structural loss function, the unlabeled training dataset, and the initial teacher network model.

[0026] According to a target segmentation model training method provided by the present invention, the step of determining the first student network model based on the first classification cross-entropy loss function, the second classification cross-entropy loss function, the first structural loss function, the unlabeled training dataset, and the initial teacher network model includes:

[0027] The second sample image from the unlabeled training dataset is input into the initial teacher network model to obtain the fourth segmentation image output by the initial teacher network model;

[0028] Based on the fourth segmented image, determine the first probability and pseudo-label data;

[0029] Based on the fourth segmented image, the first probability, and the pseudo-label data, calculate the second classification cross-entropy loss function and the second structural similarity loss function;

[0030] Based on the first classification cross-entropy loss function, the structural similarity loss function, and the second classification cross-entropy loss function, the target loss function is determined;

[0031] Based on the target loss function, the first student network model is determined.

[0032] According to a target segmentation model training method provided by the present invention, the step of determining the first student network model based on the target loss function includes:

[0033] If the target loss function does not meet the preset conditions, repeat the steps for determining the target loss function during the training process described above.

[0034] If the target loss function satisfies the preset conditions, the initial student network model is determined as the first student network model.

[0035] According to a target segmentation model training method provided by the present invention, the initial student network model and the initial teacher network model have the same structure; the initial student network model includes at least one Transformer module, a multilayer perceptron, and a decoder; the Transformer module is used to extract feature information of the image; the multilayer perceptron is used to fuse the feature information extracted by each Transformer module; and the decoder is used to decode the fused feature information.

[0036] The present invention also provides a target segmentation method, comprising:

[0037] Obtain the target image of the object to be segmented;

[0038] The target image is input into the target segmentation model to obtain the segmentation result output by the target segmentation model; the target segmentation model is trained based on the target segmentation model training method described in any of the above methods.

[0039] According to a target segmentation method provided by the present invention, the method further includes:

[0040] Based on the segmentation results, determine the projected 3D shape and key point information corresponding to the segmentation results;

[0041] Align the projected 3D shape with the segmentation result;

[0042] Based on the key point information and the historical information of the target image, the tracking information corresponding to the target image is determined;

[0043] Based on the tracking information and the aligned information, the pose of the target object is determined.

[0044] The present invention also provides a target segmentation model training device, comprising:

[0045] The first acquisition module is used to acquire a labeled training dataset and an unlabeled training dataset; the labeled training dataset includes multiple first sample images and first label data for each first sample image; the unlabeled dataset includes multiple second sample images.

[0046] The first determining module is used to determine a mixed training dataset based on the labeled training dataset and the unlabeled training dataset; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images;

[0047] The training module is used to train the initial student network model based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and the initial teacher network model to obtain the first student network model.

[0048] An update module is used to update the parameters of the initial teacher network model based on the parameters of the first student network model;

[0049] The second determination module is used to determine the target segmentation model based on the parameters of the updated initial teacher network model.

[0050] The present invention also provides a target segmentation device, comprising:

[0051] The second acquisition module is used to acquire the target image of the target object to be segmented;

[0052] The segmentation module is used to input the target image into the target segmentation model and obtain the segmentation result output by the target segmentation model; the target segmentation model is trained based on the target segmentation model training method described in any of the above methods.

[0053] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the target segmentation model training method as described above, or to implement the target segmentation method as described above.

[0054] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the target segmentation model training method as described above, or implements the target segmentation method as described above.

[0055] The present invention also provides a computer program product, comprising a computer program that, when executed by a processor, implements the target segmentation model training method as described above, or implements the target segmentation method as described above.

[0056] The present invention provides a target segmentation model training method and a target segmentation method. By acquiring labeled and unlabeled training datasets, a hybrid training dataset is determined. The labeled training dataset includes multiple first sample images and first label data for each first sample image; the unlabeled dataset includes multiple second sample images; and the hybrid training dataset includes multiple third sample images and second label data for each third sample image. Then, based on the labeled training dataset, the hybrid training dataset, the unlabeled training dataset, and a pre-built initial teacher network model, a pre-built initial student network model is trained to obtain a first student network model. Next, the parameters of the initial teacher network model are updated based on the parameters of the trained first student network model, and the updated teacher network model is used as the new initial teacher network model. Finally, the target segmentation model is determined based on the parameters of the finally updated first student network model. Because a labeled training dataset, a hybrid training dataset, and an unlabeled training dataset are used, the diversity of training samples is increased. Therefore, when training the initial student network model, the training speed of the target segmentation model can be accelerated, while simultaneously improving the accuracy of the target segmentation model in segmenting surgical instruments. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0058] Figure 1 This is one of the flowcharts illustrating the target segmentation model training method provided by the present invention;

[0059] Figure 2 This is a schematic diagram of the data augmentation method provided by the present invention;

[0060] Figure 3 This is the second flowchart illustrating the target segmentation model training method provided by the present invention;

[0061] Figure 4 This is a flowchart illustrating the target segmentation method provided by the present invention;

[0062] Figure 5 This is a schematic diagram of the pose estimation process for the target object provided by the present invention;

[0063] Figure 6 This is a schematic diagram of the target segmentation model training device provided by the present invention;

[0064] Figure 7 This is a schematic diagram of the target segmentation device provided by the present invention;

[0065] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0067] The following is combined Figures 1-5 The present invention describes the target segmentation model training method and the target segmentation method.

[0068] Figure 1 This is a flowchart illustrating the target segmentation model training method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 101-105; wherein,

[0069] Step 101: Obtain a labeled training dataset and an unlabeled training dataset; the labeled training dataset includes multiple first sample images and first label data for each first sample image; the unlabeled dataset includes multiple second sample images.

[0070] It should be noted that the target segmentation model training method provided by the present invention is applicable to the segmentation of target objects, such as the segmentation of surgical instruments. The execution subject of the method can be a target segmentation model training device, such as an electronic device, or a control module in the target segmentation model training device for executing the target segmentation model training method.

[0071] Step 102: Based on the labeled training dataset and the unlabeled training dataset, determine the hybrid training dataset; the hybrid training dataset includes multiple third sample images and second label data for each of the third sample images.

[0072] Specifically, based on the obtained labeled and unlabeled training datasets, a data augmentation method is used to determine a hybrid training dataset. That is, by mixing the labeled and unlabeled training datasets, hybrid training data and corresponding second label data (pseudo-labels) are generated, which increases the diversity of training samples and improves the accuracy of semantic segmentation and pose estimation of surgical instruments.

[0073] Step 103: Based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and the pre-built initial teacher network model, train the pre-built initial student network model to obtain the first student network model.

[0074] Optionally, the initial student network model and the initial teacher network model have the same structure; the initial student network model includes at least one Transformer module, a multilayer perceptron, and a decoder; the Transformer module is used to extract feature information of the image; the multilayer perceptron is used to fuse the feature information extracted by each Transformer module; and the decoder is used to decode the fused feature information.

[0075] Specifically, an initial teacher network model and an initial student network model are pre-constructed. The initial teacher network model and the initial student network model have the same structure, which includes at least one Transformer module, a multilayer perceptron, and a decoder. The Transformer module is used to extract feature information from the image; the multilayer perceptron is used to fuse the feature information extracted by each Transformer module; and the decoder is used to decode the fused feature information.

[0076] In practice, the semi-supervised Transformer-based network framework SSFormer trains a pre-built initial student network model using labeled training datasets, mixed training datasets, unlabeled training datasets, and a pre-built initial teacher network model, resulting in a first student network model. Within the SSFormer framework, the first student network model learns knowledge from unlabeled data in the unlabeled training dataset. This first student model's parameters are then used to update the parameters of the initial teacher network model. The updated teacher network model is then used as the new initial teacher network model to train the first student network model, updating its parameters repeatedly. Finally, the target segmentation model is determined based on the updated parameters of the first student network model, achieving good segmentation performance.

[0077] Furthermore, a robust network framework is required for the Transformer. Since learning ability can be improved from domain-invariant properties, robustness is crucial for achieving good segmentation performance. The Transformer achieves competitive performance in semi-supervised learning. The weight calculation method of the Transformer differs from that of convolutional neural networks. In the self-attention mechanism, weights are dynamically calculated based on the similarity between each pair of taps. Because self-similarity operations are more general and adaptable than convolution operations, self-similarity operations are performed on the Transformer.

[0078] Step 104: Based on the parameters of the first student network model, update the parameters of the initial teacher network model, and use the updated teacher network model as the new initial teacher network model.

[0079] Specifically, the teacher network model and the student network model have the same structure, but differ in how their parameters are updated. Teacher network model f φ The parameters will be based on the student network model f θ The parameters are updated. Typically, during model training, the teacher network model f... φ The weights are set to the values ​​of the student network model f after the t-th iteration of training. θ The exponential moving average (EMA) of the weights is used to increase the stability of the forecast.

[0080] In practice, based on the parameters of the first student network model obtained through training, the parameters of the initial teacher network model are updated using formula (1); where,

[0081] φ t+1 ←αφ t +(1-α)θ t (1)

[0082] Where, φ t+1 Let α represent the parameters of the initial teacher network model during the (t+1)th iteration of training, and φ represent the adjustment coefficient of EMA. t Let θ represent the parameters of the initial teacher network model during the t-th iteration of training. t This represents the parameters of the initial student network model during the t-th iteration of training.

[0083] Student network model f θ Training augmented target data, teacher network model f φ Using semi-supervised learning on unenhanced target data to generate pseudo-labels is a highly effective method for surgical instrument segmentation.

[0084] During iterative training, the updated teacher network model is used as the new initial teacher network model, that is, the new initial teacher network model is used as the initial teacher network model for the next iteration of training, and the first student network model is trained.

[0085] Step 105: Determine the target segmentation model based on the parameters of the final updated first student network model.

[0086] Specifically, at the end of training, the parameters of the final updated first student network model are determined as the parameters of the target segmentation model, thus obtaining the target segmentation model.

[0087] The target segmentation model training method provided by this invention determines a hybrid training dataset by acquiring labeled and unlabeled training datasets. The labeled training dataset includes multiple first sample images and first label data for each first sample image; the unlabeled dataset includes multiple second sample images; and the hybrid training dataset includes multiple third sample images and second label data for each third sample image. Then, based on the labeled training dataset, the hybrid training dataset, the unlabeled training dataset, and a pre-built initial teacher network model, a pre-built initial student network model is trained to obtain a first student network model. Next, the parameters of the initial teacher network model are updated based on the parameters of the trained first student network model, and the updated teacher network model is used as the new initial teacher network model. Finally, the target segmentation model is determined based on the parameters of the finally updated first student network model. By using labeled, hybrid, and unlabeled training datasets, the diversity of training samples is increased. Therefore, when training the initial student network model, the training speed of the target segmentation model can be accelerated, while simultaneously improving the accuracy of the target segmentation model in segmenting surgical instruments.

[0088] Optionally, the specific implementation of step 102 above includes:

[0089] 1) Based on the labeled training dataset and the unlabeled training dataset, determine at least one set of sample images; each set of sample images includes the first sample image and the second sample image.

[0090] Specifically, based on the labeled training dataset and the unlabeled training dataset, a first sample image and a second sample image are randomly selected from the labeled training dataset and the unlabeled training dataset, respectively. After multiple random selections, at least one set of sample images is obtained; each set of sample images includes the first sample image and the second sample image.

[0091] 2) Based on at least one set of sample images, the initial student network model, and the initial teacher network model, a mixed training dataset is determined; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images.

[0092] Specifically, based on at least one set of sample images, an initial student network model, and an initial teacher network model, a mixed training dataset can be further determined; wherein, the mixed training dataset includes multiple third sample images and second label data (pseudo-labels) for each third sample image.

[0093] Optionally, determining the mixed training dataset based on at least one set of sample images, the initial student network model, and the initial teacher network model includes:

[0094] a) Input the first sample image from each group of sample images into the initial student network model to obtain the first segmented image output by the initial student network model.

[0095] Specifically, during model training, the first sample image from each set of sample images is first input into the initial student network model, which yields the first segmentation image S output by the initial student network model. A .

[0096] b) Input the second sample image from each group of sample images into the initial teacher network model to obtain the second segmentation image output by the initial teacher network model.

[0097] Specifically, by inputting the second sample image from each group of sample images into the initial teacher network model, the second segmentation image S output by the initial teacher network model can be obtained. B .

[0098] c) Randomly select the first label data corresponding to the target number of pixels in the first segmented image.

[0099] Specifically, the first segmented image S is randomly selected. A The first label data corresponding to the number of pixels in the target image, where the number of targets can be the first segmented image S. A Half the number of all pixels in the image.

[0100] d) Generate a binary mask based on the first label data corresponding to the target number of pixels.

[0101] Specifically, based on the first label data corresponding to the target number of pixels, a binary mask M is generated by using the maximum value of the independent variable point set (argmax) function, and the pixel values ​​of the target number of pixels are set to 1, while the pixel values ​​of other pixels are set to 0.

[0102] e) Based on the binary mask, the first sample image and the second sample image, as well as the first segmented image and the second segmented image, are mixed to obtain a mixed sample image and the second label data of the mixed sample image.

[0103] Specifically, based on the binary mask, the first sample image and the second sample image are blended into a mixed sample image X. A (Enhanced Image), where the blended sample image X A The pixels are derived from the portion of the first sample image A with a mask value of 1, and from the second sample image B. Similarly, based on the binary mask, the first segmented image S is segmented... A Second segmented image S B The data is mixed to generate the second label data Y. A Due to the hybrid strategy, artifacts may appear, but their impact will gradually decrease as the training process progresses.

[0104] f) Enhance the mixed sample image to obtain a third sample image.

[0105] Specifically, by using methods such as rotation, noise, affine transformation, elastic deformation, or cropping to enhance the mixed sample image, a third sample image, i.e., the enhanced mixed image, can be obtained.

[0106] g) Determine a hybrid training dataset based on the third sample image and the second label data.

[0107] Specifically, the mixed training dataset can be determined based on the third sample image and the second label data of the third sample image.

[0108] Figure 2 This is a schematic diagram of the data augmentation method provided by the present invention, as shown below. Figure 2 As shown, a first sample image A and a second sample image B are randomly selected from the labeled training dataset and the unlabeled training dataset, respectively. This random selection is repeated multiple times to obtain at least one set of sample images; each set of sample images includes the first sample image and the second sample image. The first sample image from each set is input into the initial student network model to obtain the first segmentation image S output by the initial student network model. A The second sample image from each group of sample images is input into the initial teacher network model to obtain the second segmentation image S output by the initial teacher network model. B Randomly select the first segmented image S A The first label data corresponding to the number of pixels in the target image, where the number of targets can be the first segmented image S. AThe first sample image is calculated as half the number of all pixels in the image. A binary mask M is generated using the argmax function to predict the maximum number of independent variable points. The pixel values ​​of the target number of pixels are set to 1, and the pixel values ​​of other pixels are set to 0. Based on the binary mask, the first sample image and the second sample image are blended into a mixed sample image X. A Then, the mixed sample image is enhanced to obtain the third sample image; the first segmented image S is then... A Second segmented image S B The data is mixed to generate the second label data Y. A The mixed training dataset is determined based on the third sample image and the second label data of the third sample image.

[0109] The target segmentation model training method provided by this invention determines at least one set of sample images using a labeled training dataset and an unlabeled training dataset. Each set of sample images includes a first sample image and a second sample image. The first sample image from each set is input into an initial student network model to obtain a first segmentation image output by the initial student network model. The second sample image from each set is then input into an initial teacher network model to obtain a second segmentation image output by the initial teacher network model. First label data corresponding to a target number of pixels in the first segmentation image is randomly selected. A binary mask is generated based on the first label data corresponding to the target number of pixels. The first sample image and the second sample image, as well as the first segmentation image and the second segmentation image, are then mixed according to the binary mask to obtain a mixed sample image and second label data for the mixed sample image. The mixed sample image is enhanced to obtain a third sample image. Finally, a mixed training dataset is determined based on the third sample image and the second label data. By pasting half of the pixel values ​​of the first sample image onto the second sample image, and pasting half of the semantic class of the first segmentation image onto the second segmentation image, a hybrid sample image and second label data are generated. This results in a hybrid sample image and second label data with diversity and novelty. When training the initial student network model, this can accelerate the training speed of the target segmentation model and improve the accuracy of the target segmentation model in segmenting surgical instruments.

[0110] Optionally, the specific implementation of step 103 above includes:

[0111] 1) Calculate the first classification cross-entropy loss function based on the first segmented image and the first label data.

[0112] Specifically, in the semi-supervised learning process of model training, in order to achieve good performance of the initial student network model, the second sample images in the unlabeled training dataset are used. Where, N UThis represents the total number of second sample images in the unlabeled training dataset. This represents the label of the i-th second sample image, i.e., the label of the second sample image. When unavailable, the initial student network model is trained using a labeled training dataset and a mixed training dataset. The training uses one-hot encoded labels (first label data) from the labeled training dataset. and the first sample image in the labeled training dataset Where, N L This represents the total number of the first sample images in the labeled training dataset. This represents the i-th first sample image. This represents the first label data of the i-th first sample image in the labeled training dataset.

[0113] To achieve clearer boundaries for surgical instrument segmentation, a hybrid loss function is proposed to train the student network f. θ The mixed loss function is expressed by formula (2):

[0114]

[0115] in, This represents the first-class cross-entropy loss function. This represents the cross-entropy loss function for the second classification. Let represent the first structural similarity loss function.

[0116] The classification cross-entropy loss function has been widely used in semantic segmentation. During the iterative training process, the first classification cross-entropy loss function is calculated based on the first segmented image and the first label data of the first sample image output by the initial student network model. The first classification cross-entropy loss function is expressed by formula (3):

[0117]

[0118] Where C represents the total number of categories in the first label data, c represents the c-th category, H represents the height of the first sample image, and W represents the width of the first sample image.

[0119] Furthermore, by inputting the second sample from the unlabeled training dataset into the initial teacher network model, pseudo-labeled data generated by the initial teacher network model can be obtained.

[0120] 2) Input the third sample image from the mixed training dataset into the initial student network model to obtain the third segmentation image output by the initial student network model.

[0121] Specifically, by inputting the third sample image from the mixed training dataset into the initial student network model, the third segmentation image output by the initial student network model can be obtained.

[0122] 3) Based on the third segmented image and the second label data, calculate the second classification cross-entropy loss function.

[0123] Specifically, based on the third segmentation image and the second label data, the second classification cross-entropy loss function can be calculated using the above formula (3).

[0124] 4) Calculate the first structural similarity loss function based on the pixel values ​​of at least one pixel block corresponding to the third segmented image and the binary mask corresponding to the third sample image.

[0125] It should be noted that the structural similarity loss function is used to extract structural features from the ground-value label data, thereby extracting the structural information of the image. Therefore, the training loss function includes the structural similarity loss function.

[0126] Specifically, the third segmented image is divided into at least one pixel block. Based on the pixel values ​​of the at least one pixel block corresponding to the third segmented image and the binary mask corresponding to the third sample image, the first structural similarity loss function is calculated using formula (4); let v = {v j :j=1,...,H×W} is the pixel value corresponding to two image blocks cropped from the predicted third segmentation image, b={b j :j=1,...,H×W} is the binary mask corresponding to the third sample image, where the image patch size is H×W. Therefore, formula (4) is expressed as:

[0127]

[0128] Where, σ vb σ represents the covariance of v and b; v and σ b Let v and b represent the standard deviations, respectively; μ v and μ b Let v and b represent the average difference between v and b, respectively; a1 is set to 0.01. 2 a2 is set to 0.03 2 To avoid situations where the value is divided by zero.

[0129] 4) Determine the first student network model based on the first classification cross-entropy loss function, the second classification cross-entropy loss function, the first structural loss function, the unlabeled training dataset, and the initial teacher network model.

[0130] Specifically, based on the first classification cross-entropy loss function, the second classification cross-entropy loss function, the first structural loss function, the unlabeled training dataset, and the initial teacher network model, the first student network model can be further determined.

[0131] Optionally, determining the first student network model based on the first classification cross-entropy loss function, the second classification cross-entropy loss function, the first structural loss function, the unlabeled training dataset, and the initial teacher network model includes:

[0132] 1) Input the second sample image from the unlabeled training dataset into the initial teacher network model to obtain the fourth segmentation image output by the initial teacher network model.

[0133] Specifically, in each iteration of training, the second sample image from the unlabeled training dataset is input into the initial teacher network model, which yields the fourth segmentation image output by the initial teacher network model.

[0134] 2) Based on the fourth segmented image, determine the first probability and pseudo-label data.

[0135] Specifically, pseudo-label data and a first probability are determined based on the fourth segmentation image, wherein the maximum normalized (softmax) probability of pixel values ​​greater than the threshold τ in the fourth segmentation image is taken as the first probability, expressed by formula (5):

[0136]

[0137] in, Let c represent the first probability, and c′ represent the number of categories in the pseudo-label data.

[0138] The pseudo-label data is represented by formula (6):

[0139]

[0140] Here, [·] represents Iverson brackets. Furthermore, confidence estimation was also used to generate pseudo-labels.

[0141] 3) Based on the fourth segmented image, the first probability, and the pseudo-label data, calculate the second classification cross-entropy loss function and the second structural similarity loss function.

[0142] Specifically, pseudo-labels, quality estimates, and structural similarity information are used to train the network f on the unlabeled dataset using a different hybrid loss function. θ The hybrid loss function is expressed by formula (7):

[0143]

[0144] in, This represents the cross-entropy loss function for the second classification. This represents the second structural similarity loss function.

[0145] Furthermore, based on the fourth segmentation image, the first probability, and the pseudo-label data, the second classification cross-entropy loss function and the second structural similarity loss function are calculated using formulas (8) and (9), where,

[0146]

[0147]

[0148] Where, v = {v j :j=1,...,H×W} is the pixel value corresponding to the image patch cropped from the prediction result of the fourth segmented image, b′={b′ j :j=1,...,H×W} is the pseudo-label corresponding to the fourth segmentation image.

[0149] When the student network is trained on augmented mixed data, the teacher network uses unaugmented mixed data for semi-supervised learning to generate pseudo-labels.

[0150] 4) Determine the target loss function based on the first classification cross-entropy loss function, the first structural similarity loss function, the second classification cross-entropy loss function, and the second structural similarity loss function.

[0151] Specifically, the target loss function is calculated using formula (10) based on the first classification cross-entropy loss function, the first structural similarity loss function, the second classification cross-entropy loss function, and the second structural similarity loss function. Formula (10) is expressed as:

[0152]

[0153] 5) Based on the target loss function, determine the first student network model.

[0154] Specifically, the first student network model can be determined based on the calculated target loss function.

[0155] Optionally, determining the first student network model based on the target loss function includes:

[0156] If the target loss function does not meet the preset conditions, repeat the steps of determining the target loss function during the training process; if the target loss function meets the preset conditions, determine the initial student network model as the first student network model.

[0157] Specifically, it is determined whether the target loss function meets the preset conditions. If the target loss function does not meet the preset conditions, the steps for determining the target loss function in the above training process are repeated until the target loss function meets the preset conditions. If the target loss function meets the preset conditions, the training ends, and the initial student network model from the last training session is determined as the first student network model.

[0158] Figure 3 This is the second flowchart illustrating the target segmentation model training method provided by this invention. Figure 3 As shown, a hybrid training dataset is determined based on the labeled and unlabeled training datasets. The initial student network model is trained using both the labeled and hybrid training datasets. The initial student network model includes Transformer module 1, Transformer module 2, Transformer module 3, Transformer module 4, a multilayer perceptron, and a decoder. The first sample image from the labeled training dataset and the third sample image from the hybrid training dataset are input into Transformer module 1 of the initial student network model. Transformer module 1 divides the first and third sample images into 4×4 image blocks, obtaining the corresponding first feature map F1 output by Transformer module 1. The size of the first feature map is... The number of channels is C1, thus preserving the details of surgical instrument segmentation; the first feature map is input to Transformer module 2 to obtain the second feature map F2 output by Transformer module 2, the size of which is... The number of channels is C2; the second feature map is input to Transformer module 3 to obtain the third feature map F3 output by Transformer module 3. The size of the third feature map is... The number of channels is C3; the third feature map is input to Transformer module 4 to obtain the fourth feature map F4 output by Transformer module 4. The size of the fourth feature map is... The number of channels is C4. To handle high-resolution features, a reduced sequence approach is used in the self-attention block. Multi-level feature maps. Generated by a Transformer encoder. Downsampling is achieved through overlapping image patches to preserve local continuity. The first, second, third, and fourth feature maps are input into a multilayer perceptron, which fuses the multiple feature maps to obtain the fifth feature map output by the multilayer perceptron. The size of the fifth feature map is... Before feature fusion, each feature map F i Embedded into the same number of channels C eIn the middle, perform 1×1 convolution, and for each feature map F i Bilinear upsampling is performed to the size of the first feature map F1, and all upsampled feature maps are concatenated; the concatenated fifth feature map is input into the decoder, and the decoder outputs the final first segmentation image and the third segmentation image.

[0159] It should be noted that in existing results, the decoder of the Transformer used for semantic segmentation only utilizes local information. Since additional contextual information can improve the robustness of semantic segmentation, the decoder uses this contextual information. In the SSFormer network framework, cross-feature contextual information from different encoder layers is utilized.

[0160] Further, the second sample image from the unlabeled training dataset is input into the initial teacher network model to obtain the fourth segmentation image (pseudo-label) output by the initial teacher network model; based on the first segmentation image and the first label data, the first classification cross-entropy loss function is calculated; based on the third segmentation image and the second label data, the second classification cross-entropy loss function is calculated; based on the pixel value of at least one pixel block corresponding to the fourth segmentation image and the binary mask corresponding to the third sample image, the first structural similarity loss function is calculated; based on the fourth segmentation image, the first probability and pseudo-label data are determined; based on the fourth segmentation image, the first probability and the pseudo-label data, the second classification cross-entropy loss function and the second structural similarity loss function are calculated; then, based on the first classification cross-entropy loss function, the first structural similarity loss function, the second classification cross-entropy loss function and the second structural similarity loss function, the target loss function is determined; it is determined whether the target loss function meets the preset conditions. If the target loss function meets the preset conditions, the initial student network model in the final training is determined as the updated first student network model, and the finally updated first student network model is determined as the target segmentation model.

[0161] Figure 4 This is a flowchart illustrating the target segmentation method provided by the present invention, as shown below. Figure 4 As shown, the method includes steps 401-402; wherein,

[0162] Step 401: Obtain the target image of the target object to be segmented.

[0163] Specifically, the target object can be surgical instruments or other objects to be segmented.

[0164] Step 402: Input the target image into the target segmentation model to obtain the segmentation result output by the target segmentation model; the target segmentation model is trained based on the target segmentation model training method described in any of the above methods.

[0165] Specifically, by inputting the target image into the target segmentation model, the segmentation result output by the target segmentation model can be obtained; wherein, the segmentation result is a segmented image including the target object.

[0166] The target segmentation method provided by this invention can obtain the segmentation result output by the target segmentation model by inputting the target image of the target object to be segmented into the target segmentation model, thereby realizing the segmentation of the target object in the target image of the target object to be segmented and improving the segmentation accuracy and efficiency.

[0167] Optionally, after obtaining the segmentation result, the method further includes:

[0168] Based on the segmentation result, the projected 3D shape and key point information corresponding to the segmentation result are determined; the projected 3D shape and the segmentation result are aligned; based on the key point information and the historical information of the target image, the tracking information corresponding to the target image is determined; based on the tracking information and the aligned information, the pose of the target object is determined.

[0169] Specifically, addressing the issue of low accuracy in pose estimation for surgical instruments in medical scenarios, a pose estimation algorithm is proposed. Based on the region image corresponding to the segmentation result, a mutual transformation between the camera coordinate system and the 3D coordinates of the surgical instrument's center point is performed to determine the projected 3D shape corresponding to the segmentation result. Simultaneously, key point information of the target object (surgical instrument) in the region image corresponding to the segmentation result is extracted. The projected 3D shape and the segmentation result are aligned. Furthermore, based on the extracted key point information and historical information of the target image, the target object is tracked frame by frame to determine the tracking information of the target object in the target image, improving the algorithm's detection performance for surgical instruments and further enhancing the accuracy of pose estimation. Finally, based on the tracking information and the aligned and registered region image information, the specific pose of the target object in the target image is further determined, enabling accurate, fast, and robust estimation of the surgical instrument's pose in the target image.

[0170] Figure 5 This is a schematic diagram of the pose estimation process for the target object provided by the present invention, as shown below. Figure 5 As shown, the target image of the target object to be segmented is obtained, and the target image is input into the target segmentation model to obtain the segmentation result (region image) output by the target segmentation model; the projected three-dimensional shape corresponding to the segmentation result is determined; the key point information corresponding to the segmentation result is extracted; the projected three-dimensional shape and the segmentation result are aligned; the tracking information corresponding to the target image is determined according to the key point information and the historical information of the target image; and the pose of the target object is determined according to the tracking information and the aligned information.

[0171] The target segmentation model training device and target segmentation device provided by the present invention are described below. The target segmentation model training device described below and the target segmentation model training method described above can be referred to in correspondence with each other. The target segmentation device described below and the target segmentation method described above can be referred to in correspondence with each other.

[0172] Figure 6 This is a schematic diagram of the target segmentation model training device provided by the present invention, as shown below. Figure 6 As shown, the target segmentation model training device 600 includes: a first acquisition module 601, a first determination module 602, a training module 603, an update module 604, and a second determination module 605; wherein,

[0173] The first acquisition module 601 is used to acquire a labeled training dataset and an unlabeled training dataset; the labeled training dataset includes multiple first sample images and first label data for each first sample image; the unlabeled dataset includes multiple second sample images.

[0174] The first determining module 602 is used to determine a mixed training dataset based on the labeled training dataset and the unlabeled training dataset; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images;

[0175] Training module 603 is used to train the initial student network model based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and the initial teacher network model to obtain the first student network model.

[0176] The update module 604 is used to update the parameters of the initial teacher network model based on the parameters of the first student network model, and use the updated teacher network model as the new initial teacher network model.

[0177] The second determining module 605 is used to determine the target segmentation model based on the parameters of the final updated first student network model.

[0178] The target segmentation model training device provided by this invention determines a hybrid training dataset by acquiring labeled and unlabeled training datasets. The labeled training dataset includes multiple first sample images and first label data for each first sample image; the unlabeled dataset includes multiple second sample images; and the hybrid training dataset includes multiple third sample images and second label data for each third sample image. Then, based on the labeled training dataset, the hybrid training dataset, the unlabeled training dataset, and a pre-built initial teacher network model, a pre-built initial student network model is trained to obtain a first student network model. The parameters of the initial teacher network model are then updated based on the parameters of the trained first student network model, and the updated teacher network model is used as the new initial teacher network model. Finally, the target segmentation model is determined based on the parameters of the finally updated first student network model. Because it uses labeled, hybrid, and unlabeled training datasets, the diversity of training samples is increased. Therefore, when training the initial student network model, the training speed of the target segmentation model can be accelerated, while simultaneously improving the accuracy of the target segmentation model in segmenting surgical instruments.

[0179] Optionally, the first determining module 602 is specifically used for:

[0180] Based on the labeled training dataset and the unlabeled training dataset, at least one set of sample images is determined; each set of sample images includes the first sample image and the second sample image.

[0181] A mixed training dataset is determined based on at least one set of sample images, the initial student network model, and the initial teacher network model; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images.

[0182] Optionally, the first determining module 602 is specifically used for:

[0183] The first sample image in each group of sample images is input into the initial student network model to obtain the first segmented image output by the initial student network model;

[0184] The second sample image from each group of sample images is input into the initial teacher network model to obtain the second segmentation image output by the initial teacher network model;

[0185] Randomly select the first label data corresponding to the target number of pixels in the first segmented image;

[0186] A binary mask is generated based on the first label data corresponding to the target number of pixels;

[0187] Based on the binary mask, the first sample image and the second sample image, as well as the first segmented image and the second segmented image, are mixed to obtain a mixed sample image and the second label data of the mixed sample image;

[0188] The mixed sample image is enhanced to obtain a third sample image;

[0189] Based on the third sample image and the second label data, a mixed training dataset is determined.

[0190] Optionally, the training module 603 is specifically used for:

[0191] Based on the first segmented image and the first label data, calculate the first classification cross-entropy loss function;

[0192] The third sample image from the mixed training dataset is input into the initial student network model to obtain the third segmentation image output by the initial student network model;

[0193] Based on the third segmented image and the second label data, calculate the second classification cross-entropy loss function;

[0194] The first structural similarity loss function is calculated based on the pixel value of at least one pixel block corresponding to the third segmented image and the binary mask corresponding to the third sample image;

[0195] The first student network model is determined based on the first classification cross-entropy loss function, the second classification cross-entropy loss function, the first structural loss function, the unlabeled training dataset, and the initial teacher network model.

[0196] Optionally, the training module 603 is specifically used for:

[0197] The second sample image from the unlabeled training dataset is input into the initial teacher network model to obtain the fourth segmentation image output by the initial teacher network model;

[0198] Based on the fourth segmented image, determine the first probability and pseudo-label data;

[0199] Based on the fourth segmented image, the first probability, and the pseudo-label data, calculate the second classification cross-entropy loss function and the second structural similarity loss function;

[0200] Based on the first classification cross-entropy loss function, the first structural similarity loss function, the second classification cross-entropy loss function, and the second structural similarity loss function, the target loss function is determined;

[0201] Based on the target loss function, the first student network model is determined.

[0202] Optionally, the training module 603 is specifically used for:

[0203] If the target loss function does not meet the preset conditions, repeat the steps for determining the target loss function during the training process described above.

[0204] If the target loss function satisfies the preset conditions, the initial student network model is determined as the first student network model.

[0205] Optionally, the initial student network model and the initial teacher network model have the same structure; the initial student network model includes at least one Transformer module, a multilayer perceptron, and a decoder; the Transformer module is used to extract feature information of the image; the multilayer perceptron is used to fuse the feature information extracted by each Transformer module; and the decoder is used to decode the fused feature information.

[0206] Figure 7 This is a schematic diagram of the target segmentation device provided by the present invention, as shown below. Figure 7 As shown, the target segmentation device 700 includes: a second acquisition module 701 and a segmentation module 702; wherein,

[0207] The second acquisition module 701 is used to acquire the target image of the target object to be segmented;

[0208] The segmentation module 702 is used to input the target image into the target segmentation model to obtain the segmentation result output by the target segmentation model; the target segmentation model is trained based on the target segmentation model training method described in any of the above methods.

[0209] The target segmentation device provided by the present invention can obtain the segmentation result output by the target segmentation model by inputting the target image of the target object to be segmented into the target segmentation model, thereby realizing the segmentation of the target object in the target image of the target object to be segmented and improving the segmentation accuracy and efficiency.

[0210] Optionally, the target segmentation device 700 further includes:

[0211] The third determining module is used to determine the projected 3D shape and key point information corresponding to the segmentation result based on the segmentation result;

[0212] An alignment module is used to align the projected 3D shape with the segmentation result;

[0213] The fourth determining module is used to determine the tracking information corresponding to the target image based on the key point information and the historical information of the target image;

[0214] The fifth determining module is used to determine the pose of the target object based on the tracking information and the aligned information.

[0215] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as... Figure 8 As shown, the electronic device 800 may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a target segmentation model training method, which includes: acquiring a labeled training dataset and an unlabeled training dataset; the labeled training dataset includes multiple first sample images and first label data for each of the first sample images; the unlabeled dataset includes multiple second sample images; determining a mixed training dataset based on the labeled training dataset and the unlabeled training dataset; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images; training a pre-constructed initial student network model based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and a pre-constructed initial teacher network model to obtain a first student network model; updating the parameters of the initial teacher network model based on the parameters of the first student network model, and using the updated teacher network model as the new initial teacher network model; and determining a target segmentation model based on the parameters of the finally updated first student network model.

[0216] The processor 810 can call logical instructions in the memory 830 to execute a target segmentation method, which includes: acquiring a target image of a target object to be segmented; inputting the target image into a target segmentation model to obtain a segmentation result output by the target segmentation model; wherein the target segmentation model is trained based on the target segmentation model training method described in any of the foregoing embodiments.

[0217] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0218] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the target segmentation model training method provided by the above methods. The method includes: acquiring a labeled training dataset and an unlabeled training dataset; the labeled training dataset includes multiple first sample images and first label data for each of the first sample images; the unlabeled dataset includes multiple second sample images; determining a mixed training dataset based on the labeled training dataset and the unlabeled training dataset; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images; training a pre-constructed initial student network model based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and a pre-constructed initial teacher network model to obtain a first student network model; updating the parameters of the initial teacher network model based on the parameters of the first student network model, and using the updated teacher network model as the new initial teacher network model; and determining a target segmentation model based on the parameters of the finally updated first student network model.

[0219] When the computer program is executed by the processor, the computer can execute the target segmentation method provided by the above methods. The method includes: acquiring a target image of the target object to be segmented; inputting the target image into a target segmentation model to obtain the segmentation result output by the target segmentation model; the target segmentation model is trained based on the target segmentation model training method described in any of the foregoing embodiments.

[0220] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a target segmentation model training method provided by the methods described above. This method includes: acquiring a labeled training dataset and an unlabeled training dataset; the labeled training dataset including multiple first sample images and first label data for each of the first sample images; the unlabeled dataset including multiple second sample images; determining a mixed training dataset based on the labeled training dataset and the unlabeled training dataset; the mixed training dataset including multiple third sample images and second label data for each of the third sample images; training a pre-constructed initial student network model based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and a pre-constructed initial teacher network model to obtain a first student network model; updating the parameters of the initial teacher network model based on the parameters of the first student network model, and using the updated teacher network model as a new initial teacher network model; and determining a target segmentation model based on the parameters of the finally updated first student network model.

[0221] When executed by a processor, the computer program implements the target segmentation method provided by the methods described above. The method includes: acquiring a target image of a target object to be segmented; inputting the target image into a target segmentation model to obtain a segmentation result output by the target segmentation model; wherein the target segmentation model is trained based on the target segmentation model training method described in any of the foregoing embodiments.

[0222] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0223] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a target segmentation model, the method comprising: include: Obtain labeled and unlabeled training datasets; The labeled training dataset includes multiple first sample images and first label data for each first sample image; The unlabeled training dataset includes multiple second sample images; Based on the labeled training dataset and the unlabeled training dataset, a hybrid training dataset is determined; the hybrid training dataset includes multiple third sample images and second label data for each of the third sample images; Based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and the pre-built initial teacher network model, the pre-built initial student network model is trained to obtain the first student network model. Based on the parameters of the first student network model, the parameters of the initial teacher network model are updated, and the updated teacher network model is used as the new initial teacher network model. Based on the parameters of the final updated first student network model, the target segmentation model is determined; The step of determining the hybrid training dataset based on the labeled training dataset and the unlabeled training dataset includes: Based on the labeled training dataset and the unlabeled training dataset, at least one set of sample images is determined; each set of sample images includes the first sample image and the second sample image. A mixed training dataset is determined based on at least one set of sample images, the initial student network model, and the initial teacher network model; the mixed training dataset includes multiple third sample images and second label data for each of the third sample images. The determination of the mixed training dataset based on at least one set of sample images, the initial student network model, and the initial teacher network model includes: The first sample image in each group of sample images is input into the initial student network model to obtain the first segmented image output by the initial student network model; The second sample image from each group of sample images is input into the initial teacher network model to obtain the second segmentation image output by the initial teacher network model; Randomly select the first label data corresponding to the target number of pixels in the first segmented image; A binary mask is generated based on the first label data corresponding to the target number of pixels; Based on the binary mask, the first sample image and the second sample image, as well as the first segmented image and the second segmented image, are mixed to obtain a mixed sample image and the second label data of the mixed sample image; The mixed sample image is enhanced to obtain a third sample image; Based on the third sample image and the second label data, a mixed training dataset is determined. 2.The target segmentation model training method of claim 1, wherein, The step of training the initial student network model based on the labeled training dataset, the mixed training dataset, the unlabeled training dataset, and the initial teacher network model to obtain the first student network model includes: Based on the first segmented image and the first label data, calculate the first classification cross-entropy loss function; The third sample image from the mixed training dataset is input into the initial student network model to obtain the third segmentation image output by the initial student network model; Based on the third segmented image and the second label data, calculate the second classification cross-entropy loss function; The first structural similarity loss function is calculated based on the pixel value of at least one pixel block corresponding to the third segmented image and the binary mask corresponding to the third sample image; The first student network model is determined based on the first classification cross-entropy loss function, the second classification cross-entropy loss function, the first structural loss function, the unlabeled training dataset, and the initial teacher network model. 3.The target segmentation model training method of claim 2, wherein, The step of determining the first student network model based on the first classification cross-entropy loss function, the second classification cross-entropy loss function, the first structural loss function, the unlabeled training dataset, and the initial teacher network model includes: The second sample image from the unlabeled training dataset is input into the initial teacher network model to obtain the fourth segmentation image output by the initial teacher network model; Based on the fourth segmented image, determine the first probability and pseudo-label data; Based on the fourth segmented image, the first probability, and the pseudo-label data, calculate the third classification cross-entropy loss function and the second structural similarity loss function; Based on the first classification cross-entropy loss function, the first structural similarity loss function, the second classification cross-entropy loss function, the third classification cross-entropy loss function, and the second structural similarity loss function, the target loss function is determined. Based on the target loss function, the first student network model is determined.

4. The target segmentation model training method according to claim 3, characterized in that, Determining the first student network model based on the target loss function includes: If the target loss function does not meet the preset conditions, repeat the steps for determining the target loss function during the training process described above. If the target loss function satisfies the preset conditions, the initial student network model is determined as the first student network model.

5. The target segmentation model training method according to any one of claims 1-4, characterized in that, The initial student network model and the initial teacher network model have the same structure; the initial student network model includes at least one Transformer module, a multilayer perceptron, and a decoder; the Transformer module is used to extract feature information of the image; the multilayer perceptron is used to fuse the feature information extracted by each Transformer module; and the decoder is used to decode the fused feature information.

6. A target segmentation method, characterized in that, include: Obtain the target image of the object to be segmented; The target image is input into the target segmentation model to obtain the segmentation result output by the target segmentation model; The target segmentation model is trained based on the target segmentation model training method described in any one of claims 1-5.

7. The target segmentation method according to claim 6, characterized in that, The method further includes: Based on the segmentation results, determine the projected 3D shape and key point information corresponding to the segmentation results; Align the projected 3D shape with the segmentation result; Based on the key point information and the historical information of the target image, the tracking information corresponding to the target image is determined; Based on the tracking information and the aligned information, the pose of the target object is determined.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the target segmentation model training method as described in any one of claims 1 to 5, or the target segmentation method as described in claim 6 or 7.

Images