Method, device and equipment for training special talent segmentation model and storage medium

Abstract

The application discloses a training method and device of a special talent segmentation model, equipment and a storage medium, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring labeled samples and unlabeled samples; inputting the labeled samples into a student model to obtain student prediction results of the labeled samples based on the student model; inputting the unlabeled samples into the student model and a teacher model to respectively obtain student prediction results of the unlabeled samples and teacher prediction results of the unlabeled samples; acquiring prototype information in the unlabeled samples, generating prompt information of an all-around talent segmentation model based on the prototype information, and generating calibrated teacher prediction results according to the prompt information through the all-around talent segmentation model; and training a special talent segmentation model based on the student prediction results of the labeled samples, the student prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples and the calibrated teacher prediction results. The application adopts the ability of the all-around talent segmentation model to assist in training to obtain a high-quality special talent segmentation model.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, device, and storage medium for a specialist segmentation model. Background Technology

[0002] Currently, AI-based image segmentation technology can be applied in the medical field. For example, image segmentation technology can be used to identify various organs and lesions in three-dimensional (3D) MRI images.

[0003] In related technologies, due to the scarcity of labeled medical images and the high cost of labeling medical images, semi-supervised learning is often used to train specialist segmentation models (such as image segmentation models). Specialist segmentation models are trained based on a small number of labeled medical images (labeled samples) and a large number of unlabeled medical images (unlabeled samples) to obtain a specialist segmentation model that can be used for medical image segmentation.

[0004] However, semi-supervised learning methods in related technologies still require high-quality, densely labeled data. When the amount of labeled data is small, serious overfitting problems will occur. Summary of the Invention

[0005] This application provides a training method, apparatus, device, and storage medium for a specialist segmentation model, the technical solution of which is as follows.

[0006] According to one aspect of this application, a method for training a specialist segmentation model is provided, the specialist segmentation model including a teacher model and a student model, the method comprising:

[0007] Acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with real labels, and the unlabeled samples are image samples without real labels;

[0008] The labeled samples are input into the student model, and the student prediction results for the labeled samples are obtained based on the student model.

[0009] The unlabeled samples are input into the student model and the teacher model to obtain the student prediction results and the teacher prediction results of the unlabeled samples, respectively.

[0010] Obtain prototype information from the unlabeled samples, generate prompt information for a generalist segmentation model based on the prototype information, and generate calibrated teacher prediction results through the generalist segmentation model based on the prompt information. The prototype information is information obtained by processing the teacher prediction results based on the unlabeled samples.

[0011] The specialist segmentation model is trained based on the student prediction results of the labeled samples, the student prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples, and the calibrated teacher prediction results.

[0012] According to one aspect of this application, a training apparatus for a specialist segmentation model is provided, the apparatus comprising:

[0013] The sample acquisition module is used to acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples with real labels and the unlabeled samples are image samples without real labels;

[0014] A labeled segmentation module is used to input the labeled samples into the student model and obtain the student prediction results of the labeled samples based on the student model;

[0015] The unlabeled segmentation module is used to input the unlabeled samples into the student model and the teacher model, and obtain the student prediction results and the teacher prediction results of the unlabeled samples, respectively.

[0016] The prompt generation module is used to obtain prototype information from the unlabeled samples, generate prompt information for the generalist segmentation model based on the prototype information, and generate calibrated teacher prediction results through the generalist segmentation model based on the prompt information. The prototype information is information obtained by processing the teacher prediction results based on the unlabeled samples.

[0017] The model training module is used to train the specialist segmentation model based on the student prediction results of the labeled samples, the student prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples, and the calibrated teacher prediction results.

[0018] According to another aspect of this application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to implement the training method of the specialist segmentation model as described above.

[0019] According to another aspect of this application, a computer storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to implement the training method of the specialist segmentation model as described above.

[0020] According to another aspect of this application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium; the computer program is read from and executed by a processor of a computer device from the computer-readable storage medium, causing the computer device to perform the training method for the specialist segmentation model as described above.

[0021] The beneficial effects of the technical solutions provided in this application include at least the following:

[0022] This application utilizes a collaborative approach between a specialist segmentation model and a generalist segmentation model. The specialist segmentation model generates prompts for the generalist segmentation model, and the generalist model provides auxiliary supervision for the training of the specialist segmentation model by responding to these prompts. Specifically, by acquiring prototype information from unlabeled samples, prompts for the generalist segmentation model are generated based on this prototype information. The generalist segmentation model then generates calibrated teacher predictions based on these prompts. These calibrated teacher predictions provide additional supervision signals to the specialist segmentation model during training, improving the utilization of information from unlabeled samples and effectively reducing the number of labeled samples required. This application employs the general image segmentation capabilities of the generalist segmentation model to assist in training a high-quality specialist segmentation model. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is an architecture diagram of a computer system provided in one embodiment of this application;

[0025] Figure 2 This is a schematic diagram illustrating the principle of the specialist-generalist segmentation model collaboration process provided in an exemplary embodiment of this application;

[0026] Figure 3 This is a flowchart of a training method for a specialist segmentation model provided in one embodiment of this application;

[0027] Figure 4 This is a flowchart of a training method for a specialist segmentation model provided in one embodiment of this application;

[0028] Figure 5 This is a schematic diagram of a training method for a specialist segmentation model provided in one embodiment of this application;

[0029] Figure 6This is a schematic diagram of a training method for a specialist segmentation model provided in one embodiment of this application;

[0030] Figure 7 This is a schematic diagram of a training method for a specialist segmentation model provided in one embodiment of this application;

[0031] Figure 8 This is a structural block diagram of a training device for a specialist segmentation model provided in one embodiment of this application;

[0032] Figure 9 This is a schematic diagram of the structure of a computer device provided in one embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0034] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0035] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0036] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0037] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0038] First, let me introduce the relevant terms used in this application:

[0039] Voxel: short for volume pixel, is the smallest unit in a three-dimensional (3D) image, similar to a pixel in a two-dimensional (2D) image. In three-dimensional space, an image is composed of voxels, each representing a small cube in three-dimensional space. Voxels are very important in fields such as medical imaging, geographic information systems, and 3D modeling. For example, in computed tomography (CT) or magnetic resonance imaging (MRI), 3D medical images are typically three-dimensional structures composed of a large number of voxels, each containing local information about a tissue or organ.

[0040] Image segmentation is a technique that assigns each pixel or voxel in an image to a semantic category. For example, it might assign each pixel in a 2D image to one of two categories: foreground and background. Or, it might assign each voxel in a 3D medical image to one of two categories: heart and non-heart. Of course, image segmentation is not limited to binary classification scenarios; it can also be used in multi-class classification scenarios.

[0041] A general segmentation model is an image segmentation model trained on a large-scale image sample set and not limited to any particular domain. It can handle various types of image segmentation tasks and can be applied to a wide range of scenarios. Furthermore, it can segment images without specific training, requiring only simple prompts. In this embodiment, the general segmentation model is also referred to as a universal segmentation model.

[0042] In this embodiment, the generalist segmentation model is class-agnostic. Class-agnostic means that the generalist segmentation model can accurately segment different semantically classified parts of an image even without knowing the actual semantic classification of each segment.

[0043] Specialized segmentation models are image segmentation models trained using image sample sets from a specific domain, suitable for that domain. They are tailored for specific tasks or application scenarios, focusing on improving segmentation performance on specific types of objects or scenes.

[0044] True labels: True labels refer to the correct and accurate labels (categories or values) for each sample in the dataset. For example, labels annotated by medical experts.

[0045] Pseudo-labels: Pseudo-labels are labels used in the absence of real labels. They are predictions generated by a trained artificial intelligence model and used as real labels.

[0046] Paradigm: This term has different meanings in different disciplines and contexts, but it usually refers to a widely accepted theoretical framework, methodology, standard, or mode of thinking in a field.

[0047] Semi-supervised learning (SSL) is a method in machine learning that falls between supervised and unsupervised learning. In supervised learning, the model learns from labeled samples, while in unsupervised learning, the model learns from unlabeled samples. Semi-supervised learning combines these two approaches, using a small number of labeled samples and a large number of unlabeled samples for training.

[0048] Semi-supervised learning has made significant progress in the field of 3D medical image segmentation by effectively reducing the tedious and intensive annotation requirements for radiologists. Previous research on semi-supervised learning has focused on model-centric innovation, namely, developing more advanced and complex semi-supervised learning algorithms. However, with the emergence of open-source generalist segmentation models, this application shifts the research focus of semi-supervised learning to a data-centric perspective, namely, how to effectively and efficiently improve the quality of supervisory signals through the general capabilities of generalist segmentation models, thereby making more effective use of information from unlabeled samples.

[0049] Figure 1 This is an architectural diagram of a computer system provided in an exemplary embodiment of this application. The computer system includes: a patient device 110, a server 120, and a doctor device 130.

[0050] The patient device 110 has a client installed and running. This client supports uploading images to be segmented, such as 3D medical images. The client can exist in various forms, including applications (APPs), mini-programs, web pages, web clients, and cloud clients. In some embodiments, the client is logged into a first user's account. The patient device 110 can be of at least one of the following types: smartphone, tablet computer, virtual reality device (VRDevice), augmented reality device (AR Device), medical equipment, e-book reader, MP3 player, MP4 player, laptop, and desktop computer. Optionally, in a medical setting, the patient device 110 is a device used by the patient or a device used by a radiologist.

[0051] The patient device 110 is connected to the server 120 via a wireless network or a wired network. It should be noted that the number of terminals described above is merely an example and is not intended to limit the computer system provided in the embodiments of this application.

[0052] Server 120 includes at least one of a single server, multiple servers, a cloud computing platform, and a virtualization center. In some embodiments, server 120 provides backend services to clients. Server 120 runs a semi-supervised trained specialist segmentation model provided in this application, which has image segmentation capabilities within a specified domain. In this application, the specialist segmentation model is trained using a generalist segmentation model. Optionally, server 120 undertakes the primary computational work, and patient device 110 undertakes secondary computational work; or, server 120 undertakes secondary computational work, and patient device 110 undertakes primary computational work; or, server 120 and patient device 110 collaborate using a distributed computing architecture.

[0053] The doctor device 130 has a client installed and running. This client supports receiving segmentation results of images, such as 3D medical images. The client can exist in various forms, including apps, mini-programs, web pages, web clients, and cloud clients. In some embodiments, the client is logged into a second user's account. The device type of the doctor device 130 includes at least one of the following: smartphones, tablets, VR devices, AR devices, medical devices, e-book readers, MP3 players, MP4 players, laptops, and desktop computers. Optionally, in a medical setting, the doctor device 130 is a device used by a diagnosing doctor to view the segmentation results of a patient's medical images.

[0054] Taking a specialized segmentation model, which is an image segmentation model in the medical field, as an example, a patient uploads a medical image to a server 120 using a patient device 110. The server 120 inputs the medical image into the specialized segmentation model and generates a segmentation result for the medical image. The server 120 then sends the segmentation result of the medical image to a doctor's device 130, so that the diagnosing doctor can view the segmentation result on the doctor's device 130.

[0055] It should be noted that, Figure 1 The description uses one patient device 110 and one doctor device 130, but in different embodiments, there can be one or more patient devices 110 and doctor devices 130. This application does not limit this.

[0056] In related technologies, due to the scarcity of labeled medical images and the high cost of labeling medical images, semi-supervised learning is typically used to train specialist segmentation models. Specifically, specialist segmentation models are trained using a small number of labeled medical images (labeled samples) and a large number of unlabeled medical images (unlabeled samples) to obtain a specialist segmentation model suitable for medical image segmentation. However, while semi-supervised learning can reduce the amount of labeled data required, it still necessitates high-quality, densely labeled data. When the amount of labeled data is limited, severe overfitting problems can occur.

[0057] When training a specialist segmentation model using semi-supervised learning, a training method combining teacher and student models is employed. Optionally, given a trainable student model, the network parameters of the teacher model are updated after updating the student model's network parameters. Specifically, labeled samples are input into the student model, and the supervised loss for labeled samples (the loss between the student's prediction and the true label) is calculated using the student prediction results generated by the student model. Student predictions are obtained by inputting unlabeled samples into the student model, and teacher predictions are obtained by inputting unlabeled samples into the teacher model. Since the predictions of the student and teacher models are expected to be as equal as possible, a consistency loss can be calculated based on the student and teacher predictions. In the current iteration, the network parameters of the student model are updated based on the supervised loss and the consistency loss. After updating the student model's network parameters, the teacher model's network parameters are then updated using the student model's network parameters. However, in the semi-supervised learning training of related techniques, the specialist segmentation model struggles to perceive the segmentation target in the early stages of training, and the teacher model's prediction accuracy for unlabeled samples is very low, thus leading to biased predictions in the specialist segmentation model.

[0058] This application proposes a semi-supervised learning paradigm driven by a specialist segmentation model and a generalist segmentation model, aiming to develop customized specialist segmentation models using limited annotation resources. The specialist segmentation model includes a student model and a teacher model; both are used during the training phase, while only the student model is used during the inference phase.

[0059] Reference Figure 2 The training method of the specialist segmentation model in the embodiments of this application is briefly described as follows:

[0060] • Obtain training sample 10, which includes labeled sample 11 and unlabeled sample 12;

[0061] Among them, labeled sample 11 is an image sample labeled with real labels, and unlabeled sample 12 is an image sample without real labels. Optionally, the image sample is a three-dimensional image sample or a two-dimensional image sample.

[0062] In some embodiments, labeled samples 11 are obtained by labeling a small number of randomly selected unlabeled samples 12. The number of labeled samples 11 is far less than that required for semi-supervised learning in related technologies, even with low labeling budget costs.

[0063] • Input labeled sample 11 into the specialist segmentation model 20 to obtain the student prediction results for labeled sample 11;

[0064] Among them, the specialist segmentation model 20 includes student model 21 and teacher model 22.

[0065] In some embodiments, labeled sample 11 is input into student model 21, and student model 21 generates student prediction results for labeled sample 11. The student prediction results for labeled sample 11 are the results predicted by student model 21 and are used to predict the category to which the pixels or voxels in labeled sample 11 belong.

[0066] • Input the unlabeled sample 12 into the student model 21 and the teacher model 22 to obtain the student prediction result and the teacher prediction result of the unlabeled sample 12, respectively. The teacher prediction result is generated by the prompt information 40 generated by the generalist segmentation model 30 based on the prototype information. The calibrated teacher prediction result includes the calibrated label 60.

[0067] The student prediction result for unlabeled sample 12 is the result of the student model 21, used to predict the category of pixels or voxels in unlabeled sample 12. The teacher prediction result for unlabeled sample 12 is the result of the teacher model 22, used to predict the category of pixels or voxels in unlabeled sample 12.

[0068] In some embodiments, the teacher model 22 has a low accuracy in predicting unlabeled samples 12, meaning the teacher predictions for unlabeled samples 12 are inaccurate, which leads to bias in the predictions of the specialist segmentation model 20. Optionally, in this embodiment, the teacher predictions for unlabeled samples 12 are calibrated using the generalist segmentation model 30, and the calibrated teacher predictions are used to provide an effective supervisory signal for the student model 21.

[0069] The general segmentation model 30 possesses a general segmentation capability for segmenting two-dimensional images based on cue information 40. The general segmentation model includes an image encoder 31, a cue encoder 32, and a mask encoder 33. The image encoder 31 extracts features from the input two-dimensional image to obtain image features; the cue encoder 32 extracts cue information from the input image to obtain cue features; and the mask decoder 33 performs image segmentation based on cue guidance from the cue features, resulting in more accurate segmentation results. Taking the unlabeled sample 12 in this embodiment as a three-dimensional image as an example, the image input to the general segmentation model 30 is a two-dimensional image. Therefore, the three-dimensional image of the unlabeled sample 12 needs to be sliced ​​according to different perspectives to obtain two-dimensional sliced ​​images. For example, the three-dimensional image of the unlabeled sample is sliced ​​according to coronal, axial, and sagittal perspectives, with each voxel in the three-dimensional image corresponding to three perspectives.

[0070] The following describes the prompt information 40 for inputting the generalist segmentation model 30.

[0071] In some embodiments, unlabeled sample 12 is used as the image to be segmented. Prototype information of the image to be segmented is obtained, and prompt information 40 for a general segmentation model is generated based on the prototype information. The general segmentation model then generates a calibrated teacher prediction result based on the prompt information 40. The prototype information indicates the average features of the target object in the image to be segmented. It can also be understood as information obtained by averaging the teacher prediction results of unlabeled sample 12. The prototype information indicates the central tendency of the internal features of unlabeled sample 12 (the image to be segmented).

[0072] Optionally, the prototype information includes first prototype information and second prototype information. The first prototype information is used to indicate the average features of the target segmentation object, and the second prototype information is used to indicate the average features of the non-segmentation object. The non-target segmentation object is the background segmentation image in the image to be segmented, excluding the target segmentation object.

[0073] In some embodiments, the generalist segmentation model 30 generates a more accurate segmentation result based on the prompt information 40. The prompt information 40 includes at least one of two-dimensional bounding box prompt information, positive point prompt information, and negative point prompt information. The two-dimensional bounding box prompt information is used to mark the region extent of the target segmented object. The positive point prompt information is used to indicate the location of key points of the target segmented object. The negative point prompt information is used to indicate the location of non-target segmented objects.

[0074] In some embodiments, multiple slice images from three perspectives are used as images to be segmented. For each slice image, cue information 40 is generated based on object prototype information. The cue information 40 for each slice image includes the three types of cue information mentioned above (bounding box cue information, positive cue information, and negative cue information). The generalist segmentation model 30 predicts pseudo-labels on the three perspectives based on the cue information 40. Optionally, a fused pseudo-label 50 is generated by fusing the pseudo-labels on the three perspectives based on a correction strategy. For example, for any cue in the 3D image, based on the majority voting principle (or the minority obeys the majority principle), if the pseudo-labels on at least two of the three perspectives belong to the first category, the fused pseudo-label 50 of the voxel is determined to belong to the first category.

[0075] In some embodiments, after obtaining the fused pseudo-label 50, a self-identification and calibration process is performed on the fused pseudo-label 50 to generate a calibrated teacher prediction result. For each voxel in the 3D image, the predicted probability of each voxel under its category is averaged to obtain the probability threshold corresponding to the category. The label type is determined based on the relationship between the predicted probability and the probability threshold. The label type of the correctly labeled voxels in the unlabeled sample 12 is determined as the category to which the fused pseudo-label 50 belongs, and the label type of the incorrectly labeled voxels in the unlabeled sample 12 is calibrated to the category to which the potential label belongs. Here, the potential label is the true label of the preset voxel.

[0076] For example, for any voxel, the fused pseudo-label of the voxel belongs to the first category, and the latent label of the voxel is assumed to belong to the second category. The predicted probability of the voxel under the second category is averaged to obtain the probability threshold corresponding to the second category. If the predicted probability of the voxel under the second category is greater than the probability threshold, the voxel is considered to belong to the second category rather than the first category.

[0077] • The specialist segmentation model 20 was trained based on supervised loss and unsupervised loss.

[0078] In some embodiments, the loss of the talent segmentation model 20 includes supervised loss and unsupervised loss. The unsupervised loss includes consistency loss and label loss. Optionally, the supervised loss for labeled sample 11 is calculated based on the student prediction results and the true labels. Optionally, the consistency loss for unlabeled sample 12 is calculated based on the student prediction results and the teacher prediction results for unlabeled sample 12, and the label loss for unlabeled sample 12 is calculated based on the student prediction results and the calibrated teacher prediction results for unlabeled sample 12. The unsupervised loss for unlabeled sample 12 is calculated by weighted summing of the consistency loss and the label loss. The talent segmentation model 20 is then trained based on the supervised and unsupervised losses.

[0079] Figure 3 This is a flowchart illustrating a training method for a specialist segmentation model provided in an exemplary embodiment of this application. The method is performed by a computer device used by the developer, or... Figure 1 The method is executed on server 120. A specialized partitioning model runs on the computer device or server 120. The method includes at least one of the following steps.

[0080] Step 210: Obtain labeled and unlabeled samples;

[0081] Labeled samples are image samples with actual labels, while unlabeled samples are image samples without actual labels. In the field of image segmentation, image samples can be three-dimensional or two-dimensional.

[0082] In some embodiments, when the image sample is a two-dimensional image sample, the ground truth label is annotation information used to indicate the category to which a pixel in the two-dimensional image belongs. When the image sample is a three-dimensional image sample, the ground truth label is annotation information used to indicate the category to which a voxel in the three-dimensional image belongs.

[0083] Optionally, the image sample is a two-dimensional image sample. For a two-dimensional image, the ground truth label is represented using a mask image of the two-dimensional image. For example, when a pixel in the mask image has a value of 1, it means that the pixel at the corresponding position in the two-dimensional image belongs to the first category; when a pixel in the mask image has a value of 0, it means that the pixel at the corresponding position in the two-dimensional image belongs to the second category. As another example, when a pixel in the mask image has a value of 0, it means that the pixel at the corresponding position in the two-dimensional image belongs to the first category; when a pixel in the mask image has a value of 1, it means that the pixel at the corresponding position in the two-dimensional image belongs to the second category.

[0084] Optionally, the image sample is a three-dimensional image sample. For three-dimensional images, the true label can be identified using a mask of voxels in the three-dimensional image. For example, a voxel mask value of 1 in the three-dimensional image represents belonging to the first category; a voxel mask value of 0 in the three-dimensional image represents belonging to the second category. As another example, a voxel mask value of 0 in the three-dimensional image represents belonging to the first category; a voxel mask value of 1 in the three-dimensional image represents belonging to the second category.

[0085] In some embodiments, labeled samples are obtained by labeling a small number of randomly selected unlabeled samples. The number of labeled samples is far less than that required for semi-supervised learning in related techniques, even with low labeling budget costs.

[0086] Step 220: Input the labeled samples into the student model, and obtain the student prediction results based on the labeled samples;

[0087] The student prediction result for labeled samples is the result predicted by the student model and is used to predict the category of pixels or voxels in the labeled samples. In this embodiment, the student prediction result for labeled samples can also be referred to as the student segmentation result for labeled samples.

[0088] In some embodiments, the specialist segmentation model includes a student model and a teacher model. The specialist segmentation model has image segmentation capabilities within a specified domain. For example, the specialist segmentation model can be used for medical image segmentation.

[0089] The talent segmentation model can be implemented using any neural network architecture with image segmentation capabilities. For example, the talent segmentation model can be implemented using any of the following: convolutional neural networks, V-Net, fully convolutional networks for semantic segmentation (FCN), convolutional networks for biomedical image segmentation (U-net), deep convolutional networks, semantic image segmentation with deep convolutional networks (Atrous Convolution, and Fully Connected CRFs, DeepLab), densely connected convolutional networks (DenseNet), and residual networks (ResNet).

[0090] This embodiment uses V-Net, a specialized image segmentation model, as an example. V-Net is characterized by combining the ideas of 3D convolutional neural networks and fully convolutional networks (FCNs), achieving accurate image segmentation through an encoder-decoder architecture. This design helps capture deeper spatial relationships in images, thereby improving segmentation accuracy. V-Net has demonstrated excellent performance in medical image segmentation tasks and is therefore widely used in the medical field.

[0091] Step 230: Input the unlabeled samples into the student model and the teacher model to obtain the student prediction results and the teacher prediction results for the unlabeled samples, respectively;

[0092] In some embodiments, unlabeled samples are input into a student model, and student prediction results for the unlabeled samples are obtained based on the student model. These student prediction results for the unlabeled samples are the results predicted by the student model and are used to predict the category of pixels or voxels in the unlabeled samples.

[0093] In some embodiments, unlabeled samples are input into a teacher model, and teacher prediction results for the unlabeled samples are obtained based on the teacher model. These teacher prediction results for the unlabeled samples are the results predicted by the teacher model and are used to predict the category of pixels or voxels in the unlabeled samples.

[0094] Step 240: Obtain prototype information from unlabeled samples, generate prompts for the generalist segmentation model based on the prototype information, and generate calibrated teacher prediction results based on the prompts using the generalist segmentation model.

[0095] In some embodiments, the teacher model has low accuracy in predicting unlabeled samples, meaning the teacher predictions for unlabeled samples are inaccurate, which leads to bias in the predictions of the specialist segmentation model. Optionally, in this embodiment, the teacher predictions for unlabeled samples are calibrated using a generalist segmentation model, and the calibrated teacher predictions are used to provide an effective supervisory signal for the student model.

[0096] Generalist segmentation models are pre-trained (also known as frozen) segmentation models that are not limited to any particular domain. Generalist segmentation models are class-agnostic. Class agnostic means that a generalist segmentation model can accurately segment various parts of an image without knowing the specific semantics of each segment. Generalist segmentation models are typically trained using large-scale image datasets.

[0097] The generalist segmentation model possesses the ability to segment input images based on prompts; that is, it can accurately segment input images under the guidance of prompts. Utilizing this capability, and using prototype information as guidance, the generalist segmentation model can be used to perform image segmentation on unlabeled samples to obtain calibrated teacher prediction results (or calibrated teacher segmentation results).

[0098] In some embodiments, prompts for a generalist segmentation model are generated based on prototype information from unlabeled samples. These prompts are then input into the generalist segmentation model, which generates calibrated teacher predictions based on the prompts. The prototype information is obtained by processing teacher predictions from unlabeled samples.

[0099] In some embodiments, the unlabeled samples mentioned above are two-dimensional images, the specialist segmentation model is a segmentation model for two-dimensional images in a specific domain, and the generalist segmentation model is a segmentation model for two-dimensional images in a general domain. Based on prototype information, prompts for the generalist segmentation model are generated, and the generalist segmentation model generates calibrated teacher prediction results based on these prompts.

[0100] In some embodiments, the unlabeled samples mentioned above are 3D images, the specialist segmentation model is a segmentation model for 3D images in a specific domain, and the generalist segmentation model is a segmentation model for 3D images in a general domain. Based on the prototype information, prompts for the generalist segmentation model are generated, and the generalist segmentation model generates calibrated teacher prediction results based on these prompts.

[0101] In some embodiments, the unlabeled samples mentioned above are three-dimensional images, the specialist segmentation model is a segmentation model for three-dimensional images in a specific domain, and the generalist segmentation model is a segmentation model for two-dimensional images in a general domain. Based on the prototype information, prompts for the generalist segmentation model are generated, and the generalist segmentation model generates calibrated teacher prediction results based on these prompts.

[0102] Step 250: Train the specialist segmentation model based on the student prediction results of labeled samples, the student prediction results of unlabeled samples, the teacher prediction results of unlabeled samples, and the calibrated teacher prediction results.

[0103] In some embodiments, the loss of the specialist segmentation model includes supervised loss and unsupervised loss. Optionally, the unsupervised loss includes consistency loss and label loss.

[0104] Optionally, a supervised loss for labeled samples is calculated based on student predictions and the true labels. Optionally, a consistency loss for unlabeled samples is calculated based on student predictions and teacher predictions, and a labeled loss for unlabeled samples is calculated based on student predictions and calibrated teacher predictions. The consistency loss and labeled loss are then weighted and summed to calculate the unsupervised loss for unlabeled samples. Based on the supervised and unsupervised losses, a semi-supervised learning approach is used to train the specialist segmentation model.

[0105] In summary, the method provided in this application utilizes a collaborative approach between a specialist segmentation model and a generalist segmentation model. The specialist segmentation model generates prompts for the generalist segmentation model, and the generalist segmentation model provides auxiliary supervision for the training of the specialist segmentation model by responding to these prompts. Specifically, by acquiring prototype information from unlabeled samples, prompts for the generalist segmentation model are generated based on this prototype information. The generalist segmentation model can then generate calibrated teacher prediction results based on these prompts. These calibrated teacher prediction results provide additional supervision signals to the specialist segmentation model during training, improving the utilization of information from unlabeled samples and effectively reducing the number of labeled samples required. This application employs the general image segmentation capabilities of the generalist segmentation model to assist in training a high-quality specialist segmentation model.

[0106] Prototype information of unlabeled samples

[0107] In some embodiments, the generalist segmentation model possesses a general segmentation capability for segmenting the input image based on prompting information; that is, the generalist segmentation model can accurately segment the input image under the guidance of prompting information. Utilizing this capability of the generalist segmentation model, using prototype information in unlabeled samples as guiding information, image segmentation of unlabeled samples using the generalist segmentation model can yield calibrated teacher prediction results (or calibrated teacher segmentation results).

[0108] like Figure 4 As shown, step 240 above can be replaced by steps 241 and 242:

[0109] Step 241: Use the unlabeled sample as the image to be segmented and obtain the prototype information of the image to be segmented;

[0110] Unlabeled samples are image samples that are not labeled with real labels. Optionally, unlabeled samples can be three-dimensional or two-dimensional image samples.

[0111] In some embodiments, taking unlabeled samples as 3D image samples as an example, the unlabeled samples are used as images to be segmented, where the image to be segmented refers to a 3D image that needs to be segmented. Optionally, the prototype information of the image to be segmented is used to indicate the characteristics of different objects or different segmentation spaces (different segmentation regions) in the image to be segmented.

[0112] In some embodiments, prototype information refers to representative features (or typical features) of the target object in unlabeled samples, used to indicate the central tendency of features within the target object. Optionally, prototype information is obtained by performing clustering or averaging processing on the features of the target object. By analyzing the feature distribution of the image to be segmented, representative features of the target object (i.e., prototype information) are extracted, and prompts are generated based on the prototype information, thereby guiding the general segmentation model to generate segmentation results more robustly.

[0113] In some embodiments, step 241 includes at least one of the following sub-steps:

[0114] Sub-step 241a: Obtain teacher prediction results for unlabeled samples;

[0115] In some embodiments, unlabeled samples are input into a teacher model, and teacher prediction results for the unlabeled samples are obtained based on the teacher model. These teacher prediction results for the unlabeled samples are the results predicted by the teacher model and are used to predict the category of pixels or voxels in the unlabeled samples.

[0116] In some embodiments, the teacher prediction results for unlabeled samples include the predicted category of each voxel in the image to be segmented. For example, each voxel is indicated to belong to either the target segmentation object or a non-target segmentation object. When the voxel is indicated to belong to the target segmentation object, the target segmentation object corresponds to the foreground category; when the voxel is indicated to belong to a non-target segmentation object, the non-target segmentation corresponds to the background category.

[0117] Sub-step 241b: Extract the features of all voxels of the target segmentation object from the sampling feature map corresponding to the image to be segmented, and process the features of all voxels of the target segmentation object to obtain the first prototype information;

[0118] Here, the sampled feature map (such as the sampled feature map above) refers to the feature map obtained through sampling operations in a segmentation network (such as U-Net). Optionally, the sampled feature map includes features of each pixel (two-dimensional image) or voxel (three-dimensional image) in the image to be segmented.

[0119] Taking a 3D image as an example, the sampled feature map includes the features of each voxel in the image to be segmented. The features of the target segmentation object are extracted from the sampled feature map corresponding to the image to be segmented; that is, the features of the voxels corresponding to the target segmentation object are extracted from the sampled feature map corresponding to the image to be segmented. Here, the target segmentation object is the target object in the image to be segmented.

[0120] For example, in medical image segmentation, classifying the target segmentation object in an image as foreground can also be understood as the target segmentation object belonging to the foreground category, or the voxel corresponding to the target segmentation object belonging to the foreground category. For instance, when segmenting the heart region, the target segmentation object is the heart region in the image to be segmented, and the heart region belongs to the foreground category.

[0121] In some embodiments, features of the target segmentation object are extracted (filtered) from the sampled feature map corresponding to the image to be segmented. That is, features of all voxels corresponding to the target segmentation object are extracted (filtered) from the sampled feature map corresponding to the image to be segmented. The confidence score of each voxel feature in the target segmentation object is calculated (the teacher prediction result of the unlabeled sample includes the prediction probability of each voxel). After weighted averaging of the features of all voxels using the confidence score of each voxel feature, the first prototype information of the target segmentation object is obtained. The first prototype information is used to reflect the central tendency of the target segmentation object in the image to be segmented. In the embodiments of this application, the first prototype information can also be called the object prototype.

[0122] Optionally, the confidence level of a voxel's feature in the target segmentation object is positively correlated with its contribution to the first prototype information. The higher the confidence level of a voxel's feature in the target segmentation object, the greater its contribution to the first prototype information; that is, the higher the confidence level of a voxel's feature in the target segmentation object, the greater its contribution to the first prototype information. Conversely, the lower the confidence level of a voxel's feature in the target segmentation object, the smaller its contribution to the first prototype information; that is, the lower the confidence level of a voxel's feature in the target segmentation object, the smaller its contribution to the first prototype information.

[0123] In some embodiments, the first prototype information is used to indicate the characteristics of the target segmentation object. The first prototype information is used to indicate the central tendency of the target segmentation object. Optionally, the first prototype information is used to indicate the average characteristics of the target segmentation object, or, in other words, the representative characteristics of the target segmentation object, or, in other words, the central characteristics of the target segmentation object, or, in other words, the typical characteristics of the target segmentation object. This application does not limit this to any particular aspect.

[0124] Sub-step 241c: Extract the features of all voxels of the non-target segmentation object from the sampled feature map corresponding to the image to be segmented, and process the features of all voxels of the non-target segmentation object to obtain the second prototype information.

[0125] Taking a 3D image as an example, the sampled feature map includes the features of each voxel in the image to be segmented. Features of non-target segmentation objects are extracted from the sampled feature map corresponding to the image to be segmented; that is, features of voxels corresponding to non-target segmentation objects are extracted from the sampled feature map corresponding to the image to be segmented. Here, non-target segmentation objects are the background segmentation image in the image to be segmented, excluding the target object.

[0126] For example, in medical image segmentation, the target segmentation object in the image is classified as the foreground, while other regions are classified as the background. This can also be understood as the target segmentation object belonging to the foreground category, and non-target segmentation objects belonging to the background category. For instance, when segmenting the heart region, the target segmentation object is the heart region in the image to be segmented, and the heart region belongs to the foreground category, while other tissues in the image to be segmented, such as the lung region and bone region, belong to the background category.

[0127] In some embodiments, features of non-target segmentation objects are extracted (filtered) from the sampled feature map corresponding to the image to be segmented. That is, features of all voxels corresponding to non-target segmentation objects are extracted (filtered) from the sampled feature map corresponding to the image to be segmented. The confidence score of each voxel feature in the non-target segmentation object is calculated (the teacher prediction result of the unlabeled sample includes the prediction probability of each voxel). After weighted averaging of the features of all voxels based on the confidence score of each voxel feature, the second prototype information of the non-target segmentation object is obtained. The second prototype information is used to reflect the central tendency of the non-target segmentation object in the image to be segmented. In the embodiments of this application, the second prototype information can also be called the background prototype.

[0128] In some embodiments, the second prototype information is used to indicate the characteristics of non-target segmentation objects. The second prototype information is used to indicate the central tendency of non-target segmentation objects. Optionally, the second prototype information is used to indicate the average characteristics of non-target segmentation objects, or, in other words, the representative characteristics of non-target segmentation objects, or, in other words, the central characteristics of non-target segmentation objects, or, in other words, the typical characteristics of non-target segmentation objects. This application does not limit this.

[0129] In this embodiment, unlabeled samples are used as the images to be segmented. Features of all voxels of the target segmentation object and non-target segmentation objects are extracted from the images to be segmented. The features of all voxels of the target segmentation object are processed by confidence-weighted averaging to obtain the object prototype (first prototype information), and the features of all voxels of the non-target segmentation object are processed by confidence-weighted averaging to obtain the background prototype (second prototype information). These prototype information can represent the features of the target segmentation object and non-target segmentation objects, enabling the model to better understand the global and local features of the image to be segmented, which helps to perform image segmentation more accurately. Furthermore, by combining the teacher prediction results of unlabeled samples and the prototype information of the segmentation objects (target segmentation object and / or non-target segmentation objects), the model can not only learn from labeled samples but also adaptively improve its performance from unlabeled samples, improving the utilization efficiency of unlabeled samples and enhancing the accuracy and adaptability of the image segmentation model in practical applications.

[0130] Step 242: Generate prompts for the generalist segmentation model based on the prototype information, and generate calibrated teacher prediction results based on the prompts using the generalist segmentation model.

[0131] A generalist segmentation model can be an image segmentation model that has generalist segmentation capabilities for two-dimensional images, or an image segmentation model that has generalist segmentation capabilities for three-dimensional images.

[0132] In some embodiments, the unlabeled samples mentioned above are three-dimensional images, the specialist segmentation model is a segmentation model for three-dimensional images in a specific domain, and the generalist segmentation model is a segmentation model for two-dimensional images in a general domain. Based on the prototype information, prompts for the generalist segmentation model are generated, and the generalist segmentation model generates calibrated teacher prediction results based on these prompts.

[0133] Optionally, the generalized segmentation model includes an image encoder, a cue encoder, and a mask decoder. The image encoder extracts features from the input 2D image to obtain an image feature representation; the cue encoder extracts features from the input cue information (cue information generated based on prototype information) to obtain a cue feature representation; and the mask decoder, based on the cue feature representation, performs image segmentation on the image feature representation to obtain a more accurate segmentation result. However, since the input image of the generalized segmentation model is a 2D image, while the unlabeled samples are 3D images, the unlabeled samples are first sliced ​​according to different viewpoints to obtain multiple slice images corresponding to each viewpoint. Then, the generalized segmentation model performs image segmentation on each of these multiple slice images for each viewpoint.

[0134] In some embodiments, the image to be segmented is sliced ​​according to three perspectives to obtain multiple slice images corresponding to the three perspectives, and the multiple slice images are two-dimensional images. Optionally, the three perspectives correspond to the three coordinate planes of the three-dimensional image. Taking a medical three-dimensional image as an example, the three perspectives are the coronal perspective, the axial perspective, and the sagittal perspective.

[0135] The 3D image is cut into multiple 2D slices according to each viewpoint. That is, the 3D image is cut into multiple slices according to the coronal viewpoint; the 3D image is cut into multiple slices according to the axial viewpoint; and the 3D image is cut into multiple slices according to the sagittal viewpoint.

[0136] In some embodiments, step 242 includes at least one of the following sub-steps:

[0137] Sub-step 242a: Take multiple slice images from three perspectives as images to be segmented, generate prompt information for each slice image in the multiple slice images based on prototype information, and predict pseudo-labels on the three perspectives respectively based on the prompt information using a general segmentation model;

[0138] Taking the general segmentation model as an example, which is a general-domain segmentation model for two-dimensional images, the general segmentation model can perform image segmentation on the input sliced ​​image. Optionally, the image to be segmented (three-dimensional image) corresponding to the unlabeled sample is sliced ​​according to three perspectives to obtain multiple sliced ​​images corresponding to the three perspectives. The multiple sliced ​​images of the three perspectives are used as the image to be segmented (two-dimensional image) of the general segmentation model. Based on the prototype information, prompt information is generated for each sliced ​​image in the multiple sliced ​​images. The general segmentation model predicts the pseudo-labels on the three perspectives according to the prompt information.

[0139] In some embodiments, the i-th slice image of each of the three perspectives is taken as the image to be segmented, and the prompt information of the i-th slice image is generated based on the prototype information. The i-th slice image and the prompt information of the i-th slice image are input together into the general segmentation model, and the pseudo label of the i-th slice image is obtained based on the general segmentation model. The pseudo label of the i-th slice image is used to indicate the segmentation result (two-dimensional segmentation result or two-dimensional prediction result) of the i-th slice image, where i is a positive integer greater than 1.

[0140] Optionally, based on the pseudo-labels of the i-th slice image in each viewpoint, pseudo-labels for each viewpoint are generated. These pseudo-labels indicate the segmentation result for the current viewpoint. Assuming the resolution of the 3D image is H×W×D, the three viewpoints can be divided into H, W, and D slice images, respectively. Here, H represents the height of the 3D image, W represents the width of the 3D image, and D represents the depth of the 3D image.

[0141] For example, the i-th slice image from the coronal perspective is used as the image to be segmented. Based on prototype information, prompt information for the i-th slice image from the coronal perspective is generated. The segmentation result of the i-th slice image from the coronal perspective is predicted based on a general segmentation model. Here, the initial value of i is 1, and the maximum value is H. If i is not equal to H, i is updated to i+1. The above steps are repeated until the segmentation results of the general segmentation model for H slice images from the coronal perspective are obtained, and these are summarized as pseudo-labels for the coronal perspective.

[0142] For example, the i-th slice image in the axial view is taken as the image to be segmented. Based on the prototype information, prompt information for the i-th slice image in the axial view is generated. The segmentation result of the i-th slice image in the axial view is predicted based on the general segmentation model. Here, the initial value of i is 1, and the maximum value is W. If i is not equal to W, i is updated to i+1. The above steps are repeated until the segmentation results of the general segmentation model for W slice images in the axial view are obtained, and these results are summarized into pseudo-labels for the axial view.

[0143] For example, the i-th slice image in the sagittal view is taken as the image to be segmented. Based on the prototype information, prompt information for the i-th slice image in the sagittal view is generated. The segmentation result of the i-th slice image in the sagittal view is predicted based on a general segmentation model. Here, the initial value of i is 1, and the maximum value is D. If i is not equal to D, i is updated to i+1. The above steps are repeated until the segmentation results of the general segmentation model for D slice images in the sagittal view are obtained, and these are summarized as pseudo-labels in the sagittal view.

[0144] Sub-step 242b: Generate fused pseudo-labels based on the calibration strategy after fusing pseudo-labels from three perspectives, and generate calibrated teacher prediction results based on the fused pseudo-labels.

[0145] The correction strategy refers to a method for integrating pseudo-labels from different perspectives and generating fused pseudo-labels. For example, the correction strategy is a majority voting principle (or majority rule). For instance, if the pseudo-labels on at least two of the three perspectives belong to the first category, the fused pseudo-label of the voxel is determined to belong to the first category.

[0146] In some embodiments, after generating fused pseudo-labels by fusing pseudo-labels from three perspectives, self-identification and calibration processing is performed on the fused pseudo-labels to generate calibrated teacher prediction results.

[0147] In this embodiment, by segmenting the 3D image from three different perspectives (e.g., coronal, axial, and sagittal), image information from different angles can be fully utilized, obtaining more comprehensive image features from multiple dimensions and improving the robustness and accuracy of the segmentation model. After generating pseudo-labels at each perspective, the errors caused by perspective differences or pseudo-label biases can be effectively reduced through the fusion of correction strategies. This ensures that the final pseudo-labels are more consistent and accurate, providing higher-quality teacher prediction results for subsequent segmentation tasks.

[0148] In summary, the method provided in this application uses unlabeled samples as images to be segmented, obtains prototype information of the images to be segmented, and generates prompts for a general segmentation model based on the prototype information. This is equivalent to providing the general segmentation model with additional prior knowledge. The prompts help the general segmentation model focus on important features during the segmentation process, effectively guiding the learning process of the general segmentation model. The teacher model is usually responsible for providing higher-quality guidance, especially in the case of weakly supervised learning or unlabeled samples. By combining the prompts generated with the prototype information, the prediction results of the teacher model can be calibrated, thereby improving the overall performance of the segmentation task.

[0149] Prototype information generation prompt message

[0150] In some embodiments, the generalist segmentation model can accurately segment an input two-dimensional image (a slice image of a three-dimensional image sliced ​​from different perspectives) with the guidance of prompts. The prompts for inputting the generalist segmentation model are described below.

[0151] In some embodiments, the prompt message includes at least one of the following:

[0152] • Two-dimensional bounding box hints: These hints are used to mark the extent of the target segmentation object.

[0153] • On-time alert information: On-time alert information is used to indicate the location of key points of the target segmentation object;

[0154] • Negative dot information: Negative dot information is used to indicate the location of non-target segmentation objects.

[0155] Two-dimensional bounding box cue information is extracted based on the enhanced teacher prediction results.

[0156] In some embodiments, the prompt information includes two-dimensional bounding box prompt information, which is obtained based on the enhanced teacher prediction results, which are teacher prediction results guided by prototype information.

[0157] In some embodiments, the prototype information includes first prototype information and second prototype information. The first prototype information is an object prototype, used to indicate the features of the target segmented object in the image to be segmented. The second prototype information is a background prototype, used to indicate the features of non-segmented objects (the background segmented image excluding the target segmented object) in the image to be segmented.

[0158] Optionally, the similarity between the features of each voxel in the image to be segmented and the first prototype information is compared, and the similarity between the features of each voxel in the image to be segmented and the second prototype information is compared, to obtain the enhanced teacher prediction result. The enhanced teacher prediction result includes the predicted probability that each voxel in the image to be segmented belongs to the target segmentation object or is not a target segmentation object.

[0159] In some embodiments, for each voxel in the image to be segmented, a first cosine similarity is calculated between the features (from the upsampled feature map) of each voxel and the first prototype information (object prototype), and a second cosine similarity is calculated between the features of each voxel (from the upsampled feature map) and the second prototype information (background prototype). After obtaining the first and second cosine similarities, the similarity values ​​(or similarity scores) are converted into a probability distribution, which indicates the enhanced teacher prediction results, including the predicted probability that each voxel belongs to the target segmentation object or a non-target segmentation object. For example, the softmax function is used to normalize the similarity values ​​(or similarity scores) into a probability distribution.

[0160] Cosine similarity is used to measure the directional consistency between two features (the features of a voxel and the features indicated by the prototype information). Optionally, the value of cosine similarity ranges from [-1, 1], and the value is positively correlated with the similarity; the larger the value, the more similar the two features are.

[0161] In some embodiments, if the voxel's features are aligned with the first prototype information (object prototype), it indicates a high similarity between the voxel's features and the first prototype information, and a high probability that the voxel belongs to the foreground category. If the voxel's features are aligned with the second prototype information (background prototype), it indicates a high similarity between the voxel's features and the second prototype information, and a high probability that the voxel belongs to the background category.

[0162] For example, the segmentation task is the segmentation of an MRI heart in a 3D image. Features of all voxels representing the heart are extracted from the sampled feature map corresponding to the image to be segmented. A confidence-weighted average is then performed on the features of all voxels representing the heart to obtain the heart prototype. Similarly, features of all voxels representing the background (non-heart tissue) are extracted from the sampled feature map corresponding to the image to be segmented. A confidence-weighted average is then performed on the features of all voxels representing the background (non-heart tissue) to obtain the background (non-heart tissue) prototype. For a voxel A in the image to be segmented, the first similarity between the features of voxel A and the heart prototype is compared (e.g., a first similarity of 0.7). The second similarity between the features of voxel A and the background (non-heart tissue) prototype is also compared (e.g., a second similarity of 0.3). This determines that voxel A belongs to the heart category.

[0163] Optionally, when calculating similarity, the probability distribution between the features of different voxels and the first prototype information is adjusted based on the temperature parameter, or the probability distribution between the features of different voxels and the second prototype information is adjusted. The temperature parameter is a parameter used to adjust the probability distribution, smoothing or amplifying the probability distribution between the features of different voxels and the prototype information. For example, a smaller temperature parameter results in a sharper probability distribution; a larger temperature parameter results in a smoother probability distribution.

[0164] In this embodiment, by calculating the cosine similarity between the features and prototype information (object prototype or background prototype) of each voxel, the similarity distribution between each voxel and different categories can be obtained. Then, a probability distribution based on prototype information is obtained through normalization calculation. Finally, the category with the highest probability is selected as the predicted label for each voxel, thereby completing the prototype-guided prediction. This approach can improve the accuracy of image segmentation by utilizing prototype-guided prediction, especially when dealing with noisy labels or a large amount of unlabeled data. This method combines prior knowledge of prototype information with dynamic information from sampled feature maps, providing a robust semi-supervised learning method.

[0165] Extracting positive and negative point hints based on prototype feature similarity map

[0166] In some embodiments, the prompt information includes positive prompt information and negative prompt information, which are obtained based on the prototype feature similarity map of the target segmented object.

[0167] Optionally, a prototype feature similarity map of the target segmentation object is obtained based on the similarity between the sampled feature map corresponding to the image to be segmented and the first prototype information. The points in the prototype feature similarity map are sorted from high to low according to the similarity between the sampled feature map (features in the sampled feature map) and the first prototype information. Optionally, the prototype feature similarity map reflects the degree of similarity between each point in the image and the first prototype information (object prototype). Regions with high similarity are more likely to belong to the target segmentation object, while regions with low similarity are more likely to belong to non-target segmentation objects.

[0168] In one possible implementation, features in the sampled feature map that have high similarity to the prototype information of the first object are placed at the front of the prototype feature similarity map; conversely, features in the sampled feature map that have low similarity to the prototype information of the first object are placed at the back of the prototype feature similarity map.

[0169] In this embodiment, the prototype feature similarity map is obtained by calculating the similarity between the features of each voxel and the first prototype information (object prototype). The points in the prototype feature similarity map represent the degree of matching with the foreground category. By utilizing positive and negative cue information during training, the general segmentation model's focus on the target segmentation object is strengthened, and the target segmentation object is effectively distinguished from non-target segmentation objects (background segmentation image).

[0170] In some embodiments, multiple slice images from three perspectives are used as images to be segmented, and prompt information for each slice image is generated based on prototype information. The prompt information for each slice image includes at least one of two-dimensional bounding box prompt information, positive point prompt information, and negative point prompt information.

[0171] • Tooltip information for 2D bounding boxes

[0172] The two-dimensional bounding box cue information is derived from the enhanced teacher prediction results.

[0173] After obtaining the enhanced teacher prediction results, a selection operation is performed on them to obtain the prototype information label corresponding to each voxel in the image to be segmented. The prototype information label is then decomposed into sub-labels from three perspectives. The prototype information label is the prediction label with the highest prediction probability among the teacher prediction results guided by the prototype information. Optionally, each of the three perspectives corresponds to multiple slice images.

[0174] In some embodiments, for any voxel, the prototype information label corresponding to the voxel is decomposed according to the coronal view to obtain the sub-label of the prototype information label in the coronal view; the prototype information label corresponding to the voxel is decomposed according to the axial view to obtain the sub-label of the prototype information label in the axial view; the prototype information label corresponding to the voxel is decomposed according to the sagittal view to obtain the sub-label of the prototype information label in the sagittal view.

[0175] In some embodiments, for each of a plurality of slice images, a connected component of each slice image is extracted based on a sub-label, and a two-dimensional bounding box cue is determined based on the connected component. Optionally, for each slice image, the two-dimensional bounding box coordinates of each slice image are extracted based on a sub-label, and a connected component of each slice image is determined based on the two-dimensional bounding box coordinates. The connected component is used to indicate the set of pixels in the slice image predicted to belong to the target category (such as the foreground category).

[0176] In some embodiments, for each connected region of a sliced ​​image, its minimum bounding rectangle (i.e., a two-dimensional bounding box) is calculated, and the coordinates of the minimum bounding rectangle are recorded. The coordinates of the minimum bounding rectangle include the coordinates of the top-left corner and the bottom-right corner of the two-dimensional bounding box; or, the coordinates of the minimum bounding rectangle include the coordinates of the bottom-left corner and the top-right corner of the two-dimensional bounding box; or, the coordinates of the minimum bounding rectangle include the coordinates of the top-left corner, the bottom-left corner, the bottom-right corner, and the top-right corner of the two-dimensional bounding box; or, the coordinates of the minimum bounding rectangle include at least three of the following coordinates: top-left corner, bottom-left corner, bottom-right corner, and top-right corner.

[0177] In some embodiments, if multiple connected regions belong to the same target segmentation object (such as different parts of the liver), the two-dimensional bounding boxes of the multiple connected regions are merged.

[0178] • Regarding punctuality reminders

[0179] The on-time prompt information is obtained based on the prototype feature similarity map of the target segmentation object.

[0180] Optionally, the prototype feature similarity map reflects the degree of similarity between each point in the image and the first prototype information (object prototype). Regions with high similarity are more likely to belong to the target segmentation object, while regions with low similarity are more likely to belong to non-target segmentation objects.

[0181] By analyzing the prototype feature similarity map, regions in the image to be segmented that are highly similar to the target object (high similarity regions) can be identified. Optionally, points are selected from the high similarity regions as key information, and the preset points in the high similarity regions belong to the foreground category (target category) corresponding to the target object.

[0182] When the prompt message includes a timed prompt message, for each of the multiple slice images, the first h points of the prototype feature similarity map are determined as the timed prompt message, where h is a positive integer greater than or equal to 1.

[0183] The on-time prompt information is used to indicate the location of key points of the target segmentation object. The key point location refers to either the center location of the target segmentation object or a high-confidence location.

[0184] • Regarding negative point warning information

[0185] The negative point prompt information is obtained based on the prototype feature similarity map of the target segmentation object.

[0186] By analyzing the prototype feature similarity map, regions in the image to be segmented that are low in similarity to the target object (low similarity regions) can be identified. Optionally, points are selected from the low similarity regions as negative point prompts, and the preset points in the low similarity regions belong to the background category (non-target category) corresponding to the non-target object.

[0187] When the prompt information includes negative point prompt information, for each slice image in multiple slice images, the last g points of the prototype feature similarity map are determined as negative point prompt information, where g is a positive integer greater than or equal to 1.

[0188] Negative cue information is used to indicate the location of non-target objects to be segmented. For example, negative cue information is used to indicate the background location in the image to be segmented.

[0189] In some embodiments, the generalist segmentation model includes an image encoder, a cue encoder, and a mask decoder. The image encoder performs feature extraction on the input two-dimensional image to obtain an image feature representation; the cue encoder performs feature extraction on the input cue information (cue information generated based on prototype information) to obtain a cue feature representation; and the mask decoder performs image segmentation on the image feature representation based on the cue guidance provided by the cue feature representation, resulting in a more accurate segmentation result.

[0190] Optionally, the input prompt information in the generalist segmentation model includes two-dimensional bounding box prompt information, positive prompt information, and negative prompt information. The prompt encoder in the generalist segmentation model performs feature extraction on the input prompt information (two-dimensional bounding box prompt information, positive prompt information, and negative prompt information) to obtain the prompt feature representation.

[0191] In this embodiment, prompt information is generated for each slice image in multiple slice images based on prototype information. This prompt information provides the location and extent information of the target segmentation object, helping the general segmentation model to more accurately locate and segment the target object. For example, two-dimensional bounding box information provides global information to the general segmentation model by defining the approximate location of the target segmentation object in the 2D slice (e.g., organ, lesion region). Positive prompt information marks high-confidence regions within the target segmentation object, clearly indicating the center or typical region of the target segmentation object, enhancing the general segmentation model's perception of key features. Negative prompt information excludes background or easily confused regions, marking the background (e.g., adjacent organs, artifacts), preventing missegmentation by the general segmentation model. By combining the global positioning capability of two-dimensional bounding box information with the local correction capability of positive and negative point information, the prototype-guided prompt information generation mechanism significantly improves the accuracy and robustness of 3D medical image segmentation under sparse annotation. The general segmentation model can better understand the difference between the target segmentation object and non-target segmentation objects (background segmentation objects), thereby improving segmentation accuracy.

[0192] Based on the correction strategy, fused pseudo-labels are generated.

[0193] In some embodiments, the image to be segmented (3D image) corresponding to the unlabeled sample is sliced ​​according to three perspectives to obtain multiple slice images corresponding to the three perspectives respectively. The multiple slice images of the three perspectives are used as the images to be segmented (2D image) of the general segmentation model. Based on the prototype information, prompt information is generated for each slice image in the multiple slice images. The general segmentation model predicts the pseudo-labels on the three perspectives respectively according to the prompt information.

[0194] For example, the image to be segmented (3D image) corresponding to the unlabeled sample is sliced ​​according to the coronal view to obtain multiple slice images corresponding to the coronal view. The multiple slice images of the coronal view are used as the images to be segmented (2D image) of the general segmentation model. Based on the prototype information, prompt information is generated for each slice image in the multiple slice images. The general segmentation model predicts the pseudo label on the coronal view according to the prompt information.

[0195] For example, the image to be segmented (3D image) corresponding to the unlabeled sample is sliced ​​according to the axial view to obtain multiple slice images corresponding to the axial view. The multiple slice images of the axial view are used as the images to be segmented (2D image) of the general segmentation model. Based on the prototype information, prompt information is generated for each slice image in the multiple slice images. The general segmentation model predicts the pseudo label on the axial view according to the prompt information.

[0196] For example, the image to be segmented (3D image) corresponding to the unlabeled sample is sliced ​​according to the sagittal view to obtain multiple slice images corresponding to the sagittal view. The multiple slice images of the sagittal view are used as the images to be segmented (2D image) of the general segmentation model. Based on the prototype information, prompt information is generated for each slice image in the multiple slice images. The general segmentation model predicts the pseudo label on the sagittal view according to the prompt information.

[0197] In some embodiments, a fused pseudo-label is generated by fusing pseudo-labels from three perspectives (coronal, axial, and sagittal) based on a correction strategy. Here, the correction strategy refers to a method for integrating pseudo-labels from different perspectives to generate a fused pseudo-label.

[0198] Optionally, the correction strategy includes, but is not limited to, either the principle of unanimous consent or the principle of majority voting (or the principle of majority rule). This application mainly uses the principle of majority voting as an example for illustration.

[0199] Taking binary classification image segmentation as an example, the majority voting principle means that for any voxel in an unlabeled sample, if the pseudo-labels in at least two of the three viewpoints belong to the first category, the fused pseudo-label of the voxel belongs to the first category. If the pseudo-labels in at least two of the three viewpoints belong to the second category, the fused pseudo-label of the voxel belongs to the second category.

[0200] Taking a case where the first category is the foreground category and the second category is the background category as an example: If, for any voxel in an unlabeled sample, there are at least two pseudo-labels from different viewpoints that consider the voxel to belong to the foreground category, then the voxel is determined to belong to the foreground category; if there are at least two pseudo-labels from different viewpoints that consider the voxel to belong to the background category, then the voxel is determined to belong to the background category.

[0201] During the calibration of sample labels, the majority voting principle (or majority rule) allows for rapid decision-making; that is, if two perspectives reach a consensus, there is no need to examine the pseudo-labels from a third perspective. Since the number of voxels in each 3D image is extremely large, the majority rule strategy can significantly reduce the computational cost of calibrating all voxels in the entire 3D image, thus improving the efficiency of the calibration process.

[0202] Based on the generation of calibrated teacher prediction results using fused pseudo-labels

[0203] The above process, after fusing pseudo-labels from three perspectives using a correction strategy to generate fused pseudo-labels, may still contain errors. To improve the quality of the fused pseudo-labels, further fine-tuning is needed. Optionally, after generating fused pseudo-labels by fusing pseudo-labels from three perspectives, self-identification and calibration processing is performed on the fused pseudo-labels to generate calibrated teacher prediction results. The self-identification and calibration processing of the fused pseudo-labels is explained below.

[0204] In some embodiments, the predicted probability of each voxel in the image to be segmented under its category is obtained based on the teacher prediction results of unlabeled samples. Optionally, for each voxel, the predicted probability under its category is averaged to obtain a probability threshold corresponding to the category, and the label type is determined based on the relationship between the predicted probability and the probability threshold. The label type of correctly labeled voxels in the unlabeled samples is determined as the category to which the fused pseudo-label belongs, and the label type of incorrectly labeled voxels in the unlabeled samples is calibrated to the category to which the latent label belongs. The calibrated teacher prediction results are generated based on the fused pseudo-label and latent label.

[0205] In one possible implementation, the prediction results generated by the specialist segmentation model (e.g., teacher prediction results for unlabeled samples) are used as a third party for cross-validation to fine-tune the fused pseudo-labels. The Classification Noise Process (CNP) assumption is used, which assumes that each voxel has a true class label (latent label), and each latent label has a certain probability of being incorrectly labeled as another class label, i.e., there is a certain probability that it will be labeled as another class label in the fused pseudo-labels.

[0206] In some embodiments, the teacher prediction results for unlabeled samples include the predicted probability of each voxel in the unlabeled sample under its category. Optionally, a confidence average is performed on the predicted probability of each voxel under its category to obtain a probability threshold corresponding to the category. The predicted probability of each voxel under its category is compared with the probability threshold, and the label type is determined based on the relationship between the predicted probability and the probability threshold. Optionally, if the predicted probability of each voxel under its category is greater than or equal to the probability threshold, the label type of the voxel is determined to belong to the latent label, that is, the category to which the voxel belongs is determined to be the category indicated by the latent label. If the predicted probability of each voxel under its category is less than the probability threshold, the label type of the voxel is determined to belong to the fused pseudo-label, that is, the category to which the voxel belongs is determined to be the category indicated by the fused pseudo-label.

[0207] For example, for any voxel, the fused pseudo-label of the voxel belongs to the first category, and the potential label of the voxel is assumed to belong to the second category. The predicted probability of the voxel under the second category is averaged to obtain the probability threshold corresponding to the second category. If the predicted probability of the voxel under the second category is greater than the probability threshold, the voxel is considered to belong to the second category rather than the first category. That is, the fused pseudo-label of the voxel is incorrectly labeled data.

[0208] In some embodiments, for each voxel, the label type of each voxel is determined based on the relationship between the predicted probability and the probability threshold. A threshold comparison operation (comparing the predicted probability with the probability threshold) is used to filter out mislabeled voxels in the unlabeled samples, resulting in the number of voxels in the unlabeled samples whose potential label is the second category and whose fused pseudo-label is the first category (the number of mislabeled voxels).

[0209] In some embodiments, a confidence confusion matrix for unlabeled samples is determined based on the number of mislabeled voxels. The confidence confusion matrix indicates the number of mislabeled voxels in the unlabeled samples. Optionally, the confidence confusion matrix is ​​calibrated to obtain a calibrated confidence confusion matrix, where the sum of its elements equals the total number of voxels in the unlabeled samples. The calibrated confidence confusion matrix is ​​then normalized to obtain a joint probability matrix, which indicates the probability that a voxel in the unlabeled sample has a fused pseudo-label of the first category and a latent label of the second category.

[0210] In some embodiments, voxels with suspicious labels (candidate voxels) are randomly selected from the joint probability matrix to obtain a label error indication map. A suspicious label refers to a candidate voxel whose fused pseudo-label is of the first category and whose potential label is of the second category. Voxels with suspicious labels are voxels in unlabeled samples whose confidence level is lower than the confidence threshold (minimum confidence threshold). The label error indication map is used to indicate voxels in unlabeled samples that may be mislabeled.

[0211] Optionally, the label error indicator map is a binary mask (3D matrix) of the same size as the 3D image, used to mark which voxels in the unlabeled samples may have their fused pseudo-labels incorrectly labeled. For example, a voxel value of 1 in the label error indicator map indicates that the fused pseudo-label of that voxel has been incorrectly labeled; or, a voxel value of 0 in the label error indicator map indicates that the fused pseudo-label of that voxel has been incorrectly labeled.

[0212] In some embodiments, the category to which voxels in unlabeled samples belong is calibrated. Optionally, for correctly labeled voxels in unlabeled samples, the label type of the correctly labeled voxels in the unlabeled samples is determined as the category to which the fusion pseudo-label belongs. For incorrectly labeled voxels in unlabeled samples, the labels of the incorrectly labeled voxels are flipped, and the fusion pseudo-labels corresponding to the incorrectly labeled voxels in the unlabeled samples are corrected to the category to which the potential label belongs. Optionally, based on the fusion pseudo-labels corresponding to the correctly labeled voxels in unlabeled samples and the corrected voxels to the potential labels, a calibrated teacher prediction result is generated.

[0213] In this embodiment, by calculating the predicted probability of each voxel and comparing it with a probability threshold, the correctness of the voxel's label type can be determined more accurately. Specifically, using a probability threshold to determine the label type can filter out which voxels are incorrectly labeled, thereby calibrating these voxels. This helps reduce the impact of incorrect labels and improves the quality of the final labels. In unlabeled samples, correctly labeled voxels are directly assigned to the category of the fused pseudo-label, while incorrectly labeled voxels are calibrated to the category of the latent label. By fusing pseudo-labels and latent labels, label accuracy can be improved without relying on fully labeled data. This fusion strategy enhances the model's generalization ability.

[0214] Losses with or without supervision

[0215] In some embodiments, the loss of the talent segmentation model includes supervised loss and unsupervised loss. The talent segmentation model is trained based on both supervised and unsupervised losses. When labeled samples are scarce, using both labeled and unlabeled samples can significantly improve the performance of the student model, while also enhancing the diversity of training samples to improve the generalization ability of the student model.

[0216] Optionally, the supervised loss for labeled samples is calculated based on student predictions and true labels. Optionally, the consistency loss for unlabeled samples is calculated based on student predictions and teacher predictions for unlabeled samples, and the labeled loss for unlabeled samples is calculated based on student predictions and calibrated teacher predictions. The unsupervised loss for unlabeled samples is then calculated based on the consistency loss and the labeled loss.

[0217] The supervised and unsupervised losses of the specialist segmentation model will be explained below.

[0218] • Regarding supervised losses

[0219] In some embodiments, there is an error between the student prediction results of the labeled samples and the true labels. Based on the student prediction results of the labeled samples and the true labels, the supervised loss of the labeled samples is calculated.

[0220] The supervised loss is used to indicate the error between the student model's predictions of labeled samples and the true labels. The supervised loss is used to indicate the accuracy of the student model's predictions of labeled samples.

[0221] In this embodiment, the supervised loss provides the input data and its corresponding correct output, which is the key information that the semi-supervised learning algorithm can perform, enabling the student model to learn the most basic specialist segmentation ability.

[0222] • Regarding unsupervised losses

[0223] Unsupervised loss includes consistency loss and labeling loss.

[0224] Optionally, the consistency loss of unlabeled samples is calculated based on the student prediction results and the teacher prediction results of unlabeled samples.

[0225] The consistency loss is used to indicate the error between the student model's predictions of unlabeled samples and the teacher model's predictions of unlabeled samples. It can also be understood as the degree of consistency between the student model's predictions of unlabeled samples and the teacher model's predictions of unlabeled samples.

[0226] Optionally, the label loss for unlabeled samples is calculated based on the student prediction results for unlabeled samples and the calibrated teacher prediction results. The calibrated teacher prediction results have higher accuracy, and the label loss for unlabeled samples is determined based on the calibrated teacher segmentation results.

[0227] Label loss is derived from the student predictions for unlabeled samples and the calibrated teacher predictions. When there is a significant difference between the student predictions for unlabeled samples and the calibrated teacher predictions, the loss may tend to infinity, leading to overfitting to noisy labels (calibrated teacher predictions).

[0228] In some embodiments, when calculating the labeled loss, boundary constraints are applied to the predicted value vector generated by the student model. The predicted value vector is the unnormalized predicted value output by the student model. Optionally, for each voxel in the unlabeled samples, the predicted value vector corresponding to each voxel is obtained, and norm constraints are applied to the predicted value vector. If the norm of the predicted value vector is greater than a threshold, the predicted value vector is normalized to the threshold; if the norm of the predicted value vector is not greater than the threshold, the predicted value vector remains unchanged. By limiting the range of the predicted value vector, the cross-entropy loss of each voxel is prevented from reaching infinity.

[0229] While labeled loss provides effective early supervision during training, making it a crucial component of the training process, over-reliance on these noisy labels (calibrated teacher predictions) can mislead student models in later stages due to memory effects. Furthermore, the stability constraint of consistency loss becomes relatively meaningless in the early training phase because student models struggle to perceive objects. Therefore, an alternative design introduces weights related to training time to modulate the importance of the two losses during collaborative training, ensuring that the most effective loss is referenced for supervision at different stages.

[0230] In some embodiments, after calculating the consistency loss and label loss of unlabeled samples respectively, the consistency loss and label loss are weighted and summed, and the unsupervised loss of unlabeled samples is calculated based on the weighted sum of the consistency loss and label loss.

[0231] Optionally, based on the current and maximum number of training iterations, determine the first weight corresponding to the consistency loss and the second weight corresponding to the label loss. Calculate the first product of the consistency loss and the first weight, and calculate the second product of the label loss and the second weight; the sum of the first and second products is determined as the unsupervised loss for the unlabeled samples.

[0232] For example, suppose the ratio between the current number of training attempts and the maximum number of training attempts is the percentage of training attempts. The first weight is negatively correlated with the percentage of training attempts, that is, the larger the percentage of training attempts, the smaller the first weight; the second weight is positively correlated with the percentage of training attempts, that is, the larger the percentage of training attempts, the larger the second weight.

[0233] In some embodiments, the training process typically includes multiple rounds of training. Based on the supervised loss for labeled samples and the unsupervised loss for unlabeled samples, the network parameters of the student model are updated in the i-th round of training, where i is a positive integer. With the training objective of reducing the supervised loss for labeled samples and the unsupervised loss for unlabeled samples, the network parameters of the student model are updated in the i-th round of training, where i is an integer and its initial value is 0 or 1. Optionally, the network parameters of the teacher model are updated based on the network parameters of the student model before the i-th round of training and the network parameters trained in the i-th round.

[0234] The teacher-student model is based on the standard SSL design with consistency regularization. The student model is trained using the error backpropagation algorithm, while the teacher model is updated based on the student model's past and current network parameters using the Exponential Moving Average (EMA) mechanism.

[0235] The network parameters prior to the i-th training round can be network parameters from at least one previous training round. For example, the network parameters of the student model from the previous training round and the network parameters from the current training round can be summed according to their weights to obtain the network parameters of the teacher model.

[0236] In this embodiment, the unsupervised loss for unlabeled samples is calculated using labeled loss and consistency loss. This allows the student model to utilize the additional supervision provided by labeled loss and the consistency loss to help it learn representations consistent with the teacher model, thereby improving the generalization ability of the student model. In other words, it can simultaneously provide supervision signals for unlabeled samples from two training perspectives, promoting the training effect of the student model.

[0237] Active learning (actively selecting labeled samples)

[0238] In traditional semi-supervised learning, a subset of unlabeled samples is randomly selected and labeled by annotators to obtain labeled samples. However, random selection may not yield the samples most needed by the expert segmentation model, and can easily lead to noisy and redundant labeling. To better utilize the precious labeling budget, this embodiment employs active sample selection. In each training round, the student model scores each unlabeled sample, and then selects the unlabeled samples with higher scores for labeling to obtain labeled samples. This process is repeated until the labeling budget is exhausted.

[0239] Actively selecting sample labels allows for more efficient use of limited annotation resources. On one hand, by selecting the most valuable samples for labeling, and learning from the samples that best improve the student model at the current stage, the performance of the student model can be improved more quickly. On the other hand, when labeling costs are high, active learning can reduce the number of samples that need to be labeled, thereby saving costs.

[0240] In the above-described active learning embodiment, the method further includes: during the training process of the specialist segmentation model, selecting candidate labeled samples that meet the labeling conditions from the unlabeled samples, wherein the candidate labeled samples are samples with candidate labels for real labels; and filtering the candidate labeled samples based on the intensity features of the histogram to determine the samples to be labeled, and labeling the samples to be labeled with real labels.

[0241] Optionally, based on the student prediction results of the unlabeled samples in the i-th round of training, candidate labeled samples that meet the labeling conditions are selected from the unlabeled samples; the candidate labeled samples are filtered based on the intensity features of the histogram, and the samples to be labeled are determined from the candidate labeled samples. The real labels for the samples to be labeled are obtained, resulting in the labeled samples actively selected and generated in the i-th round of training, and the labeled samples are added to the set of labeled samples used in the (i+1)-th round of training. The process of labeling the samples to be labeled as labeled samples can be completed manually by people with professional skills.

[0242] Among them, unlabeled samples that meet the labeling conditions include at least one of the following.

[0243] • The top K1 unlabeled samples ranked from lowest to highest confidence level in the student prediction results;

[0244] • The top K2 unlabeled samples are sorted from high to low according to the average prediction entropy of the student prediction results of the unlabeled samples. The average prediction entropy is the average entropy of all voxels in the student prediction results of the unlabeled samples.

[0245] • The top K3 unlabeled samples are sorted from high to low according to the entropy ratio of the student prediction results of the unlabeled samples. The entropy ratio is the proportion of voxels in the student prediction results of the unlabeled samples that exceed the probability threshold to all voxels. K1, K2 and K3 are all positive integers.

[0246] For each unlabeled sample, the confidence level of the student's prediction for that unlabeled sample is used as the active score. The top K1 unlabeled samples, ranked from lowest to highest, are the actively selected samples to be labeled. Since the samples with lower confidence levels are those that the current talent segmentation model cannot accurately predict, selecting these samples for learning allows the current model to quickly improve its performance.

[0247] For each unlabeled sample, the average prediction entropy of the students' predictions for that unlabeled sample is used as the active score. The top K2 unlabeled samples, ranked from highest to lowest, are the actively selected samples to be labeled. Each voxel has its own corresponding prediction probability, representing whether it belongs to the first category. The average prediction entropy represents the uncertainty of the students' predictions for the unlabeled sample; selecting samples with higher average prediction entropy can identify samples with higher uncertainty.

[0248] For each unlabeled sample, the entropy ratio of the student's prediction results for that unlabeled sample is used as the active score. The top K3 unlabeled samples, ranked from highest to lowest, are the actively selected samples to be labeled. Compared to the average prediction entropy, the highest entropy ratio can more clearly reflect the distribution of high uncertainty in the image, which is beneficial for selecting samples with high uncertainty distributions.

[0249] In some embodiments, candidate labeled samples that meet the labeling criteria are selected from the unlabeled samples. Based on the intensity features of the histogram, the candidate labeled samples are further filtered to determine the samples to be labeled, and then the samples to be labeled are labeled with true labels. Optionally, the histogram features of the candidate labeled samples are clustered, dividing the candidate labeled samples into a predetermined number of clusters. The most representative candidate labeled sample is selected from each cluster as the sample to be labeled. Here, the histogram feature is a way to describe the voxel intensity distribution in an image. Specifically, the voxel intensity values ​​of the image are divided into multiple intensity bins, each bin representing an intensity range. This ensures that the finally selected samples to be labeled can capture various intensity distributions, thereby ensuring that the selected samples to be labeled have both uncertainty and diversity.

[0250] Reference Figure 5 , Figure 5 This is a schematic diagram illustrating the principle of an expert-model collaboration process provided in an exemplary embodiment of this application. After the i-th round of training, the student prediction results are based on the student model's predictions. An active score is calculated for each unlabeled sample using an active learning function. The top K unlabeled samples with the highest active scores are selected as samples to be labeled. Then, experts in the relevant field are consulted, and the experts perform labeling on the actively selected unlabeled samples.

[0251] Cross-labeling strategy

[0252] In some embodiments, the labeled samples are 3D images. Traditional annotation methods require labeling every voxel in a 3D image, which is extremely labor-intensive. Therefore, this application also provides an annotation method to reduce the workload. That is, the real labels of the labeled samples in this application can be annotated using a cross-labeling strategy. The cross-labeling strategy is an annotation strategy that labels key slice images from three different perspectives in a 3D image; wherein, the key slice image for each perspective is one of multiple slice images belonging to the current perspective.

[0253] Optionally, if there are n1 slice images under a viewpoint, where n1 is an odd number, then the key slice image under that viewpoint is the middle slice image, which is the (n1+1) / 2th slice image; if there are n2 slice images under a viewpoint, where n2 is an even number, then the key slice image under that viewpoint is the middle slice image, which is the n2 / 2th slice image or the (n2 / 2)+1th slice image.

[0254] Continue to refer to Figure 5 Taking a medical 3D image as an example, a key coronal slice is selected from multiple slice images in the coronal view of the 3D image; an axial key slice is selected from multiple slice images in the axial view of the 3D image; and a sagittal key slice is selected from multiple slice images in the sagittal view of the 3D image. Medical experts then annotate these three slice images (coronal, axial, and sagittal key slices) as the true labels for the 3D image after cross-annotation.

[0255] Since a 3D image has a very large number of voxels, reaching H×D×W, compared to the scheme that labels all voxels, the number of voxels required to be labeled for a 3D image in this application is H×D+H×W+D×W, which reduces the workload by 1-2 orders of magnitude. This greatly reduces the labeling workload and makes fuller use of the limited labeling budget to achieve training results similar to labeling all voxels.

[0256] Image segmentation is an important technique in computer vision and a crucial component of image understanding. The training method for the aforementioned specialized segmentation model is applicable to both 2D and 3D image segmentation scenarios. Furthermore, image segmentation can be applied in fields such as medical image analysis, autonomous driving systems, intelligent agriculture, facial recognition, video editing, augmented reality, retail image recognition, iris recognition, and machine vision.

[0257] The following example illustrates the application of image segmentation in 3D medical imaging, using the specialist segmentation model for 3D images and the generalist segmentation model for 2D images.

[0258] Problem Definition

[0259] Traditional SSL typically involves randomly pre-selecting a subset of M unlabeled samples from the training dataset D, performing dense annotation, and then using the annotated subset D. l (M labeled samples) and unlabeled subset D u (N unlabeled samples) are used for training, where the superscript l represents a labeled sample and the superscript u represents an unlabeled sample.

[0260] This embodiment explores a novel active sparse supervised learning setting that emphasizes more efficient use of the limiting annotation budget. Unlike traditional SSL, this embodiment's active sparse supervised learning incorporates a human-in-the-loop paradigm to select high-value samples for annotation during training and employs an efficient cross-labeling strategy.

[0261] Specifically, in the r-th round of active learning, this embodiment has a cross-label subset. Contains N r After cross-labeling, the labeled samples are used, and the remaining samples are treated as an unlabeled subset. Includes M r There are 10 unlabeled samples, where Cl represents cross-labeling and u represents no labeling. Represents a 3D image with height H, width W, and depth D, Y i cl ∈{0,1} H×W×D express The cross-labels (the embodiments of this application mainly focus on two-class segmentation).

[0262] Before training, embodiments of this application perform cross-labeling on a small subset of unlabeled samples to warm up the medical segmentation model, thereby constructing... As a starting point, the objective of this application is to learn an accurate medical segmentation model using actively cross-labeled and unlabeled samples by optimizing the following loss function:

[0263]

[0264] in, express Supervised loss for labeled samples, L aux This represents the auxiliary loss from all samples, i.e., the unsupervised loss. For supervised loss... This application embodiment employs partial cross-entropy loss on each voxel in the real labels. This application embodiment also employs the two data center strategies described above: specialist-generalist collaboration and expert-model collaboration.

[0265] The training process of this medical segmentation model includes at least one of the following steps.

[0266] Step 1: Obtain a large number of label-free medical samples in 3D image format;

[0267] Taking medical images, specifically radiological images, as an example, there are a large number of computed tomography (CT) images or magnetic resonance imaging (MRI) images without actual labels. These medical images are in a three-dimensional image format. Each medical sample includes H×W×D voxels.

[0268] Step 2: In the first iteration, a small number of unlabeled medical samples are randomly selected for labeling to obtain labeled medical samples;

[0269] In the first iteration, a small number of unlabeled medical samples were randomly selected. These unlabeled samples were then labeled by medical experts, resulting in labeled medical samples.

[0270] Since each medical sample comprises H×W×D voxels, labeling each voxel with a segmentation probability (the probability of belonging to the first category, ranging from 0% to 100%) would require an enormous amount of annotation work from medical experts. Therefore, this application employs a cross-labeling strategy.

[0271] Three-dimensional medical images can be viewed as a stacked structure of multiple two-dimensional images. This cross-annotation strategy is a method of labeling two-dimensional segmentation information onto key slice images from three different perspectives within a three-dimensional image format. Each key slice image is a single slice image from a set of multiple slice images belonging to that perspective. In other words, only one slice image needs to be labeled for each perspective, eliminating the need to label every single slice image.

[0272] At this point, there are a small number of labeled medical samples and a large number of unlabeled medical samples.

[0273] Step 3: Input the labeled medical samples into the student model to obtain the student prediction results for the labeled medical samples;

[0274] The student prediction results for labeled medical samples include a predicted segmentation probability for each voxel in the labeled medical sample. This segmentation probability indicates the probability that the current voxel belongs to the first category. Optionally, the first category is a certain medical organ or lesion, such as the heart region, lung region, etc.

[0275] Labeled medical samples are input into the student model, which then predicts the student's prediction results for the labeled medical samples.

[0276] Step 4: Input the unlabeled medical samples into the student model and the teacher model respectively, and obtain the student prediction results and the teacher prediction results of the unlabeled samples respectively;

[0277] Unlabeled medical samples are input into a student model, which then generates student predictions for the samples. These predictions are derived from the student model and are used to determine the category of pixels or voxels within the unlabeled medical samples.

[0278] Unlabeled medical samples are input into a teacher model, and teacher predictions for the unlabeled medical samples are obtained based on the teacher model. These teacher predictions are derived from the teacher model's predictions and are used to predict the category of pixels or voxels within the unlabeled medical samples.

[0279] Step 5: Obtain prototype information from unlabeled medical samples, generate prompts for the general segmentation model based on the prototype information, and generate calibrated teacher prediction results based on the prompts using the general segmentation model.

[0280] Because the teacher model has a very low accuracy in predicting unlabeled medical samples—that is, the teacher predictions for unlabeled medical samples are inaccurate—the specialist segmentation model's predictions become biased. Optionally, in this embodiment, the teacher predictions for unlabeled medical samples are calibrated using a generalist segmentation model, and the calibrated teacher predictions are used to provide an effective supervisory signal for the student model.

[0281] • Prototype information-guided prompt generation

[0282] Previous work intuitively used direct predictions from the teacher model to generate bounding boxes for specific objects as cues. However, under sparse supervision, the specialist segmentation model (in this example, the medical segmentation model) exhibits poor generalization ability and struggles to generate accurate predictions. Further research indicates that limited supervision often leads to conservative predictions, with training biased towards the most common or salient features in the data. Consequently, less common features and boundary details are often not adequately represented, such as... Figure 6 As shown, this can easily lead to undersegmentation. This situation poses a significant challenge to generating pseudo-labels using generalized segmentation models, because bounding box cues need to cover the entire object. Therefore, embodiments of this application design a cue generation mechanism based on prototype information. Prototype information represents the central tendency of the internal features of an object (the target segmentation object). Due to its average characteristics, it often overgeneralizes and lacks specificity in accurately describing the boundaries of complex objects. However, this imperfection can be utilized to generate bounding boxes that are more likely to cover the entire object. Furthermore, the prototype feature similarity map can also provide a basis for selecting additional positive and negative point cues.

[0283] In some embodiments, the generalist segmentation model possesses a general segmentation capability for segmenting the input image based on prompting information; that is, the generalist segmentation model can accurately segment the input image under the guidance of prompting information. Optionally, unlabeled samples are used as the images to be segmented, and prototype information of the images to be segmented is obtained. The prototype information is used to indicate the features of different segmentation regions in the images to be segmented. Prompting information for the generalist segmentation model is generated based on the prototype information, and the generalist segmentation model generates calibrated teacher prediction results based on the prompting information. The prompting information includes two-dimensional bounding box prompting information, positive point prompting information, and negative point prompting information.

[0284] In some embodiments, teacher prediction results for unlabeled medical samples are obtained, including prediction results of the category of all voxels in the image to be segmented; features of the target segmentation object are extracted from the sampling feature map corresponding to the image to be segmented, and the object prototype is obtained after processing the features of all voxels of the target segmentation object; features of non-target segmentation objects are extracted from the sampling feature map corresponding to the image to be segmented, and the background prototype is obtained after processing the features of all voxels of the non-target segmentation object.

[0285] For example, the upsampled feature map of the segmented image X before the penultimate convolutional layer in the teacher model is represented as: N c It's the number of channels. Utilizing data from F... t argmax pseudo-label Y t And the prediction probability map P t (Also known as teacher prediction results from unlabeled samples), the features of the object voxels can be cropped out, and the object prototype can be obtained by weighted averaging based on confidence level.

[0286]

[0287] Where, q obj F represents the object prototype. t(v) This represents the voxel characteristic corresponding to voxel value v. This indicates that when the voxel is v, the pseudo-label Y t Whether it belongs to an object, This represents the predicted probability that the teacher model predicts a voxel as belonging to an object when the voxel is v.

[0288] Similarly, the prototype q that provides the background information can be obtained through a confidence-weighted average. bg :

[0289]

[0290] Where, q bg F represents the background prototype. t(v)This represents the voxel characteristic corresponding to voxel value v. This indicates that when the voxel is v, the pseudo-label Y t Does it belong to the background? This represents the predicted probability that the teacher model predicts a voxel as belonging to the background when the voxel is v.

[0291] In some embodiments, the similarity between the features of each voxel in the image to be segmented and the object prototype is compared, and the similarity between the features of each voxel in the image to be segmented and the background prototype is compared to obtain an enhanced teacher prediction result. The enhanced teacher prediction result is a teacher prediction result generated based on prototype information guidance. A selection operation is performed on the enhanced teacher prediction result to obtain the prototype information label corresponding to each voxel in the image to be segmented. The prototype information label is the prediction label with the highest prediction probability in the prototype information-guided teacher prediction result.

[0292] For example, the object prototype q is successively... obj and background prototype q bg With the characteristics F of each voxel t By comparison, the enhanced teacher prediction result P was obtained. pro :

[0293]

[0294] Where sim(·,·) represents cosine similarity, and T represents the temperature parameter, empirically set to 0.05. Therefore, the prototype information label Y can be obtained. pro =argmax(P pro ).

[0295] In some embodiments, the prototype information label is decomposed into sub-labels from three perspectives. For each slice image in multiple slice images, the connected regions of each slice image are extracted based on the sub-labels, and two-dimensional bounding box prompt information is determined based on the connected regions.

[0296] In some embodiments, a prototype feature similarity map of the target segmented object is obtained based on the similarity between the sampled feature map corresponding to the image to be segmented and the object prototype. Positive and negative prompts can be obtained based on the prototype feature similarity map of the target segmented object.

[0297] For example, the prototype feature similarity map of the target segmentation object is represented as follows:

[0298]

[0299] in, sim(F) represents the prototype feature similarity map of the target segmentation object. t ,q obj F represents the characteristic of each voxel.t Cosine similarity between the object and its prototype.

[0300] For each slice image, utilize Identify the front with the highest similarity The dots may represent the center position of an object, serving as a point-of-sale notification message. h The point with the lowest similarity is selected, which may represent the background, and used as the negative point hint p. l Therefore, the input prompt (slice-level) token for the generalist segmentation model is: T prompt =Enc prompt (pb,ph,p l ). Among them, Enc prompt p represents the cue encoder in the generalist segmentation model. b p represents the 2D bounding box tooltip. h This indicates an on-time notification message, p l This indicates a negative notification message.

[0301] • Generate merged pseudo-tags (multi-view pseudo-tags)

[0302] In some embodiments, multiple slice images from three perspectives are used as images to be segmented. Based on prototype information, cue information is generated for each slice image. A general segmentation model is then used to predict pseudo-labels for each of the three perspectives based on the cue information. A fused pseudo-label is generated by fusing the pseudo-labels from the three perspectives based on a correction strategy, and a calibrated teacher prediction result is generated based on the fused pseudo-label.

[0303] Considering the challenges of generalist segmentation models in generating accurate pseudo-labels and their lack of 3D perception capabilities, this application's embodiments draw on the spirit of cross-labeling strategies to enhance pseudo-label generation based on generalist segmentation models. Specifically, this application's embodiments first obtain prompt information for each slice image based on prototype information, i.e., information from Y... pro Object-specific 2D bounding box hints, and information from S pro The exported positive and negative cue information sets span three orthogonal perspectives (coronal c, axial a, and sagittal s). These cue information sets are then used slice-by-slice to sparsely cue the frozen (pre-trained) generalist segmentation model, resulting in three 3D pseudo-labels for each scan. and For each voxel x∈X, the final fusion pseudo-label is... This can be expressed by the following formula:

[0304]

[0305] in, This indicates the fusion of pseudo-tags. This indicates a pseudo-label from a coronal perspective. This indicates a pseudo-label from an axial perspective. This represents a pseudo-label from a sagittal perspective.

[0306] like Figure 7 As shown, Figure 7 This is a schematic diagram of different labels in the pseudo-label correction process provided by an exemplary embodiment of this application. The grayscale area represents the actual left atrial region, and the blank area represents the left atrial region predicted by each label. The pseudo-labels from the three perspectives (coronal + axial + sagittal) are noisy and significantly different, but they overlap in some areas. Therefore, this embodiment of the application fuses the pseudo-labels from each perspective and obtains the fused pseudo-label based on the majority voting principle.

[0307] In this embodiment, it was found that the majority voting principle (i.e., selecting the most frequent label from the three perspectives) provides more informative labels than the strict consensus principle. Therefore, this embodiment uses the majority voting prototype (also known as the minority obeys the majority principle) as the statistical fusion function Fuse. It should be noted that, considering the inefficiency of the generalist segmentation model in 3D scanning, this embodiment strategically updates... Specifically, it is updated only once every 1000 iterations to balance its practicality and computational requirements.

[0308] Noise-tolerant collaborative learning

[0309] Due to the inherent limitations of the generalist segmentation model, the multi-view tag integration based on the majority rule in this application's embodiments still cannot avoid merging pseudo-tags. The labels contain a large number of errors, referred to as label noise (or noise labels). Label noise introduces bias into the training of medical segmentation models due to the memory effect. Therefore, to enable the specialist model to effectively absorb the knowledge from these noise labels, this application introduces two noise-resistant learning strategies to help the medical segmentation model robustly extract knowledge from these noise labels: (i) assigning the medical segmentation model during training the role of a third party to collaboratively process the fused pseudo-labels generated by the generalist segmentation model. (ii) Perform instant label noise self-identification and calibration; and (ii) reduce the inherent defects of overfitting to noise labels by constraining the logit value when calculating the supervised loss.

[0310] (i) Tag noise self-identification and calibration

[0311] In some embodiments, after generating fused pseudo-labels by fusing pseudo-labels from three perspectives, self-identification and calibration processing is performed on the fused pseudo-labels to generate calibrated teacher prediction results. The self-identification and calibration processing of the fused pseudo-labels is described below.

[0312] In some embodiments, based on the teacher prediction results of unlabeled medical samples, the predicted probability of each voxel in the image to be segmented under its respective category is obtained. Optionally, for each voxel, the predicted probability of each voxel under its respective category is averaged to obtain a probability threshold corresponding to the category, and the label type is determined based on the relationship between the predicted probability and the probability threshold. The label type of correctly labeled voxels in the unlabeled medical samples is determined as the category to which the fused pseudo-label belongs, and the label type of incorrectly labeled voxels in the unlabeled medical samples is calibrated to the category to which the latent label belongs. The calibrated teacher prediction results are generated based on the fused pseudo-label and the latent label.

[0313] In some embodiments, for each voxel, the label type of each voxel is determined based on the relationship between the predicted probability and a probability threshold. A threshold comparison operation (comparing the predicted probability to a probability threshold) filters out mislabeled voxels in the unlabeled medical samples, resulting in the number of voxels in the unlabeled medical samples whose latent label is category 2 and whose fused pseudo-label is category 1 (the number of mislabeled voxels). A confidence confusion matrix for the unlabeled medical samples is determined based on the number of mislabeled voxels, indicating the number of mislabeled voxels in the unlabeled medical samples. Optionally, the confidence confusion matrix is ​​calibrated to obtain a calibrated confidence confusion matrix, where the sum of its elements equals the total number of voxels in the unlabeled medical samples. The calibrated confidence confusion matrix is ​​then normalized to obtain a joint probability matrix, indicating the probability that a voxel in the unlabeled medical samples has a fused pseudo-label of category 1 and a latent label of category 2.

[0314] For example, significant errors may still exist in the fused pseudo-labels generated by the generalist segmentation model, requiring further fine-tuning. To achieve self-identification of label noise, this embodiment first assumes a classification noise process (CNP). Specifically, this embodiment assumes that each voxel x∈X has a true label y. * (Latent label), and each label in class j∈L may independently have a probability p = (y = i|y * =j) is incorrectly labeled as class i∈L. Then, the next step is to use a third party to estimate the joint probability p = (y = i, y * =j), to help calculate the number of voxels that may be mislabeled in the image X to be segmented. Based on the teacher model f t Weighted average prediction pt Due to its integrated features, this application embodiment uses f t As an immediate third party to estimate the required matrix.

[0315] Therefore, the embodiments of this application will use fused pseudo-labels obtained based on the generalist segmentation model. Consider it as observed noise label, and classify voxel x as... probability As its confidence level. Intuitively, if x's... satisfy Therefore, it is reasonable to suspect the potential true label y of x. * It is j, not i. The average confidence score of each image X is used as the threshold for each class, denoted as... This threshold design has been shown to be more robust to class imbalance and overconfident predictions. Assuming there are L semantic classes, these thresholds can be used to construct a confidence confusion matrix. It is an L×L matrix, obtained by counting the voxels in image X that satisfy each class threshold:

[0316]

[0317] in, The confidence confusion matrix for unlabeled medical samples indicates the number of mislabeled voxels in the unlabeled medical samples.

[0318] Because the threshold operation excludes a large number of voxels, the embodiments of this application require... Perform calibration, and then adjust the results. Normalization is performed to estimate the L×L joint probability matrix.

[0319]

[0320] in, This is the joint probability matrix, which indicates the probability that a voxel in an unlabeled medical sample has a fused pseudo-label of the first category and a latent label of the second category.

[0321] In some embodiments, voxels with suspicious labels (candidate voxels) are randomly selected from the joint probability matrix to obtain a label error indication map. A suspicious label refers to a candidate voxel whose fused pseudo-label is of category one and whose potential label is of category two. Voxels with suspicious labels are voxels in unlabeled medical samples with a confidence level below a confidence threshold (minimum confidence threshold). The label error indication map is used to indicate voxels in unlabeled medical samples that may be mislabeled.

[0322] For example, based on the concept of random walk, embodiments of this application may select... The voxels with the lowest confidence are designated as suspicious labels, where |X| is the number of voxels in the image X to be segmented.

[0323] Therefore, embodiments of this application obtain a voxel-level label error indication map E∈{0,1} H×W×D Where "1" represents This was presumed to be incorrect. Typically, at position (i,j,k), the fused pseudo-labels obtained based on the generalist segmentation model are processed as follows: Perform calibration:

[0324]

[0325] in, This indicates the calibrated fusion pseudo-label. Indicates the fusion of pseudo-tags, y * This indicates a potential label. Optionally, at position (i,j,k), if the voxel value of the label error indicator map is 1, it indicates that the fusion pseudo-label of that voxel is incorrectly marked, and the fusion pseudo-label of that voxel is calibrated as a potential label. If the voxel value of the label error indicator map is other values, it indicates that the fusion pseudo-label of that voxel is correctly marked.

[0326] For the binary classification task that this application focuses on, the calibration process can be easily implemented in the following ways:

[0327] (ii) Bounded noise tolerance learning

[0328] Even with limited expert supervision, completely identifying label noise remains challenging. Therefore, the generated label error indication map E still cannot perfectly identify all errors in the labels. For example, the cross-entropy (CE) loss used for supervision in this embodiment is not robust to label noise because it is unbounded. When there is a significant difference between the observed noisy labels and the model predictions (a huge difference between calibrated teacher predictions and student predictions for unlabeled medical samples), the CE loss may tend to infinity, leading to severe overfitting to the noisy labels. Therefore, robustness is enhanced by imposing boundary constraints on the logit vector (the unnormalized predicted value vector output by the student model). Specifically, consider the CE loss for a voxel, where z∈Z is the model logit vector for that voxel, and y is its class label. The CE loss for each voxel can be expressed as:

[0329]

[0330] Among them, Z max and Z min These are the maximum and minimum values ​​in the logit vector at that voxel, respectively. Intuitively, if Z... max -Z min If there is an upper bound, the CE loss cannot reach infinity, thereby mitigating overfitting to difficult-to-learn or mislabeled voxels. In this embodiment, the logit vector is constrained according to its norm to ensure that the logit remains bounded.

[0331]

[0332] Where ||·||² represents the norm, and τ is the upper bound of the norm (τ is empirically set to 1). Besides boundary constraints, norm-based constraints preserve the direction and relative differences between the elements of the logit vector. It is important to note that this boundary strategy is compatible with different loss functions. Here, this embodiment continues to use the CE loss to maintain consistency with the typical supervised loss for labeled voxels. The loss used for robust learning is defined as:

[0333]

[0334] Step 6: Based on the student prediction results and real labels of labeled medical samples, determine the supervised loss of labeled medical samples; based on the student prediction results of unlabeled medical samples, the teacher prediction results of unlabeled medical samples, and the calibrated teacher prediction results, determine the unsupervised loss of labeled medical samples.

[0335] • Regarding supervised losses

[0336] Since the ground truth labels only contain the segmentation probabilities of individual voxels on the key slice images from the three viewpoints, while the student predictions for labeled medical samples include the segmentation probabilities predicted for all voxels in the labeled medical samples, when calculating the supervised loss, it is not necessary to consider only the cross-entropy loss of individual voxels on the key slice images from the three viewpoints, instead of considering all voxels.

[0337] That is, for each voxel on the key slice images from three perspectives, the cross-entropy loss between the segmentation probability of each voxel in the real label and the segmentation probability in the student prediction results is calculated, which serves as the supervised loss for labeled medical samples.

[0338] • Regarding unsupervised losses

[0339] Unsupervised loss includes consistency loss and labeling loss.

[0340] In some embodiments, the consistency loss of unlabeled medical samples is calculated based on student predictions and teacher predictions for unlabeled medical samples.

[0341] In some embodiments, the labeling loss of unlabeled medical samples is calculated based on student predictions and calibrated teacher predictions.

[0342] In some embodiments, after calculating the consistency loss and labeling loss of unlabeled medical samples respectively, the consistency loss and labeling loss are weighted and summed, and the unsupervised loss of unlabeled samples is calculated based on the weighted sum of the consistency loss and labeling loss.

[0343] Optionally, based on the current and maximum number of training iterations, a first weight corresponding to the consistency loss and a second weight corresponding to the label loss are determined. A first product of the consistency loss and the first weight, and a second product of the label loss and the second weight are calculated; the sum of the first and second products is determined as the unsupervised loss for the unlabeled medical samples.

[0344] In some embodiments, label loss can provide effective early supervision, and the training process relies more heavily on label loss. However, in subsequent training, over-reliance on these noisy labels (calibrated teacher predictions) may mislead the student model due to memory effects. Therefore, embodiments of this application aim to gradually reduce the weight of noise loss supervision as training progresses. To this end, embodiments of this application introduce a time-dependent tradeoff weight λ. g To adjust

[0345]

[0346] Among them, iter and iter max Let represent the current iteration number and the maximum iteration number, respectively. Therefore, the total loss can be written as:

[0347]

[0348] Where L represents the total loss, Indicates a supervisory loss, L ssl λ represents the consistency loss. ssl This represents the first weight corresponding to the consistency loss. λ represents the label loss. g This represents the second weight corresponding to the label loss. This indicates unsupervised loss.

[0349] Step 7: Update the network parameters of the student model in the i-th round of training based on supervised loss and unsupervised loss;

[0350] The training process typically involves multiple rounds of training. With the training objective of reducing both supervised and unsupervised loss, the network parameters of the student model are updated in the i-th training round, where i is an integer and its initial value is either 0 or 1.

[0351] Step 8: Update the network parameters of the teacher model based on the network parameters of the student model before the i-th round of training and the network parameters of the i-th round of training.

[0352] Optionally, the teacher-student model is based on a standard SSL design with consistency regularization. Specifically, the network parameters of the trainable student model... Optimization is performed using the standard backpropagation algorithm, while the network parameters of the teacher model are... The student model is updated using the exponential moving average (EMA) over past and current iterations.

[0353] Let θ represent the weights corresponding to the student model. Let these be the weights of the teacher model. At the t-th iteration, The update formula is Where α is the EMA coefficient, which is set to 0.99 based on experience.

[0354] Step 9: Based on the second segmentation results of the unlabeled medical samples in the i-th round of training, select medical samples to be labeled from the unlabeled medical samples;

[0355] To better utilize the precious annotation budget, this embodiment employs an active sample selection strategy during training. In each i-th training round, each unlabeled medical sample is scored by the student model, and then the unlabeled medical sample with the highest active score is selected for cross-labeling to increase the set of labeled medical samples (also known as the cross-label set). This process is repeated until the annotation budget is exhausted.

[0356] Optionally, based on the student prediction results of unlabeled medical samples in the i-th round of training, candidate labeled samples that meet the labeling conditions are selected from the unlabeled medical samples; the candidate labeled samples are filtered based on the intensity features of the histogram, and the samples to be labeled are determined from the candidate labeled samples. The real labels for the samples to be labeled are obtained, resulting in the labeled medical samples actively selected and generated in the i-th round of training, and the labeled medical samples are added to the set of labeled medical samples used in the (i+1)-th round of training. The process of labeling the samples to be labeled as labeled medical samples can be completed manually by personnel with professional skills.

[0357] (1) Sampling based on the High Entropy Ratio (HER)

[0358] Active sample selection strategies include at least one of the following three sampling strategies.

[0359] Least Confidence (LC) sampling: Calculate the average of the predicted confidence (probability) and select the K1 samples with the lowest scores;

[0360] High Entropy (HE) Sampling: Select the Top-K2 samples with the largest average predicted entropy;

[0361] High Entropy Ratio (HER) sampling: Calculate the proportion of predictions exceeding a threshold relative to the total, and select the top-K3 samples with the highest HER values. For example, K3 is set to 2.

[0362] For HER, Let denoted as the predicted probability of the student model at voxel v, and its normalized entropy ne v Calculated using the following formula:

[0363]

[0364] Here, c∈C represents the true label. High entropy indicates, to some extent, higher uncertainty. In HER sampling, the ne of unlabeled medical samples is calculated. v The number of voxels >β (β is a threshold determined by the Gaussian ramp function from 0.25 to 0.75), then calculate ne v The ratio of the number of voxels >β to the total number of voxels yields the active score of the unlabeled medical sample. Compared to HE, HER more clearly reflects the distribution of high uncertainty in unlabeled medical samples.

[0365] (2) Histogram-based diversity selection

[0366] In this embodiment, to ensure diversity among these highly uncertain samples, a histogram-based intensity feature is calculated for each retained image (candidate annotation sample), and the sample to be annotated is determined from the candidate annotation samples based on the histogram intensity feature. This ensures that the finally selected sample to be annotated can capture various intensity distributions, thereby ensuring that the selected sample to be annotated contains both uncertainty and diversity.

[0367] In some embodiments, candidate labeled samples that meet the labeling criteria are selected from unlabeled medical samples. Based on the intensity features of the histogram, the candidate labeled samples are further screened to determine the samples to be labeled, and the samples to be labeled are labeled with real labels. Optionally, the histogram features of the candidate labeled samples are clustered to divide the candidate labeled samples into a preset number of clusters, and the most representative candidate labeled samples are selected from each cluster as the samples to be labeled.

[0368] For example, histogram features are a way to describe the distribution of voxel intensity in an image. Specifically, the voxel intensity values ​​of the image are divided into multiple intensity bins, each bin representing an intensity range. Specifically, for each intensity bin i, the histogram feature H(i) is expressed as:

[0369]

[0370] Where, n i Let S represent the number of voxels in bin i, and B be the total number of bins (here, B = 20). Then, K-means clustering is performed on these histogram features, dividing the images corresponding to the candidate labeled samples into K clusters, each cluster representing a unique intensity feature. Therefore, each cluster can be represented by a sample S. k This indicates that the sample is the closest to the cluster center.

[0371]

[0372] Among them, C k It is the sample set in cluster k, u k It is the center of cluster k. This ensures that the final selected samples can capture various intensity distributions, thus ensuring that the selected samples to be labeled have both uncertainty and diversity.

[0373] Step 10: Obtain the real labels for the medical samples to be labeled, obtain the labeled medical samples actively selected and generated in the i-th round of training, and add the labeled medical samples to the set of labeled medical samples used in the (i+1)-th round of training.

[0374] The beneficial effects of this plan

[0375] This application's embodiments underwent extensive evaluation on two commonly used SSL benchmark datasets: the Left Atrium (LA) dataset, containing 100 3D gadolinium-enhanced magnetic resonance imaging (GE-MRI) images; and the Brain Tumor (BT) MRI dataset, containing 335 3DT2-FLAIR preoperative magnetic resonance imaging (MRI) images from glioma patients. After preprocessing, the Left Atrium (LA) images had an isotropic resolution of 0.625x0.625x0.625 mm^3, and the Brain Tumor (BT) images had a resolution of 1x1x1 mm^3. For the LA dataset, 80 samples were used for training and 20 for testing. For the BT dataset, 250 samples were used for training, 25 for validation, and 60 for testing.

[0376] Implementation details and evaluation indicators

[0377] The framework utilizes an NVIDIA GeForce RTX 3090 GPU with 24GB of memory and is implemented using Python and PyTorch. In all experiments, the embodiments of this application use the same 3D V-Net as the backbone for fair comparison. For the general segmentation model, the embodiments of this application employ the Mobile SegmentAnything Model (Mobile SAM), a more efficient pre-trained base model distilled from Meta's Segment Anything Model (SAM), employing a TinyVision Transformer (ViT-Tiny) architecture. The embodiments of this application train the network using a Stochastic Gradient Descent Optimizer (SGD) optimizer (weight decay = 0.0001, momentum = 0.9). The batch size is set to 4, including 2 cross-labeled images and 2 unlabeled images. The maximum training steps are set to 20000. The learning rate is initialized to 0.01 and decays by a power of 0.9 after each step. In this embodiment, randomly cropped blocks of 112×112×80 voxels (left ventricle segmentation dataset) or 96×96×96 voxels (brain tumor BT dataset) are used as network input. Standard data augmentation, including random cropping, flipping, and rotation, is also applied. This embodiment uses a sliding window strategy with a voxel step size of 18×18×4 (left ventricle LA) or 64×64×64 (brain tumor BT) for the testing phase. Then, this embodiment uses four metrics for comprehensive evaluation: Dice Coefficient (DICE), Jaccard Index (JI), Average Surface Distance (ASD), and 95% Hausdorff Distance (95HD).

[0378] • Baseline method

[0379] To ensure fairness, the SAM-based interactive method also employs a majority-voting multi-view ensemble strategy, denoted as "3V". Hints (such as 2D bounding box voxels and positive / negative point voxels, if applicable) are generated from the ground truth labels. Since SAM-Med3D does not support bounding boxes, this embodiment instead randomly samples 10 points from the ground truth labels. In addition to the SAM-based method, this embodiment also compares it with supervised baseline (Sup) and some recently high-performing SSL methods, including: Multi-Task Learning (MT), Uncertainty-Aware Multi-Task Learning (UA-MT), Interpolation Consistency Training (ICT), Cyclic Prototype Consistency Learning (CPCL), Cross Pseudo-Supervision (CPS), Correlation-Aware Mutual Learning (CAML), Adaptive Cascade Multi-Task Learning (ACMT), DeSCO, and SemiSAM.

[0380] The aforementioned DeSCO reference is: "Orthogonal annotation benefits barely-supervised medical image segmentation," by Cai Heng et al., from the proceedings of the 2023 IEEE / CVF conference on Computer Vision and Pattern Recognition.

[0381] The above SemiSAM can be found in the reference: Exploring SAM for enhancing semi-supervised medical image segmentation with extremely limited annotations (Y. Zhang, Y. Cheng, and Y. Qi, “SemiSAM:exploring SAM for enhancing semi-supervised medical image segmentation with extremely limited annotations,” arXiv preprint arXiv:2312.06316,2023.).

[0382] Experimental Results

[0383] Table 1 shows the performance of the method in this application and other state-of-the-art semi-supervised methods when only 60 left ventricular slices are labeled (20 samples, one slice labeled in 3 directions for each sample).

[0384] Table 2 shows the performance of the method of this application embodiment and other state-of-the-art semi-supervised methods on 150 brain tumor slices (50 samples). Under the same partitioning protocol, the method of this application embodiment can achieve a significant improvement compared to the methods of related technologies, demonstrating the superiority and robustness of the method proposed in this application embodiment.

[0385] Under the standard random sampling protocol, GM-ABS consistently outperforms supervised baselines and recent methods, demonstrating the effectiveness of specialist-generalist collaboration in sparsely supervised scenarios. Furthermore, by employing an active learning strategy, the performance of GM-ABS is further enhanced, highlighting the benefits of specialist-generalist model collaboration.

[0386] Table 1

[0387]

[0388] Table 1. Quantitative Comparison. * indicates p ≤ 0.05 (the preferred embodiment of this application, GM-ABS(+AL), is compared with other methods using the Wilcoxon signed-rank test). The Wilcoxon signed-rank test is a nonparametric statistical test used to examine whether there is a significant difference between the medians of two related samples. p is the statistical indicator in the Wilcoxon signed-rank test. Standard deviations are shown in parentheses.

[0389] Table 2

[0390]

[0391] Table 2. Quantitative Comparison. * indicates p ≤ 0.05 (the preferred version of this application, GM-ABS(+AL), is compared with other methods using the Wilcoxon signed-rank test). The Wilcoxon signed-rank test is a nonparametric statistical test used to examine whether there is a significant difference between the medians of two related samples. p is the statistical indicator in the Wilcoxon signed-rank test. Standard deviations are shown in parentheses.

[0392] Under standard random sampling protocols, the methods of this application (GM-ABS, GM-ABS(+AL)) consistently outperform other methods, demonstrating the effectiveness of the expert-generalist collaboration in sparse supervision. Generally, using majority rule (MV) to calibrate pseudo-labels based on multi-view generalists yields excellent results. Furthermore, by employing an active learning strategy, the performance of the embodiments of this application is further enhanced, highlighting the benefits of expert-model collaboration.

[0393] Application Scenario Description

[0394] The training method for the specialist segmentation model can be applied to the field of medical image analysis. A medical segmentation model (a specialist segmentation model) is trained based on a medical image sample set.

[0395] Application Scenario 1

[0396] Taking the medical image sample set as the MRI image sample set, and the medical segmentation model used to identify the left atrium in the MRI image sample as an example, this will be explained.

[0397] Step 1: Obtain labeled and unlabeled MRI image samples;

[0398] Among them, labeled MRI image samples are MRI image samples with real labels, while unlabeled MRI image samples are MRI image samples without real labels.

[0399] In some embodiments, the MRI image samples are three-dimensional medical image samples. For three-dimensional medical images, the true label can be identified using a mask of voxels in the three-dimensional medical image. For example, when the mask value of a voxel in a three-dimensional medical image is 1, it represents belonging to the first category, such as the voxel belonging to the left atrium category, that is, the voxel belongs to the foreground category; when the mask value of a voxel in a three-dimensional medical image is 0, it represents belonging to the second category, such as the voxel belonging to the other tissue category, that is, the voxel belongs to the background category.

[0400] Step 2: Input the labeled MRI image samples into the student model, and obtain the student prediction results for the labeled MRI image samples based on the student model;

[0401] The student prediction result for the labeled MRI image samples is the result of the student model's prediction, used to predict the category of voxels in the labeled MRI image samples. In this embodiment, the student prediction result for the labeled MRI image samples can also be referred to as the student segmentation result for the labeled MRI image samples.

[0402] In some embodiments, the medical segmentation model can be implemented using any neural network architecture with image segmentation capabilities. For example, the medical segmentation model can be any one of convolutional neural networks, V-Net, FCN, U-net, deep convolutional networks, DeepLab, DenseNet, and ResNet. This embodiment uses V-Net as an example for illustration.

[0403] Step 3: Input the unlabeled MRI image samples into the student model and the teacher model to obtain the student prediction results and the teacher prediction results for the unlabeled MRI image samples, respectively.

[0404] In some embodiments, unlabeled MRI image samples are input into a student model, and student prediction results for the unlabeled MRI image samples are obtained based on the student model. The student prediction results for the unlabeled MRI image samples are the results predicted by the student model and are used to predict the category of voxels in the unlabeled MRI image samples.

[0405] In some embodiments, unlabeled MRI image samples are input into a teacher model, and teacher prediction results for the unlabeled MRI image samples are obtained based on the teacher model. The teacher prediction results for the unlabeled MRI image samples are the results predicted by the teacher model and are used to predict the category of voxels in the unlabeled MRI image samples.

[0406] Step 4: Obtain prototype information from unlabeled MRI image samples, generate prompts for the general segmentation model based on the prototype information, and generate calibrated teacher prediction results based on the prompts using the general segmentation model.

[0407] The prototype information is derived from teacher predictions based on unlabeled MRI image samples. The general segmentation model possesses the ability to segment input slice images based on prompts; that is, the general segmentation model can accurately segment input slice images under the guidance of prompts.

[0408] In some embodiments, unlabeled MRI image samples are used as images to be segmented, prototype information of the MRI images to be segmented is obtained, prompt information of the general segmentation model is generated based on the prototype information, and the general segmentation model generates calibrated teacher prediction results based on the prompt information.

[0409] In some embodiments, the use of a medical segmentation model for segmenting the left atrium is illustrated as an example. Prototype information includes a left atrial prototype and a background prototype. Optionally, teacher predictions for unlabeled MRI image samples are obtained, including predictions for the category of each voxel in the image to be segmented.

[0410] Features of all voxels of the left atrium are extracted from the sampling feature map corresponding to the MRI image to be segmented. After processing the features of all voxels of the left atrium, the prototype of the left atrium is obtained. The prototype of the left atrium is used to indicate the average features (or central features, representative features, typical features) of the left atrium.

[0411] The features of all voxels of the background are extracted from the sampling feature map corresponding to the MRI image to be segmented. After processing the features of all voxels of the background, the background prototype is obtained. The background prototype is used to indicate the average features of the background. The background is the image other than the left atrium in the MRI image to be segmented. For example, the background includes bone tissue, blood vessel tissue, liver tissue, etc.

[0412] In this model, the MRI image to be segmented is a three-dimensional medical image, while the input image of the general segmentation model is a two-dimensional slice image. The MRI image to be segmented needs to be sliced ​​according to three perspectives, resulting in multiple slice images corresponding to each perspective. These multiple slice images are two-dimensional slice images. For example, the three perspectives are the coronal perspective, the axial perspective, and the sagittal perspective.

[0413] Optionally, the generalized segmentation model includes an image encoder, a cue encoder, and a mask decoder. The image encoder performs feature extraction on the input two-dimensional sliced ​​image to obtain an image feature representation; the cue encoder performs feature extraction on the input cue information (cue information generated based on prototype information) to obtain a cue feature representation; and the mask decoder performs image segmentation on the image feature representation based on the cue guidance provided by the cue feature representation, resulting in a more accurate segmentation result.

[0414] In some embodiments, multiple two-dimensional slice images from three perspectives are used as slice images to be segmented. Based on prototype information (left atrial prototype and background prototype), cue information is generated for each of the multiple two-dimensional slice images. A general segmentation model is then used to predict pseudo-labels for each of the three perspectives based on the cue information. A fused pseudo-label is generated by fusing the pseudo-labels from the three perspectives based on a correction strategy, and a calibrated teacher prediction result is generated based on the fused pseudo-label.

[0415] In some embodiments, the prompt message includes at least one of the following:

[0416] • Two-dimensional bounding box hints: These hints are used to mark the extent of the left atrium.

[0417] • Punctual reminder information, used to indicate the location of key points in the left atrium;

[0418] • Negative dot hints: Negative dot hints are used to indicate the position of the background.

[0419] In some embodiments, the prompt information includes two-dimensional bounding box prompt information, which is obtained based on the enhanced teacher prediction results, which are teacher prediction results generated based on prototype information.

[0420] The prototype information includes a left atrial prototype and a background prototype. Optionally, the similarity between the features of each voxel in the MRI image to be segmented and the left atrial prototype is compared, and the similarity between the features of each voxel in the MRI image to be segmented and the background prototype is also compared, to obtain the enhanced teacher prediction result. The enhanced teacher prediction result includes the predicted probability that each voxel in the MRI image to be segmented belongs to either the left atrium or the background.

[0421] After obtaining the enhanced teacher prediction results, a selection operation is performed on the enhanced teacher prediction results to obtain the prototype information label corresponding to each voxel in the MRI image to be segmented. The prototype information label is then decomposed into sub-labels from three perspectives. Among them, the prototype information label is the prediction label with the highest prediction probability in the teacher prediction results guided by the prototype information. Optionally, each of the three perspectives corresponds to multiple slice images.

[0422] In some embodiments, for each of a plurality of two-dimensional slice images, a connected region of each two-dimensional slice image is extracted based on a sub-label, and two-dimensional bounding box cue information is determined based on the connected region. Optionally, for each two-dimensional slice image, the two-dimensional bounding box coordinates of each two-dimensional slice image are extracted based on a sub-label, and a connected region of each two-dimensional slice image is determined based on the two-dimensional bounding box coordinates. The connected region is used to indicate the set of elements in the two-dimensional slice image predicted to belong to the left atrium (foreground category).

[0423] In some embodiments, the prompt information includes positive prompt information and negative prompt information, which are obtained based on the prototype feature similarity map of the left atrium.

[0424] Optionally, a prototype feature similarity map of the left atrium is obtained based on the similarity between the sampled feature map corresponding to the MRI image to be segmented and the prototype of the left atrium. The points in the prototype feature similarity map are sorted from high to low according to the similarity between the sampled feature map (features in the sampled feature map) and the prototype of the left atrium. Optionally, the prototype feature similarity map reflects the degree of similarity between each point in the image and the prototype of the left atrium. Regions with high similarity are more likely to belong to the left atrium, and regions with low similarity are more likely to belong to the background.

[0425] By analyzing the prototype feature similarity map of the left atrium, the first h points of the prototype feature similarity map are designated as positive cue information, where h is a positive integer greater than or equal to 1. The last g points of the prototype feature similarity map are designated as negative cue information, where g is a positive integer greater than or equal to 1. The positive cue information indicates the location of key points in the left atrium, representing the center of the left atrium. The negative cue information indicates the location of the background.

[0426] In some embodiments, for any voxel in an unlabeled MRI image sample, if the pseudo-labels in at least two of the three views belong to the left atrium category (foreground category), the fused pseudo-label of the voxel is determined to belong to the left atrium category (foreground category); if the pseudo-labels in at least two of the three views belong to the background category, the fused pseudo-label of the voxel is determined to belong to the background category.

[0427] In some embodiments, based on teacher predictions of unlabeled MRI image samples, the predicted probability of each voxel in the MRI image to be segmented under its class is obtained. For each voxel, the predicted probability of each voxel under its class is averaged to obtain a probability threshold corresponding to the class. The label type is determined based on the relationship between the predicted probability and the probability threshold. The label type of correctly labeled voxels in the unlabeled MRI image samples is determined as the class to which the fused pseudo-label belongs. The label type of incorrectly labeled voxels in the unlabeled MRI image samples is calibrated to the class to which the latent label belongs. The calibrated teacher prediction results are generated based on the fused pseudo-label and latent label.

[0428] Step 5: Train the medical segmentation model based on the student prediction results of labeled MRI image samples, the student prediction results of unlabeled MRI image samples, the teacher prediction results of unlabeled MRI image samples, and the calibrated teacher prediction results.

[0429] Optionally, a supervised loss for labeled MRI image samples is calculated based on student predictions and the true labels.

[0430] Optionally, based on the student prediction results and teacher prediction results of unlabeled MRI image samples, a consistency loss for the unlabeled MRI image samples is calculated; and based on the student prediction results and calibrated teacher prediction results of the unlabeled MRI image samples, a label loss for the unlabeled MRI image samples is calculated. The unsupervised loss for the unlabeled MRI image samples is then calculated based on the weighted sum of the consistency loss and the label loss.

[0431] Based on the aforementioned supervised loss and unsupervised loss, a medical segmentation model is trained.

[0432] In some embodiments, when calculating the unsupervised loss, a first weight corresponding to the consistency loss and a second weight corresponding to the label loss are determined based on the current number of training iterations and the maximum number of training iterations; a first product of the consistency loss and the first weight is calculated, and a second product of the label loss and the second weight is calculated; the sum of the first product and the second product is determined as the unsupervised loss of the unlabeled MRI image sample.

[0433] In some embodiments, when calculating the label loss, boundary constraints are applied to the predicted value vector generated by the student model, which is the unnormalized predicted value output by the student model.

[0434] In some embodiments, during the training of the medical segmentation model, candidate labeled MRI images that meet the labeling criteria are selected from unlabeled MRI image samples. These candidate labeled MRI images are MRI images with potential ground truth labels. Based on the intensity features of the histogram, the candidate labeled MRI images are screened to determine the MRI images to be labeled, and ground truth labels are then assigned to these images.

[0435] In some embodiments, the labeled sample is a three-dimensional medical image, and the true label of the labeled MRI image sample is annotated using a cross-labeling strategy. The cross-labeling strategy is an annotation strategy that labels two-dimensional slice images from three different perspectives in a three-dimensional medical image.

[0436] In summary, this application utilizes a collaborative approach between a medical segmentation model and a generalist segmentation model. The medical segmentation model generates prompts for the generalist segmentation model, and the generalist model, through its response to these prompts, provides auxiliary supervision for the training of the specialist segmentation model. Specifically, by acquiring prototype information from unlabeled MRI image samples, prompts for the generalist segmentation model are generated based on this prototype information. The generalist segmentation model then generates calibrated teacher predictions based on these prompts. These calibrated teacher predictions provide additional supervisory signals to the medical segmentation model during training, improving the utilization of information from unlabeled MRI image samples and effectively reducing the number of labeled MRI image samples required. This application employs the general image segmentation capabilities of the generalist segmentation model to assist in training a high-quality medical segmentation model.

[0437] Application Scenario 2

[0438] The following example illustrates how a medical image sample set is used to identify brain tumors in CT image samples, with the medical segmentation model serving as the sample set.

[0439] Step 1: Obtain labeled CT samples and unlabeled CT samples;

[0440] Among them, labeled CT samples are CT samples with real labels, while unlabeled CT samples are CT samples without real labels.

[0441] In some embodiments, CT samples are three-dimensional medical image samples. For three-dimensional medical images, the true label can be identified using a mask of voxels in the three-dimensional medical image. For example, when the mask value of a voxel in a three-dimensional medical image is 1, it represents belonging to the first category, such as a voxel belonging to the brain tumor category, that is, the voxel belongs to the foreground category; when the mask value of a voxel in a three-dimensional medical image is 0, it represents belonging to the second category, such as a voxel belonging to the other tissue category, that is, the voxel belongs to the background category.

[0442] Step 2: Input the labeled CT samples into the student model, and obtain the student prediction results based on the labeled CT samples;

[0443] The student prediction results for labeled CT samples are predicted by the student model and are used to predict the category of voxels in the labeled CT samples. In this embodiment, the student prediction results for labeled CT samples can also be referred to as the student segmentation results for labeled CT samples.

[0444] In some embodiments, the medical segmentation model can be implemented using any neural network architecture with image segmentation capabilities. For example, the medical segmentation model can be any one of convolutional neural networks, V-Net, FCN, U-net, deep convolutional networks, DeepLab, DenseNet, and ResNet. This embodiment uses V-Net as an example for illustration.

[0445] Step 3: Input the unlabeled CT samples into the student model and the teacher model to obtain the student prediction results and the teacher prediction results for the unlabeled CT samples, respectively.

[0446] In some embodiments, unlabeled CT samples are input into a student model, and student prediction results for the unlabeled CT samples are obtained based on the student model. These student prediction results for the unlabeled CT samples are the results predicted by the student model and are used to predict the category of voxels in the unlabeled CT samples.

[0447] In some embodiments, unlabeled CT samples are input into a teacher model, and teacher prediction results for the unlabeled CT samples are obtained based on the teacher model. These teacher prediction results for the unlabeled CT samples are the results predicted by the teacher model and are used to predict the category of voxels in the unlabeled CT samples.

[0448] Step 4: Obtain prototype information from unlabeled CT samples, generate prompts for the general segmentation model based on the prototype information, and generate calibrated teacher prediction results based on the prompts using the general segmentation model.

[0449] The prototype information is derived from teacher predictions based on unlabeled CT samples. The general segmentation model possesses the ability to segment input slice images based on prompts; that is, the general segmentation model can accurately segment input slice images under the guidance of prompts.

[0450] In some embodiments, unlabeled CT samples are used as images to be segmented, prototype information of the CT to be segmented is obtained, prompts for a general segmentation model are generated based on the prototype information, and the general segmentation model generates calibrated teacher prediction results based on the prompts.

[0451] In some embodiments, a medical segmentation model is used to segment a brain tumor as an example. Prototype information includes a brain tumor prototype and a background prototype. Optionally, teacher predictions for unlabeled CT samples are obtained, including predictions for the category of each voxel in the image to be segmented.

[0452] Features of all voxels of the brain tumor are extracted from the sampling feature map corresponding to the CT scan to be segmented. After processing the features of all voxels of the brain tumor, the brain tumor prototype is obtained. The brain tumor prototype is used to indicate the average features (or central features, representative features, typical features) of the brain tumor.

[0453] The features of all voxels of the background are extracted from the sampled feature map corresponding to the CT to be segmented. After processing the features of all voxels of the background, the background prototype is obtained. The background prototype is used to indicate the average features of the background. The background is other images in the CT to be segmented except for brain tumors. For example, the background includes bone tissue, blood vessel tissue, liver tissue, etc.

[0454] In this model, the CT scan to be segmented is a three-dimensional medical image, while the input image of the general segmentation model is a two-dimensional slice image. The CT scan needs to be sliced ​​according to three perspectives to obtain multiple slice images corresponding to each perspective. These multiple slice images are two-dimensional slice images. For example, the three perspectives are the coronal perspective, the axial perspective, and the sagittal perspective.

[0455] Optionally, the generalized segmentation model includes an image encoder, a cue encoder, and a mask decoder. The image encoder performs feature extraction on the input two-dimensional sliced ​​image to obtain an image feature representation; the cue encoder performs feature extraction on the input cue information (cue information generated based on prototype information) to obtain a cue feature representation; and the mask decoder performs image segmentation on the image feature representation based on the cue guidance provided by the cue feature representation, resulting in a more accurate segmentation result.

[0456] In some embodiments, multiple two-dimensional slice images from three perspectives are used as slice images to be segmented. Based on prototype information (brain tumor prototype and background prototype), cue information is generated for each of the multiple two-dimensional slice images. A general segmentation model is then used to predict pseudo-labels for each of the three perspectives based on the cue information. A fused pseudo-label is generated by fusing the pseudo-labels from the three perspectives based on a correction strategy, and a calibrated teacher prediction result is generated based on the fused pseudo-label.

[0457] In some embodiments, the prompt message includes at least one of the following:

[0458] • Two-dimensional bounding box hints are used to mark the extent of brain tumors;

[0459] • Punctual alert messages are used to indicate the key locations of brain tumors;

[0460] • Negative dot hints: Negative dot hints are used to indicate the position of the background.

[0461] In some embodiments, the prompt information includes two-dimensional bounding box prompt information, which is obtained based on the enhanced teacher prediction results, which are teacher prediction results generated based on prototype information.

[0462] The prototype information includes a brain tumor prototype and a background prototype. Optionally, the similarity between the features of each voxel in the CT scan to be segmented and the brain tumor prototype, and the similarity between the features of each voxel in the CT scan to be segmented and the background prototype are compared to obtain the enhanced teacher prediction result. The enhanced teacher prediction result includes the predicted probability that each voxel in the CT scan to be segmented belongs to either a brain tumor or the background.

[0463] After obtaining the enhanced teacher prediction results, a selection operation is performed on the enhanced teacher prediction results to obtain the prototype information label corresponding to each voxel in the CT to be segmented. The prototype information label is then decomposed into sub-labels from three perspectives. Among them, the prototype information label is the prediction label with the highest prediction probability in the teacher prediction results guided by the prototype information. Optionally, each of the three perspectives corresponds to multiple slice images.

[0464] In some embodiments, for each of a plurality of two-dimensional slice images, a connected component of each two-dimensional slice image is extracted based on a sub-label, and two-dimensional bounding box cue information is determined based on the connected component. Optionally, for each two-dimensional slice image, the coordinates of the two-dimensional bounding box of each two-dimensional slice image are extracted based on a sub-label, and a connected component of each two-dimensional slice image is determined based on the coordinates of the two-dimensional bounding box. The connected component is used to indicate the set of elements in the two-dimensional slice image predicted to belong to a brain tumor (foreground category).

[0465] In some embodiments, the prompt information includes positive and negative prompt information, which are obtained based on a similarity map of the prototype features of brain tumors.

[0466] Optionally, a prototype feature similarity map of the brain tumor is obtained based on the similarity between the sampled feature map corresponding to the CT scan to be segmented and the prototype brain tumor. The points in the prototype feature similarity map are sorted from high to low according to the similarity between the sampled feature map (features in the sampled feature map) and the prototype brain tumor. Optionally, the prototype feature similarity map reflects the degree of similarity between each point in the image and the prototype brain tumor. Regions with high similarity are more likely to belong to the brain tumor, while regions with low similarity are more likely to belong to the background.

[0467] By analyzing the similarity map of the prototype features of brain tumors, the first h points of the similarity map are designated as positive cue information, where h is a positive integer greater than or equal to 1. The last g points of the similarity map are designated as negative cue information, where g is a positive integer greater than or equal to 1. Positive cue information indicates the location of key points in the brain tumor, representing its center. Negative cue information indicates the location of the background.

[0468] In some embodiments, for any voxel in an unlabeled CT sample, if the pseudo-labels in at least two of the three views belong to the brain tumor category (foreground category), the fused pseudo-label of the voxel is determined to belong to the brain tumor category (foreground category); if the pseudo-labels in at least two of the three views belong to the background category, the fused pseudo-label of the voxel is determined to belong to the background category.

[0469] In some embodiments, based on teacher predictions from unlabeled CT samples, the predicted probability of each voxel in the CT scan to be segmented is obtained under its respective category. For each voxel, the predicted probability under its category is averaged to obtain a probability threshold corresponding to that category. The label type is determined based on the relationship between the predicted probability and the probability threshold. The label type of correctly labeled voxels in the unlabeled CT samples is determined as the category to which the fused pseudo-label belongs. The label type of incorrectly labeled voxels in the unlabeled CT samples is calibrated to the category to which the latent label belongs. The calibrated teacher prediction results are generated based on the fused pseudo-label and latent label.

[0470] Step 5: Train the medical segmentation model based on the student prediction results of labeled CT samples, the student prediction results of unlabeled CT samples, the teacher prediction results of unlabeled CT samples, and the calibrated teacher prediction results.

[0471] Optionally, the supervised loss for labeled CT samples is calculated based on the student predictions and the true labels of the labeled CT samples.

[0472] Optionally, the consistency loss of unlabeled CT samples is calculated based on the student prediction results and the teacher prediction results of unlabeled CT samples, and the label loss of unlabeled CT samples is calculated based on the student prediction results and the calibrated teacher prediction results of unlabeled CT samples. The unsupervised loss of unlabeled CT samples is calculated based on the weighted sum of the consistency loss and the label loss.

[0473] Based on the aforementioned supervised loss and unsupervised loss, a medical segmentation model is trained.

[0474] In some embodiments, when calculating the unsupervised loss, a first weight corresponding to the consistency loss and a second weight corresponding to the label loss are determined based on the current number of training iterations and the maximum number of training iterations; a first product of the consistency loss and the first weight is calculated, and a second product of the label loss and the second weight is calculated; the sum of the first product and the second product is determined as the unsupervised loss of the unlabeled CT sample.

[0475] In some embodiments, when calculating the label loss, boundary constraints are applied to the predicted value vector generated by the student model, which is the unnormalized predicted value output by the student model.

[0476] In some embodiments, during the training of the medical segmentation model, candidate CT labeled samples that meet the labeling criteria are selected from unlabeled CT samples. These candidate CT labeled samples are CT samples that are candidate labels for ground truth. Based on the intensity features of the histogram, the candidate labeled CT samples are screened to determine the CT samples to be labeled, and ground truth labels are then assigned to these CT samples.

[0477] In some embodiments, the labeled sample is a three-dimensional medical image, and the real label of the labeled CT sample is annotated using a cross-labeling strategy. The cross-labeling strategy is an annotation strategy that labels two-dimensional slice images from three different perspectives in a three-dimensional medical image.

[0478] In summary, this application utilizes a collaborative approach between a medical segmentation model and a generalist segmentation model. The medical segmentation model generates prompts for the generalist segmentation model, and the generalist model, through its response to these prompts, provides auxiliary supervision for the training of the specialist segmentation model. Specifically, by acquiring prototype information from unlabeled CT samples, prompts for the generalist segmentation model are generated based on this prototype information. The generalist segmentation model then generates calibrated teacher predictions based on these prompts. These calibrated teacher predictions provide additional supervisory signals to the medical segmentation model during training, improving the utilization of information from unlabeled CT samples and effectively reducing the need for labeled CT samples. This application employs the general image segmentation capabilities of the generalist segmentation model to assist in training a high-quality medical segmentation model.

[0479] Figure 8 This diagram illustrates a structural block diagram of a training apparatus for a specialist segmentation model according to an embodiment of this application. The apparatus has the functionality to implement the training method example of the aforementioned specialist segmentation model; this functionality can be implemented in hardware or by hardware executing corresponding software. Figure 8 As shown, the device may include: a sample acquisition module 810, a labeled segmentation module 820, an unlabeled segmentation module 830, a prompt generation module 840, and a model training module 850.

[0480] The sample acquisition module 810 is used to acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with real labels, and the unlabeled samples are image samples without real labels;

[0481] The labeled segmentation module 820 is used to input the labeled samples into the student model and obtain the student prediction results of the labeled samples based on the student model;

[0482] The unlabeled segmentation module 830 is used to input the unlabeled samples into the student model and the teacher model, and obtain the student prediction results and the teacher prediction results of the unlabeled samples, respectively.

[0483] The prompt generation module 840 is used to obtain prototype information from the unlabeled samples, generate prompt information for the generalist segmentation model based on the prototype information, and generate calibrated teacher prediction results through the generalist segmentation model based on the prompt information. The prototype information is information obtained by processing the teacher prediction results based on the unlabeled samples.

[0484] The model training module 850 is used to train the specialist segmentation model based on the student prediction results of the labeled samples, the student prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples, and the calibrated teacher prediction results.

[0485] In some embodiments, the prompt generation module 840 further includes an acquisition submodule and a prompt generation submodule.

[0486] In an optional example, a submodule is used to obtain prototype information of the image to be segmented, taking the unlabeled sample as the image to be segmented.

[0487] The prompt generation submodule is used to generate prompt information for the generalist segmentation model based on the prototype information, and to generate the calibrated teacher prediction results by the generalist segmentation model based on the prompt information.

[0488] In some embodiments, the acquisition submodule further includes an acquisition unit and a generation unit.

[0489] In an optional example, the acquisition unit is used to acquire the teacher prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples including the prediction results of the category to which each voxel in the image to be segmented belongs;

[0490] The generation unit is used to extract the features of all voxels of the target segmentation object from the sampling feature map corresponding to the image to be segmented, and to process the features of all voxels of the target segmentation object to obtain first prototype information, which is used to indicate the features of the target segmentation object.

[0491] The generation unit is used to extract the features of all voxels of the non-target segmentation object from the sampled feature map corresponding to the image to be segmented, and to process the features of all voxels of the non-target segmentation object to obtain second prototype information. The second prototype information is used to indicate the features of the non-target segmentation object, which is the background segmentation image in the image to be segmented other than the target segmentation object.

[0492] In some embodiments, the apparatus includes a slicing module, and the prompt generation submodule further includes a generation unit.

[0493] In an optional example, the slicing module is used to slice the image to be segmented according to three perspectives to obtain multiple slice images corresponding to the three perspectives, and the multiple slice images are two-dimensional images;

[0494] The generation unit is used to take multiple slice images from the three perspectives as the images to be segmented, generate prompt information for each slice image in the multiple slice images based on the prototype information, and predict the pseudo-labels on the three perspectives respectively based on the prompt information by the general segmentation model.

[0495] The generation unit is used to generate fused pseudo-labels after fusing pseudo-labels from the three perspectives based on the correction strategy, and to generate the calibrated teacher prediction results based on the fused pseudo-labels.

[0496] In one optional example, the prompt message includes at least one of the following:

[0497] Two-dimensional bounding box hint information, which is used to mark the region range of the target segmentation object;

[0498] On-time prompt information, which is used to indicate the key point positions of the target segmentation object;

[0499] Negative dot information is used to indicate the location of the non-target segmentation object.

[0500] In some embodiments, the generation unit includes a decomposition subunit and a determination subunit.

[0501] In an optional example, the prompt information includes the two-dimensional bounding box prompt information, which is obtained based on the enhanced teacher prediction result, which is a teacher prediction result generated based on the prototype information.

[0502] The decomposition subunit is used to perform a selection operation on the enhanced teacher prediction result to obtain the prototype information label corresponding to each voxel in the image to be segmented. The prototype information label is decomposed into sub-labels of the three perspectives. The prototype information label is the prediction label with the highest prediction probability in the teacher prediction result guided by the prototype information. Each of the three perspectives corresponds to multiple slice images.

[0503] A subunit is defined for extracting the connected regions of each slice image based on the sub-label for each of the plurality of slice images, and determining the two-dimensional bounding box prompt information based on the connected regions.

[0504] In an optional example, the prompt information includes positive prompt information and negative prompt information, which are obtained based on the prototype feature similarity map of the target segmentation object;

[0505] A subunit is defined for determining the first h points of the prototype feature similarity map as the positive prompt information for each of the plurality of slice images, where h is a positive integer greater than or equal to 1;

[0506] A subunit is defined for determining the last g points of the prototype feature similarity map as the negative point prompt information for each of the plurality of slice images, where g is a positive integer greater than or equal to 1.

[0507] In some embodiments, the apparatus further includes a comparison module.

[0508] In an optional example, a comparison module is used to compare the similarity between the features of each voxel in the image to be segmented and the first prototype information, and to compare the similarity between the features of each voxel in the image to be segmented and the second prototype information, to obtain the enhanced teacher prediction result.

[0509] The enhanced teacher prediction result includes the predicted probability that each voxel in the image to be segmented belongs to the target segmentation object or the non-target segmentation object.

[0510] In an optional example, the comparison module is used to obtain a prototype feature similarity map of the target segmentation object based on the similarity between the sampled feature map corresponding to the image to be segmented and the first prototype information;

[0511] The points in the prototype feature similarity map are sorted from high to low according to the similarity between the sampled feature map and the prototype information of the first object.

[0512] In an optional example, a subunit is defined for determining, for any voxel in the unlabeled sample, if the pseudo-labels in at least two of the three views belong to the first category, that the fused pseudo-label of the voxel belongs to the first category; and if the pseudo-labels in at least two of the three views belong to the second category, that the fused pseudo-label of the voxel belongs to the second category.

[0513] In some embodiments, the generation unit includes a processing subunit and a determination subunit.

[0514] In an optional example, a processing subunit is configured to perform an averaging process on the predicted probability of each voxel under its category to obtain a probability threshold corresponding to the category, and determine the label type based on the relationship between the predicted probability and the probability threshold.

[0515] A subunit is defined to determine the label type of correctly labeled voxels in the unlabeled samples as the category to which the fusion pseudo-label belongs, to calibrate the label type of incorrectly labeled voxels in the unlabeled samples as the category to which the potential label belongs, and to generate the calibrated teacher prediction result based on the fusion pseudo-label and the potential label.

[0516] In some embodiments, the model training module 850 includes a computation submodule and a training submodule.

[0517] In an optional example, a computation submodule is used to calculate the supervised loss of the labeled samples based on the student prediction results of the labeled samples and the true labels;

[0518] The calculation submodule is used to calculate the consistency loss of the unlabeled samples based on the student prediction results and the teacher prediction results of the unlabeled samples, and to calculate the label loss of the unlabeled samples based on the student prediction results and the calibrated teacher prediction results of the unlabeled samples.

[0519] The calculation submodule is used to calculate the unsupervised loss of the unlabeled samples based on the weighted sum between the consistency loss and the label loss;

[0520] The training submodule is used to train the specialist segmentation model based on the supervised loss and the unsupervised loss.

[0521] In some embodiments, the calculation submodule includes a determination unit and a calculation unit.

[0522] In an optional example, a determining unit is used to determine the first weight corresponding to the consistency loss and the second weight corresponding to the label loss based on the current number of training iterations and the maximum number of training iterations.

[0523] The calculation unit is used to calculate the first product of the consistency loss and the first weight, and to calculate the second product of the label loss and the second weight;

[0524] A determining unit is configured to determine the sum of the first product and the second product as the unsupervised loss of the unlabeled sample.

[0525] In some embodiments, the apparatus further includes a constraint module.

[0526] In an optional example, a constraint module is used to apply boundary constraints to the predicted value vector generated by the student model when calculating the label loss, the predicted value vector being the unnormalized predicted values ​​output by the student model.

[0527] In some embodiments, the apparatus further includes a selection module.

[0528] In an optional example, a selection module is used to select candidate labeled samples that meet the labeling conditions from the unlabeled samples during the training process of the specialist segmentation model, wherein the candidate labeled samples are samples with candidate labels of the real labels.

[0529] The selection module is used to filter the candidate labeled samples based on the intensity features of the histogram, determine the samples to be labeled, and label the samples to be labeled with real labels.

[0530] In an optional example, the labeled sample is a 3D image, and the real labels of the labeled sample are annotated using a cross-labeling strategy, which is an annotation strategy that labels slices of the 3D image from three different perspectives.

[0531] It should be noted that the specific limitations of the embodiments of the training device for one or more specialist segmentation models provided above can be found in the limitations of the training method for specialist segmentation models above, and will not be repeated here. Each module of the above device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in the processor of the computer device in hardware form or independent of the processor, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0532] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0533] This application also provides a computer device, which includes a processor and a memory. The memory stores at least one instruction, at least one program, code set, or instruction set. The processor loads and executes the at least one instruction, at least one program, code set, or instruction set to implement the training method of the specialist segmentation model provided in the above-described method embodiments.

[0534] For example, Figure 9 This is a schematic diagram of the structure of a computer device provided in an exemplary embodiment of this application.

[0535] The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including random access memory (RAM) 1002 and read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 to facilitate information transfer between various components within the computer device, and a mass storage device 1007 for storing the operating system 1013, application programs 1014, and other program modules 1015.

[0536] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009 for user input, such as a mouse or keyboard. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include the input / output controller 1010 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0537] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable storage media provide non-volatile storage for the computer device 1000. That is, the mass storage device 1007 may include computer-readable storage media (not shown), such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0538] Without loss of generality, the computer-readable storage medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable storage instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage devices, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage medium is not limited to the above-mentioned types. The system memory 1004 and mass storage device 1007 described above can be collectively referred to as memory.

[0539] The memory stores one or more programs, which are configured to be executed by one or more central processing units 1001. The one or more programs contain instructions for implementing the above method embodiments. The central processing unit 1001 executes the one or more programs to implement the training method of the specialist segmentation model provided by the above method embodiments.

[0540] According to various embodiments of this application, the computer device 1000 can also be connected to a remote computer device on a network, such as the Internet. That is, the computer device 1000 can be connected to the network 1012 via the network interface unit 1011 connected to the system bus 1005, or the network interface unit 1011 can be used to connect to other types of networks or remote computer device systems (not shown).

[0541] The memory further includes one or more programs stored in the memory, and the one or more programs include steps executed by a computer device in the training method for the talent segmentation model provided in the embodiments of this application.

[0542] This application also provides a computer-readable storage medium storing at least one instruction, at least one program, code set, or instruction set. When the at least one instruction, at least one program, code set, or instruction set is loaded and executed by the processor of a computer device, the training method of the specialist segmentation model provided in the above-described method embodiments is implemented.

[0543] This application also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the training method for the specialist segmentation model provided in the above-described method embodiments.

[0544] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0545] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent switching, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a specialist segmentation model, characterized in that, The specialist segmentation model includes a teacher model and a student model, and the method includes: Acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples labeled with real labels, and the unlabeled samples are image samples without real labels; The labeled samples are input into the student model, and the student prediction results for the labeled samples are obtained based on the student model. The unlabeled samples are input into the student model and the teacher model to obtain the student prediction results and the teacher prediction results of the unlabeled samples, respectively. Obtain prototype information from the unlabeled samples, generate prompt information for a generalist segmentation model based on the prototype information, and generate calibrated teacher prediction results through the generalist segmentation model based on the prompt information. The prototype information is information obtained by processing the teacher prediction results based on the unlabeled samples. The specialist segmentation model is trained based on the student prediction results of the labeled samples, the student prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples, and the calibrated teacher prediction results.

2. The method according to claim 1, characterized in that, The steps of obtaining prototype information from the unlabeled samples, generating prompts for a generalist segmentation model based on the prototype information, and generating calibrated teacher prediction results using the generalist segmentation model based on the prompts include: The unlabeled sample is used as the image to be segmented, and the prototype information of the image to be segmented is obtained. Based on the prototype information, the generalist segmentation model generates prompt information, and the generalist segmentation model generates the calibrated teacher prediction results based on the prompt information.

3. The method according to claim 2, characterized in that, The prototype information includes first prototype information and second prototype information; The step of using the unlabeled sample as the image to be segmented and obtaining the prototype information of the image to be segmented includes: Obtain the teacher prediction results for the unlabeled samples, which include the prediction results for the category of each voxel in the image to be segmented; The features of all voxels of the target segmentation object are extracted from the sampling feature map corresponding to the image to be segmented. After processing the features of all voxels of the target segmentation object, first prototype information is obtained. The first prototype information is used to indicate the features of the target segmentation object. Features of all voxels of non-target segmentation objects are extracted from the sampling feature map corresponding to the image to be segmented. After processing the features of all voxels of non-target segmentation objects, second prototype information is obtained. The second prototype information is used to indicate the features of the non-target segmentation objects, which are the background segmentation images in the image to be segmented other than the target segmentation objects.

4. The method according to claim 2, characterized in that, The image to be segmented is a three-dimensional image, and the method further includes: The image to be segmented is sliced ​​according to three perspectives to obtain multiple slice images corresponding to the three perspectives, and the multiple slice images are two-dimensional images. The step of generating prompts for the generalist segmentation model based on the prototype information, and generating the calibrated teacher prediction results based on the prompts using the generalist segmentation model, includes: Multiple slice images from the three perspectives are used as the images to be segmented. Based on the prototype information, prompt information is generated for each slice image in the multiple slice images. The general segmentation model predicts the pseudo-labels on the three perspectives based on the prompt information. The system generates a fused pseudo-label by fusing the pseudo-labels from the three perspectives based on the correction strategy, and generates the calibrated teacher prediction results based on the fused pseudo-labels.

5. The method according to any one of claims 1 to 4, characterized in that, The prompt message includes at least one of the following: Two-dimensional bounding box hint information, which is used to mark the region range of the target segmentation object; On-time prompt information, which is used to indicate the key point positions of the target segmentation object; Negative dot information is used to indicate the location of the non-target segmentation object.

6. The method according to claim 5, characterized in that, The prompt information includes the two-dimensional bounding box prompt information, which is obtained based on the enhanced teacher prediction result, and the enhanced teacher prediction result is a teacher prediction result generated based on the prototype information. The step of generating prompt information for each of the plurality of slice images based on the prototype information includes: A selection operation is performed on the enhanced teacher prediction results to obtain the prototype information label corresponding to each voxel in the image to be segmented. The prototype information label is decomposed into sub-labels of the three perspectives. The prototype information label is the prediction label with the highest prediction probability in the teacher prediction results guided by the prototype information. Each of the three perspectives corresponds to multiple slice images. For each of the plurality of slice images, the connected regions of each slice image are extracted according to the sub-labels, and the two-dimensional bounding box prompt information is determined based on the connected regions.

7. The method according to claim 5, characterized in that, The prompt information includes positive prompt information and negative prompt information, which are obtained based on the prototype feature similarity map of the target segmentation object; The step of generating prompt information for each of the plurality of slice images based on the prototype information includes: For each of the plurality of slice images, the first h points of the prototype feature similarity map are determined as the positive point prompt information, where h is a positive integer greater than or equal to 1; For each of the plurality of slice images, the last g points of the prototype feature similarity map are determined as the negative point prompt information, where g is a positive integer greater than or equal to 1.

8. The method according to claim 6, characterized in that, The method further includes: The enhanced teacher prediction result is obtained by comparing the similarity between the features of each voxel in the image to be segmented and the first prototype information, and by comparing the similarity between the features of each voxel in the image to be segmented and the second prototype information. The enhanced teacher prediction result includes the predicted probability that each voxel in the image to be segmented belongs to the target segmentation object or the non-target segmentation object.

9. The method according to claim 7, characterized in that, The method further includes: Based on the similarity between the sampled feature map corresponding to the image to be segmented and the first prototype information, a prototype feature similarity map of the target segmentation object is obtained; The points in the prototype feature similarity map are sorted from high to low according to the similarity between the sampled feature map and the prototype information of the first object.

10. The method according to claim 4, characterized in that, The process of generating fused pseudo-labels by fusing the pseudo-labels from the three perspectives based on a correction strategy includes: For any voxel in the unlabeled sample, if the pseudo-labels in at least two of the three perspectives belong to the first category, the fused pseudo-label of the voxel is determined to belong to the first category; if the pseudo-labels in at least two of the three perspectives belong to the second category, the fused pseudo-label of the voxel is determined to belong to the second category.

11. The method according to claim 4, characterized in that, The process of generating the calibrated teacher prediction results based on the fused pseudo-labels includes: Based on the teacher prediction results of the unlabeled samples, the predicted probability of each voxel in the image to be segmented under its category is obtained; For each voxel, the predicted probability of each voxel under its category is averaged to obtain the probability threshold corresponding to the category, and the label type is determined based on the relationship between the predicted probability and the probability threshold. The label type of correctly labeled voxels in the unlabeled samples is determined as the category to which the fusion pseudo-label belongs. The label type of incorrectly labeled voxels in the unlabeled samples is calibrated as the category to which the potential label belongs. The calibrated teacher prediction result is generated based on the fusion pseudo-label and the potential label.

12. The method according to any one of claims 1 to 11, characterized in that, The process of training the specialist segmentation model based on the student prediction results of the labeled samples, the student prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples, and the calibrated teacher prediction results includes: Based on the student prediction results of the labeled samples and the real labels, calculate the supervised loss of the labeled samples; Based on the student prediction results and teacher prediction results of the unlabeled samples, the consistency loss of the unlabeled samples is calculated, and based on the student prediction results and the calibrated teacher prediction results of the unlabeled samples, the label loss of the unlabeled samples is calculated. The unsupervised loss of the unlabeled samples is calculated based on the weighted sum of the consistency loss and the label loss. The specialist segmentation model is trained based on the supervised loss and the unsupervised loss.

13. The method according to claim 12, characterized in that, The calculation of the unsupervised loss of the unlabeled samples based on the weighted sum of the consistency loss and the label loss includes: Based on the current number of training iterations and the maximum number of training iterations, determine the first weight corresponding to the consistency loss and the second weight corresponding to the label loss; Calculate the first product of the consistency loss and the first weight, and calculate the second product of the label loss and the second weight; The sum of the first product and the second product is determined as the unsupervised loss of the unlabeled sample.

14. The method according to claim 12 or 13, characterized in that, The method further includes: When calculating the label loss, boundary constraints are applied to the predicted value vector generated by the student model, which is the unnormalized predicted value output by the student model.

15. The method according to any one of claims 1 to 14, characterized in that, The method further includes: During the training process of the talent segmentation model, candidate labeled samples that meet the labeling conditions are selected from the unlabeled samples. The candidate labeled samples are samples with candidate labels and real labels. Based on the intensity characteristics of the histogram, the candidate labeled samples are screened to determine the samples to be labeled, and the samples to be labeled are labeled with real labels.

16. The method according to any one of claims 1 to 14, characterized in that, The labeled sample is a three-dimensional image, and the real label of the labeled sample is annotated using a cross-labeling strategy. The cross-labeling strategy is an annotation strategy that labels the slice images from three different perspectives in the three-dimensional image.

17. A training device for a specialist segmentation model, characterized in that, The device includes: The sample acquisition module is used to acquire labeled samples and unlabeled samples, wherein the labeled samples are image samples with real labels and the unlabeled samples are image samples without real labels; A labeled segmentation module is used to input the labeled samples into the student model and obtain the student prediction results of the labeled samples based on the student model; The unlabeled segmentation module is used to input the unlabeled samples into the student model and the teacher model, and obtain the student prediction results and the teacher prediction results of the unlabeled samples, respectively. The prompt generation module is used to obtain prototype information from the unlabeled samples, generate prompt information for the generalist segmentation model based on the prototype information, and generate calibrated teacher prediction results through the generalist segmentation model based on the prompt information. The prototype information is information obtained by processing the teacher prediction results based on the unlabeled samples. The model training module is used to train the specialist segmentation model based on the student prediction results of the labeled samples, the student prediction results of the unlabeled samples, the teacher prediction results of the unlabeled samples, and the calibrated teacher prediction results.

18. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to implement the message bubble-based interactive method as described in any one of claims 1 to 16.

19. A computer storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the training method of the specialist segmentation model as described in any one of claims 1 to 16.

20. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium; the computer program is read from and executed by a processor of a computer device, causing the computer device to perform a training method for the specialist segmentation model as described in any one of claims 1 to 16.

Images