Passive active field adaptive polyp semi-supervised segmentation and classification system and method

Abstract

The invention provides a passive active field self-adaptive polyp semi-supervised segmentation and classification system and a passive active field self-adaptive polyp semi-supervised segmentation and classification method. The segmentation and classification system comprises a data collection module, a data preprocessing module, a source domain sample screening module, a target domain sample screening module, a data labeling module, a polyp segmentation and classification model construction module and a polyp segmentation and classification model semi-supervised training module. According to the method, a certain number of representative samples are screened out through a source domain sample screening module, and a certain number of representative samples, inter-domain similar samples with relatively high similarity with source domain features and inter-domain difference samples with relatively low similarity with the source domain features are screened out through a target domain sample screening module; and marking the screened samples, and subsequently training a polyp segmentation and classification model through a cross pseudo-supervision consistency strategy to obtain final model parameters. According to the method, the dependence of the polyp segmentation and classification model on the labeled data volume can be greatly improved, and the method has good cross-medical center robustness.

Description

Technical Field

[0001] The invention relates to the field of medical image processing, in particular to a passive active domain adaptive polyp semi-supervised segmentation and classification system and method. Background Art

[0002] Colorectal cancer (CRC) is one of the malignant tumors with high morbidity and mortality worldwide. Early detection and treatment of colorectal cancer can significantly reduce mortality, and polyps are the early form of colorectal cancer, so early diagnosis and removal are particularly critical.

[0003] During endoscopic examination, accurate identification of the boundary range and type of polyps is crucial for subsequent lesion resection and treatment plan formulation. At present, the precise segmentation and pathological type judgment of polyps under endoscopy mainly rely on the clinical experience of doctors, who manually draw the lesion boundaries and classify them based on visual features. This method, which is highly dependent on personal experience, is inevitably subjective and uncertain, especially when the clinical workload is large and the cases are complex. It is easily affected by the doctor's fatigue and environmental factors, thus affecting the accuracy and consistency of the diagnosis.

[0004] With the development of deep learning technology, although some progress has been made in the field of endoscopic polyp segmentation and classification, it still faces severe challenges. In terms of data annotation, the annotation of medical images requires professionally qualified physicians to invest a lot of time for accurate description and classification judgment. This process not only occupies precious medical resources and significantly increases the workload of clinicians, but also the annotation results are often subject to the professional experience level and subjective cognitive differences of individual doctors. Therefore, obtaining consistent high-quality annotated data is both time-consuming and labor-intensive, and faces extremely high manpower and time costs. In terms of task integration, existing research shows a clear task separation phenomenon in polyp diagnosis: either focusing on the detection and segmentation of polyps, or focusing on the classification of pathological types. This fragmented research idea leads to the lack of a framework system that can uniformly handle these key tasks, which not only increases the computational overhead of the system, but is also likely to cause contradictions in the final diagnosis results due to inconsistent processing logic between different modules, seriously affecting the reliability and practicality of clinical applications. At the same time, in terms of cross-center applications, different medical institutions have significant differences in the brand and model of endoscopic equipment, image acquisition conditions (such as lighting, angle, distance, etc.), image processing procedures and quality control specifications. In addition, the disease characteristics and manifestations of patient groups in different regions have regional differences. These factors make it difficult to directly promote deep learning models trained in a single medical center to other medical institutions, and model performance often degrades significantly. Even if traditional transfer learning strategies are adopted, a large amount of resources still need to be invested in the target medical center to collect and annotate data for model adaptation, which seriously restricts the large-scale application of deep learning technology in clinical practice.

[0005] In summary, it is extremely difficult to implement a polyp segmentation and classification model that can effectively generalize across medical centers while reducing the reliance on labeled data. Summary of the invention

[0006] The purpose of the present invention is to provide a passive active domain adaptive polyp semi-supervised segmentation and classification system and method to solve the problems of large data annotation dependence and poor generalization ability of existing polyp segmentation and classification methods.

[0007] The present invention is achieved in that:

[0008] A passive active domain adaptive polyp semi-supervised segmentation and classification system, comprising:

[0009] The data collection module is used to collect images of polyp lesions obtained by multiple medical institutions during routine endoscopic examinations, and to collect the corresponding pathological biopsy results and expert diagnosis reports to establish an accurate correspondence between images and pathological types;

[0010] The data preprocessing module is used to anonymize the collected source domain data set and target domain data set, divide them into training data set and test data set according to a certain ratio, and randomly change the processed images in the training data set to perform data enhancement to increase the richness of samples;

[0011] The source domain sample screening module is used to construct a representative feature set of the source domain and screen a certain number of representative samples from the source domain training data set;

[0012] The target domain sample screening module is used to screen a certain number of representative samples from the target domain training data set, as well as inter-domain similar samples with high similarity to the source domain features and inter-domain difference samples with low similarity;

[0013] A data annotation module, used for annotating polyp regions in representative samples screened by the source domain sample screening module and the target domain sample screening module;

[0014] A polyp segmentation and classification model building module is used to build a polyp segmentation and classification model, and use the built model to achieve accurate segmentation of polyps and classification of pathological types;

[0015] The semi-supervised training module for polyp segmentation and classification models is used to mix labeled image samples and unlabeled image samples in the source domain or target domain training dataset, and train the polyp classification and segmentation model using a cross-pseudo-supervised consistency strategy to obtain the final model parameters.

[0016] Furthermore, the collected data including endoscopic images with polyp lesions and corresponding pathological types are from two centers, where the data from one center is used as a source domain dataset and the data from the other center is used as a target domain dataset.

[0017] Furthermore, the collected source domain datasets and target domain datasets were anonymized by removing personal identification information such as patient name, ID, age, and gender from the images, and randomly divided into training set and test set in a ratio of 4:1. The anonymized data in the training set were randomly flipped, scaled, cropped, illuminated, and color transformed to perform data augmentation and increase the richness of the samples.

[0018] Furthermore, a variational autoencoder is used to map the samples in the source domain training dataset from the original image space to a high-dimensional latent feature space, and a clustering operation is performed in the feature space to obtain the central samples of each cluster to form a representative feature set of the source domain, and then a certain number of representative samples are screened out from each cluster.

[0019] Furthermore, a variational autoencoder is used to map the samples in the target domain training dataset from the original image space to a high-dimensional latent feature space, and a clustering operation is performed in the feature space. A certain number of representative samples are screened out from each cluster, and then the similarity between the target domain sample features and each feature in the representative feature set of the source domain is calculated, and the average is taken to obtain the average similarity. The samples are sorted according to the similarity, and the inter-domain similar samples with higher similarity and the inter-domain difference samples with lower similarity are screened out.

[0020] Furthermore, for the samples screened out in the source domain sample screening module and the target domain sample screening module, LabelMe labeling software is used to label the polyp areas in the images.

[0021] Furthermore, the constructed polyp segmentation and classification model includes:

[0022] an encoder for extracting shallow and deep features from endoscopic images; and

[0023] A segmentation decoder is used to aggregate features at different levels through a feature pyramid structure or multi-scale feature fusion, and finally output pixel-level polyp region segmentation; and

[0024] The classification decoder is used to integrate multi-level features through operations such as feature aggregation and global pooling to predict the pathological category of polyps.

[0025] Furthermore, the training process of the polyp segmentation and classification model is as follows: the labeled image samples and the unlabeled image samples in the source domain or target domain dataset are mixed as the initial training set of the model; two different parameters are initialized for the constructed polyp segmentation and classification model to obtain two polyp segmentation and classification models A and B with exactly the same structure but different parameters; during the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions respectively, and the high confidence prediction results generated by the segmentation branch of model A will be used as the pseudo label of the segmentation branch of model B, and similarly, the segmentation prediction of model B will also be used as the pseudo label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, the classification prediction of model A is used as the pseudo label of the classification branch of model B, and the classification prediction of model B is used as the pseudo label of the classification branch of model A; the model parameters are iteratively trained for a certain number of rounds, and the parameters that perform best on the test set of models A and B are returned as the final model parameters for the segmentation and classification of polyps in endoscopic images.

[0026] The present invention can significantly improve the reliance of polyp segmentation and classification models on the amount of labeled data, and has good cross-medical center robustness.

[0027] The purpose of the present invention can also be achieved like this:

[0028] A passive active domain adaptive polyp semi-supervised segmentation and classification method, comprising the following steps:

[0029] S1. Construct the passive active domain adaptive polyp semi-supervised segmentation and classification system of the present invention.

[0030] S2. Collect images of polyp lesions obtained during routine endoscopic examinations at two-level medical institutions, and collect the corresponding pathological biopsy results and expert diagnosis reports to establish an accurate correspondence between images and pathological types; the data from one center is used as the source domain dataset, and the data from the other center is used as the target domain dataset.

[0031] S3. Using the data preprocessing module, the collected source domain dataset and target domain dataset are anonymized by removing personal identification information such as patient name, ID, age, gender, etc. from the images, and randomly divided into training set and test set in a ratio of 4:1. The anonymized data in the training set is randomly flipped, scaled, cropped, illuminated and color transformed to perform data enhancement and increase the richness of the samples.

[0032] S4. Using the source domain sample screening block, the samples in the source domain training data set are mapped from the original image space to the high-dimensional latent feature space, and clustering operations are performed in the feature space to obtain the central samples of each cluster to form a representative feature set of the source domain, and then a certain number of representative samples are screened out from each cluster.

[0033] S5. Using the dataset annotation module, the samples screened out in the source domain sample screening module are annotated with the LabelMe annotation software to mark the polyp areas in the images.

[0034] S6. Use the polyp segmentation and classification model construction module to construct a source domain polyp segmentation and classification model, and use the constructed model to achieve accurate segmentation and pathological type classification of source domain polyps.

[0035] S7. Using the semi-supervised training module of the polyp segmentation and classification model, the labeled image samples and unlabeled image samples of the source domain training set are mixed as the initial training set of the source domain polyp segmentation and classification model; two different parameters are initialized for the constructed source domain polyp segmentation and classification model to obtain two source domain polyp segmentation and classification models A and B with exactly the same structure but different parameters; during the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions respectively, and the high-confidence prediction results generated by the segmentation branch of model A will be used as the pseudo-label of the segmentation branch of model B, and similarly, the segmentation prediction of model B will also be used as the pseudo-label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, and the classification prediction of model A is used as the pseudo-label of the classification branch of model B, and the classification prediction of model B is used as the pseudo-label of the classification branch of model A; the model parameters are iteratively trained for a certain number of rounds, and the parameters that perform best on the test set of models A and B are returned as the final source domain model parameters for the segmentation and classification of polyps in source domain endoscopic images.

[0036] S8. Using the target domain screening module, the samples in the target domain training data set are mapped from the original image controls to the high-dimensional latent feature space, and clustering operations are performed in the feature space. A certain number of representative samples are screened out from each cluster, and then the similarity between the target domain sample features and each feature in the source domain representative feature set is calculated, and the average is taken to obtain the average similarity. The samples are sorted according to the similarity, and the inter-domain similar samples with higher similarity and the inter-domain difference samples with lower similarity are screened out; the screened samples are labeled.

[0037] S9. Use the polyp segmentation and classification model construction module to construct a target domain polyp segmentation and classification model, and use the constructed model to achieve accurate segmentation and pathological type classification of target domain polyps.

[0038] S10. Using the semi-supervised training module of the polyp segmentation and classification model, the labeled image samples and unlabeled image samples of the target domain training set are mixed as the initial training set of the target domain polyp segmentation and classification model; two different parameter initializations are performed on the constructed target domain polyp segmentation and classification model, model A uses the parameters of the source domain model as initialization, while model B uses the exponential moving average (EMA) of the source domain parameters for initialization, and two target domain polyp segmentation and classification models A and B with exactly the same structure but different parameters are obtained; during the training process, for labeled samples, both models A and B use the real segmentation mask and classification mark supervised training with labels; for unlabeled samples, the two models will make predictions separately, and the high-confidence prediction results generated by the segmentation branch of model A will be used as the pseudo-label of the segmentation branch of model B, and similarly, the segmentation predictions of model B will be used as the pseudo-label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, the classification predictions of model A are used as the pseudo-labels of the classification branch of model B, and the classification predictions of model B are used as the pseudo-labels of the classification branch of model A; the model parameters are iteratively trained for a certain number of rounds, and the parameters of models A and B that perform best on the test set are returned as the final target domain model parameters for the segmentation and classification of polyps in target domain endoscopic images.

[0039] The purpose of the present invention is to provide a passive active domain adaptive polyp semi-supervised segmentation and classification for polyp segmentation and classification in endoscopic images. This method solves the problems of high data annotation cost, fragmentation of segmentation and classification tasks, and poor cross-center generalization ability in the prior art, while improving the overall performance and robustness of the model. The present invention adopts a source domain and target domain sample screening strategy and a semi-supervised training strategy based on active learning, which reduces the data annotation cost and improves the model training efficiency. Accurate segmentation and pathological type classification of polyps are achieved simultaneously through a unified deep learning framework, and the generalization ability of the model in different medical centers is improved through domain adaptation technology.

[0040] The present invention has the following beneficial effects:

[0041] (1) This paper innovatively proposes a source domain and target domain sample screening strategy based on active learning, selects representative samples for manual annotation, and combines it with a semi-supervised training strategy to significantly reduce the data annotation cost.

[0042] (2) The present invention realizes the unified processing of polyp segmentation and classification tasks, avoids the problems caused by task splitting in traditional methods, and improves the overall efficiency and diagnostic consistency of the system.

[0043] (3) Through an innovative domain adaptation strategy, the present invention significantly improves the generalization ability of the model in different medical centers, enabling the model to quickly adapt to new application scenarios and has important clinical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is the data preprocessing flow chart.

[0045] Figure 2 This is the flowchart for screening representative samples of the source domain.

[0046] Figure 3 is a schematic diagram of the polyp segmentation and classification model.

[0047] Figure 4 Schematic diagram of cross-pseudo-supervision of polyp segmentation and classification models.

[0048] Figure 5 This is the flowchart for screening representative samples of the target domain. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to specific embodiments and drawings.

[0050] Example 1

[0051] The passive active domain adaptive polyp semi-supervised segmentation and classification system of the present invention comprises a data collection module, a data preprocessing module, a source domain sample screening module, a target domain sample screening module, a data annotation module, a polyp segmentation and classification model construction module and a polyp segmentation and classification model semi-supervised training module. The data collection module is used to collect polyp lesion images and corresponding pathological types from different medical institutions. The data preprocessing module is used to anonymize the collected data, divide the training set, the test set and the training set enhancement processing. The source domain sample screening module is used to construct a representative feature set of the source domain, and screen out a certain number of representative samples of the source domain. The target domain sample screening module is used to screen out a certain number of representative samples of the target domain, as well as inter-domain similar samples with high similarity to the source domain features and inter-domain difference samples with low similarity. The data annotation module is used to annotate the polyp area in the screened representative samples. The polyp segmentation and classification model construction module is used to construct a polyp segmentation and classification model. The polyp segmentation and classification model semi-supervised training module is used to train the polyp classification and segmentation model using a cross pseudo-supervised consistency strategy to obtain the final model parameters.

[0052] The constructed polyp segmentation and classification model includes an encoder, a segmentation decoder and a classification decoder; the encoder is used to extract shallow and deep features in endoscopic images; the segmentation decoder is used to aggregate features at different levels through a feature pyramid structure or multi-scale feature fusion, and finally output pixel-level polyp area segmentation; the classification decoder is used to integrate multi-level features through operations such as feature aggregation and global pooling to predict the pathological category of polyps.

[0053] The training process of the polyp segmentation and classification model is as follows: the labeled image samples and unlabeled image samples in the source domain or target domain dataset are mixed as the initial training set of the model; two different parameters are initialized for the constructed polyp segmentation and classification model to obtain two polyp segmentation and classification models A and B with exactly the same structure but different parameters; during the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions separately, and the high-confidence prediction results generated by the segmentation branch of model A will be used as the pseudo-label of the segmentation branch of model B, and similarly, the high-confidence predictions generated by the segmentation of model B will also be used as the pseudo-label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, the classification prediction of model A is used as the pseudo-label of the classification branch of model B, and the classification prediction of model B is used as the pseudo-label of the classification branch of model A; the model parameters are iteratively trained for a certain number of rounds, and the parameters that perform best on the test set of models A and B are returned as the final model parameters for the segmentation and classification of polyps in endoscopic images.

[0054] Example 2

[0055] The passive active domain adaptive polyp semi-supervised segmentation and classification method of the present invention is to construct the passive active domain adaptive polyp semi-supervised segmentation and classification system of the present invention, and then perform the following steps:

[0056] 1. Data Collection

[0057] We collected polyp lesion images obtained during routine endoscopic examinations from two medical institutions, as well as the corresponding pathological biopsy results and expert diagnosis reports, to establish an accurate correspondence between images and pathological types. The first institution collected data from 1,000 patients as the source domain dataset, and the second institution collected data from 500 patients as the target domain dataset.

[0058] 2. Data preprocessing

[0059] like Figure 1As shown in the figure, the collected data are preprocessed as follows: first, the personal identification information such as name, ID, age, and gender in the collected data set is anonymized to protect the privacy of the patient, and then the data set is divided into two parts, the training data set and the test data set, in a ratio of 4:1. The processed images in the training data set are randomly flipped, scaled, cropped, and the illumination and color are transformed to enhance the data and increase the richness of the samples; in terms of random flipping, the image is flipped horizontally and vertically with a probability of 0.5; in terms of scaling, the image is randomly scaled to between 0.8 and 1.2 times the original size; for random cropping, the image is first scaled to 384×384, and then a 320×320 area is randomly cropped for training. In terms of illumination enhancement, the brightness, contrast, and saturation of the image are randomly adjusted, where the adjustment range of brightness is 0.8-1.2 times the original value, the adjustment range of contrast is 0.8-1.2 times the original value, and the adjustment range of saturation is 0.8-1.2 times the original value. In terms of color transformation, color dithering is performed on the three RGB channels respectively, and the dithering range is ±10% of the original value. At the same time, the image is hue shifted, and the shift angle range is ±10 degrees.

[0060] 3. Screening and labeling of representative samples in the source domain

[0061] like Figure 2 As shown in the figure, a variational autoencoder based on a convolutional neural network is used to process the training data set of the source domain, and all samples in the source domain training set are mapped from the original image space to the high-dimensional latent feature space. After the mapping is completed, the K-means clustering algorithm is used to perform clustering operations in the latent space to identify and calculate the centers of each cluster cluster, which represent the core features of the source domain data. Next, the top 5% of samples closest to the cluster center are selected from each cluster as representative samples, which are used as representative features of the source domain. Finally, the selected representative samples of the source domain and the test set are manually annotated. The test set is annotated to verify the model.

[0062] 4. Construct source domain polyp segmentation and classification model

[0063] The polyp segmentation and classification model used in the present invention is obtained by adding a classification decoder to the U-Net image segmentation model, which can be written in Python (version 3.11.1) and uses the open source Pytorch library (version 1.6.1) as the backend of the algorithm. Figure 3As shown in the figure, the model consists of three parts: the first part is the encoder, which is used to extract shallow and deep features in the endoscopic image; the second part is the segmentation decoder, which is used to aggregate features at different levels through a feature pyramid structure or multi-scale feature fusion, and finally output pixel-level polyp area segmentation; the third part is the classification decoder, which is used to integrate multi-level features through operations such as feature aggregation and global pooling to predict the pathological category of polyps. The main categories are hyperplasia, adenoma and serrated adenoma.

[0064] 5. Semi-supervised training of source domain polyp segmentation and classification model

[0065] The labeled image samples and unlabeled image samples of the source domain training set are mixed as the initial training set of the source domain polyp segmentation and classification model. Two different parameter initializations are performed on the constructed source domain polyp segmentation and classification model to obtain two source domain polyp segmentation and classification models A and B with exactly the same structure but different parameters. Figure 4 As shown in the figure, during the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions separately, and the high-confidence prediction results generated by the segmentation branch of model A will be used as the pseudo-label of the segmentation branch of model B, and the high-confidence predictions generated by the segmentation branch of model B will also be used as the pseudo-label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, the classification prediction of model A is used as the pseudo-label of the classification branch of model B, and the classification prediction of model B is used as the pseudo-label of the classification branch of model A. The model parameters are iteratively trained for a certain number of rounds, and the parameters of models A and B that perform best on the test set are returned as the final source domain model parameters for the segmentation and classification of polyps in source domain endoscopic images.

[0066] Specifically, in the training process of the source domain polyp segmentation and classification model, the corresponding hyperparameters are as follows: 100 iterations, 8 mini-batch size, including 4 labeled samples and 4 unlabeled samples, the initial learning rate is 0.01 and decays to one-tenth of the original every 30 iterations, the segmentation loss function uses the cross entropy loss function and the Dice loss function, the classification loss function uses the cross entropy loss function, and the optimizer uses the stochastic gradient descent method. At the same time, batch-normalization and drop out are used to minimize the risk of overfitting. The Dice score and intersection-union comparison are used to quantify the polyp region segmentation results of the model on the source domain test set, and the sensitivity and specificity are used to quantify the polyp pathology classification results of the model on the source domain test set.

[0067] 6. Screening and labeling of representative samples in the target domain

[0068] like Figure 4As shown in the figure, a variational autoencoder based on a convolutional neural network is used to process the training data set of the target domain, and all samples in the target domain training set are mapped from the original image space to the high-dimensional latent feature space. After the mapping is completed, the K-means clustering algorithm is used to perform clustering operations in the latent space to identify and calculate the centers of each cluster cluster, which represent the core features of the target domain data. Then, the top 5% of samples closest to the cluster center are selected from each cluster as representative samples. Subsequently, the similarity between the target domain sample features and each feature in the representative feature set of the source domain is calculated and the average is taken to obtain the average similarity. The samples are sorted according to the similarity, and 5% of the inter-domain similar samples with higher similarity and 5% of the inter-domain difference samples with lower similarity are screened out. Finally, the selected representative samples of the target domain, inter-domain similar samples, inter-domain difference samples, and test set samples are manually labeled.

[0069] 7. Build a target domain polyp segmentation and classification model

[0070] The polyp segmentation and classification model used in the present invention is obtained by adding a classification decoder to the U-Net image segmentation model, which can be written in Python (version 3.11.1) and uses the open source Pytorch library (version 1.6.1) as the backend of the algorithm. Figure 3 As shown in the figure, the model consists of three parts: the first part is the encoder, which is used to extract shallow and deep features in the endoscopic image; the second part is the segmentation decoder, which is used to aggregate features at different levels through a feature pyramid structure or multi-scale feature fusion, and finally output pixel-level polyp area segmentation; the third part is the classification decoder, which is used to integrate multi-level features through operations such as feature aggregation and global pooling to predict the pathological category of polyps. The main categories are hyperplasia, adenoma and serrated adenoma.

[0071] 8. Semi-supervised training of target domain polyp segmentation and classification model

[0072] The labeled image samples and unlabeled image samples of the target domain training set are mixed as the initial training set of the target domain polyp segmentation and classification model. Two different parameter initializations are performed on the constructed target domain polyp segmentation and classification model. Model A uses the parameters of the source domain model as initialization, while model B uses the exponential moving average (EMA) of the source domain parameters for initialization, resulting in two target domain polyp segmentation and classification models A and B with exactly the same structure but different parameters. During the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions separately, and the high confidence prediction results generated by the segmentation branch of model A will be used as the pseudo label of the segmentation branch of model B, and the segmentation prediction of model B will also be used as the pseudo label of the segmentation branch of model A; the classification branch also uses the same cross-supervision strategy, the classification prediction of model A is used as the pseudo label of the classification branch of model B, and the classification prediction of model B is used as the pseudo label of the classification branch of model A. The model parameters are iteratively trained for a certain number of rounds, and the parameters of models A and B that perform best on the test set are returned as the final target domain model parameters for the segmentation and classification of polyps in target domain endoscopic images.

[0073] Specifically, in the process of training the target domain polyp segmentation and classification model, the corresponding hyperparameters are as follows: 100 iterations, 8 mini-batch size, including 4 labeled samples and 4 unlabeled samples, the initial learning rate is 0.01 and decays to one-tenth of the original every 30 iterations, the segmentation loss function uses the cross entropy loss function and the Dice loss function, the classification loss function uses the cross entropy loss function, and the optimizer uses the stochastic gradient descent method. At the same time, batch-normalization and drop out are used to minimize the risk of overfitting. The Dice score and intersection-union comparison are used to quantify the polyp region segmentation results of the model on the target domain test set, and the sensitivity and specificity are used to quantify the polyp pathology classification results of the model on the target domain test set.

Claims

1. A passive active domain adaptive polyp semi-supervised segmentation and classification system, characterized by, include: The data collection module is used to collect polyp lesion images obtained by at least two medical institutions during routine endoscopic examinations, and collect the corresponding pathological biopsy results and expert diagnosis reports to establish an accurate correspondence between images and pathological types; the collected data is divided into a source domain data set and a target domain data set; The data preprocessing module is used to anonymize the collected source domain data set and target domain data set respectively, and then divide them into training set and test set according to a certain ratio, and randomly change the processed images in the training set to perform data enhancement to increase the richness of samples; The source domain sample screening module is used to construct a representative feature set of the source domain and screen a set number of representative samples from the source domain training set; The target domain sample screening module is used to screen out a set number of representative samples from the target domain training set, as well as inter-domain similar samples with high similarity to the source domain features and inter-domain difference samples with low similarity; A data annotation module, used to annotate polyp regions in samples screened by the source domain sample screening module and the target domain sample screening module; A polyp segmentation and classification model building module is used to build a polyp segmentation and classification model, and use the built model to achieve accurate segmentation of polyps and classification of pathological types; The semi-supervised training module for polyp segmentation and classification models is used to mix labeled image samples and unlabeled image samples in the source domain or target domain training set, and use the cross-pseudo-supervised consistency strategy to train the polyp classification and segmentation model to obtain the final model parameters.

2. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 1, characterized in that: The data preprocessing module anonymizes the collected source domain data set and target domain data set by removing the patient's name, ID, age, and gender personal identification information in the image, and then randomly divides them into training set and test set in a ratio of 4:

1. The anonymized data in the training set is randomly flipped, scaled, cropped, and the lighting and color are transformed to perform data enhancement and increase the richness of the samples.

3. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 1, characterized in that: The source domain sample screening module uses a variational autoencoder to map the samples in the source domain training set from the original image space to a high-dimensional latent feature space, and performs clustering operations in the feature space to obtain the central samples of each cluster to form a representative feature set of the source domain, and then screens out a set number of representative samples from each cluster.

4. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 1, characterized in that: The target domain sample screening module uses a variational autoencoder to map the samples in the target domain training set from the original image space to the high-dimensional latent feature space, performs clustering operations in the feature space, and screens out a set number of representative samples from each cluster. Then, the similarity between the target domain sample features and each feature in the representative feature set of the source domain is calculated, and the average is taken to obtain the average similarity. The samples are sorted according to the average similarity to screen out inter-domain similar samples with higher similarity and inter-domain difference samples with lower similarity.

5. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 1, characterized in that: The data labeling module uses LabelMe labeling software to label the polyp areas in the images of the samples screened out by the source domain sample screening module and the target domain sample screening module.

6. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 1, characterized in that: The constructed polyp segmentation and classification models include: An encoder for extracting shallow and deep features from endoscopic images; A segmentation decoder is used to aggregate features at different levels through a feature pyramid structure or multi-scale feature fusion, and finally output pixel-level polyp region segmentation; and The classification decoder is used to predict the pathological category of polyps by integrating multi-level features through feature aggregation and global pooling operations.

7. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 1, characterized in that: The process of training the polyp classification and segmentation model in the semi-supervised training module of the polyp segmentation and classification model is as follows: the labeled image samples and the unlabeled image samples in the source domain or target domain training set are mixed as the initial training set of the model; two different parameters are initialized for the constructed polyp segmentation and classification model to obtain two polyp segmentation and classification models A and B with exactly the same structure but different parameters; during the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions separately, and the high-confidence prediction results generated by the segmentation branch of model A will be used as the pseudo-label of the segmentation branch of model B, and the high-confidence prediction results generated by the segmentation branch of model B will also be used as the pseudo-label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, and the high-confidence prediction results generated by the classification branch of model A are used as the pseudo-label of the classification branch of model B, and the high-confidence prediction results generated by the classification branch of model B are used as the pseudo-label of the classification branch of model A; The model parameters are iteratively trained for a certain number of rounds, and the parameters of models A and B that perform best on the test set are returned as the final model parameters for the segmentation and classification of polyps in endoscopic images.

8. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 7, characterized in that: A source domain polyp segmentation and classification model and a target domain polyp segmentation and classification model are constructed for the source domain training set and the target domain training set respectively. When the target domain polyp segmentation and classification model is trained according to the target domain training set, two different parameter initializations are performed on the constructed target domain polyp segmentation and classification model. Model A uses the parameters of the source domain model as initialization, while model B uses the exponential moving average of the source domain parameters for initialization, thus obtaining two target domain polyp segmentation and classification models A and B with exactly the same structure but different parameters.

9. The passive active domain adaptive polyp semi-supervised segmentation and classification system according to claim 1, characterized in that: The data annotation module is also used to annotate the polyp areas in the samples of the test set so as to verify the test results after the trained model is tested.

10. A passive active domain adaptive polyp semi-supervised segmentation and classification method, characterized by: The steps include: S1. Construct the passive active domain adaptive polyp semi-supervised segmentation and classification system as described in any one of claims 1 to 9; S2. Collect polyp lesion images obtained during routine endoscopic examinations at two medical institutions, and collect the corresponding pathological biopsy results and expert diagnosis reports to establish an accurate correspondence between images and pathological types; use the data from one of the medical institutions as the source domain dataset, and the data from the other medical institution as the target domain dataset; S3. Use the data preprocessing module to remove the patient name, ID, age, gender and other personal identification information from the collected source domain dataset and target domain dataset to achieve anonymization, and randomly divide them into training set and test set in a ratio of 4:

1. The anonymized data in the training set is randomly flipped, scaled, cropped, and the lighting and color are transformed to perform data enhancement and increase the richness of the samples. S4. Use the source domain sample screening block to map the samples in the source domain training set from the original image space to the high-dimensional latent feature space, perform clustering operations in the feature space, obtain the central samples of each cluster to form a representative feature set of the source domain, and then screen out a set number of representative samples from each cluster; the data annotation module annotates the screened representative samples; S5. Use the target domain sample screening module to map the samples in the target domain training set from the original image space to the high-dimensional latent feature space, perform clustering operations in the feature space, and screen out a set number of representative samples from each cluster, then calculate the similarity between the target domain sample features and each feature in the source domain representative feature set, take the average value to obtain the average similarity, sort them according to the average similarity, screen out inter-domain similar samples with higher similarity and inter-domain difference samples with lower similarity; and use the data annotation module to annotate the screened representative samples, inter-domain similar samples, and inter-domain difference samples; S6. construct a source domain polyp segmentation and classification model using the polyp segmentation and classification model construction module, and use the constructed model to achieve accurate segmentation and pathological type classification of source domain polyps; S7. construct a target domain polyp segmentation and classification model using the polyp segmentation and classification model construction module, and use the constructed model to achieve accurate segmentation and pathological type classification of the target domain polyps; S8. Using the semi-supervised training module of the polyp segmentation and classification model, the labeled image samples and unlabeled image samples of the source domain training set are mixed as the initial training set of the source domain polyp segmentation and classification model; two different parameters are initialized for the constructed source domain polyp segmentation and classification model to obtain two source domain polyp segmentation and classification models A and B with exactly the same structure but different parameters; during the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions separately, and the high-confidence prediction results generated by the segmentation branch of model A will be used as the pseudo-label of the segmentation branch of model B, and similarly, the high-confidence prediction results generated by the segmentation branch of model B will also be used as the pseudo-label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, and the high-confidence prediction results generated by the classification branch of model A are used as the pseudo-label of the classification branch of model B, and the high-confidence prediction results generated by the classification branch of model B are used as the pseudo-label of the classification branch of model A; Iteratively train the model parameters for a certain number of rounds, and return the best parameters of models A and B on the test set as the final source domain model parameters for the segmentation and classification of polyps in source domain endoscopic images; S9. Using the semi-supervised training module of the polyp segmentation and classification model, the labeled image samples and unlabeled image samples of the target domain training set are mixed as the initial training set of the target domain polyp segmentation and classification model; two different parameter initializations are performed on the constructed target domain polyp segmentation and classification model, model A uses the parameters of the source domain model as initialization, while model B uses the exponential moving average of the source domain parameters for initialization, and two target domain polyp segmentation and classification models A and B with exactly the same structure but different parameters are obtained; during the training process, for labeled samples, both models A and B use the real segmentation mask and classification label for supervised training; for unlabeled samples, the two models will make predictions separately, and the high-confidence prediction results generated by the segmentation branch of model A will be used as the pseudo-label of the segmentation branch of model B, and similarly, the high-confidence prediction results generated by the segmentation branch of model B will also be used as the pseudo-label of the segmentation branch of model A; the classification branch also adopts the same cross-supervision strategy, the high-confidence prediction results generated by the classification branch of model A are used as the pseudo-label of the classification branch of model B, and the high-confidence prediction results generated by the classification branch of model B are used as the pseudo-label of the classification branch of model A; The model parameters are iteratively trained for a certain number of rounds, and the parameters of models A and B that perform best on the test set are returned as the final target domain model parameters for the segmentation and classification of polyps in target domain endoscopic images.

Images