Endoscope image data enhancement method and system based on adversarial network

Abstract

The present application provides an endoscope image data enhancement method and system based on a generative adversarial network, comprising: acquiring a multi-domain medical data set, importing a medical feature extraction model to obtain a sample feature vector, acquiring high-value rare samples based on the sample feature vector, and performing feature space enhancement; encapsulating the conventional samples and the high-value rare samples as a training data set; based on the training data set, constructing and training an anatomical domain encoder and a pathology instance encoder; constructing a meta-domain image enhancement model; based on the training data set, acquiring a meta task using the anatomical domain encoder and the pathology instance encoder; training the meta-domain image enhancement model based on the meta task; acquiring a new domain medical data set; and acquiring a new domain synthetic data set using the anatomical domain encoder, the pathology instance encoder and the meta-domain image enhancement model. The present application realizes an endoscope image data enhancement method that intelligently mines high-value rare samples to improve data distribution and efficiently adapts to new domain styles.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method and system for enhancing endoscopic image data based on adversarial networks. Background Technology

[0002] Acquisition and annotation of endoscopic image data is a crucial foundational step in the field of medical image analysis. However, constructing high-quality, large-scale datasets faces significant technical challenges. First, endoscopic image data inherently suffers from severe class imbalance. During clinical acquisition, the number of normal or common lesion samples is enormous, while the number of rare case images with high medical research value is relatively small. This imbalance in data distribution makes it difficult for models trained on such data to fully learn and represent rare patterns. Traditional data augmentation methods only perform low-level expansion in the image space and cannot create samples with new semantic features, thus failing to improve the data distribution imbalance problem.

[0003] On the other hand, medical imaging data has significant multi-domain differences. Endoscopic images from different sources have significant differences in color style, texture and resolution. When a dataset built on one data source is directly applied to another data source, its effectiveness will be significantly reduced due to inter-domain differences. Existing technologies usually require collecting and manually labeling a large amount of data in the new domain for adaptation, which is costly, inefficient and difficult to implement on a large scale.

[0004] In summary, existing endoscopic image data augmentation techniques are limited in their ability to construct high-quality, highly generalizable datasets due to their inability to address data distribution imbalances and adapt to differences across multiple domains. There is an urgent need in this field for a data augmentation method that can improve data distribution and efficiently adapt to new domain styles. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for enhancing endoscopic image data based on adversarial networks.

[0006] The first aspect of this invention provides a method for enhancing endoscopic image data based on adversarial networks, comprising:

[0007] Obtain a multi-domain medical dataset and import it into a medical feature extraction model to obtain sample feature vectors.

[0008] Based on the sample feature vector, the isolated forest model is used to obtain regular samples and candidate anomaly samples. The candidate anomaly samples are imported into the rare validation model, and high-value rare samples are obtained. Feature space enhancement is performed on the high-value rare samples, and regular samples and high-value rare samples are encapsulated into a training dataset.

[0009] Based on the training data set, an anatomical domain encoder and a pathology instance encoder are constructed and trained;

[0010] An element domain image enhancement model is constructed, and based on the training data set, the anatomical domain encoder and the pathology instance encoder are used to obtain a task element loss, and the element domain image enhancement model is trained based on the task element loss;

[0011] A new domain medical data set is obtained, and based on the new domain medical data set, the anatomical domain encoder is used to obtain a new domain anatomical domain code, the pathology instance encoder is used to obtain a target instance code, the new domain anatomical domain code and the target instance code are imported into the element domain image enhancement model, and a new domain synthetic data set is obtained.

[0012] According to a preferred embodiment, a multi-domain medical data set is obtained, and the multi-domain medical data set is imported into a medical feature extraction model to obtain a sample feature vector, comprising:

[0013] A multi-domain endoscopic image and corresponding image annotation information are obtained, and the multi-domain endoscopic image and corresponding image annotation information are standardized;

[0014] The image annotation information at least includes a boundary box, a segmentation mask and a pathology category label;

[0015] The multi-domain endoscopic image and the corresponding image annotation information are packaged into a multi-domain medical data set;

[0016] The pre-trained feature extraction network is used as the medical feature extraction model;

[0017] The multi-domain endoscopic image in the multi-domain medical data set is imported into the medical feature extraction model, and a sample feature vector corresponding to each multi-domain endoscopic image is obtained.

[0018] According to a preferred embodiment, based on the sample feature vector, an isolated forest model is used to obtain a regular sample and a candidate abnormal sample, the candidate abnormal sample is imported into a rare verification model, and a high-value rare sample is obtained, the high-value rare sample is subjected to feature space enhancement, and the regular sample and the high-value rare sample are packaged into a training data set, comprising:

[0019] The sample feature vector is input into the isolated forest model, an abnormal score corresponding to the sample feature vector is obtained, and a regular sample and a candidate abnormal sample are obtained according to the abnormal score;

[0020] The sample feature vector of the candidate abnormal sample is input into the rare verification model, a rareness confidence score is obtained, and a corresponding sample label is generated for the candidate abnormal sample according to a preset rareness threshold;

[0021] The rare verification model includes a reconstruction quality verification layer, a generation consistency verification layer and a clinical relevance proxy verification layer;

[0022] The sample label includes a high-value rare sample, a sample to be reviewed, and a noise sample;

[0023] The high-value rare sample is processed by using a latent space enhancement technology to obtain an enhanced high-value rare sample;

[0024] The conventional sample and the enhanced high-value rare sample are packaged as a training data set.

[0025] According to a preferred embodiment, the sample feature vector of the candidate abnormal sample is input into a rare verification model to obtain a rare confidence score, and a corresponding sample label is generated for the candidate abnormal sample according to a preset rare threshold, including:

[0026] The candidate abnormal sample is introduced into a reconstruction quality verification layer, and a pre-trained reconstruction autoencoder is used to decode the candidate abnormal sample to obtain an abnormal sample reconstruction image;

[0027] A reconstructed lesion area is obtained based on the abnormal sample reconstruction image, a corresponding original lesion area is obtained based on the multi-domain endoscopic images in the multi-domain medical data set, and a structural similarity index is obtained based on the reconstructed lesion area and the original lesion area as a first verification score;

[0028] The candidate abnormal sample is introduced into a consistency verification layer, and a plurality of verification sample feature vectors are generated by feature interpolation or adding noise based on the candidate abnormal sample, and the verification sample feature vectors are decoded into verification sample images;

[0029] A pre-trained lesion morphology classifier is used to classify the verification sample images to obtain a statistical uncertainty of the classification result, and a second verification score is generated based on the statistical uncertainty;

[0030] The candidate abnormal sample is introduced into a clinical relevance proxy verification layer, and an abnormal sample morphological index is obtained based on the candidate abnormal sample, and a reference morphological index distribution is obtained based on the conventional sample;

[0031] The Mahalanobis distance of the abnormal sample morphological index and the reference morphological index distribution is obtained, and a third verification score is generated based on the Mahalanobis distance;

[0032] Based on the first verification score, the second verification score, and the third verification score, a corresponding rare confidence score is generated for the candidate abnormal sample.

[0033] According to a preferred embodiment, based on the training data set, an anatomical domain encoder and a pathology instance encoder are constructed and trained, including:

[0034] The anatomical domain encoder is trained based on a training data set through a contrast loss and an adversarial loss, the anatomical domain encoder takes an endoscope image as input, and outputs a low-dimensional vector representing anatomical domain data as an anatomical domain feature vector;

[0035] The pathology instance encoder is trained based on a training data set through a focal loss and a correlation constraint, the pathology instance encoder takes a lesion area of an endoscope image as input, and outputs a low-dimensional vector representing pathology instance data as a pathology instance feature vector.

[0036] According to a preferred embodiment, a meta-domain image enhancement model is constructed, based on a training data set, a task meta loss is obtained by using an anatomical domain encoder and a pathology instance encoder, and the meta-domain image enhancement model is trained based on the task meta loss, comprising:

[0037] According to a meta-learning task paradigm, a training data set is sampled to obtain a plurality of meta-tasks, each meta-task including a support set and a query set;

[0038] For the current meta-task, the anatomical domain encoder is used to obtain the target domain code based on the support set;

[0039] Based on the query set and the corresponding target domain code, an image generation and adversarial training process is performed to obtain a meta-image generation and adversarial network adapted to the current meta-task, and to obtain a task meta loss of the meta-image generation and adversarial network on the query set;

[0040] The meta-image generation and adversarial networks adapted to all meta-tasks and the corresponding task meta losses are obtained, and based on all the task meta losses, a training meta loss is obtained;

[0041] Based on the training meta loss, a meta-optimization algorithm is used to update the meta parameters of the meta-domain image enhancement model, and the iteration is performed until a preset convergence criterion is met.

[0042] According to a preferred embodiment, based on the query set and the corresponding target domain code, an image generation and adversarial training process is performed to obtain a meta-image generation and adversarial network adapted to the current meta-task, and to obtain a task meta loss of the meta-image generation and adversarial network on the query set, comprising:

[0043] The meta-image generation and adversarial network includes a meta-image generator and a meta-image discriminator;

[0044] Based on the query set, a pathology instance feature vector is obtained by using a pathology instance encoder;

[0045] The target domain code and the pathology instance feature vector are input into the meta-image generator as conditions to generate an adversarial reconstruction image;

[0046] The adversarial reconstruction image and the endoscope image corresponding to the query set are input into the meta-image discriminator to obtain an adversarial loss;

[0047] The adversarial reconstruction image and the endoscope image corresponding to the query set are input into the meta-image discriminator to obtain an adversarial loss;

[0048] The adversarial loss and the reconstruction image loss are used to update the meta-image generator and the meta-image discriminator, and the meta-image generation adversarial network is obtained through current meta-task adaptation.

[0049] According to a preferred embodiment, the parameters of the meta-domain image enhancement model are updated based on the meta-loss using a meta-optimization algorithm, and the iteration is performed until a preset convergence criterion is met, including:

[0050] For each meta-task, the meta-loss gradient of the training meta-loss to the parameters of the meta-domain image enhancement model is obtained, and the parameters of the meta-domain image enhancement model are updated based on the meta-loss gradient;

[0051] When the training meta-loss triggers any one of the threshold convergence condition, the iteration convergence condition and the performance convergence condition in continuous multiple iterations, it is determined that the training of the meta-domain image enhancement model converges.

[0052] According to a preferred embodiment, a new domain medical data set is obtained, a new domain anatomical domain code is obtained based on the new domain medical data set using an anatomical domain encoder, a target instance code is obtained based on a pathological instance encoder, the new domain anatomical domain code and the target instance code are imported into the meta-domain image enhancement model, and a new domain synthetic data set is obtained, including:

[0053] The new domain endoscope image in the new domain medical data set is imported into the medical feature extraction model to obtain a new domain sample feature vector corresponding to each new domain endoscope image;

[0054] The average new domain code is obtained based on the new domain sample feature vector using the anatomical domain encoder;

[0055] The expected pathological parameter is obtained, and the expected pathological code is obtained based on the expected pathological parameter using the pre-trained semantic mapping model;

[0056] The average new domain code and the expected pathological code are imported into the meta-image generator in the meta-domain image enhancement model as conditional data to generate a new domain synthetic data set.

[0057] The second aspect of the present application also provides an endoscope image data enhancement system based on an adversarial network, including:

[0058] The data processing and enhancement module is configured to obtain a multi-domain medical data set and import the medical data set into a medical feature extraction model, obtain a sample feature vector, obtain a normal sample and a candidate abnormal sample based on the sample feature vector by using an isolation forest model, import the candidate abnormal sample into a rare verification model to obtain a high-value rare sample, and perform feature space enhancement, and encapsulate the normal sample and the high-value rare sample into a training data set;

[0059] The encoder training module is configured to construct and train an anatomical domain encoder and a pathology instance encoder based on the training data set.

[0060] The meta-domain image enhancement model module is configured to construct a meta-domain image enhancement model, obtain a task meta loss by using the anatomical domain encoder and the pathology instance encoder based on the training data set, and train the meta-domain image enhancement model based on the task meta loss.

[0061] The new domain synthetic data module is configured to obtain a new domain medical data set, obtain a new domain anatomical domain code by using the anatomical domain encoder, obtain a target instance code based on the pathology instance encoder, and import the new domain anatomical domain code and the target instance code into the meta-domain image enhancement model to obtain a new domain synthetic data set.

[0062] Based on the above aspects, the embodiments of the present application filter out high-value rare samples and perform feature space enhancement by using an isolation forest model and a rare verification model, construct a balanced training data set, and overcome the limitation that a traditional enhancement method cannot generate new semantic content by intelligently mining and enhancing rare samples, thereby effectively solving the problem of unbalanced data distribution in endoscopic data and providing a more balanced data basis for model training.

[0063] On the other hand, the image features are decoupled by constructing an anatomical domain encoder and a pathology instance encoder, and a meta-domain image enhancement model is trained by using a meta-learning framework, so that the model has the ability to quickly adapt to the data features of a new domain. This method can generate synthetic images that are highly adapted in terms of anatomical style and clear in terms of pathological features by using only a small amount of new domain samples, thereby reducing the dependence on large-scale annotation of new domain data and effectively overcoming the problem of poor data availability caused by domain shift. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 The execution flowchart of the endoscopic image data enhancement method based on the adversarial network of the present application is shown.

[0065] Figure 2 The flowchart of the meta-learning training paradigm in the endoscopic image data enhancement method based on the adversarial network of the present application is shown.

[0066] Figure 3 The operation flow chart of the meta-domain image enhancement model in the endoscope image data enhancement method based on the adversarial network is shown.

[0067] Figure 4 The schematic diagram of the endoscope image data enhancement system based on the adversarial network is shown. DETAILED DESCRIPTION

[0068] In order to enable the above-mentioned purposes, features and advantages of the present application to be more clearly understood, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0070] In the following description, many specific details are set forth in order to provide a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the present application is not limited by the specific embodiments disclosed below.

[0071] As shown in Figure 1 , Figure 2 , Figure 3 , Figure 4 shown:

[0072] The first aspect of the present application provides an endoscope image data enhancement method based on an adversarial network, comprising:

[0073] Step S1: acquiring a multi-domain medical data set, and importing the multi-domain medical data set into a medical feature extraction model to obtain a sample feature vector.

[0074] In step S1, the following steps are included:

[0075] Step S11: acquiring multi-domain endoscope images and corresponding image annotation information, and performing standardization processing on the multi-domain endoscope images and the corresponding image annotation information.

[0076] The image annotation information at least includes a boundary box, a segmentation mask and a pathological category label.

[0077] Specifically, the multi-domain endoscopic images refer to a collection of endoscopic images collected from multiple different sources, covering different medical institutions, different endoscopic equipment models, and different anatomical sites, etc., to capture the inter-domain differences existing in clinical practice, such as lighting conditions, resolution differences, and lesion manifestation diversity. The corresponding image annotation information can be obtained through professional medical image annotation databases and platforms. The image annotation information at least includes a bounding box for positioning the lesion area, a segmentation mask for pixel-level segmentation of the lesion, and a pathology category label for identifying the pathological nature, such as normal, inflammation, polyp, etc. The pathology category label is determined by the specific lesion area. For example, assuming for colonoscopy, the multi-domain endoscopic images can include colonoscopy images and corresponding image annotation information collected from hospital 1's A-type colonoscope system, colonoscopy images and corresponding image annotation information collected from hospital 2's B-type colonoscope system, and colonoscopy images and corresponding image annotation information in the public dataset Kvasir-SEG.

[0078] Further, the multi-domain endoscopic images can be scaled to a fixed resolution, such as 640x480 pixels, using a bilinear interpolation algorithm to eliminate image size differences caused by different devices. A white balance algorithm can be used to color correct the multi-domain endoscopic images to make the color distribution of the multi-domain endoscopic images consistent and reduce the impact of light source differences. For example, for each multi-domain endoscopic image, the mean value of the RGB channels is calculated and adjusted to an equalization value to standardize the brightness range of the corrected image to [0, 255]. The annotation information is converted to a unified format, such as storing the bounding box as a coordinate list in COCO format, saving the segmentation mask as a PNG binary image, and mapping the pathology category label to an integer code, such as normal = 0, inflammation = 1, polyp = 2. The standardized multi-domain endoscopic images and corresponding image annotation information are obtained, providing a consistent and standardized data basis for subsequent steps.

[0079] In some possible embodiments, for an endoscopic image from hospital 1's A-type colonoscope system with a resolution of 1280x1024 pixels, after standardization, its resolution is unified to 640x480 pixels, the color deviation is corrected, the color cast caused by the difference in color temperature of the endoscopic light source is eliminated, and the tissue color is closer to the true performance. The corresponding polyp bounding box is converted from the original annotation to the normalized coordinate format of COCO standard, such as [0.35, 0.42, 0.15, 0.12], indicating that the polyp bounding box is located at 35% of the image width, 42% of the image height, 15% of the image width, and 12% of the image height. The corresponding segmentation mask is saved as a single-channel PNG binary image, where the polyp region pixel value is 255 and the background region pixel value is 0.

[0080] Step S12: encapsulate the multi-domain endoscopic images and the corresponding image annotation information into a multi-domain medical data set.

[0081] Specifically, the multi-domain endoscopic images and the image annotation information after standardization are organized into a structured data set in a one-to-one mapping relationship, and are used as a multi-domain medical data set. The data set format can be stored in JSON or HDF5 standard format. Each piece of data in the multi-domain medical data set should include a multi-domain endoscopic image storage path, corresponding image annotation information, and a domain identifier. The domain identifier at least includes a device ID or a hospital code. The domain identifier is used to provide clear domain information for the subsequent training process of the anatomical domain encoder and the meta-domain image enhancement model.

[0082] Step S13: use the pre-trained feature extraction network as the medical feature extraction model.

[0083] Specifically, the medical feature extraction model can use a convolutional neural network pre-trained on a large general-purpose computer vision data set such as ImageNet, such as ResNet-50, EfficientNet-B4, etc. Through pre-training, the medical feature extraction model has the ability to recognize general visual features such as edges, textures, and shapes. Using such a pre-trained convolutional neural network as a medical feature extraction model and performing transfer learning can improve the initialization performance of the medical feature extraction model on medical image tasks and the generalization performance on limited endoscopic images, and can alleviate the overfitting problem caused by limited image annotation information.

[0084] Further, the medical feature extraction model is fine-tuned based on professional endoscopic data sets such as Kvasir and HyperKvasir. A smaller learning rate such as 1e-4 is used in the fine-tuning process. The medical feature extraction model is further trained on the endoscopic data set to make its weight parameters better adapt to the feature distribution of the endoscopic images. For example, a ResNet-50 model pre-trained on ImageNet is fine-tuned using the Kvasir data set, which is a medical image segmentation data set containing polyp images and corresponding pixel-level annotations, commonly used for automatic polyp segmentation research. After fine-tuning with the Kvasir data set, the model weight parameters are adjusted, so that the medical feature extraction model has a higher response to medical features such as lesion structure and texture, thereby extracting more targeted sample feature vectors.

[0085] Step S14: import the multi-domain endoscopic images in the multi-domain medical data set into the medical feature extraction model to obtain the sample feature vector corresponding to each multi-domain endoscopic image.

[0086] Specifically, the multi-domain endoscope image is input into the pre-trained and fine-tuned medical feature extraction model for forward propagation, and high-level semantic features can be extracted from the last hidden layer or global pooling layer of the medical feature extraction model, which are represented as a low-dimensional dense vector, i.e., a sample feature vector. The sample feature vector is specifically an encoding of the multi-domain endoscope image. For example, for a medical feature extraction model based on ResNet-50, a 2048-dimensional sample feature vector is obtained by inputting a multi-domain endoscope image, which represents deep features in the multi-domain endoscope image. For example, a gastroscope image with a pathological class label of polyp, the sample feature vector extracted by the medical feature extraction model highlights the medical features corresponding to the polyp. A gastroscope image labeled as normal, its corresponding sample feature vector reflects the medical features corresponding to normal. The distance between these high-dimensional feature vectors can be calculated by methods such as cosine similarity to quantify the similarity of different multi-domain endoscope images in the feature space.

[0087] Step S2: obtaining normal samples and candidate abnormal samples based on the sample feature vectors using an isolation forest model, importing the candidate abnormal samples into a rare verification model, and obtaining high-value rare samples, performing feature space enhancement on the high-value rare samples, and packaging the normal samples and the high-value rare samples as a training data set.

[0088] Step S2 includes:

[0089] Step S21: inputting the sample feature vector into the isolation forest model to obtain an abnormal score corresponding to the sample feature vector, and obtaining normal samples and candidate abnormal samples according to the abnormal score;

[0090] Specifically, the sample feature vector is input into the isolation forest model, which constructs multiple isolated trees by randomly selecting the dimension of the sample feature vector and a split value in that dimension to calculate the abnormal score of each sample feature vector. The abnormal score reflects the degree of deviation of the sample from the normal distribution, and the score ranges from 0 to 1. The closer the score is to 1, the more likely the sample is an abnormal sample. According to a preset abnormal score threshold, the sample feature vector is divided into normal samples and candidate abnormal samples. For example, set the abnormal score threshold to 0.7, and mark the samples with a score lower than 0.7 as normal samples, indicating that they conform to the normal distribution, and mark the samples with a score higher than or equal to 0.7 as candidate abnormal samples.

[0091] It can be understood that the isolation forest model is based on the principle that the isolated forest model is more likely to be randomly isolated based on abnormal data, and realizes the preliminary screening of the large-scale sample feature vector. The method avoids the calculation bottleneck caused by blindly mining high-value rare samples in large data, reduces the data amount required by the subsequent rare verification model, and thus improves the process efficiency of mining high-value rare samples from a large amount of medical data as a whole.

[0092] In some possible embodiments, assuming that there are 10000 sample feature vectors, after importing the isolation forest model for calculation, about 9500 samples with an abnormal score lower than 0.7 are divided into normal samples, and 500 samples with an abnormal score higher than 0.7 are divided into candidate abnormal samples.

[0093] Step S22: inputting the sample feature vector of the candidate abnormal sample into the rare verification model, obtaining a rareness confidence score, and generating a corresponding sample label for the candidate abnormal sample according to a preset rareness threshold.

[0094] It should be noted that the rare verification model includes a reconstruction quality verification layer, a generation consistency verification layer and a clinical relevance proxy verification layer, and the sample label includes a high-value rare sample, a sample to be reviewed and a noise sample.

[0095] Step S22 includes:

[0096] Step S22-1: importing the candidate abnormal sample into the reconstruction quality verification layer, and decoding the candidate abnormal sample based on a pre-trained reconstruction autoencoder to obtain an abnormal sample reconstruction image.

[0097] Specifically, a reconstruction autoencoder is trained based on the normal samples obtained in step S21. The reconstruction autoencoder can adopt an encoder-decoder structure. The encoder is a neural network including three fully connected layers, the input dimension of which is the same as the dimension of the sample feature vector, and an output is a latent space vector, such as 128 dimensions. The decoder is a symmetric structure, which reconstructs the latent space vector into an image with the same size as the corresponding multi-domain endoscope image, and serves as an abnormal sample reconstruction image. The training data is the sample feature vector and the corresponding original multi-domain endoscope image of all normal samples obtained in step S21. The training target is to minimize the error, such as the MSE loss, between the normal reconstruction image and the corresponding multi-domain endoscope image. The reconstruction autoencoder learns the mapping relationship from the sample feature space to the image space for the normal samples.

[0098] Further, the sample feature vector of the candidate abnormal sample is input into the reconstruction autoencoder to generate an abnormal sample reconstruction image. Since the reconstruction autoencoder is trained on normal samples, for abnormal samples, the reconstruction image may not accurately restore the corresponding multi-domain endoscopic image, thereby revealing abnormal characteristics.

[0099] Step S22-2: Obtain a reconstructed lesion area based on the abnormal sample reconstruction image, obtain a corresponding original lesion area based on the multi-domain medical data set, and obtain a structural similarity index based on the reconstructed lesion area and the original lesion area as a first verification score.

[0100] Specifically, for each candidate abnormal sample, the original lesion area is extracted based on the segmentation mask corresponding to the multi-domain endoscopic image, and the corresponding reconstructed lesion area is extracted from the reconstruction image. The structural similarity index of the original lesion area and the reconstructed lesion area is calculated. The structural similarity index can measure the similarity of the images in brightness, contrast and structure. The value of the structural similarity index ranges from 0 to 1. The structural similarity index is taken as the first verification score. A low structural similarity index indicates poor reconstruction quality, suggesting that the sample may have rare characteristics.

[0101] Step S22-3: Introduce the candidate abnormal sample into the consistency verification layer to generate a plurality of verification sample feature vectors by feature interpolation or adding noise based on the candidate abnormal sample, and decode the verification sample feature vectors into verification sample images.

[0102] Specifically, the sample feature vector of the candidate abnormal sample is slightly transformed, such as adding weak Gaussian noise to each dimension of the sample feature vector, or linearly interpolating with the nearest k neighbor sample feature vectors, to generate a plurality of validation sample feature vectors, and a consistency decoder is trained, which can adopt a similar or identical image decoder structure as the reconstruction autoencoder in step S22-1, and is trained with the mean square error of the sample feature vector corresponding to the normal sample and the multi-domain endoscope image as the reconstruction loss, to learn a stable and smooth mapping relationship from the feature space to the image space, so that the reconstruction autoencoder can decode the sample feature vectors of the same semantics into images with consistent visual content. The input of the reconstruction autoencoder is the validation sample feature vector, and the output is the validation sample image. In the generation of consistency verification, the consistency decoder is used to decode the validation sample feature vector into the validation sample image, so as to evaluate the stability of the sample in the feature space. If the validation sample image and the multi-domain endoscope image corresponding to the candidate abnormal sample remain semantic consistency, it indicates that the candidate abnormal sample has stable pathological characteristics. If the validation sample image has a large difference, it indicates that it may be a noise sample. The structural similarity index or peak signal-to-noise ratio between the validation sample image and the corresponding multi-domain endoscope image can be calculated to evaluate whether they remain semantic consistency or have a large difference. For example, assuming that the structural similarity index threshold is 0.8, if the structural similarity index between the validation sample image and the corresponding multi-domain endoscope image is 0.58, which is lower than the structural similarity index threshold, it is determined that they have a large difference, and the candidate abnormal sample may be a noise sample.

[0103] It can be understood that the candidate abnormal sample due to its actual pathological significance should be able to maintain semantic consistency in the local field of the feature space. If a slight disturbance can cause the essential features of the decoded image, such as a large variation in lesion morphology, it is inferred that the current candidate abnormal sample may be noise.

[0104] Step S22-4: The pre-trained lesion morphology classifier is used to classify the validation sample image, and the statistical uncertainty of the classification result is obtained. The second verification score is generated based on the statistical uncertainty.

[0105] Specifically, a pre-trained lesion morphology classifier is used, which can adopt a ResNet-18 model pre-trained on public datasets such as ImageNet, and replace the last fully connected layer with one that adapts to the number of task categories, such as normal, inflammation, polyp, etc. Then, on public medical image datasets such as Kvasir dataset, the lesion morphology classifier is fine-tuned using image blocks of lesion regions and corresponding pathological category labels, with the training target being to minimize the cross-entropy loss. The fine-tuned lesion morphology classifier is used to classify the validation sample images, obtain the probability distribution of each validation sample image, and obtain the entropy or variance of the predicted probability based on the probability distribution to quantify the statistical uncertainty. The higher the entropy value, the more uncertain the lesion morphology classifier is in judging the category of the validation sample image. This high uncertainty may be associated with high-value rare samples. The mean of the statistical uncertainty of all validation sample images is calculated as the second verification score. For example, assuming that the statistical uncertainty is quantified by the entropy of the predicted probability, and the average entropy of the validation sample images generated by a candidate abnormal sample is 0.8, then the second verification score of the candidate abnormal sample is 0.8.

[0106] Step S22-5: importing the candidate abnormal sample into the clinical relevance agent verification layer, obtaining an abnormal sample morphological index based on the candidate abnormal sample, and obtaining a baseline morphological index distribution based on the conventional sample.

[0107] Specifically, based on the segmentation mask of the multi-domain endoscopic image corresponding to the candidate abnormal sample, the morphological index is obtained and used as the abnormal sample morphological index. The morphological index can include the area of the lesion region, circularity, etc. Based on the segmentation mask corresponding to the conventional sample, the morphological index is obtained and can be used as the baseline morphological index distribution by calculating the covariance of the morphological index.

[0108] Step S22-6: obtaining the Mahalanobis distance of the abnormal sample morphological index and the baseline morphological index distribution, and generating a third verification score based on the Mahalanobis distance.

[0109] Specifically, the Mahalanobis distance between the abnormal sample morphological index and the baseline morphological index distribution is calculated. The Mahalanobis distance takes into account the covariance structure of the distribution and can measure the degree of deviation of the sample from the conventional morphology. The larger the Mahalanobis distance, the more likely the candidate abnormal sample is rare. The Mahalanobis distance is normalized and used as the third verification score.

[0110] Step S22-7: generating a corresponding rareness confidence score for the candidate abnormal sample based on the first verification score, the second verification score, and the third verification score.

[0111] Specifically, the first verification score, the second verification score and the third verification score are normalized to the same dimension, for example, the interval [0, 1], to ensure that the first verification score, the second verification score and the third verification score are consistent in representing the rarity. The first verification score is converted to 1-first verification score, and the greater the value, the higher the rarity. Then, through weighted fusion and other methods, the rarity confidence score is obtained. For example, the first verification score, the second verification score and the third verification score are normalized to the range of [1, 10], and the weight can be set as the first verification score 0.4, the second verification score 0.3 and the third verification score 0.3. The weight distribution aims to emphasize the basic role of reconstruction quality in the evaluation of rarity. Then, the rarity confidence score = 0.4 x the first verification score + 0.3 x the second verification score + 0.3 x the third verification score.

[0112] Further, after obtaining the rarity confidence score, the corresponding sample label is generated for the candidate abnormal sample according to the preset rarity threshold. For example, the threshold of high-value rare sample is set to 0.7, and the threshold of noise sample is set to 0.3. If the rarity confidence score is greater than or equal to 0.7, it is marked as a high-value rare sample, indicating that the sample has a highly reliable rare feature. If the rarity confidence score is less than or equal to 0.3, it is marked as a noise sample, which can be removed to purify the data set and improve the stability and performance of subsequent model training. If the rarity confidence score is between 0.3 and 0.7, it is marked as a sample to be reviewed, which can be retained for further review by experts in the field to balance the efficiency of automated processing and the accuracy of decision-making.

[0113] Step S23: The high-value rare sample is processed using a latent space enhancement technique to obtain an enhanced high-value rare sample.

[0114] Specifically, the latent space enhancement technique increases the diversity of each high-value rare sample by transforming it in the feature space. For example, a generative adversarial network or a variational autoencoder can be used to interpolate or add noise near the sample feature vector of the high-value rare sample to generate an enhanced feature vector, which is then decoded into an enhanced image. The enhancement process preserves the rare characteristics of the high-value rare sample and generates more variants to balance the distribution of the data set. This step is used to expand the number of rare samples. For example, for a sample feature vector of a high-value rare sample, five enhanced feature vectors are generated through linear interpolation, and five enhanced images are decoded. These enhanced images are similar to the high-value rare sample in the feature space but have subtle changes.

[0115] Step S24: The conventional samples and the enhanced high-value rare samples are packaged as a training data set.

[0116] Specifically, the conventional samples obtained in step S21 are merged with the enhanced high-value rare samples obtained in step S23 to form a training data set, the training data set is encapsulated in a standard format such as HDF5 or JSON, and the training data set contains multi-domain endoscopic image storage paths, corresponding image annotation information and domain identifiers. The training data set is used for subsequent model training to ensure that the data has both conventionality and rarity, maintains the overall data distribution close to the real world to maintain the basic performance of the model, and balances the class distribution of the data by introducing a proper amount of enhanced high-value rare samples to improve the recognition ability of the model to rare pathological patterns and the overall generalization ability. For example, the conventional samples account for 90% of the training data set, and the enhanced high-value rare samples account for 10%. This proportion setting aims to effectively alleviate the scarcity of rare samples in the data set, while avoiding distorting the model's learning of conventional pathological patterns due to the high proportion of rare samples, and is not intended to accurately simulate the real clinical incidence rate distribution.

[0117] It should be noted that during the encapsulation process, the conventional samples retain the corresponding multi-domain endoscopic image paths, image annotation information and domain identifiers. For the enhanced high-value rare samples generated in step S23, the image annotation information and domain identifiers are inherited from the corresponding high-value rare samples to ensure that the enhanced samples are consistent with the enhanced high-value rare samples in terms of semantics and domain attributes.

[0118] Step S3: Based on the training data set, an anatomical domain encoder and a pathological instance encoder are constructed and trained.

[0119] Step S3 includes:

[0120] Step S31: The anatomical domain encoder is trained based on the training data set by comparing the loss and the adversarial loss. The anatomical domain encoder takes the endoscopic image as input and outputs a low-dimensional vector representing the anatomical domain data as an anatomical domain feature vector.

[0121] It should be noted that the anatomical domain encoder adopts a convolutional neural network structure, and its input is a standardized multi-domain endoscopic image and a corresponding domain identifier, such as an RGB image with a size of 224x224 pixels. The output is a low-dimensional vector used to represent the anatomical domain features of the image, such as scene light, imaging device characteristics, and overall anatomical structure of the shooting part. The anatomical domain encoder is trained using a combination of contrast loss and adversarial loss to ensure that the output low-dimensional vector can distinguish between different domains while maintaining the compactness of the features within the domain.

[0122] Specifically, for each image sample in the training dataset, it is compared with other image samples in the same domain and image samples in different domains, and the optimization goal is to minimize the distance, such as the cosine distance, of the sample feature vectors of the same domain identifier in the feature space, while maximizing the distance of the sample feature vectors of different domain identifiers, so that the anatomical domain encoder learns a feature representation sensitive to domain changes.

[0123] Further, the adversarial loss is realized by introducing a domain discriminator, which takes the low-dimensional vector output by the anatomical domain encoder as input to determine the domain of the vector, and the training goal of the anatomical domain encoder is to generate a low-dimensional vector that is difficult for the domain discriminator to determine the domain, which promotes the anatomical domain encoder to learn anatomical structure features that are more general and irrelevant to specific domain information, and suppresses noise information that is strongly related to a specific device or hospital.

[0124] By combining the contrast loss and the adversarial loss for end-to-end training, the anatomical domain encoder can finally extract a low-dimensional vector that has certain domain discrimination ability and does not rely too much on domain-specific details. The dimension of the vector can be set according to requirements, such as 128 or 256.

[0125] Step S32: The pathological instance encoder is trained based on the training dataset by focal loss and correlation constraint. The pathological instance encoder takes the lesion area of the endoscopic image as input and outputs a low-dimensional vector representing the pathological instance data as a pathological instance feature vector.

[0126] It should be noted that the pathological instance encoder adopts a convolutional neural network structure, and its input is the lesion area obtained based on the segmentation mask or the bounding box in the image annotation information, and its output is a low-dimensional vector used to represent the features of the pathological instance data, i.e., the local attributes of the lesion area itself, such as its morphology, texture, and boundary, which are closely related to the pathological properties.

[0127] Specifically, the training of the pathological instance encoder is mainly based on the pathological category label corresponding to the lesion area, and the focal loss is used as the main supervision signal. The focal loss is an improved form of cross-entropy loss, which effectively solves the problem of uneven number of samples of different categories in the training data by introducing an adjustable focusing parameter. For example, the number of normal tissue regions is much larger than that of rare lesion regions. The focal loss reduces the loss contribution of correctly classified and easily classified samples, so that the model pays more attention to difficult-to-classify samples during the training process. The difficult-to-classify samples may be rare classes or atypical lesions, thereby improving the feature extraction ability of the pathological instance encoder for rare or complex pathological instances.

[0128] Further, a correlation constraint is introduced in the training process, which is that in the feature space, the sample feature vectors of the training data with the same pathological class label should have high similarity, i.e. small intra-class distance, while the sample feature vectors of the training data with different pathological class labels should have great difference, i.e. large inter-class distance. The correlation constraint can be realized by adding a term based on the distance of sample feature vectors in the loss function, such as triplet loss or center loss. By introducing the correlation constraint, the representation ability of the output low-dimensional vector for the pathological instance data is improved.

[0129] It can be understood that through the joint action of the focus loss and the correlation constraint, the pathological instance encoder can learn a feature representation that is discriminative for the lesion area, which can highlight the key characteristics of the pathological instance and remain sensitive to the class variation. The dimension of the output low-dimensional vector can be set, for example, 64 or 128.

[0130] In summary, step S3 trains two functionally complementary encoders. The anatomical domain encoder focuses on global, domain-related anatomical background features, and the pathological instance encoder focuses on local, lesion area-related pathological features. The low-dimensional vectors output by the anatomical domain encoder and the pathological instance encoder will provide key conditional inputs for the construction of the meta-domain image enhancement model in subsequent step S4.

[0131] Step S4: Constructing a meta-domain image enhancement model, based on the training data set, using the anatomical domain encoder and the pathological instance encoder to obtain the task meta-loss, and training the meta-domain image enhancement model based on the task meta-loss.

[0132] Wherein, step S4 includes:

[0133] Step S41: Sampling the training data set according to the meta-learning task paradigm to obtain multiple meta-tasks, each meta-task containing a support set and a query set.

[0134] It should be noted that before sampling starts, the training data set is randomly divided into a meta-training set and a meta-validation set. The meta-training set is used for meta-task sampling and meta-parameter updating in the training process of the meta-domain image enhancement model, the meta-validation set does not participate in the training, and is used to determine whether the meta-domain image enhancement model is over-fitted and to decide when the training converges.

[0135] Specifically, the meta-learning task paradigm adopts an optimized meta-learning framework, such as model-agnostic meta-learning, and the core goal is to enable the meta-domain image enhancement model to quickly adapt to the image enhancement task of the target domain through a small number of iterative steps by training with a small amount of samples of the target domain. When sampling the meta-training set, a domain identifier-based sampling strategy is adopted, specifically, all samples in the support set and query set of the meta-task must come from the same domain, that is, have the same domain identifier, such as the same device ID or hospital code. Each meta-task simulates an image enhancement scenario for a specific domain, the support set is used to perform a small number of gradient updates on the meta-domain image enhancement model within the meta-task, so that it quickly adapts to the specific domain represented by the current meta-task, and the query set is used to evaluate the performance of the adapted meta-domain image enhancement model on the same specific domain, obtain the evaluation result, that is, the task loss, and use it as a feedback signal to update the meta-parameters of the meta-domain image enhancement model outside the meta-task, so as to train the model to obtain better fast adaptation capability. If the support set and the query set come from different domains, the adaptation effect of the meta-domain image enhancement model to the specific domain cannot be accurately evaluated. The samples in the support set and the query set are randomly and disjointly sampled from the data in the same domain to ensure the fairness of the evaluation.

[0136] It can be understood that a domain is randomly selected from the training data set, and a plurality of pathological category labels such as normal, inflammation, and polyp are randomly selected from the multi-domain medical data corresponding to the domain, the support set is extracted from a small number of samples in these categories, such as 5 samples per category, to simulate the actual situation of lack of image annotation information in the new domain, and the query set can extract 15 samples per category from the remaining data for performance evaluation, thereby forming an N-way K-shot meta-task. The total number of meta-tasks can be set according to the size of the training data set and the number of domains, for example, 50 meta-tasks are generated for each of the 20 different domains available, a total of 1000 meta-tasks are constructed to ensure that the meta-domain image enhancement model can access enough domain characteristics during training, thereby learning a universal fast adaptation capability.

[0137] Further, the same process of obtaining corresponding meta-tasks based on the meta-training set is adopted, the corresponding meta-tasks based on the meta-validation set are obtained, and a validation meta-task set is formed.

[0138] Step S42: For the current meta-task, the target domain code is obtained based on the support set using the anatomical domain encoder.

[0139] Specifically, the multi-domain endoscope images corresponding to the support set in the meta task are obtained and input into the anatomic domain encoder trained in step S3 to obtain an anatomic domain feature vector corresponding to each multi-domain endoscope image, which represents the global domain-related attributes of the multi-domain endoscope image, such as imaging device characteristics, scene lighting, anatomic structure background, etc. All anatomic domain feature vectors corresponding to the support set are aggregated, such as taking the mean or weighted average, to generate a comprehensive, low-dimensional target domain code, which represents the overall style features of the specific domain represented by the current meta task. The target domain code is input into the meta image generator to guide it to generate images that conform to the domain style features. For example, for a meta task in which the support set comes from the A-type colonoscope system of hospital 1, a 128-dimensional target domain code is obtained through the anatomic domain encoder, which represents the overall style features of the multi-domain endoscope images output by the A-type colonoscope system of hospital 1, such as color bias, brightness, and texture style, etc.

[0140] Step S43: Based on the query set and the corresponding target domain code, an image generation adversarial training process is performed to obtain a meta image generation adversarial network adapted to the current meta task and obtain a task meta loss of the meta image generation adversarial network on the query set.

[0141] The meta image generation adversarial network includes a meta image generator and a meta image discriminator.

[0142] It should be noted that the meta image generation adversarial network can be constructed based on a conditional generative adversarial network framework, and the meta image generator can adopt a U-Net variant, etc. The input of the meta image generator is conditional information and an optional noise vector. The conditional information is specifically the target domain code and the pathology instance feature vector. The noise vector is used to generate diverse output images with subtle changes under the same conditions, thereby enhancing the richness of the synthetic data set. The output of the meta image generator is an adversarial reconstruction image. The meta image discriminator can be a convolutional neural network, and its input is a multi-domain endoscope image, an adversarial reconstruction image, and corresponding conditional information. The multi-domain endoscope image, the adversarial reconstruction image, and the corresponding conditional information are input into the meta image discriminator for discrimination, and the output is the possibility that the input image is a multi-domain endoscope image.

[0143] Step S43 includes:

[0144] Step S43-2: The pathology instance feature vector is obtained based on the query set using the pathology instance encoder.

[0145] Specifically, each multi-domain endoscopic image in the query set of the current meta-task is input to the pathology instance encoder to extract a pathology instance feature vector corresponding to the lesion area of the multi-domain endoscopic image. The pathology instance feature vector encodes local pathological features of the multi-domain endoscopic image, such as the morphology, texture, and boundary of the lesion, to ensure that the pathological features of the generated adversarial reconstruction image are consistent with each multi-domain endoscopic image in the query set.

[0146] Step S43-3: input the target domain encoding and the pathology instance feature vector as conditions to the meta-image generator to generate an adversarial reconstruction image.

[0147] Specifically, the target domain encoding and the corresponding pathology instance feature vector are fused, for example, by splicing or by a cross-attention mechanism, and input to the meta-image generator to generate a corresponding synthetic image, i.e., an adversarial reconstruction image. For example, the meta-image generator combines the target domain encoding representing the style of hospital 1 with a pathology instance feature vector representing the pathological features of a polyp to generate an adversarial reconstruction image with the style of hospital 1 and containing the pathological features of a polyp.

[0148] Step S43-4: input the adversarial reconstruction image and the endoscopic image corresponding to the query set to the meta-image discriminator to obtain an adversarial loss.

[0149] Specifically, the meta-image discriminator receives both the multi-domain endoscopic image in the query set and the adversarial reconstruction image and judges their authenticity. The multi-domain endoscopic image is a real image, and the adversarial reconstruction image is a generated image. The adversarial loss can use a standard GAN loss function, such as a least squares loss or a hinge loss, to drive the meta-image generator to produce a realistic image that can deceive the meta-image discriminator. The loss of the meta-image discriminator specifically includes judging a real image as real and a generated image as fake. The loss of the meta-image generator is to make the adversarial reconstruction image be judged as real by the meta-image discriminator.

[0150] Step S43-5: obtain a reconstruction image loss based on the adversarial reconstruction image and the endoscopic image corresponding to the query set.

[0151] Specifically, the reconstructed image loss is used to constrain the similarity between the generated image and the real image at the pixel level and the perception level to ensure content fidelity. The reconstructed image loss at least includes a pixel level loss and a perception loss. The pixel level loss can adopt an L1 loss or an L2 loss. The pixel level loss directly measures the difference between the adversarial reconstructed image and the multi-domain endoscope image in pixel intensity. The perception loss can be obtained based on a pre-trained deep network, such as VGG. The adversarial reconstructed image and the multi-domain endoscope image are respectively input into the pre-trained deep network, a feature map is extracted at a certain intermediate layer, and an L1 loss or an L2 loss between the feature maps is calculated as the perception loss. The perception loss measures the similarity between the adversarial reconstructed image and the multi-domain endoscope image at the semantic feature level, and is more effective in maintaining details and textures.

[0152] Further, a weighted sum of the pixel level loss and the perception loss is calculated as the reconstructed image loss. The weighting weight can be empirically optimized according to specific task requirements or through performance feedback on the meta task.

[0153] Step S43-6: updating the meta image generator and the meta image discriminator based on the adversarial loss and the reconstructed image loss, and serving as the meta image generative adversarial network adapted to the current meta task.

[0154] Specifically, in the current meta task, the parameters of the meta image generator and the meta image discriminator are updated for a small number of iterations, such as 5-10 times, by minimizing the weighted sum of the adversarial loss and the reconstructed image loss through a gradient descent algorithm such as the Adam optimizer. The network parameters after this update are the meta image generative adversarial network adapted to the current meta task. The total loss of the updated image generative adversarial network is calculated on the query set, for example, the weighted sum of the adversarial loss and the reconstructed loss, as the task meta loss of the current meta task. The task meta loss reflects the performance of the meta image generative adversarial network after adaptation in generating realistic images that conform to the current domain style and content.

[0155] Step S44: obtaining the meta image generative adversarial networks adapted to all meta tasks and the corresponding task meta losses, and obtaining the training meta loss based on all the task meta losses.

[0156] Specifically, for all meta tasks generated by sampling from the meta training set, steps S42 to S43 are repeatedly executed to obtain the meta image generative adversarial network adapted to each meta task and the corresponding task meta loss. The average of all the task meta losses is calculated as the training meta loss, which represents the average performance of the current parameters of the meta model on all training tasks and is the basis for meta optimization.

[0157] Step S45: updating the meta parameters of the meta-domain image enhancement model based on the training meta loss using a meta optimization algorithm, and iteratively executing until a preset convergence criterion is met.

[0158] wherein, the step S45 comprises:

[0159] Step S45-1: for each meta-task, obtaining a meta-loss gradient of the training meta-loss with respect to the parameters of the meta-domain image enhancement model, and updating the parameters of the meta-domain image enhancement model based on the meta-loss gradient.

[0160] Specifically, a model-agnostic meta-learning algorithm can be used to calculate the gradient of the training meta-loss with respect to the initial parameters of the meta-image generator, which is obtained by backpropagating the task meta-loss of all meta-tasks, indicating how to adjust the meta-parameters to make the meta-domain image enhancement model perform better when facing new tasks, and then the gradient descent algorithm is used to update the initial parameters of the meta-image generator.

[0161] Step S45-2: when the training meta-loss triggers any one of the threshold convergence condition, the iteration convergence condition and the performance convergence condition in continuous multiple iterations, it is determined that the training of the meta-domain image enhancement model converges.

[0162] Specifically, the threshold convergence condition is that the decrease amplitude of the meta-loss in continuous P iterations is less than a threshold T, and the specific values of P and T can be set according to the training stability requirement of the training process, such as P=10 and T=0.001, the iteration convergence condition is that the number of training iterations reaches a maximum preset value, such as 10000 times, and the performance convergence condition is that the average task loss is calculated on the validation meta-task set constructed based on the meta-validation set, and is used as the validation meta-loss, when the validation meta-loss no longer decreases significantly or starts to rise, this situation indicates overfitting, and any one of the conditions is met to stop training, save the meta-parameters at this time, and the training of the meta-domain image enhancement model is completed.

[0163] Step S5: obtaining a new domain medical data set, obtaining a new domain anatomical domain code based on the new domain medical data set using the anatomical domain encoder, obtaining a target instance code based on the pathology instance encoder, importing the new domain anatomical domain code and the target instance code into the meta-domain image enhancement model, and obtaining a new domain synthetic data set.

[0164] wherein, the step S5 comprises:

[0165] Step S51: importing the new domain endoscopic images in the new domain medical data set into the medical feature extraction model to obtain a new domain sample feature vector corresponding to each new domain endoscopic image.

[0166] It can be understood that the new domain medical data set refers to a target application scenario, for example, a new cooperative hospital or a new model of endoscope device collects endoscopic image sets that do not participate in the training process of steps S1 to S4.

[0167] Specifically, the new-domain endoscopy image is subjected to the same standardization preprocessing as step S11, such as resolution unification and color correction, and the standardized new-domain endoscopy image is input into the medical feature extraction model pre-trained and fine-tuned via step S1, and forward propagation is performed to extract high-level semantic features, thereby obtaining a new-domain sample feature vector corresponding to each new-domain endoscopy image. The new-domain sample feature vector is a high-level encoding of the content of the new-domain endoscopy image. For example, a colon endoscopy image collected by a C-type colonoscope system of hospital 3 is input into the medical feature extraction model, and a 2048-dimensional new-domain sample feature vector is obtained.

[0168] Step S52: obtaining an average new-domain encoding based on the new-domain sample feature vector using the anatomical domain encoder.

[0169] Specifically, the new-domain sample feature vector is input into the anatomical domain encoder trained via step S3, and the anatomical domain encoder obtains a corresponding low-dimensional anatomical domain feature vector for each new-domain sample feature vector. The anatomical domain feature vector represents the global domain attributes of each new-domain endoscopy image, such as device characteristics and overall illumination. The anatomical domain feature vectors corresponding to all new-domain sample feature vectors are subjected to an aggregation operation, such as mean calculation, to generate a comprehensive average new-domain encoding. This encoding condenses the domain-shared features of the entire new-domain dataset, and is used as a consistency condition for generating the domain attributes of the control image. For example, 100 colon endoscopy images collected by a C-type colonoscope system of hospital 3 are input into the medical feature extraction model to obtain 100 corresponding new-domain sample feature vectors. The low-dimensional anatomical domain feature vectors corresponding to the 100 new-domain sample feature vectors are obtained using the anatomical domain encoder, and the average new-domain encoding is obtained by averaging all the anatomical domain feature vectors. The average new-domain encoding represents the typical color and texture style of the C-type colonoscope system of hospital 3.

[0170] Step S53: obtaining an expected pathological parameter, and obtaining an expected pathological encoding based on the expected pathological parameter using the pre-trained semantic mapping model.

[0171] Specifically, the desired pathological parameters are specified by the user according to the downstream task, such as training a polyp classifier, and the form of the desired pathological parameters can be a text description, a pathological category label or a morphological index. Based on a pre-trained semantic mapping model, the high-level semantic desired pathological parameters are mapped to the desired pathological code. The semantic mapping model can be a fully connected neural network, the input of which is the encoding of the desired pathological parameters, such as one-hot encoding, and the output is a low-dimensional vector with the same dimension as the pathological instance feature vector output by the pathological instance encoder. The training data of the semantic mapping model is the pathological category label of all samples in the training data set and the corresponding pathological instance feature vector extracted by the pathological instance encoder. The training target is to minimize the mean square error loss between the desired pathological code output by the semantic mapping model and the pathological instance feature vector. Through training, the model learns to map high-level semantic pathological parameters to corresponding coordinates in the feature space. For example, the pathological category label is mapped to an integer code, specifically normal = 0, inflammation = 1, polyp = 2. Input the label 2 into the semantic mapping model, and output a 64-dimensional desired pathological code. The desired pathological code is similar to the sample feature vector of the real polyp image in pathological features.

[0172] Step S54: Import the average new domain code and the desired pathological code as conditional data into the meta-image generator in the meta-domain image enhancement model to generate a new domain synthetic data set.

[0173] Specifically, the average new domain code can be used to control the domain style of the generated image, and the desired pathological code can be used to control the pathological content of the generated image. By inputting the average new domain code and the desired pathological code as conditional input into the meta-domain image enhancement model, the meta-image generator in the meta-domain image enhancement model generates a large number of synthetic images based on the conditional input. These synthetic images are visually consistent with the new domain, and at the same time meet the requirements in pathological characteristics, thereby forming a new domain synthetic data set.

[0174] It can be understood that the new domain synthetic data set solves the data distribution imbalance by specifying to generate rare pathological characteristics, and adapts to multi-domain differences by injecting new domain styles, and can be used to enhance the training data of the downstream model under the new domain, such as a medical image segmentation model, to improve the performance and generalization ability of the downstream model.

[0175] The second aspect of the present application provides an endoscopic image data enhancement system based on an adversarial network, comprising:

[0176] The data processing and enhancement module is used for acquiring a multi-domain medical data set and importing the medical feature extraction model, acquiring a sample feature vector, acquiring a regular sample and a candidate abnormal sample based on the sample feature vector, importing the candidate abnormal sample to a rare verification model to acquire a high-value rare sample, and performing feature space enhancement, and packaging the regular sample and the high-value rare sample as a training data set.

[0177] The encoder training module is used for constructing and training an anatomical domain encoder and a pathology instance encoder based on the training data set.

[0178] The meta-domain image enhancement model module is used for constructing a meta-domain image enhancement model, acquiring a task meta loss based on the training data set, the anatomical domain encoder and the pathology instance encoder, and training the meta-domain image enhancement model based on the task meta loss.

[0179] The new domain synthetic data module is used for acquiring a new domain medical data set, acquiring a new domain anatomical domain code based on the anatomical domain encoder, acquiring a target instance code based on the pathology instance encoder, and importing the new domain anatomical domain code and the target instance code into the meta-domain image enhancement model to acquire a new domain synthetic data set.

[0180] The specific use and role of the application will be described below:

[0181] The embodiments of the present application can acquire a multi-domain medical data set, filter out high-value rare samples and perform feature space enhancement by using an isolated forest model and a rare verification model, construct a balanced training data set, intelligently mine and enhance rare samples, overcome the limitation that a traditional enhancement method cannot generate new semantic content, effectively solve the problem of data distribution imbalance in endoscopic data, and provide a more balanced data basis for model training.

[0182] On the other hand, the image features are decoupled by constructing an anatomical domain encoder and a pathology instance encoder, and a meta-domain image enhancement model is trained by using a meta-learning framework, so that the model has the ability to quickly adapt to the data features of a new domain. This method can generate synthetic images that are highly adapted in anatomical style and clear in pathological features only with a small amount of new domain samples, reduces the dependence on large-scale annotation of new domain data, and effectively overcomes the problem of poor data availability caused by domain shift.

Claims

1. A method for enhancing endoscopic image data based on adversarial networks, characterized in that, The method includes: Obtain a multi-domain medical dataset and import it into a medical feature extraction model to obtain sample feature vectors. Based on the sample feature vector, the isolated forest model is used to obtain regular samples and candidate anomaly samples. The candidate anomaly samples are imported into the rare validation model, and high-value rare samples are obtained. Feature space enhancement is performed on the high-value rare samples, and regular samples and high-value rare samples are encapsulated into a training dataset. Based on the training dataset, construct and train the anatomical domain encoder and the pathological instance encoder; A meta-domain image enhancement model is constructed based on a training dataset. Task-specific meta-losses are obtained using an anatomical domain encoder and a pathological instance encoder. The model is then trained based on these task-specific meta-losses, including: sampling the training dataset according to a meta-learning task paradigm to obtain multiple meta-tasks, each containing a support set and a query set; for the current meta-task, obtaining the target domain encoding using an anatomical domain encoder based on the support set; performing an image generation adversarial training process based on the query set and the corresponding target domain encoding; obtaining the meta-image generation adversarial network adapted for the current meta-task and acquiring the task-specific meta-loss of the meta-image generation adversarial network on the query set; and obtaining the meta-image generation adversarial networks adapted for all meta-tasks and their corresponding meta-losses. The task-specific meta-loss is calculated by obtaining the training meta-loss based on all task-specific meta-losses. The meta-image generative adversarial network includes a meta-image generator and a meta-image discriminator. A pathological instance feature vector is obtained using a pathological instance encoder based on a query set. The target domain encoder and the pathological instance feature vector are used as conditions and input into the meta-image generator to generate an adversarial reconstructed image. The adversarial reconstructed image and the corresponding endoscopic image from the query set are input into the meta-image discriminator to obtain the adversarial loss. A reconstructed image loss is obtained based on the adversarial reconstructed image and the corresponding endoscopic image from the query set. The meta-image generator and meta-image discriminator are updated based on the adversarial loss and the reconstructed image loss, and this update serves as the meta-image generative adversarial network adapted for the current meta-task. Obtain a new domain medical dataset. Based on the new domain medical dataset, use an anatomical domain encoder to obtain the new domain anatomical domain encoding and use a pathological instance encoder to obtain the target instance encoding. Import the new domain anatomical domain encoding and the target instance encoding into the meta-domain image enhancement model to obtain the new domain synthetic dataset.

2. The endoscopic image data enhancement method based on adversarial networks according to claim 1, characterized in that, Obtain a multi-domain medical dataset, import it into a medical feature extraction model to obtain sample feature vectors, including: Acquire multi-domain endoscopic images and their corresponding image annotation information, and perform standardization processing on the multi-domain endoscopic images and their corresponding image annotation information; The image annotation information includes at least bounding boxes, segmentation masks, and pathological category labels; Multi-domain endoscopic images and their corresponding image annotation information are encapsulated into a multi-domain medical dataset; The pre-trained feature extraction network is used as the medical feature extraction model; Multi-domain endoscopic images from a multi-domain medical dataset are imported into a medical feature extraction model to obtain the sample feature vector corresponding to each multi-domain endoscopic image.

3. The endoscopic image data enhancement method based on adversarial networks according to claim 1, characterized in that, Based on sample feature vectors, an isolated forest model is used to obtain regular samples and candidate anomaly samples. The candidate anomaly samples are then imported into a rare validation model, and high-value rare samples are obtained. Feature space augmentation is performed on the high-value rare samples. The regular samples and high-value rare samples are then packaged into a training dataset, including: The sample feature vector is input into the isolated forest model to obtain the anomaly score corresponding to the sample feature vector, and then the normal samples and candidate anomaly samples are obtained based on the anomaly score; The sample feature vector of the candidate abnormal sample is input into the rare verification model to obtain the rareness confidence score, and corresponding sample labels are generated for the candidate abnormal sample according to the preset rareness threshold. The rare validation model includes a reconstructed quality validation layer, a generated consistency validation layer, and a clinically relevant proxy validation layer. The sample labels include high-value rare samples, samples to be reviewed, and noisy samples; We employ latent spatial augmentation techniques to process high-value rare samples, thereby obtaining augmented high-value rare samples. The regular samples and enhanced high-value rare samples are packaged into a training dataset.

4. The endoscopic image data enhancement method based on adversarial networks according to claim 3, characterized in that, The sample feature vectors of candidate anomalies are input into the rareness verification model to obtain rareness confidence scores, and corresponding sample labels are generated for the candidate anomalies based on a preset rareness threshold, including: Candidate anomaly samples are imported into the reconstruction quality verification layer, and a pre-trained reconstruction autoencoder is used to decode the anomaly sample to obtain the reconstructed image of the anomaly sample. The reconstructed lesion region is obtained from the image reconstructed based on the abnormal sample, the corresponding original lesion region is obtained from the multi-domain endoscopic image, and the structural similarity index is obtained from the reconstructed lesion region and the original lesion region as the first verification score. The candidate anomaly samples are imported into the consistency verification layer. Based on the candidate anomaly samples, multiple verification sample feature vectors are generated by feature interpolation or adding noise. The verification sample feature vectors are then decoded into verification sample images. A pre-trained lesion morphology classifier is used to classify the verification sample images, and the statistical uncertainty of the classification results is obtained. A second verification score is generated based on the statistical uncertainty. Candidate abnormal samples are imported into the clinical relevance proxy verification layer. Morphological indicators of abnormal samples are obtained based on candidate abnormal samples, and the distribution of baseline morphological indicators is obtained based on regular samples. Obtain the Mahalanobis distance between the distributions of morphological indicators of abnormal samples and baseline morphological indicators, and generate a third verification score based on the Mahalanobis distance; Based on the first verification score, the second verification score, and the third verification score, a rareness confidence score is generated for the candidate abnormal sample.

5. The endoscopic image data enhancement method based on adversarial networks according to claim 1, characterized in that, Based on the training dataset, an anatomical domain encoder and a pathological instance encoder were constructed and trained, including: The anatomical domain encoder is trained based on a training dataset using contrastive loss and adversarial loss. The anatomical domain encoder takes endoscopic images as input and outputs a low-dimensional vector representing the anatomical domain data, which serves as the anatomical domain feature vector. The pathological instance encoder is trained based on the training dataset using focus loss and correlation constraints. The pathological instance encoder takes the lesion area of ​​the endoscopic image as input and outputs a low-dimensional vector representing the pathological instance data, which serves as the feature vector of the pathological instance.

6. The endoscopic image data enhancement method based on adversarial networks according to claim 1, characterized in that, A meta-domain image enhancement model is constructed. Based on the training dataset, the task-specific meta-loss is obtained using an anatomical domain encoder and a pathological instance encoder. The meta-domain image enhancement model is then trained based on the task-specific meta-loss, including: Based on the training meta-loss, a meta-optimization algorithm is used to update the meta-parameters of the meta-domain image enhancement model, and the process is iteratively executed until the preset convergence criterion is met.

7. The endoscopic image data enhancement method based on adversarial networks according to claim 6, characterized in that, The parameters of the meta-domain image enhancement model are updated using a meta-optimization algorithm based on meta-loss, and the process is iteratively executed until a preset convergence criterion is met, including: For each meta-task, obtain the meta-loss gradient of the training meta-loss on the parameters of the meta-domain image augmentation model, and update the parameters of the meta-domain image augmentation model based on the meta-loss gradient. When the training meta-loss triggers any one of the threshold convergence condition, iterative convergence condition, or performance convergence condition in multiple consecutive iterations, it is determined that the training of the meta-domain image enhancement model has converged.

8. The endoscopic image data enhancement method based on adversarial networks according to claim 1, characterized in that, A new domain medical dataset is obtained. Based on this dataset, an anatomical domain encoder is used to obtain the new domain anatomical domain encoding, and a target instance encoding is obtained based on the pathology instance encoder. The new domain anatomical domain encoding and the target instance encoding are then imported into a meta-domain image enhancement model to obtain a new domain synthetic dataset, including: Import the new domain endoscope images from the new domain medical dataset into the medical feature extraction model to obtain the new domain sample feature vector corresponding to each new domain endoscope image; The average new domain code is obtained by using an anatomical domain encoder based on the feature vector of the new domain sample. Obtain the expected pathological parameters, and use a pre-trained semantic mapping model to obtain the expected pathological code based on the expected pathological parameters; The average new domain code and the expected pathological code are imported as conditional data into the meta-image generator within the meta-domain image enhancement model to generate a new domain synthetic dataset.

9. An endoscopic image data enhancement system based on adversarial networks, applied to the method described in any one of claims 1 to 8, characterized in that, include: The data processing and enhancement module is used to acquire multi-domain medical datasets and import them into a medical feature extraction model. It acquires sample feature vectors, uses an isolated forest model based on the sample feature vectors to acquire regular samples and candidate abnormal samples, imports the candidate abnormal samples into a rare validation model to acquire high-value rare samples, performs feature space enhancement, and encapsulates regular samples and high-value rare samples into a training dataset. An encoder training module is used to construct and train an anatomical domain encoder and a pathological instance encoder based on the training dataset. Metadomain image enhancement model module, which is used to construct a metadomain image enhancement model, obtains task meta-loss using the anatomical encoder and the pathological instance encoder based on the training dataset, and trains the metadomain image enhancement model based on the task meta-loss. The new domain synthetic data module is used to acquire a new domain medical dataset, and uses the anatomical domain encoder to acquire the new domain anatomical domain code, and uses the pathological instance encoder to acquire the target instance code, and imports the new domain anatomical domain code and the target instance code into the meta-domain image enhancement model to acquire the new domain synthetic dataset.

Images