An Atrophic Gastritis Automatic Recognition System Based on Generative Adversarial Networks
Through the dual sampling method and joint recognition model of generating adversarial networks, the data imbalance in the automatic identification of atrophic gastritis is solved, and the high accuracy and stability of gastritis severity judgment is achieved, reducing the diagnosis cost of doctors.
Patent Information
- Application Number
- CN202411782973.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-06
- Publication Date
- 2025-08-01
- Estimated Expiration
- 2044-12-06
AI Technical Summary
The existing technology lacks an effective system to automatically identify and judge the severity of atrophic gastritis, and faces data imbalance, which leads to the model's prediction results that tend to be in categories with more data, making it difficult to accurately identify atrophic gastritis.
The dual sampling method based on the generative adversarial network is adopted, and the oversampling and partitioned undersampling are oversampled through feature screening, and combined with the Kimura Takemoto typing model and atrophy area recognition model, automatic identification and severity judgment of gastroscopic images are performed.
It improves the accuracy and stability of the identification of atrophic gastritis, reduces the time and labor cost of doctors' diagnosis, solves the problem of data imbalance, and reduces the rate of missed diagnosis.
Smart Images
Figure CN119251604B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of automatic recognition of medical images, and particularly relates to an automatic recognition system for atrophic gastritis based on a generative adversarial network. Background Art
[0002] The statements in this part only provide background technical information related to the present invention and do not necessarily constitute prior art.
[0003] Gastric cancer usually develops from atrophic gastritis. Patients with advanced atrophic gastritis have a relatively high probability of developing gastric cancer. Detecting and treating atrophic gastritis as early as possible helps prevent the occurrence of gastric cancer. However, the levels of endoscopists vary greatly, and it takes a lot of time and energy for manual diagnosis of atrophic gastritis. It is very necessary to use an intelligent system to automatically identify atrophic gastritis to assist medical staff in diagnosis. Nowadays, deep learning is increasingly widely used in the fields of medical image classification and segmentation. Using deep learning can achieve automatic recognition of patients, reduce the burden on doctors, and help inexperienced doctors make more accurate diagnoses of atrophic gastritis. However, there is no perfect system for automatically identifying atrophic gastritis based on gastroscopy images in the prior art to simultaneously automatically judge the severity of atrophic gastritis and automatically identify the atrophic area.
[0004] For the gastroscopy data sets that can be collected, the data of non-atrophic gastritis are often more than those of atrophic gastritis, and the disease probabilities of different parts are different and the difficulty of collecting gastroscopy data is different, so the data quantities of each part also vary greatly, which results in the problem of data imbalance. When there is data imbalance between positive and negative samples and different parts in the gastroscopy data set, in order to ensure that the prediction results of the whole model are as accurate as possible, the prediction results of the model will tend to the category with more data, thus producing incorrect predictions.
[0005] Data augmentation for the class with less data is a method to solve data imbalance, but it is difficult to collect the class with less atrophic gastritis data. Manual data augmentation requires doctors to spend a lot of human and time costs, and the feasibility is not high.
[0006] The generative adversarial network is an unsupervised learning method in the field of computer vision and can be used to generate the required images. In recent years, the generative adversarial network has received great attention and wide application in various fields and has been applied to directions such as medical image generation and medical image enhancement in the medical field. However, the existing technology only uses simple data enhancement methods for augmenting unbalanced atrophic gastritis data, such as flipping, rotating, translating, etc. The image data obtained through these spatial transformations cannot ensure the variability of the data and cannot effectively solve the data imbalance problem. Summary of the Invention
[0007] To solve the above problems, the present invention proposes an automatic recognition system for atrophic gastritis based on a generative adversarial network, which can automatically recognize atrophic gastritis in gastroscopy images, has high accuracy, and can better assist doctors in diagnosis.
[0008] According to some embodiments, the present invention adopts the following technical solutions:
[0009] An automatic recognition system for atrophic gastritis based on a generative adversarial network, comprising:
[0010] A data acquisition module, configured to acquire gastroscopy images of the required parts, and label the parts, whether there is atrophy, and atrophy region information of the gastroscopy images, so as to obtain a gastroscopy image dataset;
[0011] A dual-sampling module, configured to preprocess the gastroscopy image dataset and perform dual-sampling of feature screening based on a generative adversarial network to obtain a balanced gastroscopy image dataset;
[0012] A model training module, configured to respectively train a Kimura-Takemoto classification model and an atrophy region recognition model by using the balanced gastroscopy image dataset;
[0013] A joint recognition module, configured to respectively process the gastroscopy image to be measured by using the Kimura-Takemoto classification model and the atrophy region recognition model, and perform joint recognition based on the outputs of the two models to determine the severity of atrophic gastritis in the gastroscopy image.
[0014] As an alternative implementation, the gastroscopy images of the required parts include gastroscopy images of the gastric antrum, gastric angle, lower part of the lesser curvature of the gastric body, middle and upper parts of the lesser curvature of the gastric body, cardia, gastric fundus, and greater curvature of the gastric body.
[0015] As an alternative implementation, the dual-sampling module includes a preprocessing module, configured to preprocess the gastroscopy image dataset, remove blurred, defocused, and halo images in the gastroscopy image dataset, as well as texts, thumbnails, and black edges in the images, and crop the images to the main body parts of the images.
[0016] As an alternative implementation, the dual sampling module includes a judgment module configured to further divide closed and open atrophic gastritis into two categories: atrophy and non-atrophy, forming sub-categories. A threshold u is set. If the number of images in a sub-category is greater than g×(1 + u), then this sub-category is the category that needs under-sampling, and under-sampling by partitioning is performed on the images in this sub-category; if the number of images in a sub-category is less than g×(1 - u), then this sub-category is the category that needs over-sampling, and over-sampling based on feature screening using a generative adversarial network is performed on the images in this sub-category; if the number of images in a sub-category is greater than or equal to g×(1 - u) and less than or equal to g×(1 + u), then no under-sampling or over-sampling operation is required. g is the ratio of the total number of gastroscopy images to n, and n is the total number of sub-categories.
[0017] As a further implementation, the dual sampling module includes an under-sampling module configured to: extract features from each image in the category that needs under-sampling using a network model to obtain feature vectors;
[0018] Calculate the Euclidean distance between every two feature vectors. For each feature vector, select the smallest m values among the distances between this feature vector and other feature vectors;
[0019] Add the reciprocals of the selected m distance values to obtain the score of this image, sort according to the score, and divide the images into a retention area and an under-sampling area according to a set ratio;
[0020] Perform random under-sampling on the images in the under-sampling area. The number of images retained after under-sampling is g×(1 - u), and the images retained after sampling and the images in the retention area are added to the training set.
[0021] As a further implementation, the dual sampling module includes an over-sampling module configured to: construct a generator and a discriminator. The generator is used to generate fake samples, and the discriminator is used to distinguish between the samples generated by the generator and real samples;
[0022] Train the generator and the discriminator. During the training process, first adjust and enhance the images;
[0023] Perform feature screening. For the images generated by the trained generator and discriminator and the original images, perform feature extraction to obtain the feature vectors of the generated images and the feature vectors of the original images. Calculate the distance between the feature vectors of the generated images and the feature vectors of each original image, select the smallest n distance values among them, add the reciprocals of the selected n distance values to obtain the score of this image. Set a threshold h. If the score of this image is less than h, then this image cannot be used as an image for training. If the score of this image is greater than or equal to h, add this image to the training set;
[0024] Repeat the above feature screening process until the number of images in the training set is greater than or equal to g×(1 - u), and then stop.
[0025] As an alternative implementation, the Kimura-Takemoto classification model includes a feature extraction layer, an image patch embedding layer, a position embedding layer, an encoder, and a multi-layer perceptron connected in sequence. The feature extraction layer is used to extract features from the images in the balanced gastroscope image dataset after resizing. The image patch embedding layer receives the feature extraction results and performs convolution and flattening operations on them. The position embedding layer is used to add position embeddings after merging the output of the image patch embedding layer and the class token. The encoder is used to perform normalization processing on the output of the position embedding layer after the dropout operation and extract the class token. The multi-layer perceptron is used to make predictions based on the extracted results.
[0026] As an alternative implementation, when training the Kimura-Takemoto classification model, each image labeled as atrophic but predicted as non-atrophic is divided into small patch images of multiple grids, the order of the image patches in the grid is randomly shuffled, and the shuffled images are added to the original training set to retrain the Kimura-Takemoto classification model.
[0027] As an alternative implementation, the atrophic region recognition model includes multiple transformer layers of different sizes connected in sequence. Each transformer layer contains an efficient self-attention layer, a hybrid fully connected feed-forward network layer, and an overlapping image patch merging layer. The multiple multi-scale features obtained are input into the multi-layer perceptron layer to obtain the prediction output.
[0028] When training the atrophic region recognition model, each image labeled as atrophic but with a predicted atrophic area less than the set value and its mask image are divided into small patch images of multiple grids, the order of the image patches in the grid is randomly shuffled, and the shuffled images are added to the original training set to retrain the atrophic region recognition model.
[0029] As an alternative implementation, the joint recognition module is used to input the gastroscope image to be tested into the Kimura-Takemoto classification model to obtain the image-level classification prediction result, and input the gastroscope image to be tested into the atrophic region recognition model to obtain the pixel-level atrophic region and the atrophic region area.
[0030] When the output result of the Kimura-Takemoto classification model is non-atrophic and the output atrophic region area of the atrophic region recognition model is less than the set value, it is judged as non-atrophic gastritis; when the output result of the Kimura-Takemoto classification model is atrophic or the output atrophic region area of the atrophic region recognition model is greater than the set value, it is judged as having atrophic gastritis.
[0031] If it is judged as having atrophic gastritis, the severity of atrophic gastritis is judged based on the output result of the Kimura-Takemoto classification model.
[0032] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0033] The present invention can automatically identify atrophic gastritis from gastroscopy images to assist doctors in judging the severity of atrophic gastritis, improving the accuracy and stability of identification, and greatly reducing the labor and time costs required for doctors' diagnosis.
[0034] In view of the characteristics of the atrophic gastritis dataset, the present invention simultaneously uses the methods of feature screening oversampling and partition undersampling based on the generative adversarial network to solve the previously difficult problem of data imbalance, increase the proportion of data in the category with less data volume, and effectively improve the accuracy, stability and robustness of model prediction.
[0035] Since each data collected and labeled in the present invention is very precious, in order to avoid wasting the effective information lost in the undersampling process, partition undersampling is used to retain more effective information, and a judgment mechanism is set to determine whether to use oversampling or undersampling for such data.
[0036] In view of the problem that it is more difficult to collect data for the category with less data volume, the use of the oversampling method to automatically generate data can greatly reduce the labor and time costs of professional doctors in collecting data; and a feature filter is added to the oversampling method to ensure the effectiveness of the training set images for training.
[0037] The present invention adds a mechanism to prevent misprediction as non-atrophic in the Kimura-Takemoto classification model and the atrophic region recognition model, which not only strengthens the learning of the model for mispredicted samples, but also effectively reduces the missed diagnosis rate. Moreover, a joint recognition module is used for auxiliary recognition. The recognition basis comes from the Kimura-Takemoto classification on the one hand and the atrophic region predicted by the model on the other hand. The recognition result is more comprehensive, further reducing the missed diagnosis rate and improving the stability of the model.
[0038] To make the above objects, features and advantages of the present invention more obvious and understandable, the following specific preferred embodiments are given, and in conjunction with the accompanying drawings, the detailed description is as follows. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The specification drawings forming a part of the present invention are used to provide a further understanding of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation to the present invention.
[0040] Figure 1 It is the overall flowchart of the automatic atrophic gastritis recognition system based on the generative adversarial network in the embodiment of the present invention;
[0041] Figure 2 It is the schematic flowchart of the dual sampling module in the embodiment of the present invention;
[0042] Figure 3 It is a schematic flow diagram of zonal undersampling in the embodiments of the present invention;
[0043] Figure 4 It is a structural diagram of a feature screening oversampling model based on a generative adversarial network in the embodiments of the present invention;
[0044] Figure 5 It is a structural diagram of the Kimura-Takemoto classification model in the embodiments of the present invention;
[0045] Figure 6 It is a schematic flow diagram of the joint recognition module in the embodiments of the present invention. Specific Embodiments
[0046] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.
[0047] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present invention belongs.
[0048] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments of the present invention. As used herein, unless the context clearly dictates otherwise, the singular forms are also intended to include the plural forms. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of the described features, steps, operations, devices, components, and / or combinations thereof.
[0049] In the case of no conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.
[0050] Embodiment 1
[0051] An automatic recognition system for atrophic gastritis based on a generative adversarial network, as Figure 1 shown, includes:
[0052] A data acquisition module, configured to acquire gastroscopy images of the required part, and label the part, whether there is atrophy, and atrophy area information of the gastroscopy images, so as to obtain a gastroscopy image dataset;
[0053] A dual sampling module, configured to preprocess and perform dual sampling on the gastroscopy image dataset to obtain a balanced gastroscopy image dataset;
[0054] A model training module, configured to use the balanced gastroscopy image dataset to train a Kimura-Takemoto classification model and an atrophy area recognition model respectively;
[0055] A combined recognition module is used to input the gastroscope images to be measured into the Kimura-Takemoto classification model and the atrophy region recognition model respectively, and input the outputs of the two models into the combined recognition model together, and automatically recognize the severity of atrophic gastritis in the gastroscope images according to the model output results to assist doctors in diagnosis.
[0056] Among them, obtaining the gastroscope image dataset includes:
[0057] Obtain gastroscope images of the gastric antrum, gastric angle, lower part of the lesser curvature of the gastric body, middle and upper parts of the lesser curvature of the gastric body, cardia, fundus of the stomach, and greater curvature of the gastric body. Professional physicians with rich experience annotate the location, whether there is atrophy, and atrophy region information of the gastroscope images. The location and whether there is atrophy of the gastroscope images are obtained by image-level annotation of the images, and the atrophy region information is obtained by pixel-level annotation of the images.
[0058] Correspond the locations of the gastroscope images with the classifications according to the Kimura-Takemoto classification method. The corresponding relationships are C1: gastric antrum, C2: gastric angle, lower part of the lesser curvature of the gastric body, C3: middle and upper parts of the lesser curvature of the gastric body, O1: cardia, O2: fundus of the stomach, O3: greater curvature of the gastric body, to obtain the gastroscope image dataset.
[0059] Among them, the Kimura-Takemoto classification method can be used to identify atrophic gastritis. This method divides atrophic gastritis into closed type (C) and open type (O). The basis for differentiation is that the atrophy boundary of the closed type does not reach the cardia, and the atrophy boundary of the open type has extended above the cardia. There are horizontal and color differences in the gastric mucosa on both sides of the atrophy boundary. According to the location of the atrophy boundary, the closed type and the open type are each divided into 3 subtypes, namely C1, C2, C3 and O1, O2, O3.
[0060] Preprocess and double-sample the gastroscope image dataset to obtain a balanced gastroscope image dataset, including:
[0061] First, remove the blurred, defocused, and halo images in the gastroscope image dataset, as well as the text, thumbnail, and black edges in the images, and crop the images to the main body part of the images.
[0062] Experiments found that the gastroscope image dataset has a strong dependence on data balance. Only using undersampling will waste useful information in the class with a large amount of data, and only using oversampling will reduce the generalization ability of the model for the class with a small amount of data. As Figure 2 shown, use the double-sampling method of undersampling and oversampling in turn to achieve data balance. The undersampling method is partition undersampling, and the oversampling method is feature screening oversampling based on the generative adversarial network. And divide the gastroscope image dataset into a training set, a validation set, and a test set according to the ratio of 8:1:1. Perform image double-sampling on the training set, and keep the validation set and the test set unchanged to obtain a balanced gastroscope image dataset.
[0063] The steps of using undersampling and oversampling methods in sequence are as follows:
[0064] C1, C2, C3, O1, O2, and O3 are each further divided into two categories: atrophic and non-atrophic, resulting in a total of 12 subcategories. Let g = the total number of gastroscopy images / 12, and set the threshold u. Here, u = 0.2 is set. If the number of images in a subcategory is greater than g×(1 + u), the subcategory is considered a category that requires undersampling, and regional undersampling is performed on the images in the corresponding category; if the number of images in a subcategory is less than g×(1 - u), it is considered a category that requires oversampling, and feature screening oversampling based on a generative adversarial network is performed on the corresponding images; if the number of images in a subcategory is greater than or equal to g×(1 - u) and less than or equal to g×(1 + u), then no undersampling or oversampling operation is required.
[0065] The process of regional undersampling is as Figure 3 shown. For each image in the category that requires undersampling, use the efficientnet network to extract features, remove the last layer of the network for output, and obtain the feature vector.
[0066] Calculate the Euclidean distance between every two feature vectors. For each feature vector, select the smallest m values among the distances between this feature vector and other feature vectors.
[0067] Add the reciprocals of the selected m distance values to obtain the score of this image. Sort the images in this category according to the scores, and divide the images into a retention area and an undersampling area according to a 1:1 ratio. The images with the top 0.5 scores are divided into the retention area, and the images with the bottom 0.5 scores are divided into the undersampling area.
[0068] Perform random undersampling on the images in the undersampling area. The number of images retained after undersampling is g×(1 - u), and add the images retained after sampling and the images in the retention area to the training set.
[0069] The steps of feature screening oversampling based on a generative adversarial network are as Figure 4 shown:
[0070] Construct a feature screening oversampling model based on the StyleGAN2 ADA network. Different from traditional generative adversarial networks, the StyleGAN2 ADA network can generate high-quality artificial images using a small amount of data, thus better solving the problem of sample imbalance in the atrophic gastritis dataset. Construct the StyleGAN2 ADA generator G and discriminator D. The generator generates fake samples, and the discriminator is used to distinguish the samples generated by the generator from the real samples.
[0071] When training the discriminator, the latent variable is input into the generator to obtain a fake sample image. The real sample image and the fake sample image are subjected to the same augmentation transformation. The transformed images are input into the discriminator, and the non-saturating logistic loss is used to train the discriminator. Its calculation formula is:
[0072] ;
[0073] When training the generator, the latent variable is input into the generator to obtain a fake sample image. The fake sample image is subjected to an augmentation transformation. The transformed image is input into the discriminator, and the non-saturating logistic loss is used to train the generator.
[0074] During the training process of the discriminator and the generator, the images of the class that needs to be oversampled are first resized to 256×256 pixel size. The augmentation transformation of the images includes color transformation, pixel operation, graphic transformation, adding noise, image spatial filtering, and cropping. And the adaptive discriminator augmentation technique is used to adaptively adjust the probability of the image augmentation transformation. This technique effectively avoids the overfitting phenomenon and can also achieve good results on datasets with a small number of images.
[0075] Construct a feature filter. The steps of feature filtering are as follows: The number of images generated by using the trained generative adversarial network is g - the number of images of this class. The generated images and the original images are input into the efficientnet network. Remove the last layer of the network and output the feature vectors to obtain the feature vectors of the generated images and the original images. Calculate the distance between the feature vector of the generated image and the feature vector of each original image, and select the smallest n distance values among them. Add the reciprocals of the selected n distance values to obtain the score of this image. Set a threshold h. If the score of this image is less than h, this image cannot be used as an image for training. If the score of this image is greater than or equal to h, add this image to the training set.
[0076] Repeat the above feature filtering operation until the number of images in the training set is greater than or equal to g×(1 - u) and then stop.
[0077] Train the Kimura-Takemoto classification model, as Figure 5 shown, including:
[0078] The balanced gastroscopy image dataset is divided into 7 categories according to {no atrophy, C1 atrophy, C2 atrophy, C3 atrophy, O1 atrophy, O2 atrophy, O3 atrophy}. The size of the images in the balanced gastroscopy image dataset is adjusted to 224×224×3, and the input is the resnet50 layer for feature extraction. The size of the first-layer feature matrix obtained is 14×14×1024. The first-layer feature matrix is input into the image patch embedding layer, which includes a 1×1 convolution and flattening process. The size of the second-layer feature matrix obtained is 196×768. After merging the second-layer feature matrix with the class label and adding the position embedding, the size of the third-layer feature matrix obtained is 197×768. The third-layer feature matrix is input into the transformer encoder after a dropout operation and undergoes normalization processing. The size of the fourth-layer feature matrix obtained is 197×768. The class label with a size of 1×768 in the fourth-layer feature matrix is extracted, and finally, it is input into the multi-layer perceptron to obtain the prediction result.
[0079] Each image labeled as atrophy but predicted as non-atrophy is divided into small images of 3×3 grids. The order of the image patches in the grid is randomly shuffled, and the shuffled images are added to the original training set to retrain the Kimura-Takemoto classification model. Gastroscopy images have distinct structural characteristics. Adding the shuffled images to the original training set can better train for the mucosal feature differences of atrophic gastritis and effectively reduce the missed diagnosis rate.
[0080] In this embodiment, the evaluation metrics of the Kimura-Takemoto classification model can be selected as accuracy, precision, recall, and / or F-score.
[0081] Training the atrophy region recognition model includes:
[0082] The size of the images in the balanced gastroscopy image dataset is adjusted to 256×256×3. After dividing the images into multiple image patches of 4×4×3 size, they are sequentially passed through 4 different-sized transformer layers. Each transformer layer contains an efficient self-attention layer, a hybrid fully-connected feed-forward network layer, and an overlapping image patch merging layer. The output multi-scale features of each transformer layer are 64×64×C1, 32×32×C2, 16×16×C3, and 8×8×C4 respectively. The 4 multi-scale features obtained are input into the multi-layer perceptron layer to obtain a prediction output of 64×64×6.
[0083] Each image labeled as atrophy but with a predicted atrophy area of 0 and its mask image are divided into small images of 3×3 grids. The order of the image patches in the grid is randomly shuffled, and the shuffled images are added to the original training set to retrain the atrophy region recognition model.
[0084] The evaluation indicators of the atrophy region recognition model are the intersection over union, Dice coefficient, and ROC curve.
[0085] Input the gastroscopy image to be measured into the Kimura-Takemoto classification model and the atrophy region recognition model, and input the outputs of the two models into the joint recognition model together, as Figure 6 shown, to obtain the recognition result of atrophic gastritis to assist doctors in diagnosis, including:
[0086] Input the gastroscopy image to be measured into the Kimura-Takemoto classification model to obtain the image-level classification prediction result, and input the gastroscopy image to be measured into the atrophy region recognition model to obtain the pixel-level atrophy region and the area of the atrophy region.
[0087] Construct a joint recognition model, input the obtained results into the joint recognition model together to obtain the multi-level atrophic gastritis recognition result. The steps are as follows:
[0088] First, judge whether there is atrophy. When the output result of the Kimura-Takemoto classification model is non-atrophy and the output atrophy region area of the atrophy region recognition model is 0, it is judged as no atrophic gastritis; when the output result of the Kimura-Takemoto classification model is atrophy or the output atrophy region area of the atrophy region recognition model is not 0, it is judged as having atrophic gastritis.
[0089] If it is judged as having atrophic gastritis, judge the severity of atrophic gastritis through the output result of the Kimura-Takemoto classification model. The severity of atrophic gastritis is classified as: mild (C1-C2), moderate (C3-O1), and severe (O2-O3).
[0090] Those skilled in the art should understand that the embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD - ROM optical storage, etc.).
[0091] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram can be implemented by computer program instructions, and the combination of processes and / or blocks in the flowchart and / or block diagram can also be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate for implementing the processes in Figure 1one or more processes and / or blocks Figure 1 means for the functions specified in one or more blocks
[0092] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction means that implements the functions in the process Figure 1 one or more processes and / or blocks Figure 1 the functions specified in one or more blocks
[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus, such that a series of operation steps are executed on the computer or other programmable apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions in the process Figure 1 one or more processes and / or blocks Figure 1 the steps of the functions specified in one or more blocks
[0094] The foregoing is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the present invention. Any modifications, equivalent substitutions, improvements, etc. made by those skilled in the art without creative efforts within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. An automatic recognition system for atrophic gastritis based on a generative adversarial network, characterized in that Including: A data acquisition module, configured to acquire gastroscopy images of the required part, and label the part, atrophy status, and atrophy region information of the gastroscopy images, so as to obtain a gastroscopy image dataset; A dual sampling module, configured to preprocess the gastroscopy image dataset, and perform feature screening dual sampling based on a generative adversarial network to obtain a balanced gastroscopy image dataset; A model training module, configured to use the balanced gastroscopy image dataset to train a Kimura-Takemoto classification model and an atrophy region recognition model respectively; A joint recognition module, configured to use the Kimura-Takemoto classification model and the atrophy region recognition model to process the gastroscopy image to be measured respectively, and perform joint recognition based on the outputs of the two models to determine the severity of atrophic gastritis in the gastroscopy image; A mechanism for preventing misprediction as non-atrophic is added to the Kimura-Takemoto classification model and the atrophy region recognition model; The dual sampling module includes an undersampling module, which is configured to: extract features of each image in the class that needs to be undersampled using a network model to obtain feature vectors; Calculate the Euclidean distance between every two feature vectors, and for each feature vector, select the smallest m values among the distances between this feature vector and other feature vectors; Add the reciprocals of the selected m distance values to obtain the score of this image, sort according to the score, and divide the images into a retention area and an undersampling area according to a set ratio; Perform random undersampling on the images in the undersampling area, and the number of images retained by undersampling is g×(1 - u), and add the images retained by sampling and the images in the retention area to the training set; The dual sampling module includes an oversampling module, which is configured to: construct a feature screening oversampling model based on the StyleGAN2 ADA network; construct a generator and a discriminator, where the generator is used to generate fake samples, and the discriminator is used to distinguish the samples generated by the generator from real samples; Train the generator and the discriminator. During the training process, first adjust and enhance the images; When training the discriminator, input the latent variable into the generator to obtain a fake sample image, perform the same enhancement changes on the real sample image and the fake sample image, input the changed images into the discriminator, and use the non-saturating logistic loss to train the discriminator: ； When training the generator, input the latent variable into the generator to obtain a fake sample image, perform enhancement changes on the fake sample image, input the changed image into the discriminator, and use the non-saturating logistic loss to train the generator; Perform feature screening. For the images generated by the trained generator and discriminator and the original images, perform feature extraction to obtain the feature vectors of the generated images and the feature vectors of the original images, calculate the distances between the feature vectors of the generated images and the feature vectors of each original image, select the smallest n distance values among them, add the reciprocals of the selected n distance values to obtain the score of this image, set a threshold h, if the score of this image is less than h, then this image cannot be used as an image for training, if the score of this image is greater than or equal to h, add this image to the training set; Repeat the above feature screening process until the number of images in the training set is greater than or equal to g×(1 - u), and then stop.
2. The automatic recognition system for atrophic gastritis based on a generative adversarial network according to claim 1, wherein The gastroscope images of the required parts include gastroscope images of the gastric antrum, gastric angle, lower part of the lesser curvature of the gastric body, middle and upper parts of the lesser curvature of the gastric body, cardia, gastric fundus, and greater curvature of the gastric body.
3. The automatic atrophic gastritis recognition system based on a generative adversarial network according to claim 1, characterized in that, The dual-sampling module includes a preprocessing module for preprocessing the gastroscope image dataset, removing blurred, defocused, and halo images in the gastroscope image dataset, as well as text, thumbnails, and black edges in the images, and cropping the images to the main body part of the images.
4. The automatic recognition system for atrophic gastritis based on a generative adversarial network according to claim 3, wherein The dual-sampling module includes a judgment module configured to further divide closed-type and open-type atrophic gastritis into two categories: atrophy and non-atrophy, forming sub-categories, setting a threshold u. If the number of images in a sub-category is greater than g×(1 + u), then this sub-category is the category that needs under-sampling, and under-sampling is performed on the images in this sub-category; if the number of images in a sub-category is less than g×(1 - u), then this sub-category is the category that needs over-sampling, and feature screening over-sampling based on a generative adversarial network is performed on the images in this sub-category; if the number of images in a sub-category is greater than or equal to g×(1 - u) and less than or equal to g×(1 + u), then no under-sampling or over-sampling operation is required, where g is the ratio of the total number of gastroscope images to n, and n is the total number of sub-categories.
5. The automatic recognition system for atrophic gastritis based on a generative adversarial network according to claim 1, characterized in that, The Kimura-Takemoto classification model includes a feature extraction layer, an image patch embedding layer, a position embedding layer, an encoder, and a multi-layer perceptron connected in sequence. The feature extraction layer is used to extract features from the resized and balanced gastroscope image dataset images. The image patch embedding layer receives the feature extraction results and performs convolution and flattening processing on them. The position embedding layer is used to add position embedding after merging the output of the image patch embedding layer with the class token. The encoder is used to perform normalization processing on the output of the position embedding layer after dropout operation and extract the class token. The multi-layer perceptron is used to make predictions based on the extracted results.
6. The automatic recognition system for atrophic gastritis based on a generative adversarial network according to claim 1 or 5, characterized in that, in When training the Kimura-Takemoto classification model, each image labeled as atrophy but predicted as non-atrophy is divided into small patch images of multiple grids, the order of the image patches in the grid is randomly shuffled, and the shuffled images are added to the original training set to retrain the Kimura-Takemoto classification model.
7. The automatic recognition system for atrophic gastritis based on a generative adversarial network according to claim 1, characterized in that, The atrophy region recognition model includes multiple transformer layers of different sizes connected in sequence. Each transformer layer contains an efficient self-attention layer, a hybrid fully connected feed-forward network layer, and an overlapping image patch merging layer. The obtained multiple multi-scale features are input into the multi-layer perceptron layer to obtain the prediction output. When training the atrophy region recognition model, each image labeled as atrophy but with a predicted atrophy area less than the set value and its mask image are divided into small patch images of multiple grids, the order of the image patches in the grid is randomly shuffled, and the shuffled images are added to the original training set to retrain the atrophy region recognition model.
8. The automatic recognition system for atrophic gastritis based on a generative adversarial network according to claim 1, characterized in that, The joint recognition module is used to input the gastroscope image to be tested into the Kimura-Takemoto classification model to obtain the image-level classification prediction result, and input the gastroscope image to be tested into the atrophy region recognition model to obtain the pixel-level atrophy region and the atrophy region area. When the output result of the Kimura-Takemoto classification model is non-atrophic and the area of the atrophic region identified by the atrophic region identification model is less than the set value, it is judged as non-atrophic gastritis; when the output result of the Kimura-Takemoto classification model is atrophic or the area of the atrophic region identified by the atrophic region identification model is greater than the set value, it is judged as having atrophic gastritis. If it is judged as having atrophic gastritis, the severity of atrophic gastritis is judged based on the output result of the Kimura-Takemoto classification model.