Method, device, and medium for classifying gastroscopic images of multitask autoimmune gastritis
By combining image registration and grouping with the RepLKNet model, Swin Transformer, and DCNN networks, the problem of low accuracy in gastroscopic image diagnosis was solved, accurate classification of gastroscopic images was achieved, and accurate diagnosis of gastric anatomical locations and lesions was provided.
Patent Information
- Application Number
- CN202510241954.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-03
- Publication Date
- 2025-10-10
- Estimated Expiration
- 2045-03-03
AI Technical Summary
In the existing technology, the automated recognition accuracy of gastroscopic image diagnosis of autoimmune gastritis is low, and it is difficult to accurately identify the gastric anatomical location and lesion conditions at the same time.
A multi-task autoimmune gastritis endoscopic image classification method was adopted. Through image registration and grouping, the RepLKNet model, Swin Transformer, and DCNN networks were combined to process the anatomical location and lesion conditions of the endoscopic images, respectively. Large kernel convolution and cross entropy loss function were used to improve the classification accuracy.
It achieves accurate classification of gastroscopic images, provides accurate diagnosis of gastric anatomical location and pathological conditions, and improves the accuracy of automated interpretation.
Smart Images
Figure CN120088567B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and in particular to a multi-task autoimmune gastritis gastroscope image classification method, device and medium based on image registration and image grouping in a patient unit. BACKGROUND
[0002] Autoimmune gastritis is considered to have great relevance to diseases such as gastric neuroendocrine tumor (NET) and gastric cancer, and thus timely and accurate diagnosis of autoimmune diseases is of great significance for preventing gastric cancer developed from autoimmune diseases. As a chronic atrophic gastritis, the main lesion manifestations of autoimmune diseases under gastroscopy are significantly different from the atrophy caused by Helicobacter pylori infection in the disease site, and thus in diagnosing autoimmune diseases based on gastroscope images, not only the normal and atrophic gastric mucosa needs to be distinguished, but also the anatomical site of the gastric mucosa needs to be clearly understood.
[0003] Therefore, there is an urgent need to provide an automatic autoimmune gastritis gastroscope image recognition method to efficiently and accurately give gastroscope image analysis results by learning the diagnosis experience of professional gastroenterologists. SUMMARY
[0004] In view of this, the present application provides a multi-task autoimmune gastritis gastroscope image classification method, device and medium for interpreting autoimmune-related gastroscope images.
[0005] In a first aspect, the embodiments of the present application provide a multi-task autoimmune gastritis gastroscope image classification method, which comprises:
[0006] Obtaining a plurality of gastroscope images of a patient, registering all gastroscope images to obtain a similarity matrix, and clustering gastroscope images according to the similarity matrix to group gastroscope images to obtain a plurality of gastroscope image groups.
[0007] Each gastroscope image group is input into a first classification network and a second classification network to obtain a first classification label and a second classification label, and a (first classification label, second classification label) result pair corresponding to the current gastroscope image group is obtained, wherein the first classification label includes anatomical sites of the stomach such as the fundus, body and antrum, and the second classification label includes lesion conditions such as normal and atrophy.
[0008] In a second aspect, the embodiments of the present application provide a multi-task autoimmune gastritis gastroscope image classification system for implementing the above-mentioned multi-task autoimmune gastritis gastroscope image classification method, which comprises:
[0009] The gastroscopic image registration and grouping module is used to obtain a number of gastroscopic images of a patient, register all the gastroscopic images, and obtain a similarity matrix; based on the similarity matrix, the gastroscopic images are clustered and grouped to obtain a number of gastroscopic image groups;
[0010] The gastroscopic image classification module is used to input each gastroscopic image group into the first classification network and the second classification network respectively to obtain the first classification label and the second classification label; wherein the first classification label includes the fundus, body and antrum, and the second classification label includes normal and atrophy.
[0011] In a third aspect, an embodiment of the present invention provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned multi-task autoimmune gastritis endoscopic image classification method.
[0012] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned multi-task autoimmune gastritis endoscopic image classification method.
[0013] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned multi-task autoimmune gastritis endoscopic image classification method.
[0014] Compared with the prior art, the present invention has the following beneficial effects:
[0015] The present invention provides a multi-task autoimmune gastritis gastroscopic image classification method based on patients. The present invention combines gastroscopic image registration and grouping with gastroscopic image automatic classification. In response to the low accuracy of automatic image interpretation in such tasks, a gastroscopic image grouping method based on image registration is proposed. The correlation between gastroscopic images is extracted and used to assist image interpretation. The RepLKNet model based on large kernel convolution is used to take into account the performance requirements of the model for gastric anatomical location and lesion classification. The accuracy of gastroscopic image classification is guaranteed on the basis of using one model to solve two problems. In addition, two branch networks are added to further improve the accuracy of the two tasks based on the different characteristics of the two classification tasks of gastric anatomical location and gastric mucosal lesion. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0017] Figure 1 A flowchart of a multi-task autoimmune gastritis gastroscopic image classification method provided by an embodiment of the present invention;
[0018] Figure 2 A schematic diagram of a multi-task autoimmune gastritis gastroscopic image classification method provided by an embodiment of the present invention;
[0019] Figure 3 A schematic diagram of an image registration result provided by an embodiment of the present invention;
[0020] Figure 4 A schematic diagram of a similarity matrix obtained through image registration and subsequent processing according to an embodiment of the present invention;
[0021] Figure 5 A schematic diagram of the RepLKNet model structure provided in an embodiment of the present invention;
[0022] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0024] It should be noted that, unless there is any conflict, the features in the following embodiments and implementations may be combined with each other.
[0025] like Figure 1 and Figure 2 As shown, an embodiment of the present invention provides a multi-task autoimmune gastritis gastroscopic image classification method, the method comprising the following steps:
[0026] Step S1, obtaining a plurality of gastroscopic images of a patient, registering all the gastroscopic images to obtain a similarity matrix; clustering the gastroscopic images according to the similarity matrix to group the gastroscopic images to obtain a plurality of gastroscopic image groups.
[0027] Specifically, if Figure 3 and Figure 4 As shown in Figure 2, the process of registering all gastroscopic images and obtaining a similarity matrix includes:
[0028] Scale-invariant feature transform (SIFT) is used to find qualified registration points between all gastroscopic images.
[0029] Set the maximum and minimum thresholds for qualified registration points. In this example, the minimum threshold for qualified registration points is set to 12 and the maximum threshold is set to 128.
[0030] When the number of qualified registration points between two gastroscopic images is less than the minimum threshold, the similarity between the two gastroscopic images is recorded as 0; when the number of qualified registration points between two gastroscopic images is greater than the maximum threshold, the similarity between the two gastroscopic images is recorded as 1; when the number of qualified registration points between two gastroscopic images is within the range of the minimum threshold and the maximum threshold, the number of qualified registration points is normalized to convert it into a similarity between 0 and 1;
[0031] The calculated similarity is added into a matrix with the image as the horizontal and vertical axes to obtain a similarity matrix;
[0032] Based on the similarity matrix, a hierarchical clustering algorithm was used to cluster the patients' gastroscopic images to obtain several gastroscopic image groups reflecting the same gastroscopic image features.
[0033] Furthermore, the process of clustering the patient's gastroscopic images using a hierarchical clustering algorithm includes:
[0034] Specifically, the hierarchical clustering algorithm first calculates the distance between samples, merges the points with the closest distance into the same class, then calculates the distance between classes, merges the classes with the closest distance and distance less than a threshold into a larger class, and repeats the operation until all classes with inter-class distance less than the threshold are combined into one large class. In this example, the distance between classes is defined as the shortest distance between samples between classes, and the threshold of the inter-class distance is set to 1 / 0.1=10.
[0035] In step S2, each gastroscopic image group is input into the first classification network and the second classification network respectively to obtain the first classification label and the second classification label, and obtain the (first classification label, second classification label) result pair corresponding to the current gastroscopic image group; wherein, the first classification label includes the gastric anatomical parts including the fundus, body, and antrum, and the second classification label includes the pathological conditions including normal and atrophy.
[0036] Among them, the first classification network includes a first network branch module and a backbone network module; the second classification network includes a second network branch module and a backbone network module; among them, the backbone network module in the first classification network is the same as the backbone network module in the second classification network, and both adopt the RepLKNet model; the first network branch module adopts Swin Transformer; the second network branch module adopts deep convolutional neural network DCNN.
[0037] It should be noted that if Figure 5 As shown in the figure, the RepLKNet model significantly improves the model's receptive field by introducing large kernel convolutions while maintaining efficient computation and parameter optimization. During training, a composite module combining large kernel decomposition and a multi-branch structure significantly reduces the computational burden, using cross-entropy as the loss function. The model's final output includes a classification prediction and a confidence score for the classification result.
[0038] Furthermore, the first classification network includes:
[0039] The gastroscopic image group is preprocessed to reduce its resolution by half (from 512*512 to 256*256 in this example), and is input into the first network branch module and the backbone network module respectively;
[0040] The feature vector output by the first network branch module is fused with the feature vector output by the backbone network module, and the fused features are input into the multi-layer perceptron MLP to obtain a first classification label; the first classification label includes the anatomical parts of the stomach including the fundus, body, and antrum.
[0041] It should be noted that the first network branch module adopts SwinTransformer with strong global information extraction capability, which can analyze images in a larger receptive field, thereby obtaining macroscopic anatomical part information in the image.
[0042] Furthermore, the second classification network includes:
[0043] Obtaining the repeated areas obtained by image registration in the gastroscopic image group, and inputting them into the second network branch module and the backbone network module respectively after image cropping and deformation;
[0044] The feature vector of the repeated area of the gastroscopic image group output by the second network branch module is fused with the feature vector output by the backbone network module, and the fused features are input into a multi-layer perceptron to obtain a second classification label; the second classification label includes gastric mucosal lesions including normal and atrophic.
[0045] It should be noted that the second network branch module adopts a deep convolutional neural network DCNN to deeply mine the gastric mucosal lesions in the repeated areas, and the second network branch module outputs a feature vector containing gastric mucosal lesion information.
[0046] Furthermore, in the process of obtaining the first classification label and the second classification label, and thus obtaining the (first classification label, second classification label) result pair corresponding to the current gastroscopic image group, the process includes: setting all gastroscopic images in the same gastroscopic image group to the same prediction result, and the final set prediction result is based on the prediction result with the largest weight in the image group; wherein the specific calculation method of the weight of the prediction result is: the average value of the confidence score output by the calculation model of each prediction category.
[0047] In summary, the present invention discloses a multi-task autoimmune gastritis gastroscopic image classification network based on image registration and image grouping with patients as units, combining gastroscopic image registration and grouping, and gastroscopic image anatomical region and lesion condition classification network. Similar images in multiple gastroscopic images of a patient are integrated, and the correlation between gastroscopic images is extracted to provide additional auxiliary information for the classification of gastroscopic images; the gastroscopic image anatomical region and lesion condition classification network uses a central network based on RepLKNet with large kernel convolution and two branch networks based on Swin Transformer and DCNN respectively, taking into account the requirements of large receptive field, local fine-grained information and inductive bias in anatomical region classification and lesion condition classification, accurately classifying gastroscopic images, and providing gastric anatomical site information and gastric mucosal lesion condition information required for the diagnosis of autoimmune gastritis. The present invention realizes the automatic analysis of gastroscopic images of autoimmune gastritis.
[0048] On the other hand, an embodiment of the present invention provides a multi-task autoimmune gastritis gastroscopic image classification system, characterized in that it is used to implement the above-mentioned multi-task autoimmune gastritis gastroscopic image classification method, and the system includes:
[0049] The gastroscopic image registration and grouping module obtains several gastroscopic images of a patient, registers all the gastroscopic images, and obtains a similarity matrix; based on the similarity matrix, the gastroscopic images are clustered and grouped to obtain several gastroscopic image groups;
[0050] The gastroscopic image classification module inputs each gastroscopic image group into the first classification network and the second classification network respectively to obtain the first classification label and the second classification label, and obtains the (first classification label, second classification label) result pair corresponding to the current gastroscopic image group; wherein, the first classification label includes the gastric anatomical parts including the fundus, body and antrum, and the second classification label includes the gastric mucosal lesions including normal and atrophic.
[0051] Regarding the system in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0052] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. A person of ordinary skill in the art can understand and implement it without paying any creative work.
[0053] Accordingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task autoimmune gastritis gastroscopic image classification method as described above. Figure 6 As shown in FIG, a hardware structure diagram of any device with data processing capability in which the multi-task autoimmune gastritis gastroscopic image classification method provided by the embodiment of the present invention is located, except Figure 6 In addition to the processor, memory, and network interface shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, generally based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0054] Accordingly, the present application also provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the multi-task autoimmune gastritis gastroscopic image classification method as described above. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities as described in any of the aforementioned embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), an SD card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium can also include both an internal storage unit and an external storage device of any device with data processing capabilities. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store data that has been output or is to be output.
[0055] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only.
[0056] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.
Claims
1. A multi-task autoimmune gastritis gastroscopic image classification method, characterized in that: The method comprises: Acquire a plurality of gastroscopic images of a patient, register all the gastroscopic images, and obtain a similarity matrix; cluster the gastroscopic images according to the similarity matrix to group the gastroscopic images, and obtain a plurality of gastroscopic image groups; Input each gastroscopic image group into the first classification network and the second classification network respectively to obtain a first classification label and a second classification label, and obtain a result pair of the first classification label and the second classification label corresponding to the current gastroscopic image group; wherein the first classification label includes gastric anatomical parts including fundus, body, and antrum, and the second classification label includes pathological conditions including normal and atrophic; Among them, the process of inputting each gastroscopic image group into the first classification network and the second classification network respectively includes: preprocessing the gastroscopic image group to reduce its resolution by half of its original resolution, and inputting it into the first network branch module; obtaining the repeated area in the gastroscopic image group obtained by image registration, and inputting it into the second network branch module after image cropping and deformation.
2. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 1, characterized in that: The process of registering all gastroscopic images and obtaining a similarity matrix includes: The qualified registration points between all gastroscopic images are found through the scale-invariant feature transformation matching algorithm; Set the maximum and minimum thresholds corresponding to qualified registration points; When the number of qualified registration points between two gastroscopic images is less than the minimum threshold, the similarity between the two gastroscopic images is recorded as 0; when the number of qualified registration points between two gastroscopic images is greater than the maximum threshold, the similarity between the two gastroscopic images is recorded as 1; when the number of qualified registration points between two gastroscopic images is within the range of the minimum threshold and the maximum threshold, the number of qualified registration points is normalized to convert it into a similarity ranging from 0 to 1; The calculated similarity is added into a matrix with the image as the horizontal and vertical axes to obtain a similarity matrix; Based on the similarity matrix, a hierarchical clustering algorithm was used to cluster the patients' gastroscopic images to obtain several gastroscopic image groups reflecting the same gastroscopic image features.
3. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 1, characterized in that: The first classification network includes a first network branch module and a backbone network module; the second classification network includes a second network branch module and a backbone network module; wherein the backbone network module in the first classification network is the same as the backbone network module in the second classification network, and both adopt the RepLKNet model.
4. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 3, characterized in that: The first classification network includes: Preprocessing the gastroscopic image group to reduce its resolution by half, and inputting it into the first network branch module and the backbone network module respectively; The feature vector output by the first network branch module is fused with the feature vector output by the backbone network module, and the fused features are input into a multi-layer perceptron to obtain a first classification label; the first classification label includes the anatomical parts of the stomach including the fundus, body, and antrum.
5. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 3, characterized in that: The second category network includes: Obtaining the repeated areas obtained by image registration in the gastroscopic image group, and inputting them into the second network branch module and the backbone network module respectively after image cropping and deformation; The feature vector output by the second network branch module is fused with the feature vector output by the backbone network module, and the fused features are input into a multi-layer perceptron to obtain a second classification label; the second classification label includes gastric mucosal lesions including normal and atrophy.
6. The multi-task autoimmune gastritis gastroscopic image classification method according to claim 4, characterized in that: The first network branch module adopts Swin Transformer; the second network branch module adopts deep convolutional neural network DCNN.
7. A multi-task autoimmune gastritis gastroscopic image classification system, characterized by: For implementing the multi-task autoimmune gastritis gastroscopic image classification method according to any one of claims 1 to 6, the system comprises: The gastroscopic image registration and grouping module obtains several gastroscopic images of a patient, registers all the gastroscopic images, and obtains a similarity matrix; based on the similarity matrix, the gastroscopic images are clustered and grouped to obtain several gastroscopic image groups; The gastroscopic image classification module inputs each gastroscopic image group into the first classification network and the second classification network respectively to obtain the first classification label and the second classification label, and obtains the result pair of the first classification label and the second classification label corresponding to the current gastroscopic image group; wherein, the first classification label includes the anatomical parts of the stomach including the fundus, body and antrum, and the second classification label includes the pathological conditions including normal and atrophy.
8. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-task autoimmune gastritis gastroscopic image classification method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-task autoimmune gastritis gastroscopic image classification method according to any one of claims 1 to 6 is implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multi-task autoimmune gastritis gastroscopic image classification method according to any one of claims 1 to 6 is implemented.