Multitask autoimmune gastritis gastroscope image classification method, device and medium

Abstract

The invention discloses a multi-task autoimmune gastritis gastroscope image classification method and device and a medium, and the method comprises the steps: obtaining a plurality of gastroscope images of a patient, and carrying out the registration of all gastroscope images, and obtaining a similarity matrix; according to the similarity matrix, the gastroscope images are clustered, so that the gastroscope images are grouped, and a plurality of gastroscope image groups are obtained; respectively inputting each gastroscope image group into a first classification network and a second classification network to obtain a first classification label and a second classification label, and obtaining a result pair (the first classification label and the second classification label) corresponding to the current gastroscope image group; wherein the first classification label comprises gastric anatomical parts including the stomach fundus, the stomach body and the gastric antrum, and the second classification label comprises lesion conditions including normal and atrophy. According to the method, the requirements for a large receptive field, local fine-grained information and inductive bias in anatomical region classification and lesion condition classification are considered, and gastroscope images are accurately classified.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a multi-task autoimmune gastritis gastroscopy image classification method, device, and medium based on image registration and image grouping for each patient. Background Art

[0002] Autoimmune gastritis is considered to have a great correlation with diseases such as gastric neuroendocrine tumor (NET) and gastric cancer. Therefore, timely and accurate diagnosis of autoimmunity is of great significance for preventing gastric cancer developed from autoimmunity. As a type of chronic atrophic gastritis, the main pathological manifestations of autoimmune gastritis under gastroscopy are significantly different from those of general atrophy caused by Helicobacter pylori infection in the lesion site. Therefore, when diagnosing autoimmune gastritis based on gastroscopy images, it is not only necessary to distinguish normal and atrophic gastric mucosa, but also a clear understanding of the gastric anatomical location where the gastric mucosa is located is required.

[0003] Therefore, there is an urgent need to provide an automated method for identifying autoimmune gastritis gastroscopy images to efficiently and accurately give the analysis results of gastroscopy images by learning the diagnostic experience of professional gastroenterologists. Summary of the Invention

[0004] In view of this, the present invention provides a multi-task autoimmune gastritis gastroscopy image classification method, device, and medium for the interpretation of gastroscopy images related to autoimmunity.

[0005] In a first aspect, an embodiment of the present invention provides a multi-task autoimmune gastritis gastroscopy image classification method, the method comprising:

[0006] Obtain a plurality of gastroscopy images of a patient, register all the gastroscopy images to obtain a similarity matrix; according to the similarity matrix, cluster the gastroscopy images to group the gastroscopy images, and obtain a plurality of groups of gastroscopy images;

[0007] Input each group of gastroscopy images into a first classification network and a second classification network respectively to obtain a first classification label and a second classification label, and obtain a result pair of (first classification label, second classification label) corresponding to the current group of gastroscopy images; wherein, the first classification label includes gastric anatomical locations such as the fundus of the stomach, the body of the stomach, and the antrum of the stomach, and the second classification label includes lesion conditions such as normal and atrophy.

[0008] In a second aspect, an embodiment of the present invention provides a multi-task autoimmune gastritis gastroscopy image classification system for implementing the above multi-task autoimmune gastritis gastroscopy image classification method, the system comprising:

[0009] A gastroscopy image registration and grouping module, which is used to obtain a number of gastroscopy images of a patient, register all the gastroscopy images to obtain a similarity matrix, and cluster the gastroscopy images according to the similarity matrix to group the gastroscopy images, obtaining a number of gastroscopy image groups.

[0010] A gastroscopy image classification module, which is used to input each gastroscopy image group into a first classification network and a second classification network respectively to obtain a first classification label and a second classification label. Among them, the first classification label includes the gastric fundus, gastric body, and gastric antrum, and the second classification label includes normal and atrophy.

[0011] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, and the memory is coupled to the processor. Among them, the memory is used to store program data, and the processor is used to execute the program data to implement the above multi-task autoimmune gastritis gastroscopy image classification method.

[0012] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the above multi-task autoimmune gastritis gastroscopy image classification method is implemented.

[0013] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program / instructions, and when the computer program / instructions are executed by a processor, the above multi-task autoimmune gastritis gastroscopy image classification method is implemented.

[0014] Compared with the prior art, the beneficial effects of the present invention are:

[0015] The present invention provides a multi-task autoimmune gastritis gastroscopy image classification method based on patients. The present invention combines gastroscopy image registration grouping and automated gastroscopy image classification. Aiming at problems such as low accuracy of automated image interpretation in this type of task, an idea of grouping gastroscopy images based on image registration is proposed, and the correlation between gastroscopy images is extracted and used to assist image interpretation. Using the RepLKNet model based on large kernel convolution, it takes into account the performance requirements of the model for classifying the gastric anatomical parts and lesion conditions. On the basis of using one model to solve two problems, the accuracy of gastroscopy image classification is ensured. In addition, aiming at the different characteristics of the two classification tasks of the gastric anatomical parts and the gastric mucosal lesion conditions, two branch networks are added to further improve the accuracy of the two tasks. Description of the Drawings

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0017] Figure 1 It is a flowchart of the multi-task autoimmune gastritis gastroscopy image classification method provided by the embodiment of the present invention;

[0018] Figure 2 It is a schematic diagram of the multi-task autoimmune gastritis gastroscopy image classification method provided by the embodiment of the present invention;

[0019] Figure 3 It is a schematic diagram of the image registration result provided by the embodiment of the present invention;

[0020] Figure 4 It is a schematic diagram of the similarity matrix obtained through image registration and subsequent processing provided by the embodiment of the present invention;

[0021] Figure 5 It is a schematic diagram of the RepLKNet model structure provided by the embodiment of the present invention;

[0022] Figure 6 It is a schematic diagram of an electronic device provided by the embodiment of the present invention. Detailed implementation manners

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0024] It should be noted that, without conflict, the features in the following embodiments and implementation manners can be combined with each other.

[0025] As Figure 1 and Figure 2 shown, the embodiment of the present invention provides a multi-task autoimmune gastritis gastroscopy image classification method, and the method includes the following steps:

[0026] Step S1, obtain a plurality of gastroscopy images of a patient, register all the gastroscopy images to obtain a similarity matrix; according to the similarity matrix, cluster the gastroscopy images to group the gastroscopy images, and obtain a plurality of groups of gastroscopy images.

[0027] Specifically, asFigure 3 and Figure 4 As shown in Figure 4 , the process of registering all gastroscopy images to obtain a similarity matrix includes:

[0028] Finding qualified registration points between all pairs of gastroscopy images through the Scale-invariant feature transform (SIFT) matching algorithm;

[0029] Setting the maximum threshold and minimum threshold corresponding to the qualified registration points; in this example, the minimum value of the qualified registration points is set to 12 and the maximum value is set to 128.

[0030] When the number of qualified registration points between two gastroscopy images is less than the minimum threshold, the similarity between these two gastroscopy images is recorded as 0; when the number of qualified registration points between two gastroscopy images is greater than the maximum threshold, the similarity between these two gastroscopy images is recorded as 1; when the number of qualified registration points between two gastroscopy images is within the range of the minimum threshold and the maximum threshold, the number of these qualified registration points is normalized to convert it into a similarity within the range of 0 to 1.

[0031] Recording the calculated similarity into a matrix with images as the horizontal and vertical axes to obtain a similarity matrix;

[0032] Based on the similarity matrix, using the hierarchical clustering algorithm to cluster the patient's gastroscopy images to obtain several groups of gastroscopy images that reflect the same gastroscopy image features.

[0033] Furthermore, the process of using the hierarchical clustering algorithm to cluster the patient's gastroscopy images includes:

[0034] Specifically, the hierarchical clustering algorithm first calculates the distances between samples, merges the points with the closest distance into the same class, then calculates the distances between classes, merges the classes with the closest distance and a distance less than the threshold into a larger class, and repeats the operation until all classes with a distance between classes less than the threshold are merged into a large class. In this example, the distance between classes is defined as the shortest distance between samples between classes, and the threshold of the distance between classes is set to 1 / 0.1 = 10.

[0035] Step S2: Input each group of gastroscopy images into the first classification network and the second classification network respectively to obtain the first classification label and the second classification label, and obtain the (first classification label, second classification label) result pair corresponding to the current group of gastroscopy images; wherein, the first classification label includes gastric anatomical parts such as the gastric fundus, gastric body, and gastric antrum, and the second classification label includes lesion conditions such as normal and atrophy.

[0036] Among them, the first classification network includes a first network branch module and a backbone network module; the second classification network includes a second network branch module and a backbone network module; among them, the backbone network module in the first classification network is the same as the backbone network module in the second classification network, and both adopt the RepLKNet model; the first network branch module adopts the Swin Transformer; the second network branch module adopts the deep convolutional neural network DCNN.

[0037] It should be noted that as Figure 5 shown, the RepLKNet model significantly improves the receptive field of the model by introducing large kernel convolutions, while maintaining computational efficiency and the efficiency of parameter optimization. A composite module containing large kernel decomposition and a multi-branch structure is used in the training stage to significantly reduce the computational burden, and cross-entropy is used as the loss function. The final output of the model includes the prediction results of classification and the confidence score of the classification results.

[0038] Furthermore, the first classification network includes:

[0039] Preprocess the gastroscope image group to reduce its resolution to half of the original (from 512*512 to 256*256 in this example), and input it into the first network branch module and the backbone network module respectively;

[0040] Fuse the feature vectors output by the first network branch module and the feature vectors output by the backbone network module, and input the fused features into the multi-layer perceptron MLP to obtain the first classification label; the first classification label includes gastric anatomical parts such as the fundus of the stomach, the body of the stomach, and the antrum of the stomach.

[0041] It should be noted that the first network branch module adopts the Swin Transformer with strong global information extraction ability, which can analyze images in a larger receptive field, so as to obtain macroscopic anatomical part information in the images.

[0042] Furthermore, the second classification network includes:

[0043] Obtain the repeated regions obtained by image registration in the gastroscope image group, and input them into the second network branch module and the backbone network module respectively after image cropping and deformation;

[0044] Fuse the feature vectors of the repeated regions of the gastroscope image group output by the second network branch module and the feature vectors output by the backbone network module, and input the fused features into the multi-layer perceptron to obtain the second classification label; the second classification label includes the gastric mucosal lesion conditions such as normal and atrophy.

[0045] It should be noted that the second network branch module uses a deep convolutional neural network (DCNN) to deeply mine the gastric mucosal lesions in the repeated regions, and the second network branch module outputs a feature vector containing gastric mucosal lesion information.

[0046] Furthermore, in the process of obtaining the first classification label and the second classification label, and thus obtaining the (first classification label, second classification label) result pair corresponding to the current gastroscopy image group, it includes: setting all the gastroscopy images in the same gastroscopy image group to the same prediction result, and finally setting the prediction result to be the one with the largest weight in the image group; wherein, the specific calculation method of the weight of the prediction result is: calculating the average value of the confidence scores output by the model for each prediction category.

[0047] In summary, the present invention discloses a multi-task autoimmune gastritis gastroscopy image classification network based on image registration and image grouping with patients as the unit, which combines gastroscopy image registration and grouping, and a classification network for gastroscopy image anatomical regions and lesion conditions. Integrating similar images among multiple gastroscopy images of a patient, extracting the relevant relationships between gastroscopy images, and providing additional auxiliary information for the classification of gastroscopy images; the classification network for gastroscopy image anatomical regions and lesion conditions uses the central network of RepLKNet based on large kernel convolution and two branch networks based on Swin Transformer and DCNN respectively, taking into account the requirements for large receptive fields, local fine-grained information and inductive bias in anatomical region classification and lesion condition classification, accurately classifying gastroscopy images, and providing the gastric anatomical site information and gastric mucosal lesion condition information required for the diagnosis of autoimmune gastritis. The present invention realizes the automatic analysis of autoimmune gastritis gastroscopy images.

[0048] On the other hand, an embodiment of the present invention provides a multi-task autoimmune gastritis gastroscopy image classification system, which is characterized in that it is used to implement the above-mentioned multi-task autoimmune gastritis gastroscopy image classification method, and the system includes:

[0049] A gastroscopy image registration and grouping module, which acquires a number of gastroscopy images of a patient, registers all the gastroscopy images to obtain a similarity matrix; according to the similarity matrix, clusters the gastroscopy images to group the gastroscopy images, and obtains a number of gastroscopy image groups;

[0050] A gastroscopy image classification module, which inputs each gastroscopy image group into the first classification network and the second classification network respectively to obtain the first classification label and the second classification label, and obtains the (first classification label, second classification label) result pair corresponding to the current gastroscopy image group; wherein, the first classification label includes the gastric anatomical sites such as the fundus of the stomach, the body of the stomach, and the antrum of the stomach, and the second classification label includes the gastric mucosal lesion conditions such as normal and atrophy.

[0051] Regarding the system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0052] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the descriptions in the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0053] Correspondingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the multi-task autoimmune gastritis gastroscope image classification method as described above. As Figure 6 shown, it is a hardware structure diagram of any device with data processing capabilities where the multi-task autoimmune gastritis gastroscope image classification method provided by the embodiment of the present invention is located. Except for Figure 6 the processors, memory, and network interfaces shown, any device with data processing capabilities where the device in the embodiment is located usually includes other hardware according to the actual functions of the device with data processing capabilities, which will not be elaborated here.

[0054] Correspondingly, this application also provides a computer-readable storage medium, on which computer instructions are stored. When the instructions are executed by a processor, the multi-task autoimmune gastritis gastroscope image classification method as described above is implemented. The computer-readable storage medium can be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium can also include both the internal storage unit of any device with data processing capabilities and the external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and can also be used to temporarily store the data that has been output or will be output.

[0055] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the content disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only to be considered as exemplary.

[0056] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A multi-task autoimmune gastritis gastroscopic image classification method, characterized in that: The method comprises: Acquire a number of gastroscopic images of a patient, register all the gastroscopic images, and obtain a similarity matrix; cluster the gastroscopic images according to the similarity matrix to group the gastroscopic images, and obtain a number of gastroscopic image groups; Each gastroscopic image group is input into the first classification network and the second classification network respectively to obtain the first classification label and the second classification label, and obtain the (first classification label, second classification label) result pair corresponding to the current gastroscopic image group; wherein the first classification label includes the anatomical parts of the stomach including the fundus, body and antrum, and the second classification label includes the pathological conditions including normal and atrophy.

2. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 1, characterized in that: The process of registering all gastroscopic images and obtaining a similarity matrix includes: Find qualified registration points between all gastroscopic images through scale-invariant feature transformation matching algorithm; Set the maximum and minimum thresholds corresponding to qualified registration points; When the number of qualified registration points between two gastroscopic images is less than the minimum threshold, the similarity between the two gastroscopic images is recorded as 0; when the number of qualified registration points between two gastroscopic images is greater than the maximum threshold, the similarity between the two gastroscopic images is recorded as 1; when the number of qualified registration points between two gastroscopic images is within the range of the minimum threshold and the maximum threshold, the number of qualified registration points is normalized to convert into a similarity ranging from 0 to 1; The calculated similarity is added into a matrix with the image as the horizontal and vertical axes to obtain a similarity matrix; Based on the similarity matrix, a hierarchical clustering algorithm was used to cluster the patients' gastroscopic images to obtain several gastroscopic image groups reflecting the same gastroscopic image features.

3. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 1, characterized in that: The first classification network includes a first network branch module and a backbone network module; the second classification network includes a second network branch module and a backbone network module; wherein the backbone network module in the first classification network is the same as the backbone network module in the second classification network, and both adopt the RepLKNet model.

4. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 3, characterized in that: The first classification network includes: Preprocessing the gastroscopic image group to reduce its resolution by half, and inputting it into the first network branch module and the backbone network module respectively; The feature vector output by the first network branch module is fused with the feature vector output by the backbone network module, and the fused features are input into a multi-layer perceptron to obtain a first classification label; the first classification label includes gastric anatomical parts including the fundus, body, and antrum.

5. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 3, characterized in that: The second category network includes: Obtaining repeated areas in the gastroscopic image group obtained by image registration, and inputting them into the second network branch module and the backbone network module respectively after image cropping and deformation; The feature vector output by the second network branch module is fused with the feature vector output by the backbone network module, and the fused features are input into a multi-layer perceptron to obtain a second classification label; the second classification label includes gastric mucosal lesions including normal and atrophy.

6. A multi-task autoimmune gastritis gastroscopic image classification method according to claim 4, characterized in that: The first network branch module adopts Swin Transformer; the second network branch module adopts deep convolutional neural network DCNN.

7. A multi-task autoimmune gastritis gastroscopic image classification system, characterized in that: Used to implement the multi-task autoimmune gastritis gastroscopic image classification method according to any one of claims 1 to 6, the system comprises: The gastroscopic image registration and grouping module obtains a number of gastroscopic images of a patient, registers all the gastroscopic images, and obtains a similarity matrix; according to the similarity matrix, the gastroscopic images are clustered to group the gastroscopic images, and a number of gastroscopic image groups are obtained; The gastroscopic image classification module inputs each gastroscopic image group into the first classification network and the second classification network respectively, obtains the first classification label and the second classification label, and obtains the (first classification label, second classification label) result pair corresponding to the current gastroscopic image group; wherein the first classification label includes the anatomical parts of the stomach including the fundus, body and antrum, and the second classification label includes the pathological conditions including normal and atrophy.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multi-task autoimmune gastritis gastroscopic image classification method described in any one of claims 1-6 above.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-task autoimmune gastritis gastroscopic image classification method as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the multi-task autoimmune gastritis gastroscopic image classification method described in any one of claims 1-6 is implemented.

Images