A gastric inflammation image analysis method based on an integrated network structure and multi-device migration

Abstract

The application discloses a gastric inflammation image analysis method based on an integrated network structure and multi-device migration, designs a local attention grouping method to avoid losing a large amount of original image feature information of a traditional down-sampling input image, and designs a double migration strategy to fully utilize information of multi-device endoscope images according to the similar characteristics of internal physical imaging between the multi-device endoscopes, and improves the accuracy of a computer image recognition system for gastric atrophy and intestinal metaplasia.

Description

Technical Field

[0001] This invention relates to, and more specifically to, a method for analyzing gastritis images based on an integrated network structure and multi-device migration. Background Technology

[0002] Gastric cancer is the third leading cause of cancer death worldwide, with gastric atrophy (GA) and intestinal metaplasia (IM) being the most common stages of its development. Therefore, early detection of gastritis, focusing on GA and IM, is crucial for the diagnosis and prevention of early-stage gastric cancer. In actual diagnosis, the false negative rate for GA and IM is as high as 42%, making the design of computer-aided intelligent diagnostic systems related to gastric atrophy and intestinal metaplasia using deep learning a cutting-edge and important task. However, the design of deep learning models for GA and IM detection is limited by existing data. Currently, there are two main endoscopic techniques: white light imaging (WLI) and linked color imaging (LCI). Traditional computer-aided systems designed for gastritis are trained based on traditional WLI endoscopic images. In contrast, LCI endoscopy enhances the differences between different color spaces, making lesions easier to distinguish under such light, thus improving the identification of subtle gastrointestinal lesions and the diagnosis of GA boundaries. Furthermore, under LCI, the mucosa of IM appears pale purple, clearly distinguishing it from the commonly seen red mucosa. Therefore, we further designed and developed a next-generation deep learning model to assist in the real-time detection of GA and IM based on images collected by multi-device WLI and LCI endoscopes.

[0003] Meanwhile, existing computer-aided intelligent diagnostic methods for gastritis directly downsample the original images, causing them to lose much of the relevant and important information. Therefore, it is necessary to design an efficient model structure that can fully utilize the original high-resolution gastritis endoscopic images to improve the model's diagnostic accuracy. Traditional intelligent gastroscopy detection systems are developed and designed based on single devices, ignoring the principle of similarity between inline imaging of multiple devices. This means that the corresponding deep learning models can only rely on images acquired by a single device, limiting the expansion of the gastritis image dataset. WLI and LCI endoscopes have similarities in physical imaging and color space. Therefore, designing an efficient transfer learning method based on the imaging similarity of multiple devices has practical application value, which can effectively alleviate the scarcity of medical images and improve model performance.

[0004] Therefore, it is crucial to design an efficient deep learning model that combines traditional WLI, advanced LCI endoscope equipment, and deep learning technology. Summary of the Invention

[0005] To address the aforementioned shortcomings in the existing technology, this invention provides a gastritis image analysis method based on an integrated network structure and multi-device migration.

[0006] To achieve the above-mentioned objectives, the technical solution adopted by this invention is as follows:

[0007] A gastritis image analysis method based on integrated network structure and multi-device migration includes the following steps:

[0008] S1. Construct an image processing system based on integrated network structure and multi-device migration to acquire original gastritis detection imaging images, wherein the image processing system based on integrated network structure and multi-device migration includes a local grouping integration module, a local attention module, and a multi-device dual migration strategy module;

[0009] S2. Using the local grouping inheritance module, randomly crop a 64*64 resolution local image from the original gastritis detection imaging image as input;

[0010] S3. Use the local attention module to extract local features from the input local image, and then aggregate the extracted local features and global features.

[0011] S4. Utilize a dual-transfer strategy based on multiple devices to construct a training network and perform cross-domain knowledge transfer learning on aggregated local and global features.

[0012] Furthermore, step S3 specifically includes the following steps:

[0013] S31. Use the attention module to compress the local input image, compressing the two-dimensional feature information of the input image into a scalar value.

[0014] S32. Activate the output scalar value using an activation function, and return the squeezed image to its original size through a fully connected layer to obtain the original feature space of the local image.

[0015] S33. The compressed scalar values ​​are combined with the original feature space to obtain local features in the local view;

[0016] S34. Aggregate local features with global features.

[0017] Furthermore, the specific method of extrusion in S31 is as follows:

[0018]

[0019] Among them, z c u is the output scalar value. c Let W be the motivation vector, H be the image width, C be the image height, and F be the number of channels in the feature map. Let i,j be the pixel coordinates of the motivation vector. sq (u c ) is the squeezing operation function of the motivation vector.

[0020] Furthermore, the excitation operation in S32 implements a gating mechanism by using σ (Sigmoid) and δ (ReLU) activation functions.

[0021] Furthermore, the scaling operation in S33 is specifically performed as follows:

[0022]

[0023] in, For the local features of the output, F scale (u c ,s c ) is the scaling operation function, s c This represents the original feature space.

[0024] Furthermore, the aggregation operation in S34 is represented as follows:

[0025] x = cat(x0,x1,x2,x3,...,x) m )

[0026] Where x1, x2, x3, ..., x m The feature vector output for each inherited module, m is the number of sub-integrated modules, x0 is the feature vector output by the backbone network, and cat() is the aggregation function.

[0027] Furthermore, step S4 specifically includes the following steps:

[0028] S41. Construct a deep neural network based on a dual transfer strategy for multiple devices, and transfer the knowledge of visual representation from the ImageNet dataset to WLI images to complete the knowledge transfer learning between two domains with significantly different distributions.

[0029] S42. Transfer learning is performed on the LCI dataset to fine-tune the parameters of the constructed deep neural network;

[0030] S43. Use the fine-tuned deep neural network to predict the result of the input image.

[0031] The present invention has the following beneficial effects:

[0032] 1) Effectively utilize the high resolution of the original gastrointestinal endoscopy images and use the local grouping integration module to improve the accuracy of the model in the atrophic intestinal metaplasia problem.

[0033] 2) It can make full use of the limited atrophic intestinal data collected by multi-device endoscopes, overcome the scarcity of accurately labeled datasets, and find the relationship between images from multiple devices to improve the accuracy of model detection.

[0034] 3) By combining deep learning with a sufficient standard labeled dataset of gastritis atrophy and intestinal metaplasia, a real-time and accurate intelligent computer-aided system for atrophy and intestinal metaplasia can be realized. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the gastritis image analysis method based on integrated network structure and multi-device migration according to the present invention.

[0036] Figure 2 This is a schematic diagram of the local grouping integration model structure in an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of the structural components of the tail attention module in an embodiment of the present invention.

[0038] Figure 4 This is the result of actual prediction performed on real-time video in an embodiment of the present invention. Detailed Implementation

[0039] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0040] A gastritis image analysis method based on integrated network architecture and multi-device migration, such as Figure 1 As shown, it includes the following steps:

[0041] S1. Construct an image processing system based on an integrated network structure and multi-device migration to acquire raw gastritis detection imaging images. The image processing system based on the integrated network structure and multi-device migration includes a local grouping integration module, such as... Figure 2 As shown; Local attention module, such as Figure 3 As shown; a dual migration strategy module based on multiple devices;

[0042] Inspired by the structure of human eye cells, a local attention grouping model architecture was designed based on a global and local ensemble learning method, effectively improving the model's utilization of high-resolution endoscopic images. Simultaneously, a dual transfer training strategy was designed by fully integrating multi-device white light (WLI) and linked imaging (LCI) endoscopes, effectively improving the model's accuracy in detecting atrophy (GA) and intestinal metaplasia (IM).

[0043] S2. Using the local grouping inheritance module, randomly crop a 64*64 resolution local image from the original gastritis detection imaging image as input;

[0044] The LAG model based on the structure of the human eye proposed in this invention can effectively avoid the compression of the input image. In the human eye imaging system, the world seen is not "completely clear and whole," but rather "partially clear and whole." The ganglion cells of the human eye are mainly connected to photoreceptors in two ways. On the one hand, if a rod cell detects a change in light, it activates the connected bipolar cell, which in turn stimulates the connected ganglion cells. Ganglion cells connected to rod cells are easily activated but not precisely enough to capture local details in the visual field; this rod-cell-based connection structure is suitable for capturing broad global information. On the other hand, cone cells and ganglion cells are connected in pairs. Although this connection is more expensive (one cone occupies one ganglion cell), cone cells provide more precise visual information; this cone-cell-based connection structure is suitable for receiving locally accurate information. Inspired by the structure of human eye cells, an efficient local grouping integration module is designed to supplement the local pixel information of the original input. We first randomly crop 64x64 resolution local images from the original image as the input to each subset module. Second, we add a squeeze and excitation (SE) attention module at the end of the submodule to enhance the network's focus on extracting local lesion information. Third, the compressed entire image is input as global visual information into a backbone neural network corresponding to the rod cells of the human eye. We use SE-ResNet to extract global pixel information and introduce an ensemble submodule to focus on local information.

[0045] S3. Use the local attention module to extract local features from the input local image, and then aggregate the extracted local features and global features.

[0046] Design of the Local Attention Module. We introduce the SE attention module at the tail of the sub-model, which enables the model to focus on key local features. The SE attention module mainly performs three operations: squeezing, firing, and scaling. Specifically, the squeezing operation compresses the information in the space into a "global descriptor" through global average pooling, that is, the original two-dimensional feature information is compressed and represented as a scalar value. The shape of the input feature map of the SE attention module is represented as W×H×C, where W is the width, H is the height, C is the number of channels of the feature map, and u c Let be the motivation vector. Then, the output of the corresponding squeeze operation function is:

[0047]

[0048] The activation operation is similar to the gate control module in a recurrent memory network, primarily used to learn the nonlinear relationships between channels. Based on this, the gating mechanism is implemented using σ(Sigmoid) and δ(ReLU) activation functions, and the original dimensions are returned through a fully connected layer, as shown below.

[0049] s = F ex (Z,W)=σ(W2δ(W1Z))

[0050] The scaling operation multiplies the scale value obtained from the squeezing operation with the original feature space. This operation simulates the effect of calibrating the filter output using key features, which are the focal points that should be of key interest in the local view.

[0051]

[0052] The output feature vector of each sub-integration module is represented as x1, x2, x3, ..., x m , where m is the number of sub-ensemble modules. The output feature vector of the backbone network is represented as x0. Because the features learned by each sub-ensemble module are local features of the original image, we aggregate the local and global features at the end of the model.

[0053] x = Cat(x0,x1,x2,x3,...,x) m )

[0054] S4. Utilize a dual-transfer strategy based on multiple devices to construct a training network and perform cross-domain knowledge transfer learning on aggregated local and global features.

[0055] This invention proposes a dual-transfer strategy based on multiple devices. The dual-transfer (DTL) strategy addresses the challenge of training deep neural networks with limited medical image samples by leveraging the correlation between the source and target domains in visual tasks. This strategy comprises two transfer phases: In the first phase, the model performs transfer learning from the ImageNet dataset (a large-class dataset) to WLI images, completing knowledge transfer learning between two domains with significantly different distributions. In the second phase, the model continues transfer learning to the LCI image dataset, fine-tuning the model to learn new knowledge provided by LCI samples, completing knowledge transfer learning between two domains with similar distributions. Therefore, the proposed DTL strategy enables the final model to fully utilize the limited knowledge from WLI and LCI samples, preventing knowledge forgetting during the transfer process and ensuring the model's learning efficiency. Prediction results are as follows... Figure 4 As shown.

[0056] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0057] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0058] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0059] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0060] Those skilled in the art will recognize that the embodiments described herein are intended to help the reader understand the principles of the invention, and should be understood that the scope of protection of the invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific modifications and combinations based on the technical teachings disclosed in this invention without departing from the spirit of the invention, and these modifications and combinations are still within the scope of protection of this invention.

Claims

1. A method for analyzing gastritis images based on an integrated network structure and multi-device migration, characterized in that, Includes the following steps: S1. Construct an image processing system based on integrated network structure and multi-device migration to acquire original gastritis detection imaging images, wherein the image processing system based on integrated network structure and multi-device migration includes a local grouping integration module, a local attention module, and a multi-device dual migration strategy module; S2. Using the local grouping integration module, randomly crop a 64*64 resolution local image from the original gastritis detection imaging image as input; S3. Use the local attention module to extract local features from the input local image, and aggregate the extracted local features and global features. Specifically, this includes the following steps: S31. Use the attention module to compress the local input image, compressing the two-dimensional feature information of the input image into a scalar value. S32. Activate the output scalar value using an activation function, and return the squeezed image to its original size through a fully connected layer to obtain the original feature space of the local image. S33. The scaling operation multiplies the compressed scalar value with the original feature space to obtain the local features in the local view. S34. Aggregate local features with global features; S4. Construct a training network using a multi-device dual-transfer strategy, and perform cross-domain knowledge transfer learning on the aggregated local and global features. This includes the following steps: S41. Construct a deep neural network based on a dual transfer strategy for multiple devices, and transfer the knowledge of visual representation from the ImageNet dataset to WLI images to complete the knowledge transfer learning between two domains with significantly different distributions. S42. Continue transfer learning on the LCI image dataset and fine-tune the parameters of the constructed deep neural network; S43. Use the fine-tuned deep neural network to predict the result of the input image.

2. The gastritis image analysis method based on integrated network structure and multi-device migration according to claim 1, characterized in that, The specific method of extrusion in S31 is as follows: in, The output scalar value. For motivation vectors, W Image width, H Where C is the image height and C is the number of channels in the feature map. The coordinates of the pixel point of the motivation vector. This is the squeezing operation function for the motivation vector.

3. The gastritis image analysis method based on integrated network structure and multi-device migration according to claim 1, characterized in that, The excitation operation in S32 is performed by using... and Activation functions are used to implement gating mechanisms.

4. The gastritis image analysis method based on integrated network structure and multi-device migration according to claim 1, characterized in that, The scaling operation in S33 is specifically performed as follows: in, For the output local features, For scaling operation functions, This represents the original feature space.

5. The gastritis image analysis method based on integrated network structure and multi-device migration according to claim 1, characterized in that, The aggregation operation in S34 is represented as follows: in, The feature vector output by each integration module, m For the number of sub-integrated modules, The backbone network outputs feature vectors. It is an aggregate function.

Images