A digestive endoscope image diagnosis system based on average multi-head self-attention mechanism

Abstract

The application relates to the technical field of medical image processing, and discloses a digestive endoscope image diagnosis system based on an average multi-head self-attention mechanism, which comprises a global feature information extraction module, a local feature information extraction module and a feature fusion module.The global feature information extraction module is used for performing multi-stage global feature information extraction on a digestive endoscope image through convolution operation and the average multi-head self-attention mechanism, and the average multi-head self-attention mechanism is adopted in each stage.The average multi-head self-attention mechanism adopts an average filter to generate keys and values of an input feature map, and obtains an output feature map through the multi-head self-attention mechanism based on queries, keys and values.The local feature information extraction module is used for performing local feature information extraction on the digestive endoscope image.The feature fusion module is used for fusing global feature information and local feature information.A rough diagnosis module is used for classifying to obtain a disease type.The system can capture the overall characteristics of the digestive endoscope image, provide more abundant and comprehensive semantic information, and improve the precision of digestive endoscope image diagnosis.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and in particular to a digestive endoscopy image diagnostic system based on an average multi-head self-attention mechanism. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] Gastrointestinal endoscopy has become a common diagnostic method used to detect and diagnose digestive tract diseases such as gastric ulcers, colitis, and esophagitis. Endoscopic images are crucial for physicians in making diagnostic and treatment decisions. However, due to the complex structure of the digestive tract, image quality is affected by various factors, such as lesion location, lighting conditions, and lens quality, posing challenges for physicians in image analysis. Therefore, developing a system capable of automatically diagnosing gastrointestinal endoscopic images has significant clinical importance.

[0004] In recent years, deep learning technology has made significant progress in the field of medical imaging, especially convolutional neural networks (CNNs), which have demonstrated outstanding performance in image recognition and classification tasks. CNNs can automatically learn image features, enabling efficient image analysis and recognition, and have been widely applied in the automatic diagnosis and disease classification of medical images. In the field of gastrointestinal endoscopic image diagnosis, CNNs can learn the features of lesion areas, helping doctors quickly and accurately identify the type and location of lesions, thus improving diagnostic efficiency and accuracy.

[0005] However, current deep learning models for digestive endoscopy images still face several challenges. Due to the diversity and complexity of digestive endoscopy images, deep learning models may lack generalization ability on new datasets, leading to performance degradation. Furthermore, digestive endoscopy images typically have high resolution and complex structures, requiring significant computational resources for processing, leaving room for improvement in both performance and speed. The shape and size of lesions in digestive endoscopy images vary considerably due to differences in the types of digestive tract diseases, varying endoscopy angles and distances, and the uncertain size and location of lesions, necessitating strong robustness and generalization ability from deep learning models. Additionally, the presence of numerous noise and interference factors in digestive endoscopy images, such as air bubbles and food residue, makes the extraction and segmentation of lesion areas by deep learning models challenging. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a gastrointestinal endoscopy image diagnostic system based on an average multi-head self-attention mechanism. This system proposes an average multi-head self-attention mechanism to extract global feature information from the input image. Unlike traditional multi-head self-attention mechanisms, the average multi-head self-attention mechanism inputs the input feature map into an average filter to generate keys and values. Therefore, its length is shorter than the input feature map, resulting in stronger global perception capabilities. This enhances the robustness and generalization ability of convolutional neural network models, enabling the capture of the overall characteristics of gastrointestinal endoscopy images, providing richer and more comprehensive semantic information, and improving the accuracy of gastrointestinal endoscopy image diagnosis.

[0007] In a first aspect, the present invention provides a digestive endoscopy image diagnostic system based on an average multi-head self-attention mechanism;

[0008] A gastrointestinal endoscopic image diagnostic system based on average multi-head self-attention mechanism includes an image acquisition module, a global feature information extraction module, a local feature information extraction module, a feature fusion module, and a coarse diagnosis module.

[0009] The image acquisition module is used to acquire digestive endoscopy images to be diagnosed;

[0010] The global feature information extraction module is used to extract global feature information from the digestive endoscopy image in multiple stages through convolution operations and an average multi-head self-attention mechanism. Each stage adopts the average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through the multi-head self-attention mechanism.

[0011] The local feature information extraction module is used to extract local feature information from the digestive endoscopy image through convolution operations;

[0012] The feature fusion module is used to fuse the global feature information and the local feature information to obtain multi-scale information;

[0013] The coarse diagnostic module is used to classify the disease type of the digestive endoscopy image based on the multi-scale information.

[0014] Furthermore, each stage of the global feature information extraction employs a sequentially connected large kernel convolutional block, image patch embedding layer, encoder, and convolutional layer;

[0015] The encoder is composed of a first residual structure and a second residual structure connected together. The first residual structure includes a normalization layer and an average multi-head attention module, and the average multi-head attention module adopts the average multi-head self-attention mechanism. The second residual structure includes a normalization layer and a feedforward layer.

[0016] Furthermore, the large kernel convolutional block contains several identical convolutional modules, and each convolutional module adopts a residual structure consisting of a batch normalization layer, a convolutional layer, a large kernel depth convolutional layer, and an activation function.

[0017] Furthermore, the image block embedding layer divides the feature map output by the large kernel convolution block into multiple image blocks, and maps each image block to one dimension through a linear mapping layer, which is then used as the input of the encoder.

[0018] Furthermore, the local feature information extraction module includes several sequentially connected convolutional modules, and each convolutional module includes a sequentially connected convolutional layer, a batch normalization layer, and an activation function.

[0019] Furthermore, the feature fusion module obtains the multi-scale information by upsampling the global feature information and then concatenating it with the local feature information.

[0020] Furthermore, the coarse diagnosis module compresses the multi-scale information through a global average pooling layer and then uses a linear classifier to obtain the disease type to which the digestive endoscopy image belongs.

[0021] Secondly, the present invention also provides an electronic device, comprising:

[0022] Memory, used for non-transitory storage of computer-readable instructions; and

[0023] Processor, for executing the computer-readable instructions,

[0024] When the computer-readable instructions are executed by the processor, the following steps are performed:

[0025] Obtain endoscopic images of the digestive tract to be diagnosed;

[0026] The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism.

[0027] Local feature information is extracted from the digestive endoscopy image through convolution operation;

[0028] The global and local feature information are fused to obtain multi-scale information;

[0029] Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

[0030] Thirdly, the present invention also provides a storage medium for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the following steps are performed:

[0031] Obtain endoscopic images of the digestive tract to be diagnosed;

[0032] The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism.

[0033] Local feature information is extracted from the digestive endoscopy image through convolution operation;

[0034] The global and local feature information are fused to obtain multi-scale information;

[0035] Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

[0036] Fourthly, the present invention also provides a computer program product, which is a computer program product containing a computer program, wherein when the processor executes the computer program, the following steps are performed:

[0037] Obtain endoscopic images of the digestive tract to be diagnosed;

[0038] The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism.

[0039] Local feature information is extracted from the digestive endoscopy image through convolution operation;

[0040] The global and local feature information are fused to obtain multi-scale information;

[0041] Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] This invention proposes an average multi-head self-attention mechanism to extract global feature information from input images. Unlike traditional multi-head self-attention mechanisms, the average multi-head self-attention mechanism inputs the input feature map into an average filter to generate keys and values. Therefore, its length is smaller than that of the input feature map, giving it a stronger global perception capability. This can enhance the robustness and generalization ability of convolutional neural network models, capture the overall characteristics of digestive endoscopy images, provide richer and more comprehensive semantic information, and improve the accuracy of digestive endoscopy image diagnosis.

[0044] The encoder of the present invention is composed of a first residual structure consisting of a normalization layer and an average multi-head attention module, and a second residual structure consisting of a normalization layer and a feedforward layer. It is suitable for extracting global feature information of digestive endoscopy images, can activate the complete object range, and capture the overall information of digestive endoscopy images.

[0045] The large kernel convolutional block of this invention constructs a multi-layer global information extraction path through convolution operations to capture global feature information. Each convolutional module adopts a residual structure composed of batch normalization layers, convolutional layers, large kernel depth convolutional layers and activation functions, which can preserve the global features of digestive endoscopy images to the maximum extent and enrich the global feature representation.

[0046] The local feature information extraction module of this invention constructs a multi-layer local information extraction path through convolution operations to capture local feature information. Each convolution module consists of a convolutional layer, a batch normalization layer and an activation function connected in sequence. It can effectively extract local feature information of digestive endoscopy images while reducing the computational complexity and processing time of convolutional neural network models, enriching the local feature representation and helping to avoid the influence of noise and interference factors.

[0047] The advantages of additional aspects of the invention will be set forth in part in the description which follows, or may be learned by practice of the invention. Attached Figure Description

[0048] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0049] Figure 1 This is a structural diagram of a digestive endoscopy image diagnostic system based on an average multi-head self-attention mechanism, according to Embodiment 1 of the present invention.

[0050] Figure 2 This is a structural diagram of the global feature information extraction module and the local feature information extraction module according to Embodiment 1 of the present invention;

[0051] Figure 3This is a structural diagram of the Mean-transformer encoder module according to Embodiment 1 of the present invention;

[0052] Figure 4 This is a structural diagram of the large kernel convolution module in Embodiment 1 of the present invention. Detailed Implementation

[0053] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0054] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0055] All data acquisition in this embodiment is carried out in accordance with laws and regulations and with user consent, and the data is used legally.

[0056] Example 1

[0057] This embodiment provides a gastrointestinal endoscopic image diagnostic system based on an average multi-head self-attention mechanism.

[0058] This embodiment provides a digestive endoscopy image diagnostic system based on the average multi-head self-attention mechanism. It utilizes a digestive tract disease diagnostic model (convolutional neural network model) to classify and diagnose various diseases in the digestive tract, thereby assisting doctors in diagnostic analysis and improving the efficiency of large-scale digestive tract disease screening.

[0059] This embodiment provides a digestive endoscopy image diagnostic system based on the average multi-head self-attention mechanism, which uses two different feature extraction paths (global information extraction path and local information extraction path) to obtain global and local information of lesions.

[0060] On one hand, in the global information extraction path, global feature information of the input image is extracted through a self-attention mechanism. Furthermore, a Mean-MHSA multi-head self-attention mechanism, different from the traditional MHSA, is used in the global information extraction path. In Mean-MHSA, the query, key, and value tensors are Q, K, and V, respectively. K and V are not directly generated from the input feature map; instead, the input feature map is fed into an averaging filter to generate K and V, whose lengths are shorter than the input feature map. Therefore, Mean-MHSA is more efficient than traditional MHSA. Moreover, the large-kernel convolutional layer preceding Mean-MHSA consists of six units, each containing a large number of components, which maximizes the preservation of global features and enriches the feature representation.

[0061] On the other hand, by extracting local feature information from the input image based on convolutional neural networks, local feature information can be effectively extracted and feature representation can be enriched while reducing the computational complexity and processing time of the convolutional neural network model.

[0062] Finally, the feature fusion module is used to fuse the feature information extracted above.

[0063] This embodiment provides a gastrointestinal endoscopic image diagnostic system based on an average multi-head self-attention mechanism, such as... Figure 1 As shown, it includes an image acquisition module, a global feature information extraction module, a local feature information extraction module, a feature fusion module (feature information fusion module), and a coarse diagnosis module.

[0064] The global feature information extraction module, the local feature information extraction module, the feature fusion module, and the coarse diagnosis module constitute the convolutional neural network model in this embodiment.

[0065] (1) Image acquisition module, used to acquire digestive endoscopy images to be diagnosed.

[0066] (2) Global feature information extraction module (global feature information extraction path), based on convolutional neural network, performs multi-stage global feature information extraction on input image (gastrointestinal endoscopy image).

[0067] Because of the different types of diseases, different endoscopic shooting angles and distances, and the uncertain size and location of lesions in digestive tract diseases, some lesions occupy a large area of ​​the image, global information is particularly important. Therefore, the global feature information in the input image is extracted by the multi-head self-attention mechanism in the improved transformer structure.

[0068] like Figure 2 As shown, the global feature information extraction module (global feature branch) constructs a four-layer global information extraction path through convolution operations and a Mean-transformer encoder to extract global feature information from the input image. That is, it contains four stage modules (stage 1, stage 2, stage 3 and stage 4). Each layer (each stage module or each stage) consists of a large kernel convolution module (abbreviated as large kernel convolution block), a patch embedding layer, a Mean-transformer encoder module (abbreviated as encoder), and a convolutional layer. By adding a multi-head self-attention mechanism, more complete global feature information is obtained, further enriching the feature information representation.

[0069] like Figure 4As shown, the large-kernel convolutional module constructs a six-layer global information extraction path to capture global feature information through convolutional operations. Specifically, it contains six identical convolutional modules. Each layer (each convolutional module) includes a residual structure consisting of two batch normalization layers, four convolutional layers with a kernel size of 1, one large-kernel depthwise convolutional layer (with a kernel size of 31), and one activation function. This structure maximizes the preservation of global features and enriches feature representation. Specifically, each layer (each convolutional module) contains a batch normalization layer, a convolutional layer with a kernel size of 1, a large-kernel convolutional layer (DW Conv) with a kernel size of 31, a convolutional layer with a kernel size of 1, a batch normalization layer, a convolutional layer with a kernel size of 1, an activation function (GELU), and another convolutional layer with a kernel size of 1, connected sequentially. The input and output of each convolutional module are fused and used as the input of the next convolutional module.

[0070] The Patch Embedding layer is used to divide the input image (feature map output by large kernel convolutional blocks) into multiple patches (image patches), and then the linear mapping layer maps each patch to a one-dimensional token, which serves as the input to the Mean-Transformer encoder module.

[0071] like Figure 3 As shown, the Mean-transformer encoder module consists of a first residual structure composed of a Norm normalization layer and a Mean-MHSA (mean multi-head attention) module, and a second residual structure composed of a Norm normalization layer and a feed-forward layer. It is suitable for extracting global feature information, activating the entire object range, and capturing overall information. The input and output of the first residual structure are fused and then input into the second residual structure. The input and output of the second residual structure are then fused and reconstructed to obtain the output of the Mean-transformer encoder module.

[0072] The Mean-MHSA module uses a different Mean multi-head self-attention mechanism than the traditional MHSA. It feeds the input feature map into an averaging filter to generate keys and values, as shown in the following formula:

[0073] P = MeanFilter(X)

[0074] (Q, K, V) = (XW) q PW k PW υ )

[0075]

[0076] Where X represents the two-dimensional input of Mean-MHSA, and P represents the mean filter of the two-dimensional input of Mean-MHSA. Q, K, and V represent the query, key, and value, respectively, and W... q W k and W v d represents the weight matrices for the linear transformations that generate the query, key, and value tensors, respectively. k The channel dimension representing K, This represents approximate normalization.

[0077] The key-value pairs generated by the averaging filter are shorter than the input feature maps, giving the convolutional neural network model a stronger global perception capability. This allows it to capture the overall characteristics of the data and provide richer and more comprehensive semantic information. Specifically, the query and the key-value pairs generated by the averaging filter are input together into the MHSA to obtain the output feature map.

[0078] (3) Local feature information extraction module (local feature information extraction path), based on convolutional neural network, extracts local feature information from the input image to be suitable for the detection of small lesions in gastrointestinal diseases.

[0079] like Figure 2 As shown, the local feature information extraction module (local feature branch) constructs a four-layer local information extraction path through convolutional operations to capture local feature information. Specifically, it contains four convolutional modules with a kernel size of 3. Each layer (each convolutional module) performs a normal small-kernel convolution operation. Each layer (each convolutional module) consists of a 3×3 convolutional layer, a batch normalization layer, and an activation function. This effectively extracts local feature information and enriches feature representation while reducing the computational complexity and processing time of the convolutional neural network model. Considering that deeper convolutional neural networks have larger receptive fields, less local feature information, and higher computational and memory consumption, the traditional method of using deep convolutional neural networks to obtain local feature information is not used.

[0080] (4) The feature fusion module is used to fuse the extracted global and local feature information through convolution operations, thereby unifying the dimension and scale of the extracted features and obtaining highly abstract multi-scale information. Specifically, the feature fusion module upsamples the global feature information and then concatenates it with the local feature information to obtain multi-scale information.

[0081] The feature map outputs of the global and local feature branches differ in both channel and spatial dimensions. If the spatial dimension of the input image is H×W=224×224, the output dimension of the global feature branch is 640×7×7, and the output dimension of the local feature branch is 288×14×14. From the dimensions of the two outputs, it can be seen that these two feature maps can be fused in the channel dimension. Before fusion, the spatial dimension of the output feature map needs to be upsampled from the global branch to 640×14×14. This yields a final fused feature map with a dimension of 928×14×14, which contains highly abstracted multi-scale image information. Therefore, the proposed CNN has a stronger ability to analyze objects in images both globally and locally, making it well-suited for multi-class classification.

[0082] (5) Coarse diagnosis module, which is used to compress the fused feature information (multi-scale information) through a global average pooling layer and to perform coarse diagnosis through a linear classifier to complete the differential classification of disease types.

[0083] This embodiment provides a gastrointestinal endoscopy image diagnostic system based on a mean multi-head self-attention mechanism. It includes two feature extraction paths with different encoding methods, enabling simultaneous extraction of local and global features from endoscopic images, thus overcoming the shortcomings of transformers and CNNs. In the global information path, a large-kernel convolution module is used to maximize the preservation of global features and enrich feature representation. A Mean-transformer encoder module is also used to activate the entire object range and capture overall information. The Mean-MHSA module within this module employs a Mean multi-head self-attention mechanism, different from traditional MHSA. It inputs the input feature map to an average filter to generate keys and values, resulting in a shorter length than the input feature map. This provides strong global perception capabilities, capturing the overall characteristics of the data and providing richer and more comprehensive semantic information. In the local information extraction path, convolution operations effectively extract local feature information while reducing the computational complexity and processing time of the convolutional neural network model, enriching feature representation. Finally, convolution operations unify the dimensionality and scale of the extracted features, obtaining highly abstract multi-scale information.

[0084] The digestive tract disease diagnostic model (MeanTrans model) of this embodiment was quantitatively compared with other state-of-the-art models on a digestive tract disease classification dataset, as shown in Table 1. The data in the table shows that the proposed MeanTrans model achieved the best performance in PREC, REC, F1, and ACC metrics. It is clearly evident that the MeanTrans model proposed in this embodiment outperforms other state-of-the-art models.

[0085] Table 1. Performance Comparison with Other Advanced Models

[0086] Poolformer 0.8171 0.794 0.8027 0.8153 Swim-transformer 0.7309 0.7023 0.7117 0.735 TransMSF 0.8687 0.8434 0.8406 0.8652 Acmix-ResNet 0.7422 0.7272 0.7344 0.7468 RepVGG 0.8442 0.8158 0.8266 0.8408 ShuffleNetV2 0.8405 0.8237 0.8304 0.8382 ViT 0.7902 0.7742 0.7799 0.7945 MobileNetV3 0.8271 0.8074 0.8151 0.8266 MeanTrans 0.8981 0.9033 0.8973 0.8916

[0087] Finally, a human-machine comparison experiment was conducted. The model of this embodiment was deployed on a webpage and compared with that of six endoscopists with different experience levels. The results are shown in Table 2. The data in the table show that the proposed MeanTrans model has good results in PREC, REC, F1, ACC, Time, and P-value indicators.

[0088] Table 2. Human-Machine Comparison Experiment

[0089]

[0090] This embodiment provides a gastrointestinal endoscopic image diagnostic system based on an average multi-head self-attention mechanism. It proposes an improved transformer and CNN architecture (called MeanTrans) for classifying various gastrointestinal diseases and grading or staging specific disease types in large-scale gastrointestinal disease screening. It includes two feature extraction paths with different encoding methods, enabling the extraction of rich global and local feature information, thus overcoming the shortcomings of transformers and CNNs.

[0091] This embodiment provides a digestive endoscopy image diagnostic system based on the mean multi-head self-attention mechanism. The proposed Mean-MHSA module uses a mean multi-head self-attention mechanism that differs from the traditional MHSA. It inputs the input feature map into the mean filter to generate keys and values. Therefore, its length is smaller than the input feature map, which has a strong global perception capability and can capture the overall characteristics of the data, providing richer and more comprehensive semantic information.

[0092] Example 2

[0093] This embodiment provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the following steps:

[0094] Obtain endoscopic images of the digestive tract to be diagnosed;

[0095] The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism.

[0096] Local feature information is extracted from the digestive endoscopy image through convolution operation;

[0097] The global and local feature information are fused to obtain multi-scale information;

[0098] Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

[0099] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0100] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0101] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0102] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0103] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0104] Example 3

[0105] This embodiment provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the following steps:

[0106] Obtain endoscopic images of the digestive tract to be diagnosed;

[0107] The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism.

[0108] Local feature information is extracted from the digestive endoscopy image through convolution operation;

[0109] The global and local feature information are fused to obtain multi-scale information;

[0110] Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

[0111] Example 4

[0112] This embodiment provides a computer program product, which is a computer program product containing a computer program. When the processor executes the computer program, it performs the following steps:

[0113] Obtain endoscopic images of the digestive tract to be diagnosed;

[0114] The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism.

[0115] Local feature information is extracted from the digestive endoscopy image through convolution operation;

[0116] The global and local feature information are fused to obtain multi-scale information;

[0117] Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

[0118] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A digestive endoscopic image diagnosis system based on an average multi-head self-attention mechanism, characterized by, It includes an image acquisition module, a global feature information extraction module, a local feature information extraction module, a feature fusion module, and a coarse diagnosis module; The image acquisition module is used to acquire digestive endoscopy images to be diagnosed; The global feature information extraction module is used to extract global feature information from the digestive endoscopy image in multiple stages through convolution operations and an average multi-head self-attention mechanism. Each stage adopts the average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through the multi-head self-attention mechanism. The local feature information extraction module is used to extract local feature information from the digestive endoscopy image through convolution operations; The feature fusion module is used to fuse the global feature information and the local feature information to obtain multi-scale information; The coarse diagnostic module is used to classify the disease type of the digestive endoscopy image based on the multi-scale information.

2. The endoscopy image diagnosis system based on the average multi-head self-attention mechanism according to claim 1, wherein Each stage of the global feature information extraction process employs a sequentially connected large kernel convolutional block, image patch embedding layer, encoder, and convolutional layer. The encoder is composed of a first residual structure and a second residual structure connected together. The first residual structure includes a normalization layer and an average multi-head attention module, and the average multi-head attention module adopts the average multi-head self-attention mechanism. The second residual structure includes a normalization layer and a feedforward layer.

3. The endoscopy image diagnosis system based on the average multi-head self-attention mechanism according to claim 2, characterized in that, The large kernel convolutional block contains several identical convolutional modules, and each convolutional module adopts a residual structure consisting of a batch normalization layer, a convolutional layer, a large kernel depth convolutional layer, and an activation function.

4. The endoscopy image diagnosis system based on the average multi-head self-attention mechanism according to claim 2, characterized in that, The image block embedding layer divides the feature map output by the large kernel convolution block into multiple image blocks. Each image block is mapped to one dimension through a linear mapping layer and then used as the input of the encoder.

5. The endoscopy image diagnosis system based on average multi-head self-attention mechanism according to claim 1, characterized in that, The local feature information extraction module includes several sequentially connected convolutional modules, and each convolutional module includes a sequentially connected convolutional layer, a batch normalization layer, and an activation function.

6. The endoscopy image diagnosis system based on average multi-head self-attention mechanism according to claim 1, wherein, The feature fusion module obtains the multi-scale information by upsampling the global feature information and then concatenating it with the local feature information.

7. The gastrointestinal endoscopic image diagnostic system based on the average multi-head self-attention mechanism as described in claim 1, characterized in that, The coarse diagnostic module compresses the multi-scale information through a global average pooling layer and then uses a linear classifier to obtain the disease type of the digestive endoscopy image.

8. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, the following steps are performed: Obtain endoscopic images of the digestive tract to be diagnosed; The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism. Local feature information is extracted from the digestive endoscopy image through convolution operation; The global and local feature information are fused to obtain multi-scale information; Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

9. A storage medium, Its characteristic is that it stores computer-readable instructions non-transitory, wherein when the non-transitory computer-readable instructions are executed by a computer, the following steps are performed: Obtain endoscopic images of the digestive tract to be diagnosed; The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism. Local feature information is extracted from the digestive endoscopy image through convolution operation; The global and local feature information are fused to obtain multi-scale information; Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

10. A computer program product, characterized in that, It is a computer program product containing a computer program. When the processor executes the computer program, it performs the following steps: Obtain endoscopic images of the digestive tract to be diagnosed; The digestive endoscopy image is subjected to multi-stage global feature information extraction through convolution operation and average multi-head self-attention mechanism. Each stage adopts average multi-head self-attention mechanism. The average multi-head self-attention mechanism uses an average filter to generate the key and value of the input feature map, and generates a query based on the input feature map. Based on the query, key and value, the output feature map is obtained through multi-head self-attention mechanism. Local feature information is extracted from the digestive endoscopy image through convolution operation; The global and local feature information are fused to obtain multi-scale information; Based on the multi-scale information, the disease type to which the digestive endoscopy image belongs is classified.

Images