Capsule endoscopy image classification methods, devices, terminal equipment, and storage media

Abstract

This application discloses a capsule endoscopy image classification method, apparatus, terminal device, and computer storage medium, relating to the field of computer analysis technology for medical images. The method involves acquiring preprocessed capsule endoscopy images; extracting global image feature information from the capsule endoscopy images using a convolutional neural network; extracting self-attention weights from the capsule endoscopy images using a self-attention mechanism neural network; fusing the global image feature information and self-attention weights to obtain target image features; and classifying the target image features using a preset classifier to obtain the classification result of the capsule endoscopy image. Using the technical solution of this application, a dual-stream capsule endoscopy image classification algorithm based on a self-attention mechanism neural network and a convolutional neural network can be used to classify capsule endoscopy images, improving the recognition efficiency and accuracy of images captured by various wireless capsule endoscopy devices, thereby enhancing the diagnostic efficiency and accuracy of doctors.

Description

Technical Field

[0001] This application relates to the field of computer analysis technology for medical images, and in particular to a method, apparatus, terminal device, and computer storage medium for classifying capsule endoscopy images. Background Technology

[0002] Wireless Capsule Endoscopy (WCE) is a highly efficient and safe imaging device for the digestive tract. It uses a swallowable capsule-shaped device to capture images in the human digestive tract.

[0003] Typically, wireless capsule endoscopy takes more than 8 hours per patient, capturing an average of two images per second, resulting in over 50,000 images of the digestive tract per examination. While a large number of digestive tract images helps doctors obtain a more comprehensive diagnosis, there are often redundant images in the capsule endoscopy image sequence. These include numerous images of gas bubbles, digestive residue, and normal small intestinal walls, with only a few images of lesions. For doctors, finding the patient's lesion area among tens of thousands of digestive tract images is a very demanding task, and the difficulty in identification can easily lead to misdiagnosis and missed diagnosis.

[0004] In summary, how to efficiently and accurately identify capsule endoscopy images to assist doctors in improving diagnostic efficiency has become an urgent problem to be solved in this field. Summary of the Invention

[0005] The main objective of this application is to provide a method, apparatus, terminal device, and computer storage medium for classifying capsule endoscopy images, which aims to efficiently and accurately identify capsule endoscopy images to assist doctors in improving diagnostic efficiency.

[0006] To achieve the above objectives, this application provides a capsule endoscopy image classification method, which includes:

[0007] Acquire preprocessed capsule endoscopy images;

[0008] Global image feature information of the capsule endoscopy image is extracted using a convolutional neural network;

[0009] The self-attention weights of the capsule endoscopy images are extracted using a self-attention mechanism neural network;

[0010] The target image features are obtained by fusing the global feature information of the image and the self-attention weights, and the target image features are classified by a preset classifier to obtain the classification result of the capsule endoscopy image.

[0011] Optionally, the step of extracting the self-attention weights of the capsule endoscopy image through a self-attention mechanism neural network includes:

[0012] The first image feature of the capsule endoscopy image is extracted by the first convolutional layer, and the first image feature is divided into multiple image feature blocks;

[0013] The image feature blocks are processed by a preset PSA (Patch self attention) module to obtain intermediate image feature blocks.

[0014] The multiple intermediate image feature blocks are combined into an intermediate image feature;

[0015] The self-attention weights of the intermediate image features are extracted using a self-attention mechanism neural network.

[0016] Optionally, the PSA module includes a convolutional unit and a first softmax (normalization function) unit. The step of processing the image feature blocks using a preset PSA algorithm to obtain intermediate image feature blocks includes:

[0017] The first image feature, the second image feature, and the third image feature of the image feature block are extracted using the convolutional unit;

[0018] The first image feature is cross-multiplied with the second image feature to obtain the fourth image feature, and the feature point score of the fourth image feature is calculated through the first softmax unit.

[0019] The intermediate image feature block is obtained by cross-multiplying the third image feature with the feature point.

[0020] Optionally, the self-attention mechanism neural network includes a flattening module, a position encoding module, a first droppath (a type of regularization) module, a Layer Normalization module, an MHA (Multi-head Attention) module, and a second droppath module. The step of extracting the self-attention weights of the intermediate image features through the self-attention mechanism neural network includes:

[0021] The intermediate image features are sequentially processed through the flattening module, the position encoding module, the first drop path module, and the LayerNorm module to obtain the fifth image feature;

[0022] The self-attention weights are obtained by performing image processing on the fifth image features through the MHA module and the second drop path module, wherein the MHA module includes a fully connected layer and a second softmax unit.

[0023] Optionally, the convolutional neural network includes pooling layers, an SA (self-attention) module, and a convolutional module, wherein the convolutional module includes deformable convolutional layers, and the step of extracting global image feature information of the capsule endoscopy image through the convolutional neural network includes:

[0024] The sixth image feature of the capsule endoscopy image is extracted through the pooling layer;

[0025] The sixth image feature is sequentially convolved through multiple convolution modules with different kernels, and the SA module is used after each convolution to enhance the image feature extraction result in order to obtain global image feature information.

[0026] Optionally, the step of fusing the global image feature information and the self-attention weights to obtain the target image features includes:

[0027] The second image global feature information is obtained by upsampling and dot multiplication of the image global feature information;

[0028] The self-attention weights are reshaped, convolved, and multiplied to obtain the second self-attention weights.

[0029] The seventh image feature is obtained by multiplying the second image global feature information and the second self-attention weight;

[0030] The target image features are obtained by sequentially processing the seventh image features through the PSA module and the AAP (Adaptive Global Average Pooling) module.

[0031] Optionally, the step of acquiring the preprocessed capsule endoscopy image includes:

[0032] Acquire initial capsule endoscopy images;

[0033] The initial capsule endoscopy images are classified and used as a capsule endoscopy image dataset;

[0034] The initial capsule endoscopy images of each category in the capsule endoscopy image dataset are divided into training set, validation set and test set according to a preset ratio;

[0035] Image processing techniques are used to increase the number of images in the training set, the validation set, and the test set to obtain preprocessed capsule endoscopy images.

[0036] Furthermore, to achieve the above objectives, this application also provides a capsule endoscopy image classification device, the capsule endoscopy image classification device comprising:

[0037] The image acquisition module is used to acquire preprocessed capsule endoscopy images;

[0038] The global image feature information extraction module is used to extract the global image feature information of the capsule endoscope image through a convolutional neural network.

[0039] The self-attention weight extraction module is used to extract the self-attention weights of the capsule endoscopy image through a self-attention mechanism neural network;

[0040] The classification module is used to fuse the global feature information of the image and the self-attention weight to obtain the target image features, and to classify the target image features through a preset classifier to obtain the classification result of the capsule endoscopy image.

[0041] In this application, each functional module of the capsule endoscopy image classification device implements the steps of the capsule endoscopy image classification method as described above during operation.

[0042] In addition, to achieve the above objectives, this application also provides a terminal device, the terminal device comprising: a memory, a processor, and a capsule endoscopy image classification program stored in the memory and executable on the processor, wherein the capsule endoscopy image classification program, when executed by the processor, implements the steps of the capsule endoscopy image classification method as described above.

[0043] In addition, to achieve the above objectives, this application also proposes a computer storage medium storing a capsule endoscopy image classification program, which, when executed by a processor, implements the steps of the capsule endoscopy image classification method as described above.

[0044] This application provides a capsule endoscopy image classification method, apparatus, terminal device, and computer storage medium. The capsule endoscopy image classification method includes: acquiring a preprocessed capsule endoscopy image; extracting global image feature information of the capsule endoscopy image through a convolutional neural network; extracting self-attention weights of the capsule endoscopy image through a self-attention mechanism neural network; fusing the global image feature information and the self-attention weights to obtain target image features; and classifying the target image features through a preset classifier to obtain the classification result of the capsule endoscopy image.

[0045] Compared to traditional capsule endoscopy image classification methods, this application acquires preprocessed capsule endoscopy images, then extracts global image feature information from the capsule endoscopy images using a convolutional neural network, and extracts self-attention weights from the capsule endoscopy images using a self-attention mechanism neural network. Finally, the global image feature information and self-attention weights are fused together to obtain the target image features of the capsule endoscopy images, and the target image features are classified using a preset classifier to obtain the image classification results of the capsule endoscopy images.

[0046] Thus, this application uses a dual-stream capsule endoscopy image classification algorithm based on a self-attention mechanism neural network and a convolutional neural network to classify capsule endoscopy images. This improves the image recognition efficiency and accuracy of images captured by various wireless capsule endoscopy devices, thereby enhancing the diagnostic efficiency and accuracy of doctors. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiments of this application;

[0048] Figure 2 This is a schematic diagram illustrating the implementation process of an embodiment of the capsule endoscopy image classification method of this application;

[0049] Figure 3 This is a schematic diagram of the network structure of the convolutional feature extraction and PSA module in the Transformer (attention-based neural network model) module involved in an embodiment of the capsule endoscopy image classification method of this application.

[0050] Figure 4 This is a schematic diagram of the global self-attention weight network structure of the Transformer module for image extraction, as described in an embodiment of the capsule endoscopy image classification method of this application.

[0051] Figure 5 This is a schematic diagram of the network structure of the Convolution (convolutional neural network model) module for extracting global feature information of images in an embodiment of the capsule endoscopy image classification method of this application;

[0052] Figure 6 This is a schematic diagram illustrating the network structure of the Convolution module for extracting global feature information of images in an embodiment of the capsule endoscopy image classification method of this application;

[0053] Figure 7 This is a schematic diagram of the capsule endoscope image category detection and classification process involved in an embodiment of the capsule endoscope image classification method of this application;

[0054] Figure 8This is a schematic diagram of the network structure of the Transformer and Convolution dual-stream capsule endoscopy image recognition algorithm involved in an embodiment of the capsule endoscopy image classification method of this application;

[0055] Figure 9 This is a schematic diagram of the heatmap results of different capsule endoscopy image recognition algorithms involved in an embodiment of the capsule endoscopy image classification method of this application;

[0056] Figure 10 This is a schematic diagram of the functional modules of an embodiment of the capsule endoscope image classification device of this application.

[0057] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0058] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0060] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.

[0061] In this application, unless otherwise expressly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or an electrical connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two components or the interaction between two components, unless otherwise expressly limited. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0062] Furthermore, the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0063] This application provides a terminal device.

[0064] like Figure 1 As shown, Figure 1 This is a schematic diagram of the hardware operating environment of the terminal device involved in the embodiments of this application.

[0065] like Figure 1 As shown, in the hardware operating environment of the terminal device, the terminal device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or stable non-volatile memory, such as a disk storage device. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0066] Those skilled in the art will understand that Figure 1 The terminal device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0067] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a capsule endoscopy image classification program.

[0068] exist Figure 1In the device shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (user terminal) and communicate data with it; while processor 1001 can be used to call the capsule endoscopy image classification program stored in memory 1005 and perform the following operations:

[0069] Acquire preprocessed capsule endoscopy images;

[0070] Global image feature information of the capsule endoscopy image is extracted using a convolutional neural network;

[0071] The self-attention weights of the capsule endoscopy images are extracted using a self-attention mechanism neural network;

[0072] The target image features are obtained by fusing the global feature information of the image and the self-attention weights, and the target image features are classified by a preset classifier to obtain the classification result of the capsule endoscopy image.

[0073] Optionally, the processor 1001 can also be used to call the capsule endoscopy image classification program stored in the memory 1005 and perform the following operations:

[0074] The first image feature of the capsule endoscopy image is extracted by the first convolutional layer, and the first image feature is divided into multiple image feature blocks;

[0075] The image feature blocks are processed by a preset PSA module to obtain intermediate image feature blocks;

[0076] The multiple intermediate image feature blocks are combined into an intermediate image feature;

[0077] The self-attention weights of the intermediate image features are extracted using a self-attention mechanism neural network.

[0078] Optionally, the PSA module includes a convolutional unit and a first softmax unit. The processor 1001 can also be used to call the capsule endoscopy image classification program stored in the memory 1005 and perform the following operations:

[0079] The first image feature, the second image feature, and the third image feature of the image feature block are extracted using the convolutional unit;

[0080] The first image feature is cross-multiplied with the second image feature to obtain the fourth image feature, and the feature point score of the fourth image feature is calculated through the first softmax unit.

[0081] The intermediate image feature block is obtained by cross-multiplying the third image feature with the feature point.

[0082] Optionally, the self-attention mechanism neural network includes a flattening module, a position encoding module, a first droppath module, a Layer Norm module, an MHA module, and a second droppath module. The processor 1001 can also be used to call the capsule endoscopy image classification program stored in the memory 1005 and perform the following operations:

[0083] The intermediate image features are sequentially processed through the flattening module, the position encoding module, the first drop path module, and the LayerNorm module to obtain the fifth image feature;

[0084] The self-attention weights are obtained by performing image processing on the fifth image features through the MHA module and the second drop path module, wherein the MHA module includes a fully connected layer and a second softmax unit.

[0085] Optionally, the convolutional neural network includes pooling layers, an SA module, and a convolutional module, wherein the convolutional module includes deformable convolutional layers. The processor 1001 can also be used to call a capsule endoscopy image classification program stored in the memory 1005 and perform the following operations:

[0086] The sixth image feature of the capsule endoscopy image is extracted through the pooling layer;

[0087] The sixth image feature is sequentially convolved through multiple convolution modules with different kernels, and the SA module is used after each convolution to enhance the image feature extraction result in order to obtain global image feature information.

[0088] Optionally, the processor 1001 can also be used to call the capsule endoscopy image classification program stored in the memory 1005 and perform the following operations:

[0089] The second image global feature information is obtained by upsampling and dot multiplication of the image global feature information;

[0090] The self-attention weights are reshaped, convolved, and multiplied to obtain the second self-attention weights.

[0091] The seventh image feature is obtained by multiplying the second image global feature information and the second self-attention weight;

[0092] The target image features are obtained by sequentially processing the seventh image features through the PSA module and the AAP module.

[0093] Optionally, the processor 1001 can also be used to call the capsule endoscopy image classification program stored in the memory 1005 and perform the following operations:

[0094] Acquire initial capsule endoscopy images;

[0095] The initial capsule endoscopy images are classified and used as a capsule endoscopy image dataset;

[0096] The initial capsule endoscopy images of each category in the capsule endoscopy image dataset are divided into training set, validation set and test set according to a preset ratio;

[0097] Image processing techniques are used to increase the number of images in the training set, the validation set, and the test set to obtain preprocessed capsule endoscopy images.

[0098] Based on the above hardware structure, the overall concept of various embodiments of the capsule endoscopy image classification method of this application is proposed.

[0099] In this embodiment, the wireless capsule endoscope is a highly efficient and safe digestive tract imaging device that uses a swallowable capsule-shaped device to capture images in the human digestive tract.

[0100] Typically, wireless capsule endoscopy takes more than 8 hours per patient, capturing an average of two images per second, resulting in over 50,000 images of the digestive tract per examination. While a large number of digestive tract images helps doctors obtain a more comprehensive diagnosis, there are often redundant images in the capsule endoscopy image sequence. These include numerous images of gas bubbles, digestive residue, and normal small intestinal walls, with only a few images of lesions. For doctors, finding the patient's lesion area among tens of thousands of digestive tract images is a very demanding task, and the difficulty in identification can easily lead to misdiagnosis and missed diagnosis.

[0101] In summary, how to efficiently and accurately identify capsule endoscopy images to assist doctors in improving diagnostic efficiency has become an urgent problem to be solved in this field.

[0102] To address the aforementioned problems, this application proposes a capsule endoscopy image classification method, apparatus, terminal device, and storage medium. The capsule endoscopy image classification method includes: acquiring a preprocessed capsule endoscopy image; extracting global image feature information of the capsule endoscopy image using a convolutional neural network; extracting self-attention weights of the capsule endoscopy image using a self-attention mechanism neural network; fusing the global image feature information and the self-attention weights to obtain target image features; and classifying the target image features using a preset classifier to obtain the classification result of the capsule endoscopy image.

[0103] Compared to traditional capsule endoscopy image classification methods, this application acquires preprocessed capsule endoscopy images, then extracts global image feature information from the capsule endoscopy images using a convolutional neural network, and extracts self-attention weights from the capsule endoscopy images using a self-attention mechanism neural network. Finally, the global image feature information and self-attention weights are fused together to obtain the target image features of the capsule endoscopy images, and the target image features are classified using a preset classifier to obtain the image classification results of the capsule endoscopy images.

[0104] Thus, this application uses a dual-stream capsule endoscopy image classification algorithm based on a self-attention mechanism neural network and a convolutional neural network to classify capsule endoscopy images. This improves the image recognition efficiency and accuracy of images captured by various wireless capsule endoscopy devices, thereby enhancing the diagnostic efficiency and accuracy of doctors.

[0105] Based on the overall concept of the capsule endoscopy image classification method of this application, various embodiments of the capsule endoscopy image classification method of this application are proposed.

[0106] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the capsule endoscopy image classification method of this application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order. For ease of understanding and explanation, this embodiment uses a terminal device as the direct execution subject to illustrate the capsule endoscopy image classification method of this application.

[0107] like Figure 2 As shown, in this embodiment, the capsule endoscopy image classification method of this application may include:

[0108] Step S10: Obtain the preprocessed capsule endoscopy image;

[0109] In this embodiment, the terminal device acquires preprocessed capsule endoscope images. Specifically, the preprocessing operation may include digital image processing methods such as image rotation, image mirroring, Gaussian blur, and salt-and-pepper noise to obtain a greater number of capsule endoscope images.

[0110] Step S20: Extract global image feature information of the capsule endoscope image using a convolutional neural network;

[0111] In this embodiment, the terminal device feeds the color capsule endoscopy image into a convolutional neural network. After multiple convolution processes, the global feature information of the capsule endoscopy image is extracted. The Convolution module can use internationally mainstream network structures such as VGG16, ResNet50, and EfficientNet B4.

[0112] Step S30: Extract the self-attention weights of the capsule endoscopy image using a self-attention mechanism neural network;

[0113] In this embodiment, the terminal device extracts the self-attention weights of the capsule endoscopy image through a self-attention mechanism neural network. The Transformer module can adopt internationally mainstream network structures such as vision transformer, cross vision transformer, and max vision transformer.

[0114] Step S40: After fusing the global feature information of the image and the self-attention weight, the target image features are obtained, and the target image features are classified by a preset classifier to obtain the classification result of the capsule endoscopy image.

[0115] In this embodiment, the terminal device fuses the global image feature information obtained by image processing of the capsule endoscope image with the self-attention weight to obtain the target image feature of the capsule endoscope image, and then classifies the target image feature by a preset classifier to obtain the classification result of the capsule endoscope image.

[0116] In this embodiment, the capsule endoscopy image classification method provided by this application includes: acquiring a preprocessed capsule endoscopy image; extracting global image feature information of the capsule endoscopy image through a convolutional neural network; extracting self-attention weights of the capsule endoscopy image through a self-attention mechanism neural network; fusing the global image feature information and the self-attention weights to obtain target image features, and classifying the target image features through a preset classifier to obtain the classification result of the capsule endoscopy image.

[0117] In this embodiment, a preprocessed capsule endoscope image is acquired by a terminal device. Then, the terminal device feeds the color capsule endoscope image into a convolutional neural network. After multiple convolution processes, the global image feature information of the capsule endoscope image is extracted. Furthermore, the terminal device also extracts the self-attention weights of the capsule endoscope image through a self-attention mechanism neural network. Finally, the terminal device fuses the global image feature information and self-attention weights obtained from image processing of the capsule endoscope image to obtain the target image features of the capsule endoscope image. The target image features are then classified using a preset classifier to obtain the classification result of the capsule endoscope image.

[0118] Thus, the embodiments of this application classify capsule endoscopy images using a dual-stream capsule endoscopy image classification algorithm based on a self-attention mechanism neural network and a convolutional neural network. This improves the image recognition efficiency and accuracy of images captured by various wireless capsule endoscopy devices, thereby enhancing the diagnostic efficiency and accuracy of doctors.

[0119] Furthermore, based on the first embodiment of the capsule endoscopy image classification method of this application described above, a second embodiment of the capsule endoscopy image classification method of this application is proposed. For ease of understanding and explanation, this embodiment uses convolution modules and algorithm modules with fixed parameters for the step description. Of course, based on actual application images, the parameter settings of each convolution module and algorithm module can also be adjusted.

[0120] In this embodiment, step S10: acquiring the preprocessed capsule endoscopy image includes:

[0121] Step A10: Obtain initial capsule endoscopy images;

[0122] Step A20: Classify the initial capsule endoscopy images and use them as a capsule endoscopy image dataset;

[0123] Step A30: Divide the initial capsule endoscopy images of each category in the capsule endoscopy image dataset into a training set, a validation set, and a test set according to a preset ratio;

[0124] Step A40: Use image processing techniques to increase the number of images in the training set, the validation set, and the test set to obtain preprocessed capsule endoscopy images.

[0125] In this embodiment, the terminal device acquires initial capsule endoscopy image data, categorizes each image into different scene folders according to its category, and then classifies these scene folders into a dataset folder. Subsequently, the dataset is divided into training, validation, and test sets according to a preset ratio (e.g., 6:2:2). Digital image processing techniques such as image rotation, image mirroring, Gaussian blur, and salt-and-pepper noise are used to increase the number of images in the training, validation, and test sets by more than four times.

[0126] Furthermore, in one feasible embodiment, step S30 above includes:

[0127] Step S301: Extract the first image features of the capsule endoscope image through the first convolutional layer, and divide the first image features into multiple image feature blocks;

[0128] In this embodiment, as Figure 3As shown, the capsule endoscopy image has a size of 224*224. The terminal device sends a capsule endoscopy image of size 224*224 into the first convolutional layer with a kernel size of 16*16, a stride of 16, padding of 0, 3 input channels, and 768 output channels to extract the first image feature. The extracted image feature has a size of 768*14*14. The extracted first image feature is a high-dimensional image tensor matrix. An image tensor slicing algorithm is used to extract a specified range of high-dimensional image tensor matrix as an image feature block. In this embodiment, the set image feature block size is 768*2*2, so the above-mentioned first image feature of size 768*14*14 will extract 49 image feature blocks of size 768*2*2.

[0129] Step S302: Perform image processing on the image feature blocks using a preset PSA module to obtain intermediate image feature blocks;

[0130] In this embodiment, the terminal device sends the first extracted image feature block to the PSA module. The PSA module performs image processing on the input image feature block with a size of 768*2*2 to obtain the intermediate image feature block.

[0131] Further, in one feasible embodiment, the PSA module includes a convolutional unit and a first softmax unit, and step S302 includes:

[0132] Step S3021: Extract the first image feature, the second image feature, and the third image feature of the image feature block through the convolutional unit;

[0133] In this embodiment, the terminal device uses the PSA module to extract image features from an input image feature block of size 768*2*2 using a convolutional layer with a kernel size of 1*1, a stride of 1, padding of 0, an input channel of 768, and an output channel of 96, resulting in two image features of size 96*2*2. Then, it extracts a third image feature of size 768*2*2 using a convolutional layer with a kernel size of 1*1, a stride of 1, padding of 0, an input channel of 768, and an output channel of 768. The first image feature is then transposed and reshaped to a size of 4*96. The second and third image features are then reshaped to a size of 96*4 and 768*4, respectively.

[0134] Step S3022: Cross-multiply the first image feature with the second image feature to obtain the fourth image feature, and calculate the feature point score of the fourth image feature through the first softmax unit;

[0135] In this embodiment, the terminal device cross-multiplies the first image feature with an image size of 4*96 with the second image feature with an image size of 96*4 to obtain a fourth image feature with a size of 4*4. The first softmax function is used to calculate the score of the fourth image feature point with a size of 4*4, which is between 0 and 1. All image feature point scores are added together to get 1. The more important feature points in the image features can be analyzed through the image feature point scores.

[0136] Step S3023: Multiply the third image feature by the feature point to obtain the intermediate image feature block.

[0137] In this embodiment, the terminal device uses a matrix tensor cross product operation to obtain an intermediate image feature block of size 768*4, which is obtained from the previous convolutional layer and the score result of the fourth image feature of size 4*4.

[0138] Step S303: Combine the multiple intermediate image feature blocks into an intermediate image feature;

[0139] In this embodiment, the terminal device transposes the fused 768*4 intermediate image feature block into a 768*2*2 intermediate image feature block and places it back into the corresponding position of the original 768*14*14 image feature. This process of sending intermediate image feature blocks to the PSA module is repeated until all 49 768*2*2 image feature blocks are sent to the PSA module and placed back into the corresponding positions of the 768*14*14 image feature, generating a new 768*14*14 intermediate image feature.

[0140] Step S304: Extract the self-attention weights of the intermediate image features using a self-attention mechanism neural network.

[0141] In this embodiment, the terminal device extracts the self-attention weights of intermediate image features through a self-attention neural network.

[0142] Further, in one feasible embodiment, the self-attention mechanism neural network includes a flattening module, a position encoding module, a first drop path module, a Layer Norm module, an MHA module, and a second drop path module, and step S304 includes:

[0143] Step S3041: The intermediate image features are sequentially processed by the flattening module, the position encoding module, the first drop path module, and the Layer Norm module to obtain the fifth image feature;

[0144] In this embodiment, the Transformer module uses a portion of the vision transformer structure and further improves upon it; its network structure is as follows: Figure 4 As shown, the terminal device inputs the intermediate image features of size 768*14*14 processed by the PSA module into the flattening module, flattening them into image features of size 768*196. In the image position encoding part, a learnable tensor generation algorithm is used to generate learnable image features of size 768*196, with the initial value set to 0. This is added to the image features of size 768*196 to obtain image features of size 768*196 after position encoding. By adding position encoding information, the spatial position information of the image is effectively preserved during network training when the image features are stretched from 2D to 1D, improving the classification accuracy. Then, the terminal device uses a first drop path module with a 50% probability to freeze the image features of size 768*196 after position encoding, obtaining image features of size 768*196. The image features of size 768*196 output from the first drop path module are input into the layer normalization module. This module effectively solves the problem of image feature normalization that can only be used with small batches of samples for training due to the large network size. The fifth image feature, after being processed by the layer normalization module, has a size of 768*196.

[0145] Step S3042: The self-attention weights are obtained by performing image processing on the fifth image features through the MHA module and the second drop path module, wherein the MHA module includes a fully connected layer and a second softmax unit.

[0146] In this embodiment, as Figure 4As shown, the terminal device sends the fifth image feature to the MHA module. This module sends the fifth image feature with an input size of 768*196 to the fully connected layer module, resulting in image features with sizes of q, k, and v of 768*196. Then, the matrix tensor dimension reshaping algorithm is used to divide each image feature with a size of 768*196 into 6 attention heads. Each attention head obtains an image feature with a size of 128*196. Therefore, a total of 18 image features with a size of 128*196 can be obtained in the part where the attention heads divide the image features. The six 128*196 image features obtained from q are subjected to a matrix tensor cross product with the six 128*196 image features obtained from k. The cross product is performed by sequentially cross-multiplying the 128*196 image features between different k and q points generated from different feature points of the image and then dividing them into six attention heads. Each time, the 128*196 image feature corresponding to k is transposed to 196*128, resulting in twelve 128*128 image features. These are then multiplied by their dot product. The value rounded down is 11, resulting in image features of the same size, but all feature point values ​​are reduced by 1 / 11. For image features of size 128*128 belonging to the same attention head, a softmax algorithm is applied (i.e., the second softmax unit) to calculate their respective feature scores, while the image feature size remains unchanged. The six 128*196 image features belonging to v are cross-multiplied by the same attention head and processed by the softmax algorithm to obtain 128*196 image features. Then, a matrix tensor dimension concatenation algorithm is used to concatenate the image features output by the six attention heads to obtain an image feature of size 768*196. This 768*196 image feature is then frozen with a 50% probability using the second drop path module, resulting in an image feature of size 768*196. The MHA module is repeated 12 times to obtain an output image feature of size 768*196.

[0147] It should be noted that, in this embodiment, compared with the existing vision transformer algorithm, the number of repetitions of multi-head attention and the number of multi-head attention in the self-attention neural network have been adjusted, thereby reducing the overall computational load of the network.

[0148] Further, in a feasible embodiment, the convolutional neural network includes a pooling layer, an SA module, and a convolutional module. Step S20 above, which involves extracting global image feature information from the capsule endoscopy image using the convolutional neural network, includes:

[0149] Step S201: Extract the sixth image feature of the capsule endoscopy image through the pooling layer;

[0150] Step S202: The sixth image feature is sequentially convolved through multiple convolution modules with different convolution kernels, and the image feature extraction result is enhanced by the SA module after each convolution to obtain global image feature information.

[0151] In this embodiment, the terminal device extracts the sixth image feature of the capsule endoscopy image through a pooling layer. Then, the terminal device sequentially performs convolution operations on the sixth image feature through multiple convolution modules with different convolution kernels. After each convolution, the SA algorithm is used to enhance the extraction result of the convolution output image feature in order to obtain global image feature information.

[0152] Specifically, in this embodiment, such as Figure 5 As shown, the Convolution module uses part of the ResNet50 structure and further improves it. The terminal device feeds a capsule endoscopy image of size 224*224 into a convolutional layer with a kernel size of 7*7, a stride of 2, padding of 0, 3 input channels, and 64 output channels. Batch normalization and max pooling with a convolution sum of 3 and a stride of 2 are used to calculate a sixth image feature of size 64*56*56. Then, the terminal device divides the sixth image feature into two branches. One branch is fed into the SA block and added to the other branch before being fed into Step 1, resulting in an image feature of size 256*56*56. The SA block step is repeated, feeding the image feature into Step 2 and Step 1 to obtain an image feature of size 512*28*28. The SA block step is repeated again, feeding the image feature into two Step 2 steps and one Step 1 step, resulting in an image feature of size 1024*14*14. The SA block step is repeated again. The block step feeds the data into two Step 2 steps, two Step 3 steps, and one Step 1 step, resulting in an image feature with a size of 2048*7*7.

[0153] Among them, such as Figure 6As shown, block1 consists of convolution, batch normalization, and ReLU activation function; block2 consists of deformable convolution, batch normalization, and ReLU activation function; block3 consists of convolution and batch normalization function; and block4 consists of deformable convolution and batch normalization function. Step1 is formed by adding the outputs of two blocks1 and one block3 from the input and applying ReLU activation function. Step2 is formed by adding the outputs of two blocks1 and one block3 from the input and applying ReLU activation function. Step3 is formed by adding the outputs of two blocks2 and one block4 from the input and applying ReLU activation function. The SA block divides the feature map into three feature maps using a fully connected function. The first and second feature maps use a matrix cross product algorithm, and the size and input feature map are obtained using a softmax function. Figure 1 The self-attention score is calculated and then combined with the third feature map using a matrix cross product algorithm to obtain the final SA feature map.

[0154] In this embodiment, as Figure 7 As shown, after preprocessing the wireless capsule endoscopy images, the terminal device divides the capsule endoscopy images into two branches. One branch feeds the features of each image into a supervised convolutional neural network to extract global image feature information. The other branch first uses a convolutional module to extract global image features, then divides the extracted image feature map into multiple feature blocks, and uses the PSA algorithm to further process the feature blocks. Then, each processed image feature is fed into a supervised self-attention mechanism neural network to extract global image self-attention weights. Finally, the terminal device fuses the global image feature information and global image self-attention weights obtained from the two branches, and inputs the fused result into a preset classifier to obtain the predicted classification result of the wireless capsule endoscopy image.

[0155] Specifically, the terminal device extracts image features sequentially from left to right and top to bottom according to a 2x2 image feature size using the PSA algorithm. For each extracted 2x2 image feature patch, a traditional attention module is used to perform image feature attention weighting at the patch level, strengthening points with strong image feature representation and weakening points with weaker representation. After each patch undergoes image feature attention weighting, it is sequentially returned to its original position, ultimately generating new image features after attention weighting. Furthermore, the vision transformer algorithm is improved by adjusting the number of multi-heads to reduce the overall network computation. The ResNet 50 algorithm is also improved by using deformable convolutions in some convolutional modules to enhance the convolution effect, and a self-attention module is used after each change in feature map size and channel number to enhance the network's image feature extraction results.

[0156] Furthermore, based on the first and / or second embodiments of the capsule endoscopy image classification method of this application described above, a third embodiment of the capsule endoscopy image classification method of this application is proposed.

[0157] In this embodiment, as Figure 8 As shown, step S40, "fusing the global feature information of the image and the self-attention weights to obtain the target image features," includes:

[0158] Step S401: The global feature information of the image is upsampled and cross-multiplied to obtain the second global feature information of the image;

[0159] In this embodiment, the terminal device uses a 2x bilinear interpolation upsampling algorithm on the output of the Convolution module, i.e., upsampling, to obtain image features of size 2048*14*14. Then, a learnable tensor generation algorithm is used to generate a learnable global image feature information scaling factor, and the initial value is set to 1. This factor is then multiplied by the 2048*14*14 image features to obtain image features of size 2048*14*14, i.e., the second global image feature information.

[0160] Step S402: The self-attention weights are reshaped, convolved, and cross-multiplied to obtain the second self-attention weights;

[0161] In this embodiment, the terminal device uses a tensor reshaping algorithm on the output of the Transformer module to obtain image features of size 768*14*14. These features are then fed into a convolutional layer with a kernel of 1, a stride of 1, 768 input channels, and 2048 output channels to extract image features, resulting in image features of size 2048*14*14. Next, a learnable tensor generation algorithm is used to generate a learnable self-attention weight multiplier factor, with an initial value set to 1. This factor is then multiplied by the 2048*14*14 image features to obtain image features of size 2048*14*14, which is the second self-attention weight.

[0162] Step S403: Perform a dot product operation on the second image global feature information and the second self-attention weight to obtain the seventh image feature;

[0163] In this embodiment, the terminal device performs a dot product operation on the second image global feature information and the second self-attention weight to obtain a seventh image feature with a size of 2048*14*14.

[0164] Step S404: The target image features are obtained by sequentially processing the seventh image features through the PSA module and the AAP module.

[0165] In this embodiment, the terminal device sends the seventh image feature to the PSA module to obtain an image feature with a size of 2048*14*14. The image feature with a size of 2048*14*14 is then sent to the Adaptive Global Average Pooling Algorithm (AAP module) to obtain an image feature with a size of 2048*1*1. Finally, the classification result of the capsule endoscopy image is obtained through a fully connected layer (FC) algorithm.

[0166] Thus, the embodiments of this application classify capsule endoscopy images using a dual-stream capsule endoscopy image classification algorithm based on a self-attention mechanism neural network and a convolutional neural network. This improves the image recognition efficiency and accuracy of images captured by various wireless capsule endoscopy devices, thereby enhancing the diagnostic efficiency and accuracy of doctors.

[0167] Furthermore, in one feasible embodiment, the softmax function used in the above process is formulated as follows:

[0168]

[0169] Where z i It is a feature value for which the probability of a feature point needs to be calculated in a 4x4 image feature set, where K is the total number of image feature points, and z is the number of feature points. j The value of each feature point in the image features. The cross product formula for image features is:

[0170] A m×n ×B n×m =C m×m

[0171] Where A m×n Matrix A has m rows and n columns, B n×m Matrix B has n rows and m columns, and m can be equal to n. The cross product of matrix A and matrix B results in an m-row, m-column matrix.

[0172] Since network model training involves a backpropagation process, a loss function needs to be designed within the network. This network uses the cross-entropy loss function, the formula of which is as follows:

[0173]

[0174] Among them, y i This is the label for sample i, 1 for positive samples and 0 for negative samples; p i y is the probability that sample i is predicted as a positive sample; M is the number of sample classes; ic When the true class of sample i is the same as the class of c, y ic If y is 1, otherwise y ic p is 0; ic This is the predicted probability when sample i belongs to class c. Each loss calculation uses the image features from the last layer of the neural network, processed by the sigmoid function to obtain the probability output for each class. The predicted class probabilities from network model training are then compared with the true class labels using the aforementioned cross-entropy loss function, thus completing the backpropagation neuron weight correction process of the network.

[0175] Below are heatmaps of traditional ResNet50, Vision Transformer, and Transformer + Convolution dual-flow capsule endoscopy image recognition algorithms, such as... Figure 9 As shown. The correlation in the heatmap, from strongest to weakest, is represented by colors: red, orange, blue, and dark blue. Figure 6 It can be seen that the ResNet50 network cannot accurately focus on the lesion area in the wireless capsule endoscopy image, and there is a problem of offsetting the important information related to the category classification in the image features; the VisonTransformer network does not focus on any category classification related information in the image features of the whole image, and the network will randomly output the classification result; the dual-stream capsule endoscopy image recognition algorithm of Transformer and Convolution can accurately focus on the lesion area in the wireless capsule endoscopy image, analyze the important information related to the category classification in the image features, and output the correct classification result.

[0176] Furthermore, this application also proposes a capsule endoscopy image classification device.

[0177] Please refer to Figure 10 The capsule endoscopy image classification device of this application includes:

[0178] Image acquisition module 10 is used to acquire preprocessed capsule endoscope images;

[0179] The global feature information extraction module 20 is used to extract the global feature information of the capsule endoscope image through a convolutional neural network.

[0180] The self-attention weight extraction module 30 is used to extract the self-attention weights of the capsule endoscopy image through a self-attention mechanism neural network.

[0181] The classification module 40 is used to fuse the global feature information of the image and the self-attention weight to obtain the target image features, and to classify the target image features through a preset classifier to obtain the classification result of the capsule endoscopy image.

[0182] The functions of each module in the capsule endoscopy image classification device correspond to the steps in the capsule endoscopy image classification method embodiment, and their functions and implementation processes will not be described in detail here.

[0183] Furthermore, this application also proposes a computer storage medium storing a program for classifying capsule endoscope images, which, when executed by a processor, implements the steps of the capsule endoscope image classification method of this application as described above.

[0184] The specific embodiments of the computer storage medium in this application are basically the same as the embodiments of the capsule endoscopy image classification method described above, and will not be repeated here.

[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0186] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0188] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for classifying capsule endoscopy images, characterized in that, The capsule endoscopy image classification method includes: Acquire preprocessed capsule endoscopy images; Global image feature information of the capsule endoscopy image is extracted using a convolutional neural network; The self-attention weights of the capsule endoscopy images are extracted using a self-attention mechanism neural network; The target image features are obtained by fusing the global feature information of the image and the self-attention weights, and the target image features are classified by a preset classifier to obtain the classification result of the capsule endoscopy image. The convolutional neural network includes pooling layers, an SA module, and a convolutional module, wherein the convolutional module includes a deformable convolutional layer. The step of extracting global feature information of the capsule endoscopy image through the convolutional neural network includes: The sixth image feature of the capsule endoscopy image is extracted through the pooling layer; The sixth image feature is sequentially convolved through multiple convolution modules with different kernels, and the SA module is used to enhance the image feature extraction result after each convolution to obtain global image feature information. The step of extracting the self-attention weights of the capsule endoscopy image using a self-attention mechanism neural network includes: The first image feature of the capsule endoscopy image is extracted by the first convolutional layer, and the first image feature is divided into multiple image feature blocks; The image feature blocks are processed by a preset PSA module to obtain intermediate image feature blocks; The multiple intermediate image feature blocks are combined into an intermediate image feature; The self-attention weights of the intermediate image features are extracted using a self-attention mechanism neural network; The step of fusing the global feature information of the image and the self-attention weights to obtain the target image features includes: The second image global feature information is obtained by upsampling and dot multiplication of the image global feature information; The self-attention weights are reshaped, convolved, and multiplied to obtain the second self-attention weights. The seventh image feature is obtained by multiplying the second image global feature information and the second self-attention weight; The target image features are obtained by sequentially processing the seventh image features through the PSA module and the AAP module.

2. The capsule endoscopy image classification method according to claim 1, characterized in that, The PSA module includes a convolutional unit and a first softmax unit. The step of processing the image feature blocks using the preset PSA module to obtain intermediate image feature blocks includes: The first image feature, the second image feature, and the third image feature of the image feature block are extracted using the convolutional unit; The first image feature is cross-multiplied with the second image feature to obtain the fourth image feature, and the feature point score of the fourth image feature is calculated through the first softmax unit. The intermediate image feature block is obtained by cross-multiplying the third image feature with the feature point.

3. The capsule endoscopy image classification method according to claim 1, characterized in that, The self-attention mechanism neural network includes a flattening module, a position encoding module, a first drop path module, a Layer Norm module, an MHA module, and a second drop path module. The step of extracting the self-attention weights of the intermediate image features through the self-attention mechanism neural network includes: The intermediate image features are sequentially processed through the flattening module, the position encoding module, the first drop path module, and the LayerNorm module to obtain the fifth image feature; The self-attention weights are obtained by performing image processing on the fifth image features through the MHA module and the second drop path module, wherein the MHA module includes a fully connected layer and a second softmax unit.

4. The capsule endoscopy image classification method according to claim 1, characterized in that, The step of acquiring the preprocessed capsule endoscopy image includes: Acquire initial capsule endoscopy images; The initial capsule endoscopy images are classified and used as a capsule endoscopy image dataset; The initial capsule endoscopy images of each category in the capsule endoscopy image dataset are divided into training set, validation set and test set according to a preset ratio; Image processing techniques are used to increase the number of images in the training set, the validation set, and the test set to obtain preprocessed capsule endoscopy images.

5. A capsule endoscopy image classification device, characterized in that, The capsule endoscopy image classification device includes: The image acquisition module is used to acquire preprocessed capsule endoscopy images; The global image feature information extraction module is used to extract global image feature information of the capsule endoscope image through a convolutional neural network. The convolutional neural network includes a pooling layer, an SA module, and a convolution module. The convolution module includes a deformable convolutional layer. The global image feature information extraction module is also used to extract a sixth image feature of the capsule endoscope image through the pooling layer. The sixth image feature is then sequentially convolved through multiple convolution modules with different convolution kernels, and the SA module is used to enhance the image feature extraction result after each convolution to obtain global image feature information. The self-attention weight extraction module is used to extract the self-attention weights of the capsule endoscopy image through a self-attention mechanism neural network. The self-attention weight extraction module is also used to extract the first image feature of the capsule endoscopy image through a first convolutional layer and divide the first image feature into multiple image feature blocks; perform image processing on the image feature blocks through a preset PSA module to obtain intermediate image feature blocks; combine multiple intermediate image feature blocks into intermediate image features; and extract the self-attention weights of the intermediate image features through a self-attention mechanism neural network. The classification module is used to fuse the global image feature information and the self-attention weights to obtain target image features, and to classify the target image features using a preset classifier to obtain the classification result of the capsule endoscopy image. The classification module is also used to upsample and multiply the global image feature information to obtain second global image feature information; to reshape, convolve, and multiply the self-attention weights to obtain second self-attention weights; to multiply the second global image feature information and the second self-attention weights to obtain a seventh image feature; and to process the seventh image feature sequentially through the PSA module and the AAP module to obtain the target image feature.

6. A terminal device, characterized in that, The terminal device includes: a memory, a processor, and a capsule endoscopy image classification program stored in the memory and executable on the processor. When the capsule endoscopy image classification program is executed by the processor, it implements the steps of the capsule endoscopy image classification method as described in any one of claims 1 to 4.

7. A computer storage medium, characterized in that, The computer storage medium stores a capsule endoscopy image classification program, which, when executed by a processor, implements the steps of the capsule endoscopy image classification method as described in any one of claims 1 to 4.

Images