Lesion recognition method for capsule endoscopy images based on frequency domain combined transform attention

Abstract

The present invention relates to a method for recognizing lesions in capsule endoscopy images based on frequency domain combined transform attention. With ResNeSt as the backbone, a single frequency domain transform is established after establishing an attention base network. On the basis of the two, a combined frequency domain transform is established. A frequency domain combined transform attention mechanism based on a neural network is established to form a final model. An image prediction model is obtained through training. A test data set is input into the image prediction model for image lesion recognition and evaluation. The capsule endoscopy image is recognized using the image prediction model that meets preset conditions. The present invention can fully obtain the diverse features of the WCE image channel and better preserve image information; remove trivial frequency components without losing accuracy and retain more effective features; better obtain image features while accelerating the convergence of the network to obtain better performance; and experiments further prove that selecting the green channel for frequency domain conversion can effectively improve network performance.

Description

Technical Field

[0001] The present invention relates to image analysis, and in particular to a capsule endoscope image lesion recognition method based on frequency domain combined transformation attention. Background Art

[0002] The small intestine is typically over four meters long, making it difficult to fully examine it using traditional handheld gastroenteroscopy. Therefore, capsule endoscopy has become an effective method for diagnosing digestive tract diseases, especially small intestinal diseases. Wireless capsule endoscopy (WCE), as a diagnostic tool for digestive tract diseases, allows for direct, non-invasive, and painless examination of the patient's entire gastrointestinal tract.

[0003] However, when using a wireless capsule endoscope, the capsule endoscope will capture approximately 50,000 frames of video images within 8-12 hours from the time the patient swallows the capsule pill for examination. Clinicians usually need to spend several hours to complete the entire WCE video examination, and misdiagnosis and missed diagnosis are prone to occur under long-term fatigue examinations. Therefore, it is of great significance to design a computer-aided system to assist doctors in reading films.

[0004] In recent years, computer vision has been applied to lesion recognition in capsule endoscopy images. The neural network attention mechanism, with its remarkable effectiveness and computational simplicity, has garnered widespread attention. Generally speaking, for visual reasoning tasks, low-frequency channels typically carry more useful information than high-frequency channels. However, existing attention-based methods often extract features only from the lowest-frequency portion of the information, discarding information at other frequencies. This fails to fully capture the diverse features of WCE image channels, resulting in suboptimal detection performance. Summary of the Invention

[0005] The present invention solves the problems existing in the prior art and provides a capsule endoscopy image lesion recognition method based on frequency domain combined transform attention, which has low computational complexity and can extract more spectral information, so that the input of the attention mechanism contains richer information, thereby improving the accuracy of the overall network.

[0006] The technical solution adopted by the present invention is a method for recognizing lesions in capsule endoscopy images based on frequency domain combined transform attention, the method comprising the following steps:

[0007] S1 obtains a dataset for training wireless capsule endoscopy disease classification;

[0008] S2 uses ResNeSt as the backbone to build an attention-based network;

[0009] S3 establishes a single frequency domain transform based on the attention base network in S2;

[0010] S4 establishes a combined frequency domain transform based on S2 and S3;

[0011] S5 is based on S4 and establishes a frequency domain combination transformation attention mechanism based on a neural network to form the final model;

[0012] S6 trains the final model of S5 with the dataset of S1 to obtain an image prediction model;

[0013] S7 inputs the test data set of the wireless capsule endoscope into the image prediction model to perform image lesion recognition and evaluation;

[0014] S8 performs lesion recognition on the capsule endoscopy image using an image prediction model that meets preset conditions.

[0015] Preferably, in S1, the data set used for lesion identification is set to X = {x1, x2, ..., x n}, where X matrix represents the data set, n represents the total number of samples, and x i Represents the input image, where i is an integer from 1 to n.

[0016] Preferably, said S2 comprises the following steps:

[0017] S2.1 build ResNeSt as the basic backbone network;

[0018] S2.2 Based on ResNeSt, we add attention based on frequency domain combination transformation to build an attention basic network.

[0019] x out =F dct (x i )

[0020] Among them, F dct (·) is the mechanism for realizing the frequency domain combined transform attention, x out is the transformed output, and then x out Send it to the next layer of the ResNeSt network, and you will get the basic network of frequency domain combined transformation attention.

[0021] Preferably, in the attention-based network, x i ∈R 3×H×w For the input image, the pixel values ​​of the three channels R, G, and B are the feature vectors. The RGB channels are represented as the red channel, green channel, and blue channel respectively. H and W are the height and width of the feature map respectively.

[0022] Preferably, the step S3 includes the following steps:

[0023] S3.1 takes the input feature x i ∈R 3×H×WThe R, G, B three channels are separated into 3 single channels x r 、x g 、x b , and x g Perform discrete cosine transform, the transformation process is:

[0024]

[0025] Among them, h∈{0,1,...,H-1}, w∈{0,1,...,W-1}, for a two-dimensional feature map x of size H×W g , a total of H×W transformed spectral components are obtained, This is the discrete cosine transform result obtained; i and j are integers from 0 to H-1 and 0 to W-1 respectively;

[0026] S3.2 sets a threshold a for the green channel frequency domain information obtained in S3.1, and retains the frequency domain information whose spectral components are less than a, that is,

[0027] S3.3 According to the test results of different thresholds, take the optimal f g , f g ∈R H×W , is the result of a single frequency domain transform.

[0028] Preferably, in said S4, f obtained according to S3.3 g , through 1×1 convolution, setting the number of convolution kernels to 3, changing the dimension of the image, we get Will With the initial input x i Combine, get

[0029] Preferably, in S5, for the input image of the basic network in S2, the attention F is transformed according to the frequency domain combination dct The final output is then fed into the next layer of the network, where it is used to establish the frequency domain combination transformation attention mechanism of the neural network, forming the final model. This process establishes the frequency domain combination transformation attention mechanism of the neural network. After a series of transformations, the results are fed into the next layer of the network.

[0030] Preferably, in the method, the output of the network is a classification result of the data set, and the image is judged as having a lesion or not based on the classification result.

[0031] The present invention relates to a method for recognizing lesions in capsule endoscope images based on frequency domain combined transform attention. The method comprises the following steps: obtaining a data set for training disease classification of wireless capsule endoscopes; establishing a single frequency domain transform after establishing an attention basic network with ResNeSt as the backbone, and establishing a combined frequency domain transform based on the two, and establishing a frequency domain combined transform attention mechanism based on a neural network to form a final model; training the final model with the data set to obtain an image prediction model; inputting a test data set of the wireless capsule endoscope into the image prediction model to perform image lesion recognition and evaluation, and performing lesion recognition on capsule endoscope images with the image prediction model that meets preset conditions.

[0032] The beneficial effects of the present invention are:

[0033] (1) The frequency domain combined transform attention mechanism is introduced to retain more frequency domain information by setting a certain threshold. Compared with the traditional spatial or channel-based attention method, it can fully obtain the diverse characteristics of the WCE image channel and better preserve the image information;

[0034] (2) It can be easily used by existing neural network models that use RGB images as input, removing trivial frequency components without losing accuracy and retaining more effective features;

[0035] (3) Since the network structure is not destroyed, the basic structure and parameter requirements of the original network are retained. By using frequency domain information to better obtain image features, the network convergence is accelerated, thereby achieving better performance;

[0036] (4) A simple test of the frequency domain transformation of the RGB red channel, blue channel or other channel combinations was conducted. Through the experimental results, it was further proved that selecting the green channel for frequency domain transformation can effectively improve network performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is an overall flow chart of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described in detail below with reference to the embodiments, but the protection scope of the present invention is not limited thereto.

[0039] The present invention relates to a method for recognizing lesions in capsule endoscopy images based on frequency domain combined transform attention, comprising the following steps:

[0040] S1. Obtain a dataset for training wireless capsule endoscopy disease classification.

[0041] In step S1 of this embodiment, the data set for lesion identification is set as X={x1, x2, ..., x n}, where X matrix represents the data set, n represents the total number of samples, and x i Represents the input image.

[0042] The algorithm goal is defined as: obtaining the classification results of the data set and determining whether the image has lesions or not.

[0043] In the present invention, after obtaining the public data set, it will be divided into a training set, a validation set, and a test set according to a certain ratio; the training set and the validation set are used to train the model, and the test set is used to refer to the performance indicators of the trained model.

[0044] S2. Using ResNeSt as the backbone, establish an attention-based network.

[0045] In step S2 of this embodiment, the specific process is as follows:

[0046] S21. Build ResNeSt as the basic backbone network;

[0047] S22. Based on ResNeSt, we add attention based on frequency domain combination transformation to build an attention basic network, where x i ∈R 3×H×W Represents the input image, and the feature vector is composed of the pixel values ​​of the three channels R, G, and B. H and W are the height and width of the feature map respectively. (xi,y i ) represents the labeled sample i, y i Represents the sample category label, where a value of 0 indicates the presence of a lesion and a value of 1 indicates the absence of a lesion.

[0048] The attention mechanism is to make the following transformations on the input layer:

[0049] x out =F dct (x i )

[0050] Among them F dct (·) is the attention mechanism used to implement the frequency domain combined transformation, x out is the transformed output, and then x out Send it to the next layer of the ResNeSt network, and you will get the basic network of frequency domain combined transformation attention.

[0051] S3. Establish a single frequency domain transform based on the attention base network in S2.

[0052] In step S3 of this embodiment, the specific process is as follows:

[0053] S31, the input feature x of the network i ∈R 3×H×W The R, G, and B channels are separated into three single channels, namely xr 、x g 、x b , and the two-dimensional feature map x of the green channel (G channel) g Perform discrete cosine transform, the transformation process is:

[0054]

[0055] where h∈{0,1,...,H-1}, w∈{0,1,...,W-1};

[0056] For a two-dimensional feature map x of size H×W g , a total of H×W transformed spectral components are obtained, That is the discrete cosine transform result; i and j are integers from 0 to H-1 and 0 to W-1 respectively; the subscripts of i and j here represent traversing each pixel of the image with a height of H and a width of W, and x g It is the result of separating the G (green) channel of the three-channel image;

[0057] RGB images are composed of red (R), green (G), and blue (B) pixel components superimposed in a matrix. To analyze the differences between WCE color channels, the mean and standard deviation of the R, G, and B channel color values ​​for 50 images with and without lesions were randomly calculated (see Table 1).

[0058] Table 1 Mean and standard deviation of channel color values

[0059] Evaluation indicators There is a lesion No lesions Absolute value of difference R channel mean 148.01 150.63 2.62 R channel standard deviation 50.41 51.92 1.51 G channel mean 87.93 92.37 4.44 G channel standard deviation 37.17 35.84 1.33 B channel mean 49.25 51.17 1.92 B channel standard deviation 25.26 24.16 1.10

[0060] In Table 1, the difference in channel color means is G > R > B, indicating that the grayscale value distribution of the G channel is more diverse than that of the R and B channels, and the G channel color distribution information is richer. The difference in standard deviation is R > G > B, indicating that the R and G channels contain more color detail information than the B channel. Combining these two evaluation metrics, the G channel's mean and standard deviation perform better than the R and B channels, so we extract the G channel features for frequency domain transformation.

[0061] S32, for the green channel frequency domain information obtained in S31, set a threshold a, a is generally 1e2 ~ 1e4, and retain the frequency domain information whose spectrum components are less than a, so as to obtain more effective frequency domain information beyond the lowest frequency. According to the different threshold test results, the effect ranking is finally obtained, and the optimal value is taken, that is, f g Substituted into the model, different thresholds make the classification effect of the model different, leaving the f that makes the model best g The value of

[0062]

[0063] f g ∈R H×W This is the result of a single frequency domain transform.

[0064] Because high-frequency regions in frequency-domain feature maps generally store image boundaries and texture information, while low-frequency regions primarily store information about flat areas within the image, existing methods often only extract the lowest-frequency portion of the feature information when extracting it, discarding much of the other frequencies. To fully capture the diversity of WCE image channels and obtain more effective frequency-domain information beyond the lowest frequency, this paper uses different threshold test results to rank the results, ultimately selecting the optimal value as a = 9e3.

[0065] S4. Establish a combined frequency domain transform based on S2 and S3.

[0066] In step S4 of this embodiment, the specific process is as follows:

[0067] S41, the single frequency domain transformation result f obtained in step S33 g ∈R H×W , through 1×1 convolution, setting the number of convolution kernels to 3 to change the dimension of the image, we get

[0068] Through 1×1 convolution, we can transform the original image and obtain a new image, which can improve the generalization ability and reduce overfitting. At the same time, in this process, depending on the number of 1×1 convolutions and convolution kernels selected, cross-channel interaction and information integration can be achieved, and the dimension of the image can also be changed.

[0069] S42, the step S41 obtained With the initial input x i The process of combining is:

[0070]

[0071] You can get F dct (x i ).

[0072] S5. Based on S4, a frequency domain combination transformation attention mechanism based on a neural network is established to form the final model.

[0073] In step S5 of this embodiment, the process of establishing a wireless capsule endoscopy image lesion recognition method based on frequency domain combined transform attention is as follows:

[0074] S51, for the input image of the basic network in S2, transform the attention F according to the frequency domain combination dctThe transformation is performed to obtain the final output, which is sent to the next layer of the network, and the frequency domain combination transformation attention mechanism of the neural network is established to form the final model.

[0075] S6. Based on the training data set of S1, the final model in S5 is trained to obtain an image prediction model.

[0076] In step S6 of this embodiment, the specific process is as follows:

[0077] Based on the capsule endoscopy disease classification dataset in S1, the frequency domain combined transformation attention mechanism obtained by S2, S3 and S4 is substituted into the final model in S5 for training to obtain the image recognition prediction model.

[0078] S7. Inputting the test data set of the wireless capsule endoscope into the image prediction model to perform image lesion recognition and evaluation.

[0079] In step S7 of this embodiment, the specific process is as follows:

[0080] After obtaining the prediction model in step S6, the test set images are input to predict the prediction model. The performance of the model is evaluated using three indicators: Accuracy, F1-score, and Precision, and the capsule endoscopy image classification prediction results are obtained.

[0081]

[0082]

[0083]

[0084] Among them, TP, TN, FP and FN represent true positive, true negative, false positive and false negative, respectively. True positive means that the lesion is correctly identified, true negative means that the background pixels are correctly identified, that is, there is no lesion, false positive means that the background pixels are mistakenly identified as lesions, and false negative means that the lesion is mistakenly segmented (identified) as background pixels.

[0085] S8 Finally, the capsule endoscopy image is subjected to lesion recognition using an image prediction model that meets preset conditions.

[0086] The effects of the present invention can be further illustrated by the following simulation experiments.

[0087] (1) Simulation conditions

[0088] The experiment uses an Intel(R) Xeon(R) Silver 4210R CPU@2.40GHz

[0089] A 2.39GHz workstation (2 processors), 64GB of RAM, Windows 10 operating system, and three NVIDIA GeForce RTX2080SUPER graphics cards were used. Cuda version 10.0 was used. The model was implemented using the PyTorch deep learning framework, PyTorch version 1.6.0 + cu101, and Python version 3.7. Training was performed with a batch size of 32, using the Adam algorithm to optimize overall parameters. The learning rate was set to 0.0001, adjusted using the StepLR mechanism. The model converged after 100 epochs.

[0090] (2) Simulation results

[0091] The proposed method was compared with various classification networks on the same dataset. Accuracy, F1-score, and Precision, commonly used metrics in image classification, were used to evaluate the model, as shown in Table 2.

[0092] Table 2

[0093]

[0094] As can be seen from Table 2, the method of the present invention has better results than other classification networks, effectively improves the classification performance, can be better applied to lesion recognition and auxiliary diagnosis in wireless capsule endoscopy images, and has great practical engineering application value.

[0095] In order to achieve the above content, the present invention also relates to a computer-readable storage medium, which stores a capsule endoscope image lesion recognition program based on frequency domain combined transform attention. When the program is executed by a processor, it implements the above-mentioned capsule endoscope image lesion recognition method based on frequency domain combined transform attention.

[0096] In order to achieve the above content, the present invention also relates to a computer device, including a memory, a processor and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the above-mentioned capsule endoscopy image lesion recognition method based on frequency domain combined transform attention is implemented.

[0097] By using the computer-readable storage medium and the computer device in the medium, and by inputting the intestinal wall blood vessel image to be segmented into the device, the method has lower computational complexity and can extract more spectral information, so that the input of the attention mechanism contains richer information, thereby improving the accuracy of the overall network.

[0098] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0102] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0103] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A method for capsule endoscopy image lesion recognition based on frequency domain combined transform attention, characterized by: The method comprises the following steps: S1 obtains a dataset for training wireless capsule endoscopy disease classification; S2 uses ResNeSt as the backbone to build an attention-based network; S3 builds a single frequency domain transform based on the attention base network in S2, including the following steps: S3.1 Input features The R, G, B three channels are separated into 3 single channels x r 、x g 、x b , and x g Perform discrete cosine transform, the transformation process is: ， in, , for a two-dimensional feature map x of size H×W g , a total of H×W transformed spectral components are obtained, This is the discrete cosine transform result obtained; i and j are integers from 0 to H-1 and 0 to W-1 respectively; S3.2 Set a threshold a for the green channel frequency domain information obtained in S3.1, and retain the frequency domain information whose spectral components are less than a, that is, ; S3.3 According to different threshold test results, take the best , , is the result of a single frequency domain transform; S4 establishes a combined frequency domain transform based on S2 and S3; based on S3.3 , through 1×1 convolution, setting the number of convolution kernels to 3, changing the dimension of the image, we get ;Will With initial input Combine, get ; S5 builds a frequency domain combination transformation attention mechanism based on neural network based on S4 to form the final model; S6 trains the final model of S5 with the dataset of S1 to obtain an image prediction model; S7 inputs the test data set of the wireless capsule endoscope into the image prediction model to perform image lesion recognition and evaluation; S8 uses an image prediction model that meets preset conditions to identify lesions in capsule endoscopy images.

2. The method for capsule endoscopy image lesion recognition based on frequency domain combined transform attention according to claim 1, characterized in that: In S1, the data set used for lesion identification is set as ,in, The matrix represents the data set, n represents the total number of samples, x i Represents the input image, where i is an integer from 1 to n.

3. The method for capsule endoscopy image lesion recognition based on frequency domain combined transform attention according to claim 2, characterized in that: The S2 comprises the following steps: S2.1 Build ResNeSt as the basic backbone network; S2.2 Based on ResNeSt, we add attention based on frequency domain combination transformation to build an attention basic network. , in, is a mechanism for implementing frequency domain combined transform attention, is the transformed output, and then Send it to the next layer of the ResNeSt network, and you will get the basic network of frequency domain combined transformation attention.

4. The method for capsule endoscopy image lesion recognition based on frequency domain combined transform attention according to claim 3, characterized in that: In the attention-based network, For the input image, the pixel values ​​of the three channels R, G, and B are the feature vectors, and H and W are the height and width of the feature map respectively.

5. The method for capsule endoscopy image lesion recognition based on frequency domain combined transform attention according to claim 1, characterized in that: In S5, for the input image of the basic network in S2, the attention is transformed according to the frequency domain combination The transformation is performed to obtain the final output, which is sent to the next layer of the network, and the frequency domain combination transformation attention mechanism of the neural network is established to form the final model.

6. The method for capsule endoscopy image lesion recognition based on frequency domain combined transform attention according to claim 1, characterized in that: In the method, the output of the network is the classification result of the data set, and the image is judged as having lesions or not based on the classification result.

Images