Fiber microscopic endoscope image reconstruction method based on contrast learning and storage medium

Abstract

The application discloses a fiber microscopic endoscope image reconstruction method based on contrast learning and a storage medium. The method comprises the following steps: acquiring a batch of optical fiber microscopic endoscope pictures, pairing the pictures according to whether the pictures have a grid or not, performing pixel filtering on the grid pictures to obtain pictures similar to the grid, and performing trap wave filtering on the pictures to obtain pictures without the grid; a de-griding model is constructed based on a convolutional auto-encoder, a convolutional auto-decoder, an attention mechanism and a gated convolution, and the model is optimized; and clear pictures are generated by inputting the images into the optimized model. The application enables the neural network to learn the images without the grid and the images with the grid and the processed images thereof, combines the images with the grid and the images without the grid, learns the grid and the rough texture features without the grid, and enables the generated images to be closer to the images without the grid and farther away from the images with the grid under the supervision of the adversarial learning, so that the optical fiber microscopic endoscope images with the grid structure can be converted into the optical fiber microscopic endoscope images without the grid structure.

Description

Technical Field

[0001] This invention relates to the field of fiber optic endoscope image technology, specifically to a fiber optic endoscope image reconstruction method and storage medium based on contrastive learning. Background Technology

[0002] Fiber optic endoscopy is a novel endoscope that utilizes laser scanning confocal imaging technology and fluorescent labeling. It can present cell morphology highly consistent with biopsy pathological images, allowing physicians to accurately determine cancer, precancerous lesions, or healthy conditions in situ with minimal patient discomfort. Fiber optic endoscopy boasts micrometer-level resolution and is currently the only endoscopic instrument capable of cell imaging. Its advent represents a significant leap forward in endoscopic technology. Studies have shown that fiber optic endoscopy has immense clinical value and potential in the diagnosis of early colorectal cancer, ulcerative colitis, Barrett's esophagus, gastroesophageal reflux disease, non-erosive reflux disease, and other digestive tract diseases. However, the imaging process is affected by the fiber optic cable, resulting in a mesh structure that obscures some information in the image and causes uneven brightness, significantly impacting the effectiveness of image-based disease diagnosis and image-based artificial intelligence recognition and treatment methods. Summary of the Invention

[0003] The present invention proposes a fiber optic endoscope image reconstruction method storage medium based on contrastive learning, which can be well adapted to different fiber optic endoscope images, thereby enabling the processing of gridded images to obtain clear gridless images.

[0004] To achieve the above objectives, the present invention adopts the following technical solution:

[0005] A fiber optic endoscope image reconstruction method based on contrastive learning includes the following steps:

[0006] A batch of fiber optic endoscope images are acquired, and the images are paired according to whether they have a grid. Then, the grid images are processed by pixel filtering to obtain images with an approximate grid, and notch filtering is processed to obtain images without a grid.

[0007] A demesh model is constructed based on convolutional autoencoders, convolutional autodecoders, attention mechanisms, and gated convolutions. The parameters of the demesh model are then optimized based on the dataset to obtain an optimized model.

[0008] A clear image is generated by inputting the image to be optimized and its notch-filtered image into the optimization model.

[0009] Furthermore, the process of acquiring batches of fiber optic endoscope images, pairing them according to whether they have a grid pattern, performing pixel filtering on the grid images to obtain images with approximate grid patterns, and performing notch filtering on the grid images to obtain images without grid patterns specifically includes:

[0010] The process involves acquiring images from a fiber optic endoscope and a filtered image derived from a gridded image. The steps for obtaining the filtered image are as follows: First, the gridded image undergoes a Gaussian filter with a parameter of 1 to remove noise from images captured under dark conditions. Then, the gridded image is filtered by setting pixels with a value greater than 30 to 0, resulting in an image C that approximately contains only the grid. The filtering steps are as follows:

[0011]

[0012] Where pixel represents all the pixels in the grid image I;

[0013] Next, perform notch filtering on the gridded image with a filtering range of 200-500 to obtain an approximately grid-free image F. The filtering steps are as follows:

[0014] F1 = FFT(I)

[0015]

[0016] F = IFFT(F2)

[0017] Where FFT is Fourier transform, Pixe is the pixel in F1, and center is the center point of F1. Whether filtering is needed is determined based on the Euclidean distance between the pixel and the center point.

[0018] Furthermore, the construction of a meshless model based on convolutional autoencoders, convolutional autodecoders, attention mechanisms, and gated convolutions, followed by parameter optimization of the meshless model using the dataset to obtain an optimized model, specifically includes:

[0019] The obtained feature maps are used to optimize the de-networking model, which includes a cellular feature encoder (CFE) for recovering the occluded parts by the grid and a filter encoder (FFE) for extracting grid-free features. The inputs to the cellular feature encoder (CFE) are the pixel-filtered image C and the image I. The specific steps of the cellular encoder include:

[0020] F c =GatedConv Upsample(ResNetBlock(GatedConv Downsample(C⊙I)))

[0021] Where F cThis represents the features learned by the cellular feature encoder. `GatedConvDownsample` and `GatedConvupsample` are downsampling and upsampling performed using gated convolution. Gated convolution involves performing different convolutional encoding operations on the same feature map, then applying sigmoid activation to one of the encoded feature maps and multiplying it by the other encoded feature map. The above process can be formalized as follows:

[0022] F G (I)=Conv1(I)+Simoid(Conv2(I))

[0023] GatedConv Downsample(I)=Conv3(F G (F G (I)))

[0024]

[0025] Conv1, Conv2, and Conv3 represent different convolutional networks without padding. Conv1 and Conv2 are convolutions with a kernel size of 3*3 and a stride of 1, while Conv3 is a convolution with a kernel size of 3*3 and a stride of 4. After extracting high-level features, residual learning is performed, which includes three residual learning blocks.

[0026] The input to the filter encoder FFE is the filtered image F. The specific steps of the filter encoder FFE include:

[0027] F c =Conv Upsample(FABlock(Conv Downsample(F)))

[0028] Conv Upsample and Conv Downsample refer to ordinary upsampling / downsampling convolutional coding to reduce the size of feature maps. FABlock refers to convolutional coding with added channel attention and pixel attention mechanisms on ordinary convolutional coding, specifically including:

[0029] FABlock(F)=Pixel((Channel(F)+F))+(Channel(F)+F)

[0030] Channel(F) is the added channel attention convolutional encoding, and Pixel(F) is the added pixel attention convolutional encoding to F, which can be expressed by the following formulas:

[0031] Channel(F)=Pooling(Conv6(ReLU(Conv7(F))))

[0032] Pixel(F)=Conv7(ReLU(Conv7(F)))

[0033] Pooling is an average pooling layer, and the other two Convs are ordinary convolutional encodings with a kernel size of 3*3 and a stride of 1.

[0034] Finally, the features extracted by the Cellular Feature Encoder (CFE) and the Filtered Feature Encoder (FFE) are added and fused along channels, and a feature map I with the same size and number of channels as the original image is obtained through convolution. F And perform loss function calculation, where the loss function is:

[0035]

[0036] FFT stands for Fourier Transform. In addition to L1 loss, an adversarial loss based on Fourier Transform based on adversarial learning is added. The similarity between the learned feature function and the ground truth is adjusted by the weight hyperparameter β, and the feature function is kept away from the image containing the grid. The loss is reduced during iterative training. When the loss no longer decreases, the current network weights are saved for image degrinding.

[0037] Furthermore, the specific steps of the residual learning block are expressed by the following formula:

[0038] F ResBlock (Feature)=Conv4(Feature)+Conv5(ReLU(Conv4(Feature)))

[0039] ReLU(x) = max(0, z)

[0040] Feature is the feature map obtained by threshold convolution, and Conv4 and Conv5 are convolutions with a kernel size of 3*3 and a stride of 1.

[0041] In another aspect, a computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the steps of the method described above.

[0042] As can be seen from the above technical solution, the fiber optic endoscope image reconstruction method based on contrastive learning of the present invention enables the neural network to learn meshless images and meshed images and their processed images. By combining meshed and meshless images, in the learning of the coarse texture features of meshed and meshless images, under the supervision of adversarial learning, the generated image is made closer to the meshless image and further away from the meshed image, thus enabling the transformation of meshed fiber optic endoscope images into meshless fiber optic endoscope images. Attached Figure Description

[0043] Figure 1This is a flowchart of the steps of a fiber optic endoscope image reconstruction method based on contrastive learning according to the present invention.

[0044] Figure 2 This is a structural block diagram of a fiber optic endoscope image reconstruction system based on contrastive learning according to the present invention.

[0045] Figure 3 This is a structural block diagram of a fiber optic microendoscopic image network model based on contrastive learning according to the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0047] like Figure 1 As shown, this invention provides a fiber optic endoscope image reconstruction method based on contrastive learning, which includes the following steps:

[0048] S1. Obtain a batch of fiber optic endoscope images, pair the images according to whether they have grids, then perform pixel filtering on the grid images to obtain images with approximate grids and notch filtering to obtain images without grids.

[0049] S2. Construct a meshless model based on convolutional autoencoders, convolutional autodecoders, attention mechanisms, and gated convolutions;

[0050] S3. Optimize the parameters of the meshless model based on the dataset to obtain the optimized model;

[0051] S4. By acquiring the image to be optimized and its processed image, inputting them into the optimization model, a clear image is generated.

[0052] Specifically, it includes three stages: the first stage is to acquire images from fiber optic endoscopes and filtered images converted from gridded images; the second stage is to optimize the degrid model using the obtained feature maps; and the third stage is to convert the acquired gridded images into clear, gridless images.

[0053] The first stage involves acquiring images from a fiber optic endoscope and a filtered image derived from the gridded image. The steps for obtaining the filtered image are as follows: First, the gridded image undergoes a Gaussian filter with a parameter of 1 to remove most of the noise present in images captured under dark conditions. Then, the gridded image is filtered by setting pixels with a value greater than 30 to 0, resulting in an image C that approximately contains only the grid. The filtering steps are as follows:

[0054]

[0055] Where pixel represents all the pixels in the grid image I;

[0056] Next, perform notch filtering on the gridded image with a filtering range of 200-500 to obtain an approximately grid-free image F. The filtering steps are as follows:

[0057] F1 = FFT(I)

[0058]

[0059] F = IFFT(F2)

[0060] Here, FFT stands for Fourier Transform, Pixe represents a pixel in F1, and center is the center point of F1. The Euclidean distance between the pixel and the center point determines whether filtering is necessary. The network frequency range of 200-500 is determined by manual measurement. By filtering out signals in the 200-500 frequency range from the Fourier space, an approximately grid-free image can be obtained.

[0061] The second stage utilizes the obtained feature maps to optimize the de-network model. The de-network model includes a cellular feature encoder (CFE) for recovering the occluded parts and a filter encoder (FFE) for extracting mesh-free features. The input to the cellular feature encoder CFE is the pixel-filtered image C and image I. The specific steps of the cellular encoder include:

[0062] F c =GatedConv Upsample(ResNetBlock(GatedConv Downsample(C⊙I)))

[0063] Where F c This represents the features learned by the cellular feature encoder. `GatedConvDownsample` and `GatedConvupsample` are downsampling and upsampling performed using gated convolution. Gated convolution involves performing different convolutional encoding operations on the same feature map, then applying sigmoid activation to one of the encoded feature maps and multiplying it by the other encoded feature map. The above process can be formalized as follows:

[0064] F G (I)=Conv1(I)+Simoid(Conv2(I))

[0065] GatedConv Downsample(I)=Conv3(F G (F G (I)))

[0066]

[0067] Where Conv1, Conv2, and Conv3 represent different convolutional networks without padding, Conv1 and Conv2 are convolutions with a kernel size of 3*3 and a stride of 1, and Conv3 is a convolution with a kernel size of 3*3 and a stride of 4. After extracting high-level features, residual learning is performed. The residual learning consists of three residual blocks, and the specific steps of the residual learning blocks are expressed by the following formula:

[0068] F ResBlock (Feature)=Conv4(Feature)+Conv5(ReLU(Conv4(Feature)))

[0069] ReLU(x) = max(0, z)

[0070] Feature is the feature map obtained by threshold convolution, and Conv4 and Conv5 are convolutions with a kernel size of 3*3 and a stride of 1.

[0071] The input to the filter encoder FFE is the filtered image F. The specific steps of the filter encoder FFE include:

[0072] F c =Conv Upsample(FABlock(Conv Downsample(F)))

[0073] Conv Upsample and Conv Downsample refer to ordinary upsampling / downsampling convolutional coding to reduce the size of feature maps. FABlock refers to convolutional coding with added channel attention and pixel attention mechanisms on ordinary convolutional coding, specifically including:

[0074] FABlock(F)=Pixel((Channel(F)+F))+(Channel(F)+F)

[0075] Channel(F) is the added channel attention convolutional encoding, and Pixel(F) is the added pixel attention convolutional encoding to F, which can be expressed by the following formulas:

[0076] Channel(F)=Pooling(Conv6(ReLU(Conv7(F))))

[0077] Pixel(F)=Conv7(ReLU(Conv7(F)))

[0078] Pooling is an average pooling layer, and the other two Conv layers are ordinary convolutional encoding layers with a kernel size of 3*3 and a stride of 1.

[0079] Finally, the features extracted by the Cellular Feature Encoder (CFE) and the Filtered Feature Encoder (FFE) are added and fused along channels, and a feature map I with the same size and number of channels as the original image is obtained through convolution. F And perform loss function calculation, where the loss function is:

[0080]

[0081] FFT stands for Fourier Transform. In addition to L1 loss, an adversarial loss based on Fourier Transform based on adversarial learning is added. The similarity between the learned feature function and the ground truth is adjusted by the weight hyperparameter β, and the feature function is kept away from the image containing the grid. The loss is reduced during iterative training. When the loss no longer decreases, the current network weights are saved for image degrinding.

[0082] The third stage involves converting the obtained gridded image into a clear, gridless image. Specifically, this includes: inputting a gridded image, filtering it through pixel filtering and notch filtering to obtain an approximate gridded image and an approximate gridless image, inputting the three images into a de-grid model, and obtaining the gridless image after convolution processing at each layer.

[0083] On the other hand, embodiments of the present invention also disclose a fiber optic endoscope image reconstruction system based on contrastive learning, comprising:

[0084] The acquisition module is used to acquire images from fiber optic endoscopes and image features extracted from gridless images;

[0085] The optimization module is used to optimize the de-network model using feature maps from both meshless and meshed images;

[0086] The conversion module is used to convert the obtained gridded image into a clear image without a grid.

[0087] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0088] A fiber optic endoscope image reconstruction device based on contrastive learning:

[0089] At least one processor;

[0090] At least one memory for storing at least one program;

[0091] When the at least one program is executed by the at least one processor, the at least one processor implements a fiber optic endoscope image reconstruction method based on contrastive learning as described above.

[0092] In another aspect, the present invention also discloses a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to perform the steps of any of the methods described above.

[0093] In another aspect, the present invention also discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of any of the methods described above.

[0094] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the methods described in the above embodiments.

[0095] It is understood that the system provided in the embodiments of the present invention corresponds to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0096] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0097] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0098] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fiber optic endoscope image reconstruction method based on contrastive learning, characterized in that, Includes the following steps: A batch of fiber optic endoscope images are acquired, and the images are paired according to whether they have a grid. Then, the grid images are processed by pixel filtering to obtain images with an approximate grid, and notch filtering is processed to obtain images without a grid. A meshless model is constructed based on convolutional autoencoders, convolutional autodecoders, attention mechanisms, and gated convolutions. The parameters of the meshless model are then optimized based on the dataset to obtain an optimized model. A clear image is generated by inputting the image to be optimized and its notch-filtered image into the optimization model. The process of acquiring batches of fiber optic endoscope images, pairing images based on whether they have a grid pattern, performing pixel filtering on the grid images to obtain images with an approximate grid pattern, and performing notch filtering on the grid images to obtain images without a grid pattern, specifically includes: The process involves acquiring images from a fiber optic endoscope and a filtered image derived from a gridded image. The steps for obtaining the filtered image are as follows: First, the gridded image undergoes a Gaussian filter with a parameter of 1 to remove noise from images captured under dark conditions. Then, the gridded image is filtered by setting pixels with a value greater than 30 to 0, resulting in an image C that approximately contains only the grid. The filtering steps are as follows: Where pixel represents all the pixels in the grid image I; Next, perform notch filtering on the gridded image with a filtering range of 200-500 to obtain an approximately grid-free image F. The filtering steps are as follows: in It is a Fourier transform. yes The pixels in yes The center point is determined based on the Euclidean distance between the pixel and the center point to determine whether filtering is needed; the demeshing model is constructed based on convolutional autoencoder, convolutional autodecoder, attention mechanism and gated convolution, and the optimized model is obtained by optimizing the parameters of the demeshing model according to the dataset, specifically including: The obtained feature maps are used to optimize the de-networking model, which includes a cellular feature encoder (CFE) for recovering the occluded parts by the grid and a filter encoder (FFE) for extracting grid-free features. The inputs to the cellular feature encoder (CFE) are the pixel-filtered image C and the image I. The specific steps of the cellular encoder include: in This represents the features learned by the cellular feature encoder. and This refers to downsampling and upsampling using threshold convolution. Threshold convolution involves performing different convolutional encoding operations on the same feature map, then applying sigmoid activation to one of the encoded feature maps and performing a dot product with the other encoded feature map. The formula is as follows: in , , The different convolutional networks have no padding, among which , For convolution kernel size of 3 3 convolutions with a stride of 1, For convolution kernel size of 3 Three convolutions with a stride of 4 are used to extract high-level features, followed by residual learning. The residual learning consists of three residual learning blocks. The input to the filter encoder FFE is the filtered image F. The specific steps of the filter encoder FFE include: in and This refers to ordinary upsampling / downsampling convolutional coding used to reduce the size of feature maps. FABlock refers to convolutional coding with added channel attention and pixel attention mechanisms, specifically including: That Encode the added channel attention convolution. The pixel attention convolutional encodings added to F are expressed by the following formulas: in For average pooling, the other two... For convolution kernel size of 3 3. Ordinary convolutional coding with a stride of 1; Finally, the features extracted by the Cellular Feature Encoder (CFE) and the Filtered Feature Encoder (FFE) are added and fused along channels, and a feature map with the same size and number of channels as the original image is obtained through convolution. And perform loss function calculation, where the loss function is: Where FFT stands for Fourier Transform, in Based on the loss, an adversarial loss based on Fourier transform of adversarial learning is added. The similarity between the learned feature function and the ground truth is adjusted by the weight hyperparameter β, and it is kept away from the image containing the grid. The loss is reduced during iterative training. When the loss no longer decreases, the current network weights are saved for image degrinding.

2. The fiber optic endoscope image reconstruction method based on contrastive learning according to claim 1, characterized in that: The specific steps of residual learning blocks can be expressed by the following formula: in The feature map obtained by threshold convolution. and For convolution kernel size of 3 3 convolutions with a stride of 1.

3. A computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method as described in any one of claims 1 to 2.

Images