Gastroscope white light to narrow-band light image conversion method based on Vision Mama

Abstract

The invention relates to a gastroscope white light to narrow-band light image conversion method based on Vision Mama, and belongs to the technical field of image processing. According to the method, a generative adversarial network based on Vision Mama is provided, and detailed representation is enhanced through a long-range dependency relationship and fine-grained local features; and information interaction between the source domain image and the target domain image is maximized by using comparative learning so as to improve detail representation after the white light image is converted into the narrow-band light image. In addition, a structural consistency constraint is designed, and the organization structure consistency of the generated image and the original image is kept while information interaction is performed. In order to verify the validity of the proposed method, wide experiments are performed on a private gastroscope data set and a BraTS 2021 data set. Experimental results show that non-paired image conversion closer to a real image can be achieved through the method.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and specifically relates to a method for converting gastroscopic white light to narrow-band light images based on Vision Mamba. The method adopts unpaired image conversion to generate paired white light and narrow-band light images, which can be applied to clinical gastroscopic auxiliary diagnosis and provide protection for the prevention and treatment of gastric diseases and even gastric cancer. Background Art

[0002] The global incidence and mortality of gastric cancer have increased significantly, seriously threatening human life and health. If patients can detect and receive timely treatment in the early stage of the disease, it is entirely possible to cure gastric cancer. Gastroscopy is the preferred way to screen gastric diseases clinically. It uses a tubular gastroscope to examine the mucosa of the esophagus, stomach, and duodenum through the mouth. Existing gastroscopy mainly relies on manually operated endoscopic imaging systems to identify suspicious lesion areas. The general process is as follows: first, the doctor manually operates the endoscope white light imaging (WLI) and judges the suspicious lesion area (abnormal area) based on experience; secondly, the abnormal area is observed more closely by acetic acid staining, narrow band imaging (NBI), image enhancement, etc., to make a preliminary inference on whether it is cancerous; finally, it is recommended that patients with suspected cancer undergo a biopsy to confirm whether it is cancerous. It can be seen from the gastroscopy process that WLI and NBI are two important modalities for cancer diagnosis. Manual gastroscopy requires frequent switching of lenses and light sources, and the accuracy and efficiency of the examination depend on the professional skills of doctors in the endoscopic digestive center. Moreover, manual gastroscopy has the problems of strong subjectivity, large individual differences, and time-consuming and labor-intensive, and the detection accuracy is easily affected by the doctor's visual fatigue.

[0003] In recent years, medical image-assisted diagnosis based on computer vision has attracted extensive attention from researchers, making deep learning flourish in medical image segmentation, classification, anomaly detection and image generation tasks. In particular, multimodal medical image analysis usually uses different imaging methods of the diseased area to achieve clinical auxiliary diagnosis, and has gradually become a hot topic of researchers. At present, endoscopic image analysis based on deep learning mainly uses white light images. For example, Galdran et al. proposed a dual encoder-decoder network for gastric polyp segmentation in endoscopic white light images. Trinh et al. combined Meta-Former with Unet and introduced a multi-scale upsampling block of level-up combination in the decoder stage to achieve detailed endoscopic image polyp segmentation. Tian et al. proposed an anomaly detection method of constrained contrast learning distribution and achieved advanced performance in colonoscopy anomaly detection. Huo et al. proposed a three-branch multi-level multi-scale feature fusion network HiFuse, which achieved accurate classification of 8 types of gastrointestinal diseases by fusing multi-scale global and local features.

[0004] However, these methods ignore the important role of narrow-band light images in clinical endoscopic diagnosis. Figure 1 As shown in (a), white light is more sensitive to examining morphology, mucosa, and vascular distribution, which helps doctors find gastric ulcers and tumors. Figure 1 As shown in (b), NBI narrow-band images can highlight the mucosal features and vascular structures in suspicious areas, and can assist in the location of early gastric cancer. In clinical practice, only white light images are usually collected, and narrow-band light images are rarely collected, which poses a challenge to multimodal gastroscopy imaging tasks.

[0005] In summary, the current difficulties in converting white light images to narrow-band light images are: (1) How to generate paired white light and narrow-band light images in the absence of paired images. (2) How to ensure that the generated images are highly consistent with the original images in terms of organizational structure. (3) How to maximize the information interaction between the white light image domain and the narrow-band light image domain. Therefore, it is necessary to design a white light to narrow-band light image generation model based on unpaired image conversion. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the prior art and provide a gastroscopic white light to narrow-band light image conversion method based on Vision Mamba (abbreviated as WtNGAN), which realizes the style conversion of unpaired white light images to NBI narrow-band images, and provides a possible data basis for multimodal gastroscopic image-assisted diagnosis.

[0007] To achieve the above object, the technical solution of the present invention is: a method for converting gastroscopic white light to narrow-band light images based on Vision Mamba, comprising:

[0008] A generative adversarial network based on Vision Mamba is proposed to enhance detailed representation through long-range dependencies and fine-grained local features.

[0009] In one embodiment of the present invention, the method further includes:

[0010] Contrastive learning is used to maximize the information interaction between source domain images and target domain images to improve the detail representation after converting white light images to narrow-band light images.

[0011] In one embodiment of the present invention, the method further includes:

[0012] A structural consistency constraint is designed to maintain the organizational structure consistency between the generated image and the original image while information exchange is in progress.

[0013] In one embodiment of the present invention, a generative adversarial network based on Vision Mamba is constructed, and the specific implementation is as follows:

[0014] The input image x first passes through the deformer T to obtain the deformed image T(x); then, x and T(x) are embedded in the image block and the image features are extracted by the encoder Enc of the generator G. The generator G is constructed according to the U-net structure. The encoder has 4 layers, each layer consists of two VSS Blocks and corresponds to the decoder Dec of the generator G; the output of each layer of the encoder is recorded as E i , where i represents the number of encoder layers, and is added to the decoder input through a jump connection, and the decoder outputs the image transformation results G(x) and G(T(x)); in order to constrain the shape of the generated image, G(x) passes through the same deformer T as the input image to obtain T(G(x)), and the consistency loss is calculated; G(x) is input into the encoder again, projected into the shared patch embedding space and recorded as M i ; Then, through E i and M i The contrast loss is calculated to realize the information interaction between the source domain image and the target domain image; finally, the discriminator D distinguishes the generated image from the real image to realize adversarial training with the generator.

[0015] In one embodiment of the present invention, contrastive learning is used to maximize the information interaction between the source domain image and the target domain image, which is specifically implemented as follows:

[0016] Contrastive learning CL creates positive and negative sample pairs by bringing positive pairs closer and negative pairs farther apart to learn unsupervised feature representations; specifically, the encoder Enc calculates the feature maps of each layer of the input image x Then, E i Divided into multiple blocks Where N is the number of selected spatial positions, and the blocks corresponding to the spatial positions are recorded as The tiles at other locations are represented as in Similarly, the feature map obtained after the output image y=G(x) is input into the reused encoder is recorded as Among them The corresponding block is and The corresponding blocks are

[0017] Set the encoder E i The image blocks of each layer output feature are matched with the image blocks of the corresponding position of the multi-layer perceptron output feature in the same layer. The encoder E iEach layer outputs all patches at different positions as negative pairs. By contrastive learning, the distance between positive pairs is shortened to promote the conversion of input images to target images, and the negative pairs are kept away from each other to avoid the input images being close to their own image styles. The contrastive learning loss is defined as:

[0018]

[0019] in represents the expectation of the input image x, Represents the input image x and the tiles obtained from x expectations, obtained from the source domain image and calculated by the reused encoder; Expressed as the loss for the classification problem, it is calculated as follows:

[0020]

[0021] Among them, a represents the query value, b represents the positive sample, c represents the negative sample, τ represents the scaling temperature, and c n represents the nth negative sample.

[0022] In one embodiment of the present invention, through structural consistency constraints, the organizational structure consistency of the generated image and the original image is maintained while information is exchanged, which is specifically implemented as follows:

[0023] First generate the two-dimensional standardized coordinate grid Grid of image x X,Y (x), where X and Y represent the horizontal and vertical axes respectively; then, a sine function transformation is applied to the horizontal component of the grid and added to the vertical component to produce a distorted grid effect; finally, the original image is sampled according to the distorted grid to generate a distorted image; the formula for generating the distorted image is calculated as follows:

[0024] T(x)=Sin(Grid X (x))×ρ+Grid Y (x)

[0025] Where Sin(·) represents the sine function, and ρ is a hyperparameter that controls the degree of image deformation. In order to maintain the structural consistency between the generated image and the original image, the L1 loss is used to penalize the inconsistency of the generated model structure prediction. The structural consistency loss is expressed as:

[0026]

[0027] Among them, H represents the image height, W represents the image width, and x h,w represents the original input image, T h,w (x) represents the image after deformation.

[0028] In one embodiment of the present invention, a generative adversarial network based on Vision Mamba is composed of a patch embedding layer, a four-layer encoder-decoder, a final projection layer and a discriminator; wherein,

[0029] The patch embedding layer first divides the input image x into 4×4 tiles and maps them to the C dimension; then, the image is sent to the 4-layer encoder-decoder to extract features. Each layer of the encoder-decoder contains two VSS Blocks; the VSS Block performs layer regularization on the input image, and then enters two branches. One branch passes through the linear layer and extracts features by deep convolution, and then is activated by the SiLU function and performs two-dimensional selective scanning. After the image is divided into tiles, the image sequence is scanned along four directions, and then by Figure 6 The S6 shown is the six-step selection operation of the Selection 6 algorithm, which extracts features from the sequence and passes through a regularization layer; the other branch is the original input image, and then the features of the two branches are multiplied, passed through a linear layer, and residually connected with the original input to obtain the output of the VSS Block; after the four-layer encoder-decoder extracts the image features, a 1×1 convolution layer and the final projection layer project the features to 3 channels to output the generated image; the discriminator consists of 4 layers of 4×4 convolutional layers, and each convolutional layer is activated by the LeakyReLU function and the features are normalized by batch normalization; the 3-channel image output by the final projection layer is judged as true or false by the discriminator;

[0030] In one embodiment of the present invention, the final loss of the entire method is composed of contrastive learning loss Structural consistency loss And adversarial loss The composition is as follows:

[0031] The adversarial loss is used to encourage the Nash equilibrium between the generator and the discriminator, hoping that the output image is visually as close to the target domain image as possible, and is defined as follows:

[0032]

[0033] in, represents the expectation of the real image y, Represents the expectation of the input original image x. The final loss is expressed as:

[0034]

[0035] Among them, α, β, and γ are hyperparameters of loss weighting.

[0036] The present invention also provides a gastroscopic white light to narrow-band light image conversion system based on Vision Mamba, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0037] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0038] Compared with the prior art, the present invention has the following beneficial effects: The present invention proposes a gastroscopic white light to narrowband light image conversion method WtNGAN based on VisionMamba, and proposes a generative adversarial network based on Vision Mamba to enhance detailed representation through long-range dependencies and fine-grained local features. Contrastive learning is used to maximize the information interaction between the source domain image and the target domain image, thereby improving the detail representation of the white light image in the narrowband light image. In addition, a structural consistency constraint is designed to keep the organizational structure of the generated image consistent with that of the original image while interacting with information. In order to verify the effectiveness of the proposed method, the present invention has conducted extensive experiments on private gastroscopy datasets and BraTS2021 datasets. Experimental results show that the WtNGAN proposed in the present invention can achieve unpaired image conversion that is closer to the real image. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 Examples of two common gastroscopic imaging methods for patients with early gastric cancer: (a) white light imaging and (b) narrow-band imaging.

[0040] Figure 2 This is a framework diagram of the method of the present invention.

[0041] Figure 3 Schematic diagram of Patch wise Contrastive Loss, where the image is extracted features through a reused encoder, and then a classification task of N negative samples + 1 positive sample is constructed.

[0042] Figure 4 Examples of gastroscopic images, including (a) a flat gastroscopic image and (b) a gastroscopic image with wrinkles.

[0043] Figure 5 is the patch scanning direction in the SS2D block, where the extended patch is scanned from left to center and the merged patch is scanned from center to right.

[0044] Figure 6 Pseudo code for the S6 operation in SS2D.

[0045] Figure 7 It is the result of converting WLI to NBI by each algorithm.

[0046] Figure 8 It is the result of converting NBI to WLI by each algorithm.

[0047] Fig. 9 It is the result of each algorithm converting T1 to T2.

[0048] Fig.10 It is the result of each algorithm converting T2 to T1. DETAILED DESCRIPTION

[0049] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0050] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0051] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0052] The present invention provides a method for converting gastroscopic white light to narrow-band light images based on Vision Mamba, comprising:

[0053] A generative adversarial network based on Vision Mamba is proposed to enhance detailed representation through long-range dependencies and fine-grained local features.

[0054] Use contrastive learning to maximize the information interaction between source domain images and target domain images to improve the detail representation after converting white light images to narrow-band light images;

[0055] A structural consistency constraint is designed to maintain the organizational structure consistency between the generated image and the original image while exchanging information.

[0056] The following is the specific implementation process of the present invention.

[0057] The key symbols and their descriptions involved in the present invention are shown in Table 1.

[0058] Table 1 Key symbols and their descriptions

[0059]

[0060]

[0061] 1. Network structure

[0062] The network structure of WtNGAN is as follows Figure 2 As shown in Figure 1. The input image x first passes through the deformer T to obtain the deformed image T(x). Subsequently, x and T(x) are embedded in the image blocks and then the encoder Enc extracts the image features. The generator G is constructed according to the U-net structure. Its encoder has 4 layers, each layer consists of two VSS Blocks and corresponds to the decoder Dec. The output of each layer of the encoder is denoted as E i , where i represents the number of encoder layers, and the decoder outputs the image conversion results G(x) and G(T(x)) after adding them to the decoder input through a jump connection. In order to constrain the shape of the generated image, G(x) is transformed through the same transformer T as the input image to obtain T(G(x)), and the consistency loss is calculated. The present invention inputs the converted image G(x) into the encoder again, projects it into the shared tile embedding space and records it as M i Then, through E i and M i The contrast loss is calculated to achieve information interaction between the source domain image and the target domain image. Finally, the discriminator D distinguishes the generated image from the real image to achieve adversarial training with the generator.

[0063] 2. Contrastive Learning

[0064] Contrastive learning (CL) is one of the commonly used methods in unsupervised learning. CL creates positive and negative sample pairs by shortening the distance between positive pairs and moving negative pairs away to learn unsupervised feature representation. Specifically, the encoder Enc calculates the feature map of each layer of the input image x Then, the present invention will E i Divided into multiple blocks Where N is the number of selected spatial positions, and the present invention records the blocks corresponding to the spatial positions as The tiles at other locations are represented as in Similarly, the present invention records the feature map obtained after the output image y=G(x) is input to the reused encoder Enc as Among them The corresponding block is

[0065] like Figure 3 As shown, the present invention converts the encoder E iThe image blocks of each layer output feature are matched with the image blocks of the corresponding position of the MLP output feature of the same layer. The encoder E i Each layer outputs all patches at different positions of the feature as negative pairs. By contrastive learning, the distance between positive pairs is shortened to promote the conversion of the input image to the target image, and the negative pairs are kept away from each other to avoid the input image being close to its own image style. Therefore, the contrastive learning loss is defined as:

[0066]

[0067] in represents the expectation of the input image x, Represents the input image x and the tiles obtained from x expectations, and similar, is obtained from the source domain image and computed by the reused encoder. The loss expressed as a classification problem is calculated as follows:

[0068]

[0069] Among them, a represents the query value, b represents the positive sample, c represents the negative sample, τ represents the scaling temperature, and c n Represents the nth negative sample

[0070] 3. Structural consistency constraints

[0071] Clinically, WLI to NBI is achieved by switching the imaging light source, and the paired WLI and NBI collected are usually structurally consistent. Intuitively, the present invention hopes that the diseased tissue appears in the same position in the two modalities. On the one hand, it helps doctors to quickly locate the suspicious area. On the other hand, the two corresponding lesion images are crucial for visual multimodal tasks. The present invention adds a simple and effective deformation constraint on the basis of contrastive learning.

[0072] Specifically, the present invention first generates a two-dimensional standardized coordinate grid Grid of image x X,Y (x), where X and Y represent the horizontal and vertical axes, respectively. Then, the present invention applies a sine function transformation to the horizontal component of the grid and adds it to the vertical component to produce a distorted effect of the grid. Finally, the original image is sampled according to the distorted grid to generate a distorted image. The formula for generating the distorted image is calculated as follows:

[0073] T(x)=Sin(Grid X (x))×ρ+Grid Y (x)

[0074] Where Sin(·) represents the sine function, and ρ is a hyperparameter that controls the degree of image deformation. In order to maintain the structural consistency between the generated image and the original image, the present invention uses the L1 loss penalty model to punish the inconsistency of the generated model structure prediction, and the structural consistency loss is expressed as:

[0075]

[0076] Among them, H represents the image height, W represents the image width, and x h,w represents the original input image, T h,w (x) represents the image after deformation.

[0077] The acquisition location of gastroscopic images usually includes the esophagus, stomach, and intestines. Figure 4 As shown, the tissue structures at these locations often have irregular wrinkles. These wrinkled images pose a challenge to the generative model, so the present invention hopes to improve the structural similarity between the generated image and the original image by creating an irregular pattern.

[0078] 4. VisionMambaUnet

[0079] Recently, deep models based on the Mamba architecture have attracted extensive attention from researchers. VisionMamba based on vision provides a new idea for learning long-distance dependencies. Yu et al. believe that the four-way scanning mechanism of VisionMamba is more suitable for intensive prediction tasks. Therefore, the present invention is inspired by VM-Unet and proposes a generative adversarial network based on VisionMamba.

[0080] Specifically, the network consists of a patch embedding layer, a four-layer encoder-decoder, a final projection layer, and a discriminator. The patch embedding layer first divides the input image x into 4×4 tiles and maps it to the C dimension. Then, the image is fed into a 4-layer encoder-decoder to extract features. Each layer of the encoder-decoder contains two Visual State Space (VSS) Blocks. Figure 2 As shown, VSS Block performs LayerNormalization on the input image.

[0081] Then it enters two branches. One branch passes through the linear layer and extracts features by deep convolution, then is activated by the SiLU function and performs a two-dimensional selective scan (2D-Selective-Scan, SS2D). SS2D is as follows Figure 5 As shown, the image is divided into blocks and then the image sequence is scanned along four directions, and then Figure 6The six-step selection operation of S6, i.e., Selection 6 algorithm, extracts features from the sequence and passes through a LN layer. The feature extraction operation of S6 is as follows Figure 6 As shown in Figure 2, the features of the two branches are then multiplied, passed through a linear layer, and connected to the original input through a residual connection to obtain the final output.

[0082] The following is a differentiable explanation of the two-dimensional selective scanning operation (SS2D) in VSS Block:

[0083] Mamba-based visual models all rely on traditional continuous schemes. Assume the input is Model through intermediate implicit mapping Map it to This process is expressed as a linear ordinary differential equation as follows:

[0084] h' t =Ah t +Bx t ,

[0085] y t =Ch t ,

[0086] in represents the state matrix, and represents the projection parameter. Mamba discretizes the continuous system to adapt to the deep learning scenario. Specifically, Mamba introduces the time scale Δ and discretizes A and B as and The discretization rule uses a zero-order holder, which is defined as follows:

[0087]

[0088] The discretized state space model is calculated by linear recursion as follows:

[0089]

[0090] y t =Ch t .

[0091] 5. Loss Function

[0092] The final loss of the proposed model consists of three parts: contrastive learning loss Structural consistency loss And adversarial loss The adversarial loss is used to encourage the Nash equilibrium between the generator and the discriminator. The present invention hopes to maximize the visual similarity of the output image with the target domain image, which is defined as follows:

[0093]

[0094] in, represents the expectation of the real image y, Represents the expectation of the input original image x. So the total loss of the network can be expressed as:

[0095]

[0096] Among them, α, β, and γ are hyperparameters of loss weighting.

[0097] 6. Implementation Examples

[0098] A gastroscopic image dataset Stomach was constructed by collecting gastroscopic WLI and NBI of 103 patients through the digestive endoscopy center of a hospital. The Stomach dataset contains 900 unpaired WLI and NBI, and the main collection locations are the patient's stomach and esophagus. All data and experiments in this invention have been approved by the hospital ethics committee and the scientific research ethics committee. In addition, in order to verify the generalization ability of the model in the field of unpaired image conversion, the present invention evaluated all algorithms on the task of mutual conversion between T1 and T2 modes of brain MRI in the BraTS2021 dataset.

[0099] The comparison algorithm is as follows:

[0100] 1) CycleGAN constructs two generators, one generator converts source domain images to target domain images, and the other generator converts the generated target domain images back to the source domain to achieve bidirectional image conversion. The proposed cycle consistency loss provides a strong constraint for the image generation task and significantly improves the baseline of the unpaired image generation task.

[0101] 2) ACL-GAN proposes an adversarial consistency loss, which aims to maximize the sharing of information between source domain images and target domain images, and hopes that the generated target domain images retain more features of the source domain images.

[0102] 3) CUT extends contrastive learning to the field of unpaired image translation for the first time. By operating on patches instead of the entire image, it encourages negative patches to be mapped to similar points in the learned feature space, thereby improving the information interaction between the source domain and the target domain.

[0103] 4) AttentionGAN is an attention-guided generative adversarial network that achieves discriminative image transformation by distinguishing the foreground and background of the source domain image, while preserving the background features and transforming the style of the foreground target.

[0104] 5) NEGCUT believes that the selection of negative samples in unpaired image conversion tasks is crucial to image conversion. It improves the quality of image conversion by generating negative samples online.

[0105] 6) MSPC believes that the distortion of unpaired image translation is caused by the significant change in the space between the source domain image and the target domain image. MSPC designs a maximum spatial perturbation consistency constraint to reduce unnecessary content space deformation.

[0106] 7) FFPE++ mixed the channel-spatial attention module in GAN and designed a self-regularization loss, which effectively reduced artifacts in formalin-fixed and paraffin-embedded tissue sections and improved the accuracy of clinical pathological diagnosis.

[0107] 6.1 Implementation Details

[0108] In the training phase, the present invention sets the initial learning rate to 0.0002 and uses momentum terms β 1 = 0.5 and β 2 =0.999 Adam Optimezer. The model of the present invention was trained for a total of 200 epochs, and the batch size was set to 1. In addition, the present invention implemented all methods on Ubuntu 18.04OS with Python 3.10.13and Pytorch 2.1.1. A 12-core computer equipped with an Intel Xeon Silver 4214R 2.4GHz (128GB RAM) CPU and an NVIDIA GeForce RTX 309024GB GPU was used for training and testing.

[0109] 6.2 Qualitative Comparison

[0110] The present invention has conducted extensive experiments on private gastroscopy image datasets and publicly available BraTS2021 datasets. In the gastroscopy dataset, the present invention mainly evaluates the performance of mutual conversion between WLI and NBI. In the BraTS2021 dataset, the present invention mainly evaluates the performance of mutual conversion between T1 and T2 modalities of MRI. The present invention uses PSNR, SSIM, CIEDE as reference evaluation indicators for image conversion, and FID and KID as the main evaluation indicators.

[0111] The experimental results of each method on the Stomach dataset are shown in Table 2. First, WtNGAN achieved the best FID and KID of 111.561 and 1.724% on the WLI→NBI task, and the FID and KID of 114.172 and 3.600% on the NBI→WLI task, which shows that the distribution difference between the image generated by the method of the present invention and the real image in the feature space is the best. In addition, WtNGAN achieved the best SSIM, which also shows that the image generated by the method of the present invention maintains the best structural consistency with the original image. Then, CycleGAN performed second best, because the cycle consistency loss also constrained the structure during the conversion process between WLI and NBI. Both CUT and NEGCUT use contrastive learning to realize the information interaction between the source domain image and the target domain image. On the one hand, when the difference between the source domain image and the target domain image is large, only using contrastive loss as a constraint will result in limited performance. On the other hand, the improved negative sample selection method is not as beneficial in the gastroscopic image conversion task as in the natural image conversion task. Third, ACL-GAN performed poorly in WLI→NBI, but achieved an FID of 114.461 in NBI→WLI. One possible reason is that ACL-GAN focuses on style, and when the target domain image is more complex and the source domain image has less information, the performance of style transfer is not satisfactory. Finally, AttentionGAN only achieved an FID of 173.722 and a KID of 9.066% in the WLI→NBI task. The reason is that AttentionGAN distinguishes between foreground and background during image generation. The foreground and background in gastroscopy images are difficult to define, resulting in unsatisfactory image conversion results.

[0112] Table 2 Experimental results of various comparison methods on gastroscopy dataset

[0113]

[0114] The experimental results of each method on the BraTS2021 dataset are shown in Table 3. First, WtNGAN achieved the best results both in T1→T2 and T2→T1. The FID and KID when converting from T1 to T2 were 22.942 and 1.599%, respectively, and the FID and KID when converting from T2 to T1 were 43.068 and 2.623%, respectively. Compared with the real image, the SSIM of WtNGAN reached more than 99%, maintaining the best structural consistency. Second, CUT, MSPC, and NEGCUT achieved suboptimal results in the conversion of T1 and T2 modalities. Compared with gastroscopy images, brain MRI image conversion is simpler, and the contrastive learning-based method can fully explore the role of positive and negative examples. In addition, although MSPC did not achieve the best overall image quality, its SSIM in the two conversion tasks reached 85.24% and 79.602%, respectively, ranking second. Third, the experimental results of ACL-GAN and FFPE++ are slightly inferior to the above methods. The adversarial consistency constraint of ACL-GAN is too strong for the conversion between T1 and T2, and the regularization method used by FFPE++ to remove artifacts causes the generated image to lose a lot of details. Finally, the present invention found that CycleGAN performed poorly in the T1→T2 task, but the conversion quality of T2→T1 images was greatly improved. This is probably because when the source domain image information is richer, it is relatively simple to convert the high signal area of ​​the T2 image to the low signal area, and CycleGAN is more likely to generate high-quality T1 images.

[0115] Table 3 Experimental results of various comparison methods on the BraTS2021 dataset

[0116]

[0117] 6.3 Quantitative Analysis

[0118] The result of the unpaired conversion from WLI to NBI is as follows Figure 7 As shown, the unpaired conversion result from NBI to WLI is as follows Figure 8 As shown in the figure. AttentionGAN retains the characteristics of the source domain image to a large extent, which is mainly due to the difficulty in distinguishing the foreground and background in gastroscopic images. ACL-GAN is very sensitive to white spots in gastroscopic images, and almost all of the spots are converted incorrectly. The overall quality of the images generated by CUT, NEGCUT and FFPE++ is good and closer to the real target domain images. However, these methods still have unimportant source domain information when dealing with uneven areas. CycleGAN and WtNGAN have the best image conversion quality. CycleGAN performs best for spots, while WtNGAN is not clear enough when dealing with spots. WtNGAN generates more content in the target domain, while CycleGAN lacks detailed representation.

[0119] The conversion result from T1 to T2 is as follows Fig. 9 As shown, the conversion result from T2 to T1 is as follows Fig.10 As shown in the figure. Since the background of MRI images is significantly black, the generated images of almost all methods maintain a relatively clear brain outline, but the rich details inside the brain images are still challenging for each algorithm. The generated images of WtNGAN, CycleGAN and MSPC have the best quality, among which WtNGAN is most similar to the real target image, especially some areas with brain tumors. CycleGAN retains a lot of source domain image information that the target domain image does not have during the image conversion process. MSPC does not handle the boundaries well in a small part of the image due to the strong constraints on the shape. FFPE++, ACL-GAN, CUT and NEGCUT have fuzzy conversions in some abnormal parts, resulting in deviations from the real target domain image.

[0120] In general, the colors displayed by white light images in gastroscopy are more realistic and can initially detect suspicious lesions. Narrow-band light enhances the contrast between the mucosal surface and blood vessels through colored light sources, which helps to judge lesions. The NBI generated by WtNGAN highlights the vascular and mucosal details of WLI to a large extent. In addition, the T1 mode of brain MRI images is sensitive to fat tissue and is used to display detailed anatomical structures, and the T2 mode is sensitive to fluid and is used to display the water content of the lesion area. WtNGAN effectively identifies the characteristics of the two modes, and the generated images have good conversion to the lesion area.

[0121] The present invention also provides a gastroscopic white light to narrow-band light image conversion system based on Vision Mamba, comprising a memory, a processor, and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0122] The present invention also provides a computer-readable storage medium, on which computer program instructions that can be executed by a processor are stored. When the processor executes the computer program instructions, the method steps described above can be implemented.

[0123] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0124] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0126] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0127] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A method for converting gastroscopic white light to narrow-band light images based on Vision Mamba, characterized in that: include: A generative adversarial network based on Vision Mamba is proposed to enhance detailed representation through long-range dependencies and fine-grained local features.

2. The method for converting gastroscopic white light to narrow-band light images based on Vision Mamba according to claim 1, characterized in that: Also includes: Contrastive learning is used to maximize the information interaction between source domain images and target domain images to improve the detail representation after converting white light images to narrow-band light images.

3. The method for converting gastroscopic white light to narrow-band light images based on Vision Mamba according to claim 2, characterized in that: Also includes: A structural consistency constraint is designed to maintain the organizational structure consistency between the generated image and the original image while exchanging information.

4. The method for converting gastroscopic white light to narrow-band light images based on Vision Mamba according to claim 3, characterized in that: The generative adversarial network of Vision Mamba is constructed based on Vision Mamba. The specific implementation is as follows: The input image x first passes through the deformer T to obtain the deformed image T(x); then, x and T(x) are embedded in the image block and the image features are extracted by the encoder Enc of the generator G. The generator G is constructed according to the U-net structure. The encoder has 4 layers, each layer consists of two VSS Blocks and corresponds to the decoder Dec of the generator G; the output of each layer of the encoder is recorded as E i , where i represents the number of encoder layers, and is added to the decoder input through a jump connection, and the decoder outputs the image transformation results G(x) and G(T(x)); in order to constrain the shape of the generated image, G(x) passes through the same deformer T as the input image to obtain T(G(x)), and the consistency loss is calculated; G(x) is input into the encoder again, projected into the shared patch embedding space and recorded as M i ; Then, through E i and M i The contrast loss is calculated to realize the information interaction between the source domain image and the target domain image; finally, the discriminator D distinguishes the generated image from the real image to realize adversarial training with the generator.

5. The method for converting gastroscopic white light to narrow-band light images based on Vision Mamba according to claim 4, characterized in that: Contrastive learning is used to maximize the information interaction between source domain images and target domain images. The specific implementation is as follows: Contrastive learning CL creates positive and negative sample pairs by bringing positive pairs closer and negative pairs farther apart to learn unsupervised feature representations; specifically, the encoder Enc calculates the feature maps of each layer of the input image x Then, E i Divided into multiple blocks Where N is the number of selected spatial positions, and the corresponding spatial position block is recorded as n∈{1,2,…,N}, and the tiles at other locations are represented as in Similarly, the feature map obtained after the output image y=G(x) is input into the reused encoder is recorded as Among them The corresponding block is and The corresponding blocks are Set the encoder E i The image blocks of each layer output feature and the image blocks of the corresponding positions of the multi-layer perceptron output feature of the same layer are used as positive pairs, and the encoder E i Each layer outputs all patches at different positions as negative pairs. By contrastive learning, the distance between positive pairs is shortened to promote the conversion of input images to target images, and the negative pairs are kept away from each other to avoid the input images being close to their own image styles. The contrastive learning loss is defined as: in, represents the expectation of the input image x, Represents the input image x and the tiles obtained from x expectations, obtained from the source domain image and calculated by the reused encoder; Expressed as the loss for the classification problem, it is calculated as follows: Among them, a represents the query value, b represents the positive sample, c represents the negative sample, τ represents the scaling temperature, and c n represents the nth negative sample.

6. The method for converting gastroscopic white light to narrow-band light images based on VisionMamba according to claim 5, characterized in that: Through structural consistency constraints, the organizational structure of the generated image and the original image is kept consistent while information is exchanged. The specific implementation is as follows: First generate the two-dimensional standardized coordinate grid Grid of image x X,Y (x), where X and Y represent the horizontal and vertical axes respectively; then, a sine function transformation is applied to the horizontal component of the grid and added to the vertical component to produce a distorted grid effect; finally, the original image is sampled according to the distorted grid to generate a distorted image; the formula for generating the distorted image is calculated as follows: T(x)=Sin(Grid X (x))×ρ+Grid Y (x) Where Sin(·) represents the sine function, and ρ is a hyperparameter that controls the degree of image deformation. In order to maintain the structural consistency between the generated image and the original image, the L1 loss is used to penalize the inconsistency of the generated model structure prediction. The structural consistency loss is expressed as: Among them, H represents the image height, W represents the image width, and x h,w represents the original input image, T h,w (x) represents the image after deformation.

7. The method for converting gastroscopic white light to narrow-band light images based on VisionMamba according to claim 1 or 4, characterized in that: The generative adversarial network based on Vision Mamba consists of a patch embedding layer, a four-layer encoder-decoder, a final projection layer, and a discriminator. The patch embedding layer first divides the input image x into 4×4 tiles and maps them to the C dimension; then, the image is sent to a 4-layer encoder-decoder to extract features, and each layer of the encoder-decoder contains two VSS Blocks; the VSS Block performs layer regularization on the input image, and then enters two branches. One branch passes through a linear layer and extracts features by deep convolution, and then is activated by the SiLU function and performs a two-dimensional selective scan. After the image is divided into tiles, the image sequence is scanned along four directions, and then the sequence is extracted through a selection operation and passed through a layer regularization layer; the other branch is the original input image, and then the features of the two branches are multiplied, and then passed through a linear layer and residually connected to the original input to obtain the output of the VSSBlock; after the four-layer encoder-decoder extracts the image features, a 1×1 convolution layer and a final projection layer project the features to 3 channels to output the generated image; the discriminator consists of 4 layers of 4×4 convolutional layers, and each convolutional layer is activated by the LeakyReLU function and the features are normalized by batch normalization. ; The 3-channel image output by the final projection layer is distinguished as true or false by the discriminator.

8. The method for converting gastroscopic white light to narrow-band light images based on Vision Mamba according to claim 6, characterized in that: The final loss of the whole method is composed of contrastive learning loss Structural consistency loss And adversarial loss The composition is as follows: The adversarial loss is used to encourage the Nash equilibrium between the generator and the discriminator, hoping that the output image is visually as close to the target domain image as possible, and is defined as follows: in, represents the expectation of the real image y, Represents the expectation of the input original image x; the final loss is expressed as: Among them, α, β, and γ are hyperparameters of loss weighting.

9. A gastroscope white light to narrow-band light image conversion system based on Vision Mamba, characterized in that: The method comprises a memory, a processor and computer program instructions stored in the memory and capable of being executed by the processor. When the processor executes the computer program instructions, the method steps as claimed in any one of claims 1 to 8 can be implemented.

10. A computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, and when the processor executes the computer program instructions, the method steps according to any one of claims 1 to 8 can be implemented.

Images