Analog indigo dyeing method based on cycle adversarial generative network

Abstract

This invention discloses a simulated indigo carmine staining method based on a recurrent adversarial generative network, comprising the following steps: inputting an endoscopic image (white light image) into a trained simulated indigo carmine staining model to generate a simulated indigo carmine staining image; the simulated indigo carmine staining model is a TransUNet backbone network based on Transformer and CNN, wherein the CNN includes a generator G. A Generator G B and discriminator D x The training method for the simulated indigo carmine staining model includes: establishing a network training set; and then generating a simulated staining image P. 3 ; simulated staining image P 3 Input discriminator D x In the middle, by discriminator D x Determine the simulated staining image P 3 Is it an endoscopic image stained image P? 2 The corresponding total loss is obtained and input into the generator G. A and generator G B The present invention provides a simulated indigo carmine staining method based on a recurrent adversarial generative network, which can simulate indigo carmine staining on endoscopic images, reducing medical costs and avoiding multiple endoscopic examinations and drug use.

Description

Technical Field

[0001] This invention relates to a simulated indigo staining method based on a recurrent adversarial generative network, belonging to the field of image processing technology. Background Technology

[0002] Stomach cancer is one of the most common malignant tumors in my country, with the highest incidence rate among digestive system tumors. Early detection and treatment can significantly reduce mortality. If stomach cancer is detected early via gastroscopy and confined to a small area of ​​the mucosa, the diseased tissue can be removed endoscopically without surgery or chemotherapy. This results in less pain, better recovery, and a significantly higher survival rate. Improving endoscopic detection accuracy is crucial for helping doctors detect lesions promptly.

[0003] Gastrointestinal endoscopy is the primary method for screening for early gastrointestinal cancers. A common contrast-enhanced method for gastrointestinal endoscopy is indigo carmine staining. Normal gastrointestinal mucosa is smooth, but changes occur when inflammation, ulcers, or cancer are present. If the lesion is small or in its early stages, it is difficult to detect under ordinary endoscopy. Indigo carmine staining is then applied to the mucosa. The smooth areas do not deposit the carmine; instead, it accumulates in the grooves and folds of the gastric mucosa, revealing subtle changes in texture and three-dimensional structure under endoscopy. However, indigo carmine staining not only increases medical costs and requires multiple endoscopy sessions for patients, but also places excessive demands on the physician's expertise. In less developed regions, the required expertise of doctors may not meet the clinical needs. Furthermore, repeated endoscopic examinations and medication use can cause serious secondary harm to patients. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the defects of the prior art and provide a simulated indigo carmine staining method based on a recurrent adversarial generative network, which can simulate indigo carmine staining on endoscopic images, reduce medical costs, and avoid multiple endoscopic examinations and drug use.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0006] A simulated indigo carmine staining method based on a recurrent adversarial generative network includes the following steps:

[0007] The white light image of the endoscope is input into the trained simulated indigo carmine staining model to generate a simulated indigo carmine staining image;

[0008] The simulated indigo carmine staining model is based on the TransUNet backbone network, which incorporates a generator within the CNN. Generator and discriminator ,

[0009] The training method for the simulated indigo carmine staining model includes:

[0010] Collect endoscopic image datasets and segment unstained endoscopic images into white light images. Stained images of endoscopes stained with indigo carmine White light image of endoscopy and stained images of endoscopy The network training set is obtained by cropping the dataset.

[0011] During downsampling, white light images of endoscopes in the network training set are used. Feed into generator Encode the endoscopic image into a white light image. The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated staining image. ;

[0012] Simulated staining image Input discriminator In the middle, by the discriminator Determine the simulated staining image Is it an endoscopic image stained image? The corresponding total loss is obtained and input into the generator. and generator Training is conducted during this period;

[0013] Simulated staining image Feed into generator Encode the simulated stained image The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated white light image. ;

[0014] Simulated white light image Input discriminator In the middle, by the discriminator Determine the simulated white light image Is it an endoscopic image or a white light image? The corresponding total loss is obtained and input into the generator. and generator The generator is trained in this way to obtain a well-trained generator with network parameter weights. and generator .

[0015] White light image of endoscopy and endoscopic image staining The cropping process specifically employs a center-cropping method, which involves cropping outwards from the center point of the image using a selected size, resulting in a cropped size of [value missing]. .

[0016] The process of transforming low-level image features into feature vectors involves: using a CNN to extract features from the image, obtaining a feature map of the input image; then fusing the obtained feature map with a feature map of the same size in the decoder to obtain a fused feature map; finally, transforming the fused feature map through linear projection to obtain a two-dimensional labeled vector of size n_patch, D, i.e., the feature vector, where n_patch = 8 and D is selected according to the input image; and then using a Transformer to extract features from the feature vector. After obtaining the feature vector, the feature vector is fed into 12 Transformer layers, and the output is reshaped. , and This indicates that the height and width of the feature map are downsampled by 16 times the input image.

[0017] The decoder upsampling process uses the ReLU activation function with a size of The convolution kernel is upsampled, and the formula is expressed as follows:

[0018]

[0019] in , , , Features for image downsampling , , , Features for image upsampling, For downsampling convolution functions, For Transformer feature extraction function, This is an upsampling convolution function.

[0020] Total loss To combat losses Circular Consistency Loss The sum of .

[0021] Combat losses The calculation is as follows:

[0022]

[0023] The adversarial loss is the loss function of the discriminator, for y represents the distribution of real data The samples obtained from sampling The larger, the more it means The larger the value, the more accurately the discriminator can identify real samples as real samples; for x is from a specific distribution The samples obtained in the process, Fake samples generated by the generator The larger, the better The smaller the value, the better the discriminator can correctly distinguish fake samples. This refers to the discriminator D obtaining information about the differences between real and fake samples under the influence of sample y. This refers to the fake samples obtained using discriminator D. Information on the differences between the sample and the real sample;

[0024] Cyclic consistent loss The formula for calculating the sum is as follows:

[0025] .

[0026] A simulated indigo carmine staining device based on a recurrent adversarial generative network includes:

[0027] The image acquisition module is used to acquire white light images of endoscopic images;

[0028] A simulated indigo carmine staining model is provided, wherein the simulated indigo carmine staining model is based on a TransUNet backbone network using Transformer and CNN, wherein the CNN includes a generator. Generator and discriminator ,

[0029] The training method for the simulated indigo carmine staining model includes:

[0030] Collect endoscopic image datasets and segment unstained endoscopic images into white light images. Stained images of endoscopes stained with indigo carmine White light image of endoscopy and stained images of endoscopy The network training set is obtained by cropping the dataset.

[0031] During downsampling, white light images of endoscopes in the network training set are used. Feed into generator Encode the endoscopic image into a white light image. The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated staining image. ;

[0032] Simulated staining image Input discriminator In the middle, by the discriminator Determine the simulated staining image Is it an endoscopic image stained image? The corresponding total loss is obtained and input into the generator. and generator Training is conducted during this period;

[0033] Simulated staining image Feed into generator Encode the simulated stained image The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated white light image. ;

[0034] Simulated white light image Input discriminator In the middle, by the discriminator Determine the simulated white light image Is it an endoscopic image or a white light image? The corresponding total loss is obtained and input into the generator. and generator The generator is trained in this way to obtain a well-trained generator with network parameter weights. and generator .

[0035] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the simulated indigo staining method based on a recurrent adversarial generative network.

[0036] The beneficial effects of this invention are as follows: This invention provides a simulated indigo carmine staining method based on a recurrent adversarial generative network (ROV). The method first acquires an unstained endoscopic image (white light image) and an endoscopic image stained with indigo carmine (stained image). Then, through image processing of the white light image and the stained image, and after setting the parameters of the RRV, the network is trained to obtain a trained simulated indigo carmine staining model. This model can then be used to simulate indigo carmine staining and generate stained images, saving medical costs, reducing patient treatment costs, and lowering the technical threshold for doctors. It has broad market prospects and application value. Attached Figure Description

[0037] Figure 1 This is a flowchart of a simulated indigo carmine staining method based on a recurrent adversarial generative network according to the present invention;

[0038] Figure 2 This is a detailed structural diagram of the generator module in the simulated indigo carmine staining model of the present invention;

[0039] Figure 3 This is an example of the results of using a simulated indigo staining method based on a recurrent adversarial generative network according to the present invention. Detailed Implementation

[0040] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to illustrate the technical solution of the present invention more clearly, and should not be used to limit the scope of protection of the present invention.

[0041] This invention discloses a simulated indigo carmine staining method based on a recurrent adversarial generative network.

[0042] First, a simulated indigo carmine staining model based on a recurrent adversarial generative network (ROGGN) is trained using a training set. Then, this model is applied to obtain a simulated staining image of a white light image. The simulated indigo carmine staining model uses a TransUNet backbone network based on Transformer and CNN, where the CNN includes a generator. Generator and discriminator ,

[0043] like Figure 1 As shown, the training method for simulating the indigo carmine staining model in this invention includes the following steps:

[0044] Step 1: Collect endoscopic image dataset and segment unstained endoscopic images into white light images. Stained images of endoscopes stained with indigo carmine White light image of endoscopy and stained images of endoscopy The images are cropped to obtain the network training set. Because the initial endoscopic images have excessively high resolution and black borders, this invention first crops the images to obtain the final training set for the network. The normal, unprocessed endoscopic image size is often... Furthermore, the presence of black borders leads to insufficient network training and interferes with image generation quality. This invention uses a center-cropping method instead of simply resizing the image. Specifically, it crops the image outwards from the midpoint using a selected size to ensure the central information of the image is obtained. The cropping size used in this invention is... This not only effectively cropped the black borders of the image, but also effectively reduced the size of the training image while preserving the image's details by setting the cropping size.

[0045] Step two: CNN-based methods typically exhibit limitations in modeling explicit long-range relationships. These architectures often produce weak performance, especially for target structures that show significant inter-patient differences in texture, shape, and size. Therefore, this invention uses TransUNet, which employs a hybrid CNN-Transformer architecture to leverage detailed, high-resolution spatial information from CNN features and global context encoded by Transformers. While Transformers prioritize global information, they tend to ignore low-resolution image details, which significantly impairs the decoder's ability to recover pixel sizes, resulting in coarse segmentation results. CNNs, on the other hand, can effectively overcome this shortcoming of Transformers.

[0046] Based on CNN and Transformer networks, this invention uses the TransUNet backbone network to downsample the endoscopic image to a white light image. Generator fed into CNN convolutional network Encode the endoscopic image into a white light image. The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated staining image. .

[0047] Specifically, transforming low-level image features into feature vectors involves: using a CNN to extract features from the image, obtaining a feature map of the input image, and then fusing the obtained feature map with a feature map of the same size in the decoder. Figure 2As shown by the dashed line, the fused feature map is obtained. Then, the fused feature map is transformed by linear projection to obtain a two-dimensional label vector of size n_patch, D, which is the feature vector, where n_patch=8 and D is selected according to the input image.

[0048] The Transformer performs feature extraction on the feature vectors as follows: After obtaining the feature vectors, they are fed into 12 Transformer layers to encode less short-range information and more long-range information from the image. Finally, to prepare the upsampling path, the output is reshaped... , and This indicates that the height and width of the feature map are downsampled by 16 times the input image.

[0049] In the decoder upsampling process, this invention uses the ReLU activation function and a value of [missing information]. The convolution kernel is upsampled, and the formula is expressed as follows:

[0050]

[0051] in , , , Features for image downsampling , , , Features for image upsampling, For downsampling convolution functions, For Transformer feature extraction function, This is an upsampling convolution function.

[0052] Step 3, simulate the staining image Input discriminator In the middle, by the discriminator Determine the simulated staining image Is it an endoscopic image stained image? The corresponding total loss is obtained and input into the generator. and generator Training will be conducted in the following manner.

[0053] In this invention, the total loss To combat losses Circular Consistency Loss The sum of .

[0054] The Cyclegan model uses a binary global discriminator to classify the image. However, the simulated indigo staining based on a recurrent adversarial generative network (ROGGN) in this invention is a pixel-level prediction task. The intensity of staining required in different regions of an unstained white light endoscope image varies. Therefore, image-level classification cannot meet the requirements of the simulated indigo staining model based on a RPGGN and will lead to an imbalance in the quality of the simulated staining. Therefore, this invention uses a pixel-level discriminator from Pactgan to adapt to the need for enhanced staining in different regions. The adversarial loss employed in this invention... as follows:

[0055]

[0056] The adversarial loss is the loss function of the discriminator, for y represents the distribution of real data The sample obtained from sampling. The larger, the more it means The larger the value, the more accurately the discriminator can identify real samples as real samples; for x is from a specific distribution The samples obtained in the process, Fake samples generated by the generator The larger, the better The smaller the value, the better the discriminator can correctly distinguish fake samples.

[0057] Step four, simulate the staining image Feed into generator Encode the simulated stained image The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated white light image. The process of converting low-level image features into feature vectors and the specific process of extracting features from the feature vectors using Transformer are the same as in step two.

[0058] Because this invention uses unmatched paired datasets, endoscopic images and white light images... In the generator Endoscopic image stained image obtained from the generation mapping The image might be randomly mapped to any region, resulting in a generated image that has no relation to the original input image. Therefore, this invention uses cycle-consistent loss, which is the sum of the losses of the two generators, to prevent obtaining an image completely unrelated to the original image during transformation. The obtained image is then processed through To restore the image to its original state, we need to perform two operations. The formula is as follows:

[0059] .

[0060] Step 5: Simulate white light image Input discriminator In the middle, by the discriminator Determine the simulated white light image Is it an endoscopic image or a white light image? The corresponding total loss is obtained and input into the generator. and generator The generator is trained in this way to obtain a well-trained generator with network parameter weights. and generator .

[0061] Figure 3 This is an example of the results of using a simulated indigo staining method based on a recurrent adversarial generative network according to the present invention.

[0062] This invention also discloses a simulated indigo carmine dyeing device based on a recurrent adversarial generative network, comprising:

[0063] The image acquisition module is used to acquire white light images of endoscopic images;

[0064] A simulated indigo carmine staining model is provided, wherein the simulated indigo carmine staining model is based on a TransUNet backbone network using Transformer and CNN, wherein the CNN includes a generator. Generator and discriminator ,

[0065] The training method for the simulated indigo carmine staining model includes:

[0066] Collect endoscopic image datasets and segment unstained endoscopic images into white light images. Stained images of endoscopes stained with indigo carmine White light image of endoscopy and stained images of endoscopy The network training set is obtained by cropping the dataset.

[0067] During downsampling, white light images of endoscopes in the network training set are used. Feed into generator Encode the endoscopic image into a white light image. The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated staining image. ;

[0068] Simulated staining image Input discriminator In the middle, by the discriminator Determine the simulated staining image Is it an endoscopic image stained image? The corresponding total loss is obtained and input into the generator. and generator Training is conducted during this period;

[0069] Simulated staining image Feed into generator Encode the simulated stained image The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated white light image. ;

[0070] Simulated white light image Input discriminator In the middle, by the discriminator Determine the simulated white light image Is it an endoscopic image or a white light image? The corresponding total loss is obtained and input into the generator. and generator The generator is trained in this way to obtain a well-trained generator with network parameter weights. and generator .

[0071] Finally, this invention discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the simulated indigo staining method based on a recurrent adversarial generative network.

[0072] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A simulated indigo carmine staining method based on a recurrent adversarial generative network, characterized in that: Includes the following steps: The white light image of the endoscope is input into the trained simulated indigo carmine staining model to generate a simulated indigo carmine staining image; The simulated indigo carmine staining model is based on the TransUNet backbone network, which incorporates a generator within the CNN. Generator and discriminator , The training method for the simulated indigo carmine staining model includes: Collect endoscopic image datasets and segment unstained endoscopic images into white light images. Stained images of endoscopes stained with indigo carmine White light image of endoscopy and stained images of endoscopy The network training set is obtained by cropping the dataset. During downsampling, white light images of endoscopes in the network training set are used. Feed into generator Encode the endoscopic image into a white light image. The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated staining image. The process of transforming low-level image features into feature vectors involves: using a CNN to extract features from the image, obtaining a feature map of the input image; then fusing the obtained feature map with a feature map of the same size in the decoder to obtain a fused feature map; and finally, transforming the fused feature map through linear projection to obtain a two-dimensional labeled vector of size n_patch, D, i.e., the feature vector, where n_patch = 8 and D is selected according to the input image; the Transformer extracts features from the feature vector; after obtaining the feature vector, the feature vector is fed into 12 Transformer layers, and the output is reshaped. , and This indicates that the height and width of the feature map are downsampled by 16 times the input image. Simulated staining image Input discriminator In the middle, by the discriminator Determine the simulated staining image Is it an endoscopic image stained image? The corresponding total loss is obtained and input into the generator. and generator Training is conducted during this period; Simulated staining image Feed into generator Encode the simulated stained image The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated white light image. The decoder upsampling process uses the ReLU activation function and a size of The convolution kernel is upsampled, and the formula is expressed as follows: ； in , , , Features for image downsampling , , , Features for image upsampling, For downsampling convolution functions, For Transformer feature extraction function, It is an upsampling convolution function; Simulated white light image Input discriminator In the middle, by the discriminator Determine the simulated white light image Is it an endoscopic image or a white light image? The corresponding total loss is obtained and input into the generator. and generator The generator is trained in this way to obtain a well-trained generator with network parameter weights. and generator .

2. The simulated indigo carmine staining method based on a recurrent adversarial generative network according to claim 1, characterized in that: White light image of endoscopy and endoscopic image staining The cropping process specifically employs a center-cropping method, which involves cropping outwards from the center point of the image using a selected size, resulting in a cropped size of [value missing]. .

3. The simulated indigo carmine staining method based on a recurrent adversarial generative network according to claim 1, characterized in that: Total loss To combat losses Circular Consistency Loss The sum of .

4. The simulated indigo carmine staining method based on a recurrent adversarial generative network according to claim 3, characterized in that: Combat losses The calculation is as follows: ； The adversarial loss is the loss function of the discriminator, for y represents the distribution of real data The samples obtained from sampling The larger, the more it means The larger the value, the more accurately the discriminator can identify real samples as real samples; for x is from a specific distribution The samples obtained in the process, Fake samples generated by the generator The larger, the better The smaller the value, the better the discriminator can correctly distinguish fake samples. This refers to the discriminator D obtaining information about the differences between real and fake samples under the influence of sample y. This refers to the fake samples obtained using discriminator D. Information on the differences between the sample and the real sample; Cyclic consistent loss The formula for calculating the sum is as follows: 。 5. A simulated indigo carmine staining apparatus based on a cyclic adversarial generative network, performing the simulated indigo carmine staining method based on a cyclic adversarial generative network as described in any one of claims 1 to 4, characterized in that, include: The image acquisition module is used to acquire white light images of endoscopic images; A simulated indigo carmine staining model is provided, wherein the simulated indigo carmine staining model is based on a TransUNet backbone network using Transformer and CNN, wherein the CNN includes a generator. Generator and discriminator , The training method for the simulated indigo carmine staining model includes: Collect endoscopic image datasets and segment unstained endoscopic images into white light images. Stained images of endoscopes stained with indigo carmine White light image of endoscopy and stained images of endoscopy The network training set is obtained by cropping the dataset. During downsampling, white light images of endoscopes in the network training set are used. Feed into generator Encode the endoscopic image into a white light image. The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated staining image. ; Simulated staining image Input discriminator In the middle, by the discriminator Determine the simulated staining image Is it an endoscopic image stained image? The corresponding total loss is obtained and input into the generator. and generator Training is conducted during this period; Simulated staining image Feed into generator Encode the simulated stained image The underlying features are transformed into feature vectors, then the Transformer is used to extract features from the feature vectors to obtain the feature results, and finally the generator is used. The decoder upsamples and decodes the feature results to generate a simulated white light image. ; Simulated white light image Input discriminator In the middle, by the discriminator Determine the simulated white light image Is it an endoscopic image or a white light image? The corresponding total loss is obtained and input into the generator. and generator The generator is trained in this way to obtain a well-trained generator with network parameter weights. and generator .

6. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the simulated indigo staining method based on a recurrent adversarial generative network as described in any one of claims 1-4.

Images