Endoscopic image generation method
By generating high-quality endoscopic basic images and fusing lesion images for the adversarial neural network, the problem of difficulty in obtaining capsule endoscopic data and high labeling costs is solved, and the accuracy and diagnostic efficiency of the detection neural network are improved.
Patent Information
- Application Number
- CN202310553139.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-16
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the prior art, capsule endoscopy lacks labeled image data when used for small intestinal bleeding or vascular lesions image recognition, resulting in inaccurate training of machine learning models, increasing the difficulty of diagnosis and misdiagnosis rate, and high labeling cost.
Generate high-quality endoscopic basic images by generating adversarial neural networks (GANs), and extracting lesion images from segmented datasets for fusing, generating medical images containing lesions, increasing the number and quality of the datasets, thereby improving the detection neural network Accuracy.
The quality and quantity of capsule endoscopic data acquisition are improved, the average accuracy of the detection neural network is improved by about 7.6%, reducing diagnostic errors and doctors' workload, and improving diagnostic efficiency.
Smart Images

Figure CN119006353A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an image generating method, and in particular to an endoscope image generating method. Background Art
[0002] Endoscopes, such as capsule endoscopes, have become an important method for diagnosing small intestinal or other diseases. The images taken by capsule endoscopes in the human digestive tract are affected by gastrointestinal bubbles, endoscope lighting, and shooting angles, resulting in uneven image quality, which increases the difficulty of diagnosis. At the same time, each capsule endoscope examination in the later diagnosis will generate hundreds of thousands of images, which will increase the workload of medical staff, reduce the work efficiency of doctors, and the misdiagnosis rate will increase over time. Therefore, machine learning methods such as deep learning technology can be used to solve the problem of image recognition of small intestinal bleeding or vascular lesions in capsule endoscopes. However, due to ethical and privacy considerations and the high cost compared to traditional gastroscopy, the labeled image data of capsule endoscopes that can be used for target detection is currently lacking. Training machine learning / neural networks in the absence of a data set will make the test results inaccurate and may even mislead doctors' diagnoses. Summary of the invention
[0003] In one aspect, the present invention provides a method for generating an endoscopic image. The method comprises the following steps: obtaining a machine learning model, the machine learning model comprising model parameters; generating a plurality of unlabeled endoscopic basic images through the machine learning model; extracting a plurality of lesion images from a segmented data set; fusing each of the unlabeled endoscopic basic images with one of the plurality of lesion images to generate a plurality of endoscopic medical images containing lesions.
[0004] Preferably, each of the endoscopic medical images is an endoscopic medical image including corresponding annotations. Preferably, each of the lesion images is a lesion image including corresponding annotations. Preferably, extracting a plurality of lesion images from the segmented data set comprises: obtaining the plurality of lesion images by applying a mask to a plurality of images in the segmented data set.
[0005] In one embodiment, obtaining the machine learning model includes: performing data preprocessing on sample data to obtain corresponding training data, wherein the sample data includes multiple unlabeled sample images; inputting the training data into a generative adversarial neural network for adversarial learning to obtain the machine learning model and the model parameters.
[0006] Preferably, the inputting the training data into a generative adversarial neural network for adversarial learning comprises: inputting a training image in the training data into a generator of the generative adversarial neural network to obtain a generated image; obtaining an actual training error between the generated image and a sample image in the sample data by using a discriminator of the generative adversarial neural network; and when the actual training error meets a preset training error, obtaining the machine learning model and the model parameters.
[0007] Preferably, the step of inputting the training data into a generated adversarial neural network for adversarial learning also includes: when the actual training error does not meet the preset training error, updating the model parameters through the error function of the machine learning model; and re-performing adversarial learning through the activation function of the machine learning model and the updated model parameters.
[0008] Preferably, the generative adversarial neural network is a progressive growth GAN network.
[0009] Preferably, the data preprocessing of the sample data includes: standardizing the size of the sample image, for example, scaling it to a standard 448×448 pixels, standardizing the pixels of the sample image, standardizing the direction of the sample image, for example, by image rotation, and flipping, locally scaling, and adjusting the brightness of the sample image.
[0010] Preferably, the method further includes: generating a medical image using the machine learning model and the model parameters; and comparing the medical image with a real capsule endoscopy image to obtain a comparison error. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1 is a flow chart of a method for generating an endoscopic image according to an embodiment of the present invention;
[0012] Figure 2 is based on Figure 1 Examples of endoscopic base images generated by the method;
[0013] Figure 3 is based on Figure 1 Examples of endoscopic base images generated by the method;
[0014] Figure 4 yes Figure 1 Examples of masks in methods;
[0015] Figure 5 is based on Figure 1 Examples of lesion images extracted by the method;
[0016] Figure 6is a flowchart of obtaining a machine learning model in a method for generating an endoscopic image according to an embodiment of the present invention; and
[0017] Figure 7 The present invention is a flowchart of adversarial learning in a method for generating an endoscopic image according to an embodiment of the present invention. DETAILED DESCRIPTION
[0018] In some application examples, detection machine learning models, such as deep learning-based detection neural networks, are used to solve the problem of identifying images of small intestinal bleeding or vascular lesions in capsule endoscopy. However, the formation / training of machine learning methods for lesion detection requires a large amount of high-quality data sets as support to enable it to have high accuracy and good performance.
[0019] The present invention provides a method for generating endoscopic images based on machine learning. The method for generating endoscopic images only needs to use existing image data for training and generating machine learning models. In this article, the machine learning model may include multiple models, such as deep learning networks, generative adversarial neural networks, etc. A high-quality new endoscopic basic image can be generated using the trained generative adversarial neural network. Lesion images are extracted from instance images with lesions (including normal, inflamed, polyps, etc.) on the segmented data set. And the generated endoscopic basic image is fused with the lesion image, for example, by a pasting method, to obtain or generate a new endoscopic medical image that is available and of high quality. The new endoscopic medical image can be added to the data set to make it more sufficient and of higher quality. The use of the data set can improve the accuracy and performance of the detection neural network for training. The new endoscopic medical image is added to the data set, and the detection neural network is trained and tested again, and it is found that the experimental results are improved, and the average precision (Average Precision) is improved by about 7.6%. Therefore, the method for generating endoscopic images effectively solves the problems of difficulty in acquiring capsule endoscope data and high labeling costs, and can also help improve the doctor's diagnostic accuracy, thereby solving the problems of long time consumption, low efficiency and large workload caused by human eye observation and recognition in the later diagnosis process of capsule endoscopy.
[0020] Figure 1A method 100 for generating an endoscopic image according to an embodiment of the present invention is shown. Method 100 includes: in step 110, obtaining a machine learning model, wherein the machine learning model includes model parameters. In one embodiment, the machine learning model or deep learning network may be a generative adversarial neural network, such as a progressive growing GAN network, and the model parameters may be corresponding parameters of the progressive growing GAN network. The progressive growing GAN network may gradually increase the pixel value of the generated image as the number of training epochs increases. In step 120, a plurality of unlabeled endoscopic base images are generated through the machine learning model.
[0021] Figure 2 and Figure 3 The example 200 / 300 of the endoscope basic image according to the present invention. The endoscope basic image may contain more background information and contain as few lesion instances as possible. Based on the generated endoscope basic image, the endoscope basic image excludes the remaining interference information as much as possible. In order to be fused with the extracted lesion instances later, including too much information will cause noise in the final synthesized image.
[0022] The method 100 further includes: in step 130, extracting a plurality of lesion images from the segmented data set. The segmented data set may include diverse, accurate and high-quality lesion information. The lesion image may include various types of clear lesion information. Each lesion image may include a correspondingly annotated lesion image, such as annotated as hemorrhage, vascular lesions, ulcers, erosions, tumors, etc. In one embodiment, a mask may be used to obtain or extract the lesion image. The use of a mask may more clearly show the lesion image and delete non-relevant areas, i.e., image areas without lesions.
[0023] Figure 4 FIG. 4 shows an example of a mask 400 for marking a lesion region. Specifically, the pixel value of the region 420 containing the lesion is set to 255, which is represented as white, and the pixel value of the remaining region 410 is set to 0, which is represented as black. Figure 5 As shown, by applying mask 400 to the images in the segmentation dataset, a lesion image 500 can be obtained. Figure 4 The mask 400 shown is applied to the image in the segmentation data set for superposition. Thus, the region 520 containing the lesion in the image can be retained and the region 510 can be represented as black, thereby obtaining Figure 5The lesion image 500 shown. Subsequently, in step 140, each of the unlabeled endoscopic basic images is fused with one of the multiple lesion images to generate multiple endoscopic medical images containing lesions. In one embodiment, the fusion may include a copy and paste operation. After generating multiple endoscopic medical images containing lesions, the newly generated multiple endoscopic medical images may be saved. The newly generated multiple endoscopic medical images are labeled according to the standard of the detection neural network. Therefore, each endoscopic medical image may be an endoscopic medical image including corresponding annotations. Afterwards, the endoscopic medical images including corresponding annotations are added to the data set of endoscopic medical images, and the detection neural network is retrained to improve the accuracy and performance of the detection neural network.
[0024] Figure 6 Show Figure 1 An embodiment of step 110 of training and obtaining a machine learning model of the method shown. In step 210, data preprocessing is performed on the sample data to obtain corresponding training data. The sample data may include multiple unlabeled sample images. Similarly, the training data may include multiple unlabeled training images. Data preprocessing of the sample data may standardize the sample data. In some embodiments, data preprocessing may include the following operations: standardizing the size of the sample image (for example, scaling to 448×448 pixels), standardizing the pixels of the sample image, standardizing the direction of the sample image (for example, by image steering), and folding, locally scaling, and adjusting the brightness of the sample image. In step 220, the training data is input into a generative adversarial neural network for adversarial learning to obtain the machine learning model and the model parameters. Unlike traditional neural networks, the generative adversarial network does not need to divide the data set, so unlabeled training images can be used.
[0025] According to an example, the sample data may be a Kvasir open source dataset, which includes multiple categories of datasets for gastrointestinal disease detection and classification, and the capsule endoscopy open source dataset of the Kvasir open source dataset has been verified by experts in the field of small intestinal lesions.
[0026] In such Figure 7 In some embodiments shown, the generative adversarial neural network may include a generator and a discriminator. Figure 7 As shown, Figure 6The adversarial learning step 220 may include: in step 222, inputting the training image in the training data into the generator of the generative adversarial neural network to obtain a generated image. The generated image may be the output of the generative adversarial neural network through adversarial learning. A characteristic of the generative adversarial neural network is that the model training starts with an image of 64×64 pixels, and each time a pixel value is trained for a certain number of iterations, the training pixel is expanded to 2 times. Finally, the training is stopped and the pixel size is set to 512×512.
[0027] In step 224, by using the discriminator of the generative adversarial neural network, the actual training error between the generated image and the sample image in the sample data is obtained. In step 226, when the actual training error meets the preset training error, the machine learning model and the model parameters are obtained. Thus, the trained machine learning model and the corresponding model parameters can be obtained, which are the optimal model parameters. In the example of using the Kvasir open source data set, the generator of the generative adversarial neural network and the discriminator of the generative adversarial neural network can be trained. The training can be carried out until the image generated by the generator can be almost stably identified as a real image by the discriminator, and the training is stopped.
[0028] On the contrary, in step 228, when the actual training error does not meet the preset training error, the model parameters are updated by the error function of the machine learning model. In step 229, the adversarial learning step 220 is performed again by the activation function of the machine learning model and the updated model parameters.
[0029] After the training is completed to obtain the machine learning model and the corresponding model parameters, the machine learning model and the model parameters can be used to generate a medical image, and the medical image can be compared with the real capsule endoscopy image to obtain the error. Specifically, the number of images to be generated and the truncation rate (truncation rate is 0 to 1) when synthesizing the image and other parameters can be input, and the medical image can be generated by the generator of the generative adversarial neural network. In one embodiment, randomly sorted generated medical images and real capsule endoscopy images can be selected, and experts in the medical field can be invited to make judgments to obtain comparison errors. If the experts are also unable to judge the difference between the real data and the generated medical image, it can be determined that the error of the trained machine learning model is small. When the comparison error of the actual test is less than the set test error, it can be determined that the test of the machine learning model and model parameters is completed.
[0030] The following is an exemplary application of the method 100 for generating an endoscopic image, comprising the following steps:
[0031] Input image: The image comes from the Kvasir dataset, which is an open source multi-class dataset for gastrointestinal disease detection and classification approved by professional doctors.
[0032] Image preprocessing: Perform uniform processing on images to increase the expressive power of deep learning networks when learning image data and enhance the generalization performance of the network.
[0033] In order to increase the number and diversity of data sets and make it easier to achieve generalization effects after training, preprocessing may include:
[0034] 1) Standardization of image size: Scale the image to 448×448 pixels. Use the resizing function to resize the pixel size of all image data input to the deep learning network to 448×448. This can unify the size of the input image data to ensure the learning performance of the network, while also ensuring the training speed and reducing memory requirements;
[0035] 2) Standardization of image pixel values: After the image is represented by a tensor, the grayscale value range of each pixel is between 0 and 255. The grayscale value standardization of pixels is to divide the grayscale values of all pixels in the image by 255. The above processing can normalize the value range of image pixels to between 0 and 1. This is a processing based on data probability distribution and convex optimization theory. The processed centralized data is more in line with the data distribution law. At the same time, after the training is completed, one or more of the following image processing is performed to obtain the generalization effect after training;
[0036] 3) Standardization of image direction: Perform necessary rotation processing on the image, that is, rotate the image by a certain angle with the center of the image as the axis;
[0037] 4) Flip the image, that is, flip the image upside down or left to right;
[0038] 5) Perform local image scaling, i.e. scaling the image of the lesion site according to a certain ratio;
[0039] 6) Adjust the image brightness, that is, increase or decrease the brightness of the image.
[0040] Furthermore, a deep learning network can be built, such as a generative adversarial neural network. The deep learning network is built using the Pytorch framework. The deep learning network structure mainly includes a generator (G), a discriminator (D) and a loss function (Loss). The model parameters are optimized by training the model, that is, the data set is sent to the built deep learning network, and WGAN-GP is selected as the final optimization target. The loss function (Loss) is:
[0041] Loxs=D(real)-D(G(x))+λ×grad pen
[0042] The remaining built-in functions are expressed as follows:
[0043] grad p en=MSE(Norm-k)
[0044]
[0045] Where k represents the Lipschitz constant, which represents the minimum lower bound of the function gradient. Where MSE is the square error formula, X inter To sample X from the entire joint distribution, the gradient penalty term grad p en is the square difference between the gradient of X corresponding to D in the entire joint distribution space and k. The discriminator widens the score gap between true and false samples as much as possible, the gradient is as large as possible, and the change amplitude is as large as possible, so after the discriminator is fully trained, its gradient Norm will be near k. Therefore, the above loss function (Loss) can be changed to make the gradient close to k (k can be any value). Define k as 1, and then weighted merge it with the loss of the original discriminator of WGAN to obtain a new discriminator. In some examples, leakyRelu is selected as the activation function with a slope of 0.2 and a deviation of 0. The loss function can be optimized using the Adam optimizer, and the optimizer parameters are set to α=0.001, β_1=0, β_2=0.99. Compared with other optimizers, the Adam optimizer makes the training converge faster and the performance more stable.
[0046] Table 1: Test results of the original dataset
[0047]
[0048] Table 2: Test results of new dataset
[0049]
[0050] Tables 1 and 2 show the corresponding test result examples of the original data set and the new data set, which confirm that the endoscopic image generation method 100 can expand the size of the data set and increase the diversity of its content. In addition, the expanded data set can also improve the performance of the detection neural network. As shown in Table 1, without adding the generated image, the overall accuracy of the detection neural network is 62.5%. The detection accuracy of single-classification data of vascular (Vascular), ulcer (Ulceration), and erosion (Erosion) is 89%, 49.9%, and 48.5%, respectively. This is because the remaining data is difficult to obtain relative to the vascular (Vascular) data, and the amount of data is relatively insufficient, resulting in insufficient detection accuracy of the detection neural network. After generating new vascular (Vascular) image data, the detection neural network is retrained with the new data set. As shown in Table 2, the accuracy of the detection neural network is improved by 1.9%, and the accuracy of the single-classification data detection of vascular (Vascular) and erosion (Erosion) is improved, that is, it is increased by 1.6% and 4.3%, respectively, and the accuracy of ulcer (Ulceration) is almost unchanged.
[0051] As used herein, unless expressly stated otherwise, the singular "a" and "an" may be construed to include the plural "one or more".
[0052] The present disclosure is presented above for the purpose of illustration and description, but is not intended to be exhaustive or limiting. Many modifications and variations will be apparent to those of ordinary skill in the art. The exemplary embodiments are selected and described in order to explain the principles and practical applications, and to enable those of ordinary skill in the art to understand the various embodiments of the present disclosure, with various modifications suitable for the particular use contemplated.
[0053] Therefore, although illustrative example embodiments are described herein with reference to the accompanying drawings, it should be understood that the description is not restrictive and that those skilled in the art may implement various other changes and modifications therein without departing from the scope, innovative concepts, and technical solutions of the present disclosure.
Claims
1. A method for generating an endoscopic image, characterized in that: The method comprises the following steps: Obtaining a machine learning model, wherein the machine learning model includes model parameters; Generate a plurality of unlabeled endoscope basic images through the machine learning model; extracting multiple lesion images from the segmentation dataset; Each of the unlabeled endoscopic basic images is fused with one of the multiple lesion images to generate multiple endoscopic medical images containing lesions.
2. The method according to claim 1, characterized in that Each of the endoscopic medical images is an endoscopic medical image including corresponding annotations.
3. The method according to claim 1, characterized in that Each of the lesion images is a lesion image including corresponding annotations.
4. The method according to claim 1, characterized in that The extracting a plurality of lesion images from the segmented data set includes: obtaining the plurality of lesion images by applying a mask to a plurality of images in the segmented data set.
5. The method according to claim 1, characterized in that The obtaining of the machine learning model comprises: Performing data preprocessing on sample data to obtain corresponding training data, wherein the sample data includes a plurality of unlabeled sample images; The training data is input into a generative adversarial neural network for adversarial learning to obtain the machine learning model and the model parameters.
6. The method according to claim 5, characterized in that The step of inputting the training data into a generative adversarial neural network for adversarial learning includes: Inputting the training image in the training data into the generator of the generative adversarial neural network to obtain a generated image; Obtaining an actual training error between the generated image and a sample image in the sample data by using a discriminator of the generative adversarial neural network; and When the actual training error meets the preset training error, the machine learning model and the model parameters are obtained.
7. The method according to claim 5, characterized in that The step of inputting the training data into a generative adversarial neural network for adversarial learning further includes: When the actual training error does not meet the preset training error, updating the model parameters by using the error function of the machine learning model; Adversarial learning is performed again using the activation function of the machine learning model and the updated model parameters.
8. The method according to claim 5, characterized in that The generative adversarial neural network is a progressive growth GAN network.
9. The method according to claim 5, characterized in that The data preprocessing of the sample data includes: standardizing the size of the sample image, standardizing the pixels of the sample image, standardizing the direction of the sample image, and flipping, locally scaling, and adjusting the brightness of the sample image.
10. The method according to claim 1, characterized in that Also includes: A medical image is generated using the machine learning model and the model parameters, and the medical image is compared with a real capsule endoscopy image to obtain a comparison error.
