Image data for learning creation method and system

Abstract

To provide an image data for learning creation method and system which can create image data for learning for learning model creation without being affected by a distortion of a peripheral part of an endoscopic image and a color difference between a mucous membrane image and a mask image.SOLUTION: An image data for learning creation method comprises: creating an image for learning subjected to extension processing by performing mask extension processing of making a region which is adjacent to a mask image and in which a mucous membrane image is distorted due to the influence of a lens of a scope of an endoscope that observes a mucous membrane an extension region image of the mask image; creating image data for learning subjected to mask color replacement by performing color conversion correction on the mask image of the image for learning subjected to extension processing; creating image data for learning subjected to mucous membrane color tone correction by correcting the color tone of the mucous membrane image so as not to affect learning of the image for learning subjected to mask color replacement; and making the image data for learning subjected to mucous membrane color tone correction image data for learning.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method and system for creating learning image data by using an endoscopic image as learning image data. [Background technology]

[0002] For example, Hanna type interstitial cystitis is an inflammatory disease characterized by the presence of Hanna lesions, which are characteristic red mucosal lesions, in the bladder. However, there are no objective diagnostic indicators for Hanna lesions, and the diagnosis is made by the subjective judgment of the cystoscopist. For this reason, even experienced doctors who specialize in interstitial cystitis have difficulty distinguishing between similar red lesions seen in BCG-associated cystitis and intraepithelial carcinoma by observing cystoscopy images alone. In order to distinguish between lesions with similar mucosal characteristics, it is important to improve the accuracy of lesion discrimination in endoscopic examinations, and it is necessary to support doctors in their diagnosis by using artificial intelligence, etc., and improve diagnostic accuracy.

[0003] Conventionally, there is a technology for creating a learning model by learning endoscopic images as learning image data as artificial intelligence to support diagnosis. For example, JP 2020-141995 A (Patent Document 1) discloses a technology for generating a superimposed image by superimposing a foreground endoscopic image from which an endoscopic treatment tool is extracted and a background endoscopic image that forms the background of the foreground image, using this superimposed image to create a large amount of learning image data for image recognition, and using this learning image data to create a learning model.

[0004] Furthermore, as a technique for identifying the observation direction of the mucosa being observed by an endoscopic image, Japanese Patent No. 3720617 (Patent Document 2) discloses attaching an index to a mask image in order to identify the direction of the endoscopic image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2020-141995 A [Patent Document 2] Patent No. 3720617 Summary of the Invention [Problem to be solved by the invention]

[0006] In the past, there was no awareness that in endoscopic images, distortion of the mucosal image at the periphery of the endoscopic image due to the influence of the lens of the endoscope scope used to observe the mucosa, or color differences (differences in hue, saturation, and brightness) at the boundary between the mucosal image and the mask image affect the learning results of artificial intelligence. Therefore, in the past, learning image data was created by directly using endoscopic image data in which a mask image was placed around the mucosal image. As a result, the quality of the learning image data was poor, and even if augmented image data created using a learning image augmentation technology was used, there was a limit to improving the accuracy of the learning model. For example, in cases with very few cases such as Hanna type interstitial cystitis, the quality of the learning data cannot be covered by the amount of learning data, which directly affects the learning accuracy of the learning model.

[0007] An object of the present invention is to provide a method and system for creating learning image data that can create learning image data for creating a learning model without being affected by distortion in the peripheral parts of an endoscopic image or color differences between mucosal images and mask images. [Means for solving the problem]

[0008] The present invention is directed to a method for creating learning image data for creating a learning model based on an endoscopic image in which a mask image is present outside the mucosa image. In the method of the present invention, an extension-processed learning image is created by performing a mask extension process in which an area adjacent to the mask image and in which the mucosa image is distorted due to the influence of the lens of an endoscope for observing the mucosa is treated as an extended area image of the mask image. Then, it is preferable to perform color conversion correction on the mask image of the extension-processed learning image to create mask color-replaced learning image data, correct the color tone of the mucosa image so as not to affect the learning of the mask color-replaced learning image to create mucosa color-corrected learning image data, and use the mucosa color-corrected learning image data as the learning image data. In cases where sufficient learning data for endoscopic images cannot be prepared, it is preferable that the learning data that is the basis of the extension does not contain elements that become noise as much as possible. As a result of various studies based on such a premise, the inventor has found that the mucosa image adjacent to the mask image may be distorted due to the influence of the lens, etc. Based on this knowledge, the inventors decided to create an extension-processed learning image by performing a mask extension process in which an area adjacent to a mask image where the mucosa image is distorted due to the influence of a lens or the like is made into an extension area image of the mask image. Then, the inventors decided to replace the mask image and the extension area image in the extension-processed learning image with a pattern image of a color that does not cause distortion in the mucosa image by performing color conversion correction on the mask image and the extension area image in the extension-processed learning image to create mask color-replaced learning image data. Next, the color tone of the mucosa image is corrected so that the difference in color tone of the mucosa image in the mask color-replaced learning image data does not affect the learning results, and mucosa color-corrected learning image data is created. This color correction is to correct the difference in color tone of the image depending on the type of endoscope used. Note that color correction can also be performed before the above-mentioned color conversion correction. When color conversion correction and color correction are performed, the learning image data contains almost no noise elements, and the learning accuracy of the learning model can be improved.

[0009] In addition, the mucous membrane color-corrected learning image data may be used to create extended learning image data by image extension technology, and the extended learning image data may be used as learning image data. In this way, the learning accuracy of the learning model can be improved without compensating for the quality of the learning data with the amount of learning data.

[0010] A typical mask image includes an index for checking the up-down direction of the endoscope image. In this case, it is preferable that the extended region image is extended to a range where the presence of the index does not affect the mucosa image. In this way, the influence of the presence of the index can be reliably eliminated.

[0011] When the contour of the mucosa image is circular or polygonal, it is preferable that the extended region image is an annular image surrounding the outer periphery of the mucosa image. In this way, the image shapes of the learning image data are unified, and the influence of differences in image shapes can be reduced.

[0012] In the color conversion correction, the mask image and the extended region image can be replaced with a pattern image of a color that does not affect the mucous membrane image, based on the average and variance of the pixel values ​​of the mucous membrane image. In color correction, the most frequent hue, saturation, and brightness of the mucous membrane image are used as a standard, and color correction can be performed so that the distribution of the hue, saturation, and brightness of the entire mucous membrane image is not biased.

[0013] When a learning model trained using extended learning image data created by the method of the present invention is used to diagnose endoscopic images of the bladder of a patient with Hanna type interstitial cystitis, the accuracy of diagnosing Hanna type interstitial cystitis can be significantly improved compared to the conventional method.

[0014] The present invention can be understood as an extended learning image data creation system that uses an endoscopic image in which a mask image exists outside the mucosa image as learning image data and creates extended learning image data using a data processor. In this case, the data processor preferably includes an extension processing means for performing a mask extension process to create an extended learning image by making an extended area image of the mask image an image of an area adjacent to the mask image and in which the mucosa image is distorted due to the influence of the lens of an endoscope that observes the mucosa, a color replacement processing means for performing color conversion correction on the mask image of the extended learning image to create mask color-replaced learning image data, and a color correction processing means for correcting the color tone of the mucosa image so that the difference in color tone of the mucosa image of the mask color-replaced learning image does not affect the learning result, thereby creating mucosa color-corrected learning image data.

[0015] The data processor may further include an extension processing means for creating extended learning image data by an image extension technique using the mucous membrane color-tone corrected learning image data. [Brief description of the drawings]

[0016] [Figure 1] 1 is a block diagram showing the configuration of the main parts of an image diagnostic system equipped with an extended learning image data creation system which implements the learning image data creation method of the present invention. FIG. [Diagram 2] 2 is a flowchart showing an outline of a program algorithm for automatically creating learning image data using a computer in the system of FIG. 1. [Diagram 3] 1A to 1C are schematic diagrams showing the image processing state. [Figure 4] 1A to 1C are schematic diagrams showing the image processing state. [Diagram 5] 11 is a flowchart showing an outline of an algorithm for a program for color conversion correction. [Figure 6] 1 is a flowchart showing an outline of a program algorithm for color correction. [Figure 7]13A to 13D are diagrams showing examples of actual images obtained by image data processing. [Figure 8] FIG. 1A is a diagram showing the results of using the AUC value as an evaluation index for each learning model, and FIG. 1B is a diagram showing the results of obtaining an ROC curve as the evaluation index. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] An example of an embodiment of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the main parts of an image diagnosis system equipped with an extended learning image data creation system which implements the learning image data creation method of the present invention. FIG. 2 is a flow chart showing an outline of a program algorithm when learning image data is automatically created using a computer in the system of FIG. 1. As shown in FIG. 1, this system stores endoscopic image data captured by an endoscopic scope in endoscopic image storage means 2. The endoscopic image data stored in the endoscopic image storage means 2 is then processed by a data processor 3.

[0018] The data processor 3 creates extended learning image data using an endoscopic image in which a mask image exists outside a mucosal image as learning image data. The extended learning image data created by the data processor 3 is stored in a learning image storage unit 4, and the extended learning image data is used for AI diagnosis in an image AI diagnosis processing unit 5.

[0019] The data processor 3 is configured with a first data processing means 31 to a fourth data processing means 34 by installing a program inside. As shown in the schematic diagram of FIG. 3(A), the first data processing means 31 (extension processing means) performs a mask extension process in which the region where the mucosa image MCI adjacent to the mask image MI is distorted due to the influence of the lens of the endoscope scope 1 for observing the mucosa is set as an extended region image EI of the mask image MI to create an extension-processed learning image (steps ST2 and ST3 in FIG. 2). As shown in FIGS. 3(B) and (C), a normal mask image MI includes a direction confirmation index CI for confirming the up-down direction of the endoscope image. Therefore, when performing the mask extension process, it is preferable to extend the extended region image EI to a range where the presence of the direction confirmation index CI does not affect the mucosa image MCI. For example, when there is an inward direction confirmation index CI as shown in FIG. 3(C), it is preferable to extend the extended region image EI at least to a range where the direction confirmation index CI is hidden. 2B, when there is an outward direction confirmation index CI, it is preferable to hide the outward direction confirmation index CI and extend the inward extension area image EI by a radial dimension close to the height dimension of the direction confirmation index CI. In this embodiment, the extension amount of the extension area image EI may be determined in advance based on the presence or absence of the direction confirmation index CI, known dimension information of the direction confirmation index CI, known inner diameter information of the mask, etc., but it can be determined based on an arithmetic expression of the extension amount proportional to the inner diameter dimension of the mask obtained by a previous test.

[0020] The second data processing means (color replacement processing means) 32 performs color conversion correction on the mask image to create a mask color-replaced learning image. The second data processing means 32 acquires the average and standard deviation of RGB of the mucosa image, generates a uniformly distributed pattern image according to the average and standard deviation of RGB, and replaces the mask image MI. The pattern image here is an image with a so-called sandstorm-like pattern in which colors are distributed within the range of the standard deviation centered on the average of RGB. Specifically, as shown in Figures 4(A) and (B), in the color conversion correction, the mask image MI and the extended region image EI are replaced with a pattern image of a color that does not affect the mucosa image MCI. This color conversion correction is executed according to the program flowchart shown in Figure 5. First, the average and variance of the pixel values ​​of the mucosa image are calculated (step ST41). For this calculation, the grayscale values ​​of the pixels in the mucosa image MCI are multiplied by x j If (j=1, 2, ...M), the average μ and standard deviation σ of the gray values ​​can be expressed by the following numbers (1) and (2).

[0021]

number

[0022]

number

[0023]

number

[0024] Next, as shown in FIG. 4(C), the third data processing means (color tone correction processing means) 33 corrects the color tone of the mucosa image so that the difference in color tone of the mucosa image of the mask color-replaced learning image does not affect the learning result, to create a mucosa color tone-corrected learning image. Specifically, hue shift and saturation and brightness flattening processing are performed. First, the mucosa image consisting of the masked RGB image is converted into an HSV image and moved around the hue mode. Here, moving around the hue mode means that if the entire image is reddish, the color distribution is shifted so that the color is distributed around the red part. Next, adaptive histogram equalization is performed with the saturation and brightness of the image that has been hue shifted. Here, adaptive histogram equalization means processing so that the saturation and brightness of the image are distributed evenly. By performing such processing, even if the color tone of the mucosa surface changes due to the type of light source of the endoscope device, it is possible to create a similar color tone by flattening the bias in saturation and brightness, making it easier to see the state of the mucosa surface.

[0025] This color correction is performed according to the program flowchart shown in FIG. 6 (steps ST6 and ST7 in FIG. 2). First, the color space of the mucosa image is converted from RGB to HSV (step ST61). The red, green, and blue pixel values ​​of the color of the pixels in the mucosa image MCI are multiplied by x j =(R j ,G j ,B j ) (j=1,…,M), the following formula is used to convert the color space to HSV (H: Hue, S: Saturation, V: Value). Here, MAX j is the maximum value of each RGB color in the pixel, MIN j is the minimum value for each RGB color.

[0026]

number

[0027]

number

[0028]

number

[0029] Next, the fourth data processing means (extension processing means) creates extended learning image data by image extension technology using the mucous membrane color-tone corrected learning image data (step ST8 in FIG. 2). In this embodiment, the extended learning image data is used as learning image data. In this way, the learning accuracy of the learning model can be improved without compensating for the quality of the learning data with the amount of learning data. As the image extension technology that can be used here, a well-known data extension technology may be used. As a preferred extension technology, image rotation, inversion, blurring, translation, etc. implemented in machine learning frameworks such as Keras and PyTorch can be used.

[0030] 7(A) to (D) show examples of actual images obtained by the above image data processing. Fig. 7(A) is an endoscopic image, Fig. 7(B) is a learning image with mask extension processing, Fig. 7(C) is a learning image with mask color replacement, and Fig. 7(D) is a learning image with mucous membrane color correction. Using the learning image data with mucous membrane color correction created in the above embodiment, extended learning image data may be created by image extension technology, and the extended learning image data may be used as learning image data, or a learning model may be created using the learning image data with mucous membrane color correction.

[0031] The inventors evaluated the accuracy of learning models (A, A-2, B, C, D, D-2, and E) created using image data obtained by combining the above-mentioned "mask extension," "mask color replacement," "mucous membrane color correction," and "data expansion" as learning data based on endoscopic image data with very few cases such as Hanna-type interstitial cystitis. Figure 8 (A) shows the results of using the AUC (Area Under the Curve) value as an evaluation index for each learning model, and Figure 8 (B) shows the results of obtaining a ROC (Receiver Operating Characteristic) curve as an evaluation index. The image processing technology used for the comparison was a combination of the above-mentioned "mask extension," "mask color replacement," "mucous membrane color correction," and "data expansion." Learning model A in Figure 8 is a learning model created using endoscopic image data itself without any image processing as learning image data, and the average AUC value was 0.868. Learning model A-2 is a learning model created using endoscopic image data and expanded learning image data obtained by data expansion from endoscopic image data, and the average AUC value was 0.88. The learning model B was created using the extension-processed learning image data obtained by mask extension as the learning image data, and the average AUC value was 0.839. The learning model C was created using the image data obtained by mask extension and mucous membrane color correction as the learning image data, and the average AUC value was 0.844. The learning model D-2 was created using the image data obtained by mask extension and mask color replacement as the learning image data, and the average AUC value was shown. The learning model D was created using the image data obtained by mask extension and mask color replacement and mucous membrane color correction as the learning image data, and the average AUC value was 0.915. The learning model E was created by using the image data obtained by mask extension and mask color replacement and mucous membrane color correction as the learning image data, and the image data obtained by mask extension and mask color replacement and mucous membrane color correction as the learning image data, and the extended learning image data was created using the extension technology. The average AUC value was 0.921.

[0032] From these average AUC values, it was confirmed that learning models D and E had average AUC values ​​of 0.9 or higher, and thus achieved high learning accuracy.

[0033] The ROC curves shown in Figure 8(B) show the ROC curves of learning models A, B, C, D, and E. The horizontal axis of the ROC curve is "1-specificity" and the vertical axis is "sensitivity." The ideal learning model in terms of performance is one in which the sensitivity and specificity of the ROC curve approach the upper left corner. Therefore, from the ROC curve shown in Figure 8(B), it can be seen that learning models D and E have high accuracy. From this result, it was confirmed that by using learning models D or E, the learning accuracy of the learning model can be improved without compensating for the quality of the learning data with the amount of learning data.

[0034] Furthermore, when learning model E was used to make a diagnosis based on endoscopic images of Hanna type interstitial cystitis, it was confirmed that the diagnostic accuracy of Hanna type interstitial cystitis could be significantly improved compared to the conventional method. [Industrial Applicability]

[0035] According to the present invention, the learning image data contains almost no noise, and the learning accuracy of the learning model can be improved. [Explanation of symbols]

[0036] 1 Endoscope 2 Endoscopic image storage means 3. Data Processor 31 First data processing means (extension processing means) 32 Second data processing means (color replacement processing means) 33 Third data processing means (color correction processing means) 34 Fourth data processing means (extension processing means) 4. Learning image memory 5. Image AI diagnosis processing section

Claims

1. A method for creating learning image data for creating a learning model based on an endoscopic image in which a mask image exists outside a mucosal image, the method comprising: A mask extension process is performed to generate an extension-processed learning image data by forming an image of an extended region of the mask image in a region adjacent to the mask image and in which the mucosa image is distorted due to the influence of a lens of an endoscope for observing the mucosa, and creating the learning image data based on the extension-processed learning image data; performing color conversion correction on the mask image and the extended region image of the extension-processed learning image data to generate mask color-replaced learning image data; correcting the color tone of the mucous membrane image so that the difference in color tone of the mucous membrane image in the mask color-replaced learning image data does not affect the image data, thereby creating mucous membrane color-corrected learning image data; A method for creating learning image data, comprising using the mucous membrane color-tone corrected learning image data as the learning image data.

2. Using the mucous membrane color-corrected learning image data, extended learning image data is created by image extension technology; The method for creating image data for training according to claim 1 , wherein the extended learning image data is used as the learning image data.

3. The method for creating learning image data as described in claim 1, wherein the mask image includes an indicator for confirming the up and down direction of the endoscopic image, and the extended area image is extended to a range where the presence of the indicator affects the mucosal image.

4. The method for creating learning image data described in claim 2, wherein the mask image includes an indicator for confirming the up and down direction of the endoscopic image, and the extended area image is extended to a range where the presence of the indicator affects the mucosal image.

5. 2. The method for creating learning image data according to claim 1, wherein the contour of the mucous membrane image is circular or polygonal, and the extended region image is an annular image surrounding the outer periphery of the mucous membrane image.

6. 3. The learning image data creating method according to claim 2, wherein the contour of the mucous membrane image is circular or polygonal, and the extended region image is an annular image surrounding the outer periphery of the mucous membrane image.

7. The learning image data creation method according to claim 1 , wherein the color conversion correction replaces the mask image and the extended region image with a pattern image of a color that does not affect the mucosa image based on the average and variance of the pixel values ​​of the mucosa image.

8. The method for creating learning image data according to claim 1, wherein the color correction is performed based on the most frequent values ​​of the hue, saturation, and brightness of the mucous membrane image, so that the distribution of the hue, saturation, and brightness of the entire mucous membrane image is not biased.

9. A method for diagnosing Hanna type interstitial cystitis, comprising diagnosing endoscopic images of the bladder of a patient with Hanna type interstitial cystitis using a learning model trained using learning image data created by the method according to any one of claims 1 to 8.

10. A learning image data creation system that uses a data processor to create learning image data for creating a learning model based on an endoscopic image in which a mask image exists outside a mucosal image, The data processor includes: an extension processing means for performing a mask extension process to generate an extension-processed learning image by taking a region of the mucosa image distorted due to the influence of a lens of an endoscope for observing the mucosa adjacent to the mask image and forming an extension region image of the mask image; a color replacement processing means for performing color conversion correction on the mask image of the extension-processed learning image to generate mask color-replaced learning image data; a color correction processing means for correcting the color tone of the mucous membrane image so as to prevent the difference in color tone of the mucous membrane image in the mask color-replaced learning image data from affecting the learning image data, thereby creating mucous membrane color-tone-corrected learning image data; A learning image data creation system, characterized in that the mucous membrane color-tone corrected learning image data is used as the learning image data.

11. The learning image data creation system according to claim 10 , wherein the data processor further includes an extension processing means for creating extended learning image data by an image extension technique using the mucous membrane color-tone corrected learning image data.

12. the mask image includes an index for confirming the up-down direction of the endoscopic image, The learning image data creation system according to claim 10 , wherein the extension processing means extends the extension region image to a range where the presence of the marker affects the mucous membrane image.

13. The learning image data creation system according to claim 10 , wherein the contour of the mucous membrane image is circular or polygonal, and the extended region image is an annular image surrounding the outer periphery of the mucous membrane image.

14. The color replacement processing means is configured such that the color conversion correction performs conversion correction on the hue, saturation, and lightness of the entire mucosal image so that their distributions are not skewed, based on the mode values of the hue, saturation, and lightness of the mucosal image. The learning image data creation system according to claim 10.

Images