Learning image data creation method and system

Abstract

Provided are a method and a system for creating learning image data whereby learning image data used in creating a learning model can be generated without being affected by distortion of a peripheral portion of an endoscopic image and a color difference between a mucous membrane image and a mask image. A mask extension process is performed on a region that is adjacent to the mask image and distorts the mucous membrane image due to the influence of a lens of an endoscope for observing the mucous membrane to produce an extended region image of the mask image, thereby producing a learning image after the extension process. Performing color conversion correction processing on the mask image of the learning image after the extension processing to generate learning image data after mask color replacement, and then correcting the hue of the mucous membrane image so as not to affect learning of the learning image after mask color replacement, thereby generating learning image data after mucous membrane hue correction, and using the mucosa hue corrected learning image data as learning image data.

Description

Technical Field

[0001] The present invention relates to a learning image data creating method and system for creating learning image data by using an endoscopic image as learning image data. Background Art

[0002] For example, Hunner-type interstitial cystitis is an inflammatory disease characterized by the presence of specific red mucosal lesions called Hunner lesions identified in the bladder. However, there are no objective diagnostic indicators for Hunner lesions, and the diagnosis of Hunner lesions will be determined subjectively by the cystoscopist. Therefore, even for experienced physicians specializing in interstitial cystitis, it is difficult to distinguish between similar red mucosal lesions that appear in BCG-related cystitis or carcinoma in situ and true red mucosal lesions based solely on observation of cystoscopic images. In order to distinguish lesions with similar mucosal features as described above, it is important to improve the accuracy of lesion differentiation in endoscopic examination, and it is also necessary to improve diagnostic accuracy in the case of using physician-assisted diagnosis supported by the use of artificial intelligence, etc.

[0003] Conventionally, as artificial intelligence for supporting diagnosis, there have been some technologies for creating a learning model by learning an endoscopic image as learning image data. For example, Japanese Patent Publication 2020-141995 (Patent Document 1) discloses a technology that includes generating a superimposed image, creating a large amount of learning image data for image recognition by using the superimposed image, and creating a learning model by using the learning image data, wherein the superimposed image superimposes a foreground endoscopic image extracted using an endoscopic treatment tool with a background endoscopic image as a background of the foreground endoscopic image.

[0004] Furthermore, as a technique for identifying an observation direction of a mucosa in which a mucosa image is observed, Japanese Patent 3720617 (Patent Document 2) discloses adding an indicator on a mask image to identify the direction of an endoscopic image.

[0005] Related technical documents

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Publication No. 2020-141995

[0008] Patent Document 2: Japanese Patent 3720617 Summary of the invention

[0009] Technical issues

[0010] The following problem is not recognized in conventional technology: in the endoscopic image, the distortion of the mucosal image around the endoscopic image due to the influence of the lens of the endoscope that observes the mucosal image, and the color difference (difference in hue, saturation and brightness) of the boundary between the mucosal image and the mask image may affect the learning results of artificial intelligence. Due to the above reasons, conventional technology creates learning image data by using the endoscopic image data itself, where the mask image is located around the mucosal image in the endoscopic image. Therefore, even if the extended image data created by the extended technology using the learning image data is used, the quality of the learning image data is poor, and as a result, there are limitations in improving the accuracy of the learning model. For example, in the case of Henner's interstitial cystitis, since the quantity of learning data cannot compensate for the lack of quality of the learning data, the above fact will lead to problems that directly affect the learning accuracy of the learning model.

[0011] An object of the present invention is to provide a method and system for creating learning image data, whereby learning image data used in creating a learning model can be generated without being affected by distortion of the peripheral portion of the endoscopic image and color differences between the mucosal image and the mask image.

[0012] Technical solutions to technical problems

[0013] The present invention relates to a learning image data creation method for creating learning image data for creating a learning model based on an endoscopic image, wherein a mask image exists outside a mucosa image in the endoscopic image. The method of the present invention includes: creating learning image data after extension processing, the learning image data after extension processing having an extension area image of the mask image, the extension area image being generated by mask extension processing, the mask extension processing defining an area adjacent to the mask image and in which the mucosa image is distorted due to the influence of a lens of an endoscope for observing the mucosa as the extension area image. In addition, it is preferably included: creating learning image data after mask color replacement by performing color conversion correction on the mask image and the extension area image of the learning image data after extension processing; creating learning image data after mucosa tone correction by performing tone correction on the mucosa image so that the tone difference of the mucosa image in the learning image data after mask color replacement does not have an influence; and defining the learning image data after mucosa tone correction as the learning image data. Preferably, when there is not enough endoscopic image learning data, the learning data on which the extension is based contains as few noise elements as possible. As a result of various studies based on the above premise, the inventors of the present invention have found that the mucosa image adjacent to the mask image may be distorted due to the influence of the lens, etc. Then, based on the knowledge confirmed above, the inventors create learning image data after extension processing, which has an extended area image of the mask image, and the extended area image is generated by mask extension processing, and the mask extension processing defines the area adjacent to the mask image and in which the mucosa image is distorted due to the influence of the lens of the endoscope used to observe the mucosa as the extended area image. The inventors of the present invention replace the mask image and the extended area image with a pattern image of the following color, under which the mask color-replaced learning image data created by performing color conversion correction on the mask image and the extended area image of the learning image data after extension processing does not cause the mucosa image to be distorted. Next, the inventors of the present invention create learning image data after mucosa tone correction by performing tone correction on the mucosa image, so that the tone difference of the mucosa image in the learning image data after mask color replacement does not affect the learning result. The tone difference of the image can be corrected by tone correction corresponding to the type of endoscope used. Note that the tone correction can be performed before the above-mentioned color conversion correction. When conversion correction and tone correction are performed, the learning image data contains almost no noise elements, so that the learning accuracy of the learning model can be improved.

[0014] Note that the method of the present invention can create an extended learning image by using the learning image data after the mucosal tone correction through the image extension technology, and can define the extended learning image data as the learning image data. Therefore, even if the quality of the learning image data is not supported by the quantity of the learning image data, the learning accuracy of the learning model can be improved.

[0015] The normal mask image includes an indicator for confirming the up and down direction of the endoscopic image. In this case, it is preferred to extend the extended area image to an area where the presence of the indicator does not affect the mucosal image. In this way, the influence of the presence of the indicator can be reliably eliminated.

[0016] In the case where the outline of the mucosal image is circular or polygonal, preferably, the extended area image is an image forming a ring shape around the periphery of the mucosal image. In this way, the image shape of the learning image data is unified, and the influence of the image shape difference can be reduced.

[0017] Color conversion correction may be performed based on the average value and dispersion of pixels of the mucosa image to convert the mask image and the extended region image into pattern images of a color that has no influence on the mucosa image.

[0018] Hue correction may be performed on the hue, saturation, and lightness (brightness) of the entire mucous membrane image using the most frequent values ​​of the hue, saturation, and lightness of the mucous membrane image as a reference so that the distribution of the hue, saturation, and lightness has no deviation.

[0019] When endoscopic images of the bladder of a patient with Henner's interstitial cystitis are diagnosed by using a learning model learned using learning image data created by the method of the present invention, the diagnostic accuracy of Henner's interstitial cystitis can be significantly improved compared to conventional methods.

[0020] The present invention can be understood as an extended learning image data creation system for creating extended learning image data using a data processor based on an endoscopic image in which a mask image exists outside a mucosal image. In the above case, preferably, the data processor includes: an extension processing unit that creates extended processed learning image data, the extended processed learning image data having an extended area image of the mask image, the extended area image being generated by mask extension processing, the mask extension processing defining an area adjacent to the mask image and in which the mucosal image is distorted due to the influence of a lens of an endoscope for observing the mucosa as the extended area image; a color replacement processing unit that performs color conversion correction on the mask image of the extended processed learning image data to create learning image data after mask color replacement; and a hue correction processing unit that corrects the hue of the mucosal image so that the hue difference of the mucosal image in the learning image data after mask color replacement does not have an effect to create learning image data after mucosal hue correction.

[0021] Furthermore, the data processor includes an expansion processing section that creates expanded learning image data by an image expansion technique using the learning image data after the mucous membrane tone correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 : is a block diagram showing the configuration of a main part of an image diagnosis system equipped with an extended learning image data creation system that executes the learning image data creation method of the present invention.

[0023] Figure 2 It shows Figure 1 A flowchart showing an outline of an algorithm for automatically creating a program for learning image data by using a computer.

[0024] Figure 3 (A), (B) and (C) are schematic diagrams showing image processing states.

[0025] Figure 4 (A), (B) and (C) are schematic diagrams showing image processing states.

[0026] Figure 5 is a flowchart showing an outline of an algorithm of a procedure for color conversion correction.

[0027] Figure 6 is a flowchart showing an outline of an algorithm of a program for tone correction.

[0028] Figure 7 (A), (B), (C), and (D) in FIG. 1 are diagrams showing examples of actual images obtained through image data processing.

[0029] Figure 8 (A) in FIG. 1 is a graph showing the results when the AUC value is used as an evaluation indicator for each learning model, and Figure 8 (B) in FIG. 1 is a diagram showing the result when the ROC curve is obtained as the evaluation index. Specific embodiments

[0030] Examples of embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Figure 1 : is a block diagram showing the configuration of a main part of an image diagnosis system equipped with an extended learning image data creation system that executes the learning image data creation method of the present invention. Figure 2 It shows Figure 1 A flowchart outlining an algorithm of a program for automatically creating learning image data using a computer in the system shown. Figure 1 As shown in FIG. 1 , the system stores endoscopic image data captured by an endoscope 1 in an endoscopic image storage unit 2 . The endoscopic image data stored in the endoscopic image data storage unit 2 is processed by a data processor 3 .

[0031] The data processor 3 creates extended learning image data using the endoscopic image in which the mask image exists outside the mucosa image as learning image data. The extended learning image data created by the data processor 3 is stored in the learning image storage unit 4, and the extended learning image data is used for AI diagnosis in the image AI diagnosis processing unit 5.

[0032] By installing the program in the data processor 3, the first data processing section 31, the second data processing section 32, the third data processing section 33, and the fourth data processing section 34 are implemented in the data processor 3. The first data processing section 31 (extension processing section) creates an extension-processed learning image having an extended region image EI of the mask image MI made by a mask extension process that defines the extended region image as a region adjacent to the mask image MI and in which the mucosa image MCI is distorted due to the influence of the lens of the endoscope 1 for observing the mucosa, as Figure 3 As shown in the schematic diagram of (A) (steps ST2 and ST3 are as Figure 2 The mask image MI includes a direction confirmation indicator CI, which is used to confirm the up and down direction of the endoscopic image, as shown in FIG. Figure 3 Then, when the mask extension process is performed, the extended area image EI can be extended to the area where the presence of the direction confirmation indicator CI does not affect the mucosal image MCI. Figure 3 In the case where there is an inward direction confirmation indicator CI as shown in (C) in FIG. 1 , the extended area image EI may be extended at least to the extent that the direction confirmation indicator CI can be hidden. Figure 3 In the case where there is an outward direction confirmation indicator CI as shown in (B) in FIG. 1 , the inward extended area image EI may be extended to such an extent that the outward direction confirmation indicator CI may be hidden and the radial dimension approaches the height dimension of the direction confirmation indicator CI. In the present embodiment, the amount of the extended area image EI may be predetermined based on the presence or absence of the direction confirmation indicator CI, the known dimension information of the direction confirmation indicator CI, the known radial dimension information of the mask, etc., but the amount of the extended area image E1 may be determined based on an arithmetic expression of the extension amount proportional to the inner diameter of the mask calculated in advance by experiments.

[0033] The second data processing unit (color replacement processing unit) 32 creates learning image data after mask color replacement by performing color conversion correction on the mask image. The second data processing unit 32 obtains the average value and standard deviation of RGB of the mucosal image, generates a pattern image with uniform distribution based on the average value and standard deviation of RGB, and replaces the mask image with the pattern image. The pattern image is an image of a so-called sandstorm-like pattern in which colors are distributed within the standard deviation range centered on the average value of RGB. In particular, as Figure 4 As shown in (A) and (B) of FIG. 1 , the color conversion correction replaces the mask image MI and the extended area image EI with a pattern image having a color that does not affect the mucosal image MCI. Figure 5 The program flow chart shown in the figure performs color conversion correction. First, the average value and dispersion of the pixel values ​​of the mucosa image are calculated (step ST41). In the calculation, when the shadow value of the pixel in the mucosa image MCI is x j When (j = 1, 2, … M), the mean value µ and standard deviation σ of the shadow value can be expressed as the following formula 1 and formula 2.

[0034] [Formula 1]

[0035] [Formula 2]

[0036] Next, a pattern image is created based on the average value and the dispersion (step ST42). texture The pixel g in i (i = 1, 2, …, N) is populated with uniformly distributed random numbers in the following range defined by the calculated mean µ and standard deviation σ.

[0037] [Formula 3]

[0038] Finally, the mask part is replaced by the pattern image. texture The pixel values ​​of the mask image MI are used to replace the pixels of the area of ​​the mask image MI. The mask color replacement process is completed by the above replacement. Note that the above process is an example, and the present invention is not limited to this process.

[0039] Next, the third data processing unit (tone correction processing unit) 33 creates a mucosa tone-corrected learning image by performing tone correction on the mucosa image so that the tone difference of the mucosa image in the learning image after the mask color replacement does not affect the learning result, such as Figure 4As shown in (C) in the figure. Specifically, hue shifting, and saturation and lightness (brightness) flattening processing are performed. First, the mucosal image, including the RGB image after mask processing, is converted into an HSV image, and the most frequent value of the hue is shifted to the center. The shift of the most frequent value of the hue to the center means that if the image is completely red, the color distribution will shift so that the color is distributed around the red area. In addition, adaptive histogram equalization is performed on the saturation and lightness of the hue-shifted image. Adaptive histogram equalization means that the saturation and lightness of the image are processed to be evenly distributed. According to the above processing, even if the hue of the mucosal surface changes due to the type of light source used by the endoscopic device, a similar hue can be produced by flattening the deviation of saturation and lightness, making it easier to see the condition of the mucosal surface.

[0040] according to Figure 6 The flowchart shown in FIG. 1 performs the above-mentioned tone correction ( Figure 2 First, the color space of the mucosa image is converted from RGB to HSV (step ST61). When the pixel values ​​of red, green, and blue as the colors of the pixels in the mucosa image MCI are x j = (R j , G j , B j ) (j = 1, …, M), the pixel values ​​are converted to the HSV color space (H: hue, S: saturation, and V: value (brightness)). Here, MAX j Respectively represent the maximum value of RGB color in pixels, and MIN j Respectively represent the minimum value of the RGB color in the pixel.

[0041] [Formula 4]

[0042] Next, the distribution of the H color space is shifted around the most frequent value (step ST62). Specifically, each pixel in the HSV color space is represented as a hue value of level L, and when the number of pixels of level i (i=0, 1, ..., L) is represented as n i When , the pixel distribution in the hue color space is shifted in parallel according to the following formula, so that n i The largest most frequent value level i max Located in the center of the distribution.

[0043] [Formula 5]

[0044] Finally, adaptive histogram equalization is performed on the SV color space. That is, when the total number of pixels of the mucosal image is M = n0 + n1 + … + n L , and the maximum values ​​of the pixel values ​​in the saturation space and the brightness color space are S max and V max When , the pixel distribution of each color space is flattened according to the following formula.

[0045] [Formula 6]

[0046] If the hue of the mucosal image is corrected as described above, the hue difference of the image can be corrected corresponding to the type of endoscope used. Note that the hue correction can be performed before the above-mentioned color conversion correction. Even when the learning model is created when the learning image data is the learning image data after the mucosal hue correction, the learning accuracy of the learning model can be improved.

[0047] Furthermore, the fourth data processing unit (expansion processing unit) 34 uses the learning image data after the mucous membrane tone correction to create expanded learning image data ( Figure 2 Step ST8 shown in the figure). This embodiment uses the extended learning image data as the learning image data. Therefore, the learning accuracy of the learning model can be improved without using a large amount of training data to compensate for the insufficient quality of the learning data. The above-mentioned image expansion technology can be a well-known data expansion technology. As a preferred expansion technology, rotation, flipping, blurring and translation of images each installed on a machine learning framework (for example, Keras, PyTorch, etc.) can be used.

[0048] Figure 7 (A), (B), (C), and (D) in FIG. 1 are examples of actual images performed by the above-mentioned image data processing. Figure 7 (A) is an endoscopic image. Figure 7 (B) in the figure is the learning image after mask expansion. Figure 7 (C) in is the learning image after mask color replacement, and Figure 7 (D) in FIG. 1 is an image after the mucous membrane tone correction. Note that the extended learning image data can be created by an image extension technique using the learning image data after the mucous membrane tone correction, and the extended learning image data can be used as the learning image data. In addition, the learning model can be created when the learning image data after the mucous membrane tone correction is used as the learning image data.

[0049] The inventors of the present invention have performed accuracy evaluation on learning models (A, A-2, B, C, D, D-2, and E) created by using image data as learning image data obtained by combining the above-mentioned "mask extension", "mask color replacement", "mucosal tone correction", and "data expansion" based on endoscopic image data of very rare cases of Henner-type interstitial cystitis. Figure 8 (A) in FIG. 1 is a graph showing the results when the AUC (Area Under the Curve) value is used as an evaluation indicator for each learning model, and Figure 8 (B) in FIG. 1 is a diagram showing the result when a ROC (Receiver Operating Characteristic) curve is obtained as an evaluation index. The image processing technique used for the above comparison is a combination of the above-mentioned "mask extension", "mask color replacement", "mucous membrane tone correction", and "data expansion". Figure 8 The learning model A shown in is a learning model created by using endoscopic image data without any image processing as learning image data, and the average AUC value is 0.868. Learning model A-2 is a learning model created by using the endoscopic image data itself and the extended learning image data from the endoscopic image data, and the average AUC value is 0.88. Learning model B is a learning model created using the extended learning image data obtained by mask extension as learning image data, and the average AUC value is 0.839. Learning model C is a learning model created by performing mucosal tone correction on the extended learning image data obtained by mask extension as learning image data, and the average AUC value is 0.844. Learning model D-2 is a learning model created by performing mask color replacement on the extended learning image data obtained by mask extension as learning image data, and the average AUC value is shown. Learning model D is a learning model created by performing mask color replacement and mucosal tone correction on the extended learning image data obtained by mask extension as learning image data, and the average AUC value is 0.915. The learning model E is a learning model created by using the extended learning image data obtained by extending the extended learning image data obtained by mask extension as the learning image data using the extension technique, and the average AUC value is 0.921.

[0050] As a result of the average AUC values ​​obtained as described above, for the learning model D or the learning model E, the average AUC value was 0.9 or greater, thereby confirming that the learning model D and the learning model E achieved higher learning accuracy.

[0051] also, Figure 8The ROC curves shown in (B) in Figure 1 show the ROC curves of learning model A, learning model B, learning model C, learning model D, and learning model E. The horizontal axis of the ROC curve is "1-specificity", and the vertical axis of the ROC is "sensitivity". In terms of performance, the ideal learning model is the one whose sensitivity and specificity of the ROC curve are close to the upper left corner. Therefore, from Figure 8 From the ROC curve shown in (B), learning model D and learning model E have higher learning accuracy. As a result of the above exploration, it is confirmed that by using learning model D or learning model E, the learning accuracy of the learning model can be improved without using a large amount of learning data to compensate for the lack of quality of the learning data.

[0052] Note that it has been confirmed that when the learning model E is used to diagnose based on the endoscopic images of patients with Henner's type interstitial cystitis, the diagnostic accuracy of patients with Henner's type interstitial cystitis is significantly higher than that of the conventional method.

[0053] Industrial Applicability

[0054] According to the present invention, less noise is included in the learning image data, so that higher learning accuracy of the learning model can be obtained.

[0055] Description of Reference Numerals

[0056] 1: endoscope, 2: endoscope image storage unit, 3: data processor, 31: first data processing unit (extension processing unit), 32: second data processing unit (color replacement processing unit), 33: third data processing unit (tone correction processing unit), 34: fourth data processing unit (expansion processing unit), 4: learning image storage unit, 5: image AI diagnosis processing unit.

Claims

1. A learning image data creation method for creating learning image data for creating a learning model based on an endoscopic image, wherein in the endoscopic image, a mask image exists outside a mucosal image, the learning image data creation method comprising the following steps: creating extended-processed learning image data having an extended-region image of the mask image, the extended-region image being generated by a mask extension process that defines, as the extended-region image, a region adjacent to the mask image and in which the mucosa image is distorted due to an influence of a lens of an endoscope for observing the mucosa; creating the learning image data based on the extended learning image data; creating mask color replaced learning image data by performing color conversion correction on the mask image and the extension region image of the extension processed learning image data; creating mucous membrane tone-corrected learning image data by performing tone correction on the mucous membrane image so that a tone difference of the mucous membrane image in the mask color-replaced learning image data does not have an influence; as well as The learning image data after the mucous membrane tone correction is defined as the learning image data.

2. The learning image data creation method according to claim 1, further comprising the following steps: Using the mucous membrane tone-corrected learning image data, creating extended learning image data by an image extension technique; The extended learning image data is defined as the learning image data.

3. The learning image data creation method according to claim 1, wherein: The mask image includes an indicator confirming the up-down direction of the endoscopic image, and the extended region image is extended outside a region where the presence of the indicator affects the mucosal image.

4. The learning image data creation method according to claim 2, wherein: The mask image includes an indicator confirming the up-down direction of the endoscopic image, and the extended region image is extended outside a region where the presence of the indicator affects the mucosal image.

5. The learning image data creation method according to claim 1, wherein: The outline of the mucosa image is circular or polygonal, and the extended area image is an image forming a ring shape around the circumference of the mucosa image.

6. The learning image data creation method according to claim 2, wherein: The outline of the mucosa image is circular or polygonal, and the extended area image is an image forming a ring shape around the circumference of the mucosa image.

7. The learning image data creation method according to claim 1, wherein: Color conversion correction is performed based on an average value and dispersion of pixels of the mucous membrane image to convert the mask image and the extended region image into pattern images of colors that have no influence on the mucous membrane image.

8. The learning image data creation method according to claim 1, wherein: Using the most frequent values ​​of the hue, saturation and lightness of the mucous membrane image as a reference, hue correction is performed on the hue, saturation and lightness of the entire mucous membrane image so that the distribution of the hue, saturation and lightness has no deviation.

9. A method for diagnosing Henner's interstitial cystitis, comprising: An endoscopic image of a bladder of a patient suffering from Henner's type interstitial cystitis is diagnosed by using a learning model learned by using learning image data created by the method according to any one of claims 1 to 8.

10. A learning image data creation system for creating learning image data using a data processor based on an endoscopic image, the learning image data being used to create a learning model, wherein in the endoscopic image, a mask image exists outside a mucosal image, wherein: The data processor comprises: an extension processing section that creates extended-processed learning image data having an extended-region image of the mask image, the extended-region image being generated by a mask extension process that defines, as the extended-region image, a region that is adjacent to the mask image and in which the mucosa image is distorted due to an influence of a lens of an endoscope for observing the mucosa; a color replacement processing section that performs color conversion correction on the mask image of the learning image data after the extension process to create learning image data after the mask color is replaced; and a tone correction processing unit that corrects the tone of the mucous membrane image so that a difference in tone of the mucous membrane image in the learning image data after the mask color replacement has no influence, thereby creating the learning image data after the mucous membrane tone correction; The learning image data after the mucous membrane tone correction is defined as the learning image data.

11. The learning image data creation system according to claim 10, wherein: The data processor further includes an expansion processing section that creates expanded learning image data by an image expansion technique using the mucous membrane tone-corrected learning image data.

12. The learning image data creation system according to claim 10, wherein: The mask image includes an indicator for confirming the up and down direction of the endoscopic image, and The extension processing section extends the extended region image outside a region where the presence of the indicator affects the mucous membrane image.

13. The learning image data creation system according to claim 10, wherein: The outline of the mucosa image is circular or polygonal, and the extended area image is an image forming a ring shape around the circumference of the mucosa image.

14. The learning image data creation system according to claim 10, wherein: The color replacement processing section is configured to perform hue correction on the hue, saturation and brightness of the entire mucous membrane image based on the most frequent values ​​of the hue, saturation and brightness of the mucous membrane image as a reference so that the distribution of the hue, saturation and brightness has no deviation.

Images