Endoscopic diagnostic program, endoscopy diagnostic device, control method for endoscopy diagnostic device, and program for generating trained model for endoscopic diagnosis

Abstract

To provide an endoscope diagnosis program, an endoscope diagnosis device, a control method for an endoscope diagnosis device, and a program for generating an endoscope diagnosis trained model which each make it possible to use endoscope images that have been accumulated to date to diagnose the presence or absence of a lesion, without limitations on the type of endoscope device. [Solution] A computer is made to function as: a diagnostic-use image acquisition unit 66 that acquires a diagnostic-use image of a site that is subject to diagnosis which has been captured with an endoscope device 20; a color tone correction unit 67 that performs color tone correction on the diagnostic-use image in accordance with a reference color tone, which is a color tone of an image of a site subject to diagnosis having been captured in advance with the endoscope device; and a lesion presence / absence diagnosis unit 68 that inputs the corrected diagnostic-use image into a trained model and that performs a process for diagnosing the presence or absence of a lesion from a result output by the trained model, said trained model having being generated by machine learning of a plurality of training images which have been subjected to color tone correction with the reference color tone.

Description

[Technical Field]

[0001] The present invention relates to an endoscopic diagnostic program, an endoscopic diagnostic device, a control method for an endoscopic diagnostic device, and a trained model generation program for endoscopic diagnosis, for diagnosing the presence or absence of a lesion using endoscopic images captured by the endoscopic device. [Background technology]

[0002] Ulcerative colitis is a chronic inflammation of the mucous membrane of the large intestine. It is a serious illness that can cause symptoms such as fever, abdominal pain, diarrhea, and bloody stools. When the condition worsens, it can cause a variety of symptoms throughout the body, including fever, weight loss, and anemia. Therefore, early detection and appropriate treatment are essential.

[0003] Endoscopic devices are commonly used to diagnose ulcerative colitis. Doctors irradiate the area to be diagnosed with illumination light and view the endoscopic images captured by the endoscopic device to determine whether inflammation or ulcers are present. However, this type of visual diagnosis is heavily influenced by the doctor's skill and experience. Furthermore, even specialists have difficulty making an accurate diagnosis because early ulcerative colitis can be difficult to distinguish from normal tissue, and inflammation can also occur due to other infectious enteritis, such as Campylobacter enteritis.

[0004] Therefore, development of devices that can perform more accurate diagnosis using endoscopic images is underway. For example, Japanese Patent Application Laid-Open No. 2022-60540 proposes an endoscopic device that irradiates a diagnostic target area with light of a specific spectrum (so-called narrowband light) and can identify the presence, type, and progression of a lesion using the captured image (Patent Document 1). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2022-60540 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the invention described in Patent Document 1 is an expensive device because it uses light of a specific spectrum, and is not widely used.

[0007] However, endoscopic devices capable of capturing images using illumination light (e.g., white light) that is not limited to a specific spectrum are generally in widespread use, and thus a large number of endoscopic images captured using light that is not limited to a specific spectrum, particularly white light, have been accumulated.

[0008] Furthermore, recent advances in AI technology have made it possible to use machine learning to detect abnormalities within images. For this reason, it is hoped that a device can be developed that can automatically diagnose the presence or absence of lesions by using machine learning to learn from endoscopic images that have been accumulated up to now.

[0009] However, the color tone and image quality of endoscopic images vary depending on the model used to capture them, and therefore, although a large number of endoscopic images have been accumulated, highly accurate diagnoses cannot be achieved even when machine learning is performed using all images captured by different endoscopic devices.

[0010] While it is possible to train the system on a per-model basis, this reduces the number of images available for machine learning. Furthermore, the system must be trained again each time the endoscope system is replaced, and high accuracy cannot be expected immediately after the replacement due to the small number of images. Furthermore, once a certain number of images have been accumulated and machine learning has been performed, it is time to switch to a new endoscope system, and the system must be trained from scratch, creating a vicious cycle.

[0011] The present invention has been made to solve the above-mentioned problems, and aims to provide an endoscopic diagnostic program, an endoscopic diagnostic device, a control method for an endoscopic diagnostic device, and a trained model generation program for endoscopic diagnosis that can utilize endoscopic images accumulated to date to diagnose the presence or absence of a lesion without limiting the model of the endoscopic device. [Means for solving the problem]

[0012] The endoscopic diagnostic program of the present invention is an endoscopic diagnostic program that diagnoses the presence or absence of a lesion based on endoscopic images captured by an endoscopic device, in order to solve the problem of being able to diagnose the presence or absence of a lesion using images captured by different endoscopic devices.The endoscopic diagnostic program causes a computer to function as a diagnostic image acquisition unit that acquires a diagnostic image of the diagnostic target area captured by the endoscopic device, a color correction unit that corrects the color tone of the diagnostic image to match a reference color tone, which is the color tone of the image of the diagnostic target area previously captured by the endoscopic device, and a lesion presence / absence diagnosis unit that inputs the corrected diagnostic image into a trained model generated by machine learning multiple training images that have been color-corrected using the reference color tone, and diagnoses the presence or absence of a lesion from the output results.

[0013] Furthermore, as one aspect of the present invention, in order to solve the problem of increasing diagnostic accuracy by keeping the image quality between images within a certain range, the color correction unit may perform the color correction and high-image-quality processing on the diagnostic image, and the lesion presence / absence diagnosis unit may diagnose the presence or absence of a lesion using the trained model generated by machine learning the training image that has been subjected to the same color correction and high-image-quality processing as the color correction unit.

[0014] Furthermore, as one aspect of the present invention, in order to solve the problem of being able to diagnose the presence or absence of a lesion using a large number of accumulated images that do not reflect lesions, the lesion presence / absence diagnosis unit may input the corrected diagnostic image into a trained model generated by machine learning a plurality of training images that have been corrected with the reference color tone and do not reflect lesions into an image abnormality detection algorithm, and diagnose the presence of a lesion when an abnormality is detected in the diagnostic image as the output result.

[0015] Furthermore, as one aspect of the present invention, in order to solve the problem of being able to diagnose the presence or absence of a lesion using accumulated images regardless of whether or not a lesion is reflected, the lesion presence / absence diagnosis unit may input the corrected diagnostic image into a trained model generated by using an algorithm that detects image features through machine learning of a plurality of the training images that have been corrected with the reference color tone and do not reflect a lesion, and a plurality of the training images that have been corrected with the reference color tone and reflect a lesion, and as the output result, diagnose the presence of a lesion when the feature amount of the diagnostic image is determined to be equal to or greater than a predetermined value, and diagnose the absence of a lesion when the feature amount of the diagnostic image is determined to be less than the predetermined value.

[0016] The endoscopic diagnostic device of the present invention is an endoscopic diagnostic device that diagnoses the presence or absence of a lesion based on endoscopic images captured by endoscopic devices, in order to solve the problem of being able to diagnose the presence or absence of a lesion using images captured by different endoscopic devices, and includes a diagnostic image acquisition unit that acquires a diagnostic image of the diagnostic target area captured by the endoscopic device, a color correction unit that corrects the color tone of the diagnostic image to match a reference color tone, which is the color tone of the image of the diagnostic target area captured in advance by the endoscopic device, and a lesion presence / absence diagnosis unit that inputs the corrected diagnostic image into a trained model generated by machine learning multiple training images that have been color-corrected using the reference color tone, and performs diagnostic processing for the presence or absence of a lesion based on the output results.

[0017] The control method for an endoscopic diagnostic device according to the present invention is a control method for an endoscopic diagnostic device that diagnoses the presence or absence of a lesion based on endoscopic images captured by an endoscopic device, in order to solve the problem of being able to diagnose the presence or absence of a lesion using images captured by different endoscopic devices, and includes a diagnostic image acquisition step of acquiring a diagnostic image of a diagnostic target area captured by the endoscopic device, a color correction step of color correcting the diagnostic image to match a reference color tone, which is the color tone of an image of the diagnostic target area captured in advance by the endoscopic device, and a lesion presence / absence diagnosis step of inputting the corrected diagnostic image into a trained model generated by machine learning multiple training images that have been color-corrected using the reference color tone, and diagnosing the presence or absence of a lesion from the output result.

[0018] The trained model generation program for endoscopic diagnosis of the present invention is a trained model generation program for endoscopic diagnosis that generates a trained model to be used in endoscopic diagnosis that diagnoses the presence or absence of a lesion based on endoscopic images captured by an endoscopic device, in order to solve the problem of generating a trained model that can diagnose the presence or absence of a lesion using images captured by different endoscopic devices.The program causes a computer to function as: a training image acquisition unit that acquires multiple training images of the diagnostic target area captured by the endoscopic device; a color correction unit that color corrects each of the training images to match a reference color tone, which is the color tone of the image of the diagnostic target area captured in advance by the endoscopic device; a trained model generation unit that generates multiple trained models using a machine learning algorithm; and a trained model determination unit that inputs a verification image, in which the presence or absence of a lesion has been verified, into each of the generated trained models and determines the trained model with the highest accuracy rate as the trained model to be used for endoscopic diagnosis.

[0019] Furthermore, as one aspect of the present invention, in order to solve the problem of improving diagnostic accuracy when using a training image that reflects a lesion, the color correction unit may be configured to perform color correction using the reference color tone on an area other than the area where the lesion is present when a lesion is reflected in the training image.

[0020] Furthermore, as one aspect of the present invention, in order to solve the problem of artificially increasing the number of training images and further improving diagnostic accuracy, a computer may be made to function as an affine transformation unit that performs an arbitrary affine transformation on the training images corrected by the color correction unit, and the trained model generation unit may perform machine learning on the training images corrected by the color correction unit and the training images transformed by the affine transformation unit to generate a trained model. [Effects of the Invention]

[0021] According to the present invention, it is possible to diagnose the presence or absence of a lesion by utilizing endoscopic images that have been accumulated up to now, without limiting the model of the endoscope device. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a block diagram showing a first embodiment of an endoscopic diagnostic device according to the present invention. [Figure 2] FIG. 10 is a flowchart showing the processing flow of the program for generating a trained model for endoscopic diagnosis according to the first embodiment. [Figure 3] FIG. 4 is a flowchart showing the processing flow of the endoscope diagnostic program according to the first embodiment. [Figure 4] FIG. 1 is a schematic diagram showing a combination of learning images (training data) and test data for stratified 5-fold cross-validation used to generate a trained model and verify diagnostic accuracy in Example 1. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, a first embodiment of an endoscopic diagnostic program, an endoscopic diagnostic device, a control method for an endoscopic diagnostic device, and a trained model generation program for endoscopic diagnosis according to the present invention will be described with reference to the drawings.

[0024] The endoscopic diagnostic device 1 automatically diagnoses the presence or absence of a lesion by inputting an endoscopic image captured by an endoscopic device 20, and in this first embodiment, as shown in Fig. 1, is mainly composed of a display means 2, an input means 3, and a computer 4 equipped with a storage means 5 and an arithmetic processing means 6. The endoscopic diagnostic device 1 is connected to the endoscopic device 20 in a state in which it can receive image data.

[0025] The endoscopic diagnosis device 1 of the first embodiment also has a function of generating a trained model for endoscopic diagnosis by a trained model generation program 10a for endoscopic diagnosis.

[0026] The endoscopic device 20 is also equipped with a general medical endoscopic camera 21, and is connected so that it can output endoscopic images captured using illumination light (e.g., white light) and transmit them to the endoscopic diagnostic device 1.

[0027] In the first embodiment, the endoscopic diagnostic device 1 and the endoscope device 20 are configured as separate devices, but the endoscopic diagnostic device 1 and the endoscope device 20 may be configured as an integrated device.

[0028] Each component of the endoscopic diagnostic device 1 will be described in detail below.

[0029] The display means 2 is composed of a liquid crystal display or the like, and displays various information to the user, such as an input image for inputting calculation conditions, and diagnostic results obtained by calculation processing.

[0030] The input means 3 is made up of a mouse, keyboard, etc., and is used to input various selections and instructions by the user, and in this embodiment is used to input calculation conditions and the like.

[0031] In this first embodiment, the display means 2 and the input means 3 are provided separately, but this configuration is not limited to this, and the device may be configured with a display / input means that combines display and input functions, such as a touch panel.

[0032] The computer 4 is a device for executing the endoscopic diagnostic program 1a and the endoscopic diagnosis trained model generation program 10a, and in this embodiment, as shown in Fig. 1, includes a storage means 5 and an arithmetic processing means 6. Although not shown, the computer 4 also includes a power supply device that supplies power to the arithmetic processing means 6 and other components, as well as communication means such as a LAN (Local Area Network), Wi-Fi (Wireless Fidelity), and Bluetooth.

[0033] The storage means 5 stores various data and functions as a working area when the arithmetic processing means 6 performs arithmetic processing. In this embodiment, the storage means 5 is composed of a hard disk, a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc., and as shown in Fig. 1, has a program storage unit 51, a learning image storage unit 52 that stores learning images used in machine learning, a reference color tone storage unit 53 that stores reference colors used for color correction, a trained model storage unit 54 that stores trained models generated by the endoscopic diagnosis trained model generation program 10a, and a diagnostic image storage unit 55 that stores diagnostic images used for diagnosis.

[0034] An endoscopic diagnostic program 1a for acquiring endoscopic images captured by the endoscopic device 20 and diagnosing the presence or absence of a lesion, and an endoscopic diagnosis trained model generation program 10a for generating a trained model used in the endoscopic diagnosis program 1a are installed in the program storage unit 51. The arithmetic processing means 6 executes the endoscopic diagnostic program 1a and the endoscopic diagnosis trained model generation program 10a, causing the computer 4 to function as each of the components described below.

[0035] The use of the endoscopic diagnostic program 1a and the endoscopic diagnosis trained model generation program 10a is not limited to the above configuration. For example, the endoscopic diagnostic program 1a and the endoscopic diagnosis trained model generation program 10a may be stored on a non-transitory recording medium readable by the computer 4, such as a CD-ROM or DVD-ROM, and read and executed directly from the recording medium. The endoscopic diagnostic program 1a and the endoscopic diagnosis trained model generation program 10a may also be used from an external server or the like using a cloud computing method or an ASP (Application Service Provider) method.

[0036] The training image memory unit 52 is a memory area for storing training images for generating a trained model for endoscopic diagnosis using the trained model generation program 10a for endoscopic diagnosis. In this first embodiment, the memory area stores multiple training images (raw images) that have been captured using various types of endoscopic devices under various types of illumination light, including white light, corrected training images (color-tone corrected images) that have been corrected by the color correction unit 62 described below, and training images (affine-transformed images) obtained by affine-transforming the corrected training images.

[0037] Furthermore, the learning images in this first embodiment are images of the diagnostic target area captured by multiple types of endoscopic devices, and are images that have been confirmed in advance to not include any lesions. The illumination light used to capture the learning images is not particularly limited, but in this first embodiment, the images are captured using so-called white light, which is not limited to light of a specific spectrum.

[0038] The reference color tone storage unit 53 is a storage area for storing reference color tones that serve as a reference for color tone correction. In the first embodiment, the reference color tones are color distributions based on RGB values ​​and the like acquired by analyzing an image captured by an endoscopic device in advance. The color distributions can be acquired using existing program code or original program code. Examples of existing program code that can be used include technologies based on hue analysis, K-means clustering, color histograms, convolutional neural networks, principal component analysis, self-organizing maps, Gaussian Mixture Models, support vector machines, decision trees, OpenCV, matplotlib, and the like. Alternatively, a hue analysis tool in image processing software can be used.

[0039] The reference color tone is not limited to a color distribution based on RGB values ​​or the like, but may also be a numerical value based on the color density, contrast, or brightness.

[0040] The trained model storage unit 54 is a storage area that stores trained models generated by the trained model generation program for endoscopic diagnosis 10 a. In the first embodiment, the trained model storage unit 54 stores a plurality of trained models (candidate models) generated by the trained model generation unit 64 and a trained model (diagnostic model) to be used for endoscopic diagnosis determined by the trained model determination unit 65.

[0041] The diagnostic image storage unit 55 is a storage area for storing diagnostic images for diagnosing the presence or absence of a lesion, and in the first embodiment, stores diagnostic images (raw images) acquired from the endoscope device 20 and diagnostic images (color-tone corrected images) corrected by the color correction unit 67. The diagnostic images (raw images) acquired from the endoscope device 20 may be images captured in real time by the endoscope camera 21, or may be images captured in advance by the endoscope camera 21. The illumination light used to capture the diagnostic images is not particularly limited, but in the first embodiment, images captured using so-called white light, which is not limited to light of a specific spectrum, are used.

[0042] The calculation processing means 6 is composed of a CPU (Central Processing Unit) etc., and by executing the endoscopic diagnosis trained model generation program 10a installed in the program memory unit 51, it functions as a training image acquisition unit 61, a color correction unit 62, an affine transformation unit 63, a trained model generation unit 64 and a trained model determination unit 65, and by executing the endoscopic diagnostic program 1a, it functions as a diagnostic image acquisition unit 66, a color correction unit 67 and a lesion presence / absence diagnosis unit 68.

[0043] The learning image acquisition unit 61 acquires a plurality of learning images of a diagnostic target region photographed by an endoscope device, and in the first embodiment, it is configured to appropriately acquire learning images (raw images), corrected learning images (color-tone corrected images), and affine-transformed learning images (affine-transformed images) stored in the learning image storage unit 52.

[0044] The color correction unit 62 performs color correction on the learning images (raw images) that have been accumulated up to now and that have been acquired by the learning image acquisition unit 61, and stores the corrected learning images (color-corrected images) in the learning image memory unit 52.

[0045] The color tone correction unit 62 in the first embodiment acquires a reference color tone from the reference color tone storage unit 53, corrects the color tone of each learning image to the reference color tone, and performs image quality improvement processing on each learning image.

[0046] Color correction is performed, for example, by applying a color distribution stored as a reference color to the colors of the training image that are similar to the color of the training image. This color correction smoothes the training images, allowing images with different color tones and image quality depending on the model to be learned together. This allows for effective use of images accumulated up to now. Existing program code or original program code can be used for this color correction. Examples of existing program code that can be used include histogram equalization, retinex algorithm, deep learning networks, color transfer, automatic white balance, gamma correction, color space conversion, and style transfer. It is also possible to use a hue correction tool in image processing software.

[0047] Image quality improvement processes include, for example, noise removal, image smoothing, blur removal, mosquito noise correction, sharpening, similar color improvement, chromatic aberration correction, and enlargement. These image quality improvement processes can speed up the generation of trained models (promote convergence) and also improve the diagnostic accuracy of trained models by standardizing the image quality between images. Existing program code or original program code can be used for these image quality improvement processes. Existing program code includes ESRGAN, CUGAN, Cycle GAN, TecoGAN, pix2pix, Super-Resolution Convolutional Neural Network, SRGAN, and CodeFormer. Additionally, noise processing tools and sharpening tools in image processing software can also be used.

[0048] The order in which the color correction unit 62 performs the color correction and the image quality improvement process is not limited, and either process may be performed first.

[0049] The affine transformation unit 63 acquires the corrected training image from the training image storage unit 52, performs an affine transformation, and stores the affine transformed training image in the training image storage unit 52 as a training image separate from the training image before the affine transformation.

[0050] Affine transformation involves rotating, enlarging / reducing, translating, skewing, and other image transformations. This makes it possible to generate simulated images with different shooting angles and different distances and positions between the tip of the endoscopic camera and the area to be diagnosed, without actually taking the image. Furthermore, by storing the images as separate images, the number of images used in machine learning can be increased, improving the diagnostic accuracy of the trained model.

[0051] The affine transformation unit 63 in the first embodiment is capable of arbitrarily setting the rotation angle, enlargement / reduction ratio, translation amount, and shear (skew) angle.

[0052] Each affine transformed training image is then given an identification number or name different from that of the training image before the affine transformation, and is stored in training image storage unit 52 as a different training image.

[0053] The trained model generation unit 64 generates a trained model by performing machine learning on a plurality of training images. In the first embodiment, only images that are known in advance not to include lesions (normal images) are used as training images. Generally, there are more normal images that do not include lesions than abnormal images that include lesions among the stored images, so it is possible to train a large number of images through machine learning.

[0054] Therefore, the trained model generation unit 64 generates a trained model using a machine learning algorithm used for image anomaly detection, enabling detection of image anomalies. For example, a machine learning algorithm used for image anomaly detection, such as Patchcore, is trained using multiple training images (normal images) as training data to generate a feature set (memory bank) of normal images (normal data). Then, multiple trained models (candidate models) are generated by changing the number of times (number of epochs) that a single training image (training data) is repeatedly trained. In addition to Patchcore, other machine learning algorithms that can be used for image anomaly detection include algorithms based on an AutoEncoding model, Attention-guidance, SPADE, PaDiM, etc.

[0055] Furthermore, the trained model generation unit 64 may perform normalization to improve the convergence speed of learning and the stability of the learning results. For example, normalization involves converting pixel values ​​using an equation such as: converted pixel value = (pixel value before conversion - average value) / ((standard deviation)). Furthermore, values ​​used in training using conventional methods can be used for the average value and standard deviation. In the first embodiment, the average value = [0.485, 0.456, 0.405] and the standard deviation = [0.229, 0.224, 0.225] are used.

[0056] The trained model determination unit 65 creates a trained model through training while checking the trained model generated by the trained model generation unit 64 with the verification data, determines the model with the highest accuracy rate in the verification data as the trained model (diagnostic model) to be used in the endoscopic diagnostic program 1a, and stores the trained model (diagnostic model) in the trained model memory unit 54.

[0057] The trained model determination unit 65 in the first embodiment inputs a verification image in which the presence or absence of a lesion has been verified for each of the multiple trained models (candidate models) generated by the trained model generation unit 64, and checks whether the result obtained as output matches the verification image (whether it is correct or not).The trained model (candidate model) with the highest accuracy rate of the confirmed output result is then determined as the trained model (diagnostic model) to be used for endoscopic diagnosis.

[0058] Next, each unit executed by the endoscopic diagnostic program 1a will be described.

[0059] The diagnostic image acquisition unit 66 acquires diagnostic images of the diagnostic target area photographed by the endoscope device 20. The diagnostic images can be acquired directly from the endoscope device 20, or diagnostic images stored in the diagnostic image storage unit 55 can be acquired.

[0060] The color correction unit 67 performs color correction on the diagnostic image (raw image) acquired from the endoscopic device 20 or the diagnostic image storage unit 55, and stores the corrected diagnostic image (color-corrected image) in the diagnostic image storage unit 55.

[0061] The color correction performed here is the same as the color correction of the training images used to generate the trained model, based on reference color tones, etc. By performing the same color correction on the diagnostic images as that used to generate the trained model, the image quality and color tones that differ depending on the training images and model are unified, improving diagnostic accuracy.

[0062] The lesion presence / absence diagnosis unit 68 inputs the diagnostic image corrected by the color correction unit 67 into the trained model stored in the trained model storage unit 54, and diagnoses the presence or absence of a lesion from the output result. In the first embodiment, when the output result shows that the feature amount of the diagnostic image is 50% or more, the image is determined to be abnormal and diagnosed as having a lesion, and the diagnostic result is output. On the other hand, when the output result shows that the feature amount of the diagnostic image is less than 50%, the image is determined to be normal and diagnosed as having no lesion, and the diagnostic result is output.

[0063] Next, the functions of each component of the endoscopic diagnostic device 1, the endoscopic diagnostic program 1a, and the endoscopic diagnosis trained model generation program 10a of this first embodiment will be described together with a control method for the endoscopic diagnostic device.

[0064] 2, in the endoscopic diagnosis trained model generation program 10a, the training image acquisition unit 61 acquires a plurality of training images (raw images) captured by an endoscopic device and stored in the training image storage unit 52 (training image acquisition step: S1). The acquired training images (raw images) may include images captured by different models of endoscopic devices.

[0065] Next, the color tone correction unit 62 acquires the reference color tone stored in the reference color tone storage unit 53 and performs color tone correction on each learning image to match the reference color tone (color tone correction step: S2). If the learning images include images captured with different models of endoscope devices, this process smooths the color tones and image quality between the learning images. The color tone correction unit 62 stores the color-corrected learning images in the learning image storage unit 52.

[0066] Next, the affine transformation unit 63 acquires the corrected learning image stored in the learning image storage unit 52, performs an arbitrary affine transformation on it, and stores it as another learning image in the learning image storage unit 52 (affine transformation step: S3). This makes it possible to generate pseudo images with different shooting angles and different distances and positions between the tip of the endoscopic camera and the diagnostic target area without actually taking images, and also increases the number of images to be machine-learned.

[0067] Next, the trained model generation unit 64 acquires multiple training images after correction and affine transformation, and trains each of the acquired training images using a machine learning algorithm to generate multiple trained models (trained model generation step: S4). Specifically, Patchcore, a machine learning algorithm used for image anomaly detection, trains the multiple training images as training data to generate a group of features for normal images. Then, multiple trained models (candidate models) are generated by changing the number of times a single training image is repeatedly trained.

[0068] The trained model determination unit 65 then inputs verification images, for which the presence or absence of a lesion has already been verified, into the multiple trained models generated, and determines the trained model with the highest accuracy rate as the trained model to be used for endoscopic diagnosis (trained model determination step: S5). Specifically, for each trained model (candidate model) generated by the trained model generation unit 64, an image containing a lesion (abnormal image) is input as a verification image, for which the presence or absence of a lesion has already been verified, and checks whether the image is output as an abnormal image. The trained model (candidate model) with the highest accuracy rate for outputting an abnormal image is determined to be the trained model (diagnostic model) to be used for endoscopic diagnosis. The trained model determined to be used for endoscopic diagnosis is then stored in the trained model storage unit 54.

[0069] This ends the execution of the program 10a for generating a trained model for endoscopic diagnosis. It is preferable to end the execution of the program 10a for generating a trained model for endoscopic diagnosis before the next diagnosis of the presence or absence of a lesion is performed by the program 1a for endoscopic diagnosis.

[0070] Next, the presence or absence of a lesion is diagnosed by the endoscope diagnostic program 1a.

[0071] 3, the diagnostic image acquisition unit 66 acquires a diagnostic image (raw image) of a diagnostic target region from the endoscope device 20 or the diagnostic image storage unit 55 (diagnostic image acquisition step: SS1). The diagnostic image (raw image) acquired at this time may be from an endoscope device of a model different from that of the endoscope device that captured the learning image used to generate the trained model.

[0072] Next, the color correction unit 67 acquires the reference color tone stored in the reference color tone storage unit 53 and performs the same color correction (including image quality improvement processing) on ​​the diagnostic image as on the training image (color correction step: SS2). By performing the same color correction as on the training image used to generate the trained model, the color tone and image quality are unified with those of the training image, enabling highly accurate diagnostic processing even for images captured with a different model of endoscope device 20 from that used for the training image.

[0073] Then, lesion presence / absence diagnosis unit 68 inputs the corrected diagnostic image into the trained model stored in trained model storage unit 54, and diagnoses the presence or absence of a lesion from the output result (lesion presence / absence diagnosis steps: SS3, SS4). Specifically, the corrected diagnostic image is input into the trained model, and the feature amount of the diagnostic image is output (SS3). Then, if the feature amount of the diagnostic image is output as an output result of 50% or more (SS4: YES), the image is determined to be abnormal and diagnosed as having a lesion, and this result is output to display means 2, etc. On the other hand, if the feature amount of the diagnostic image is output as an output result of less than 50% (SS4: NO), the image is determined to be normal and diagnosed as having no lesion, and this result is output to display means 2, etc.

[0074] This completes the diagnosis. Note that the diagnosis results are not limited to those displayed on the display means 2, but may be output to the outside by output means such as a printer.

[0075] According to the first embodiment described above, the following advantageous effects can be achieved. 1. By performing color correction and image quality improvement processing on the training images that have been accumulated up to now, the color tone and image quality of the images can be smoothed out, improving the diagnostic accuracy of the trained model. 2. By using an anomaly detection algorithm in the algorithm that generates the trained model, it is possible to use images that do not contain lesions as training images, thereby increasing the number of images used for machine learning and improving diagnostic accuracy. and 3. By performing an affine transformation on the training images, it is possible to generate simulated images with different shooting angles and different distances and positions between the tip of the endoscopic camera and the area to be diagnosed without actually taking the images. By storing these as separate images, the number of images used in machine learning can be increased, improving the diagnostic accuracy of the trained model. 4. By performing the same color correction and image quality improvement processing on diagnostic images as on training images, the color tone and image quality of the training images used to generate the trained model are unified with those of the training images, thereby improving the diagnostic accuracy of the trained model.

[0076] Next, a second embodiment of the endoscopic diagnostic program, the endoscopic diagnostic device, the control method for the endoscopic diagnostic device, and the endoscopic diagnosis trained model generation program according to the present invention will be described with reference to the drawings. Note that components that are the same as or equivalent to those described in the first embodiment will be assigned the same reference numerals and will not be described again.

[0077] In the second embodiment, not only images that do not include lesions are used as learning images, but also images that include lesions. By using images that include lesions, it is possible to more effectively utilize the images that have been accumulated up to now.

[0078] In the second embodiment, if a learning image contains a lesion, the color correction unit 62 performs color correction using a reference color tone on the range other than the range where the lesion is present. By performing color correction separately on the range where the lesion is present (abnormal range) and the range other than the range where the lesion is present (normal range), it becomes easier to distinguish between the abnormal range and the normal range during machine learning, and by performing uniform color correction using the reference color tone on the normal range, images with different color tones and image quality are smoothed, thereby improving the diagnostic accuracy of the generated trained model.

[0079] The separation of the area where the lesion exists (abnormal area) from the area where the lesion does not exist (normal area) can be done using an existing program that automatically detects boundaries within the image, or by manually separating the areas using pen input, etc. Although separating the areas requires more effort than color correction for the entire area, it makes it easier to distinguish between abnormal and normal areas, which can improve the accuracy of diagnosis.

[0080] Furthermore, in order to handle both types of images, the trained model generation unit 64 in the second embodiment generates a trained model by performing machine learning on the corrected training images (including training images after affine transformation) using an algorithm that detects image features. The algorithms for acquiring features from images include SIFT (Scale-Invariant Feature Transform) and SURE (Speed ​​Up Robust Feature). Algorithms based on ORB (Oriented Fast and Rotated BRIDF), CNN (Covolutional Neural Network), etc. can be used.

[0081] The lesion presence / absence diagnosing unit 68 in the second embodiment inputs a diagnostic image into the generated trained model, diagnoses the presence of a lesion when the feature amount of the diagnostic image is determined to be equal to or greater than a predetermined value, and outputs the result to the display means 2, etc. On the other hand, when the feature amount of the diagnostic image is determined to be less than the predetermined value, it diagnoses the absence of a lesion, and outputs the result to the display means 2, etc.

[0082] As a result, in this second embodiment, it becomes possible to diagnose the presence or absence of a lesion using accumulated images regardless of whether or not a lesion is reflected, thereby achieving the advantageous effect of making more effective use of images accumulated to date. [Example]

[0083] In Example 1, an endoscopic diagnostic program according to the present invention was created, and the accuracy of diagnosing the presence or absence of a lesion was verified using diagnostic images (endoscopic images) captured by an endoscopic device.

[0084] <Endoscopic device> The endoscope devices used for the verification of this Example 1 were three models: CF-H290ECI manufactured by Olympus Corporation, CF-FH260AZI manufactured by Olympus Corporation, and EC-L600ZP7 manufactured by Fujifilm Corporation.

[0085] <Diagnostic Imaging> The diagnostic images used in the verification of this Example 1 are images of the inside of the large intestine as the diagnostic target area. Each image is previously sorted by a doctor into images of the large intestine in a normal state without any lesions such as inflammation or ulcers (hereinafter referred to as "normal images") and images of the large intestine in an abnormal state with any lesions (hereinafter referred to as "abnormal images").

[0086] In the following, diagnostic images taken with the Olympus CF-H290ECI will be referred to as "Olympus 290 images," diagnostic images taken with the Olympus CF-FH260AZI will be referred to as "Olympus 260 images," and diagnostic images taken with the Fujifilm EC-L600ZP7 will be referred to as "Fuji 600 images."

[0087] <Software used for color correction> In the verification of this Example 1, Photoshop (registered trademark), an image processing software manufactured by Adobe, was used to determine the reference color tones for color correction of diagnostic images and to perform color correction of diagnostic images based on the reference color tones.

[0088] <Determining the reference color tone (reference image)> In this Example 1, the reference color tone (reference image) was determined by the following procedure. Step 1) Randomly select five normal images from diagnostic images taken with one endoscopic device. Step 2) Select one representative image from the five diagnostic images extracted in step 1. Step 3) Perform "Color Correction" (a Photoshop function) on the representative image selected in Step 2, using the other four images as source images. Step 4) The representative image after step 3 is determined as the reference color tone (reference image) and saved.

[0089] The method for selecting the representative image in step 2 can be arbitrary, but in this Example 1, the image in which the lumen is most centrally captured was used as the representative image.

[0090] In addition, in step 3, (1) brightness, (2) color adaptability, and (3) fade can be set arbitrarily between 0 and 100, but in this Example 1, they were set to (1) 100, (2) 100, and (3) 0 (default setting values), respectively.

[0091] In this Example 1, the reference image determined using the "Olympus 290 image" was referred to as the "Olympus reference image," and the reference image determined using the "Fuji 600 image" was referred to as the "Fuji reference image."

[0092] <Color correction based on reference color (creating learning images)> In this Example 1, color correction is performed on the diagnostic image by performing "color correction" on the diagnostic image using a reference image as the source image. For example, when performing color correction on a "Fuji 600 image" to match it with an "Olympus reference image," the "Fuji 600 image" is subjected to "color correction" using the "Olympus reference image" as the source image. In this case, (1) brightness, (2) color adaptability, and (3) fade are set to (1) 100, (2) 100, and (3) 0 (default settings), respectively, as in Step 3.

[0093] <Generating a trained model> In this Example 1, Patchcore, a machine learning algorithm used for detecting anomalies in images, was used to generate the trained model. Specifically, a dataset of images to be verified was divided into learning images (training data) and test data, and machine learning was performed using Patchcore to generate the trained model.

[0094] However, it is possible that errors may occur in the diagnostic accuracy of the generated trained model depending on which data in the dataset are used as training images and which data are used as test data. Therefore, in this Example 1, a method called stratified k-fold cross validation was adopted.

[0095] As shown in Fig. 4, this stratified k-fold cross-validation is a method in which a dataset (image data group) is divided into multiple image data groups, one of which is used as a diagnostic image and the other image data groups are used as training images, and trained models are generated for each division, the diagnostic accuracy is verified, and the average value is calculated. Although the verification takes k times longer, it is possible to suppress errors in diagnostic accuracy due to the division method. In this Example 1, stratified 5-fold cross-validation with k = 5 was used.

[0096] <Method for verifying diagnostic accuracy> The diagnostic accuracy is evaluated by inputting a diagnostic image into the generated trained model to determine whether the diagnostic image is a normal image or an abnormal image. In this Example 1, the five trained models generated by stratified 5-fold cross-validation were used to discriminate between diagnostic images (validation data: all of the training images and test data used to generate the trained models), and the accuracy rate for each model and the average accuracy rate of the five accuracy rates were calculated.

[0097] The verification results will be explained below.

[0098] <Verification results using Olympus 290 images> First, we performed a verification using images taken with the Olympus 290. The Olympus 290 images used for the verification consisted of 287 images in total, including 151 abnormal images and 136 normal images.

[0099] Table 1 below shows the results of stratified 5-fold cross-validation using Olympus 290 images. In this table, the first item from the left is the fold number, the second is the loss function Loss based on the training data (training images and test data), the third is the accuracy rate based on the training data, the fourth is the loss coefficient Loss based on the diagnostic images (validation data), and the fifth is the accuracy rate for the diagnostic images. [Table 1] TIFF0007778334000001.tif63150

[0100] As shown in Table 1, the accuracy rate for the diagnostic images (verification data) for each division number was approximately 0.72 to approximately 0.82, and the average accuracy rate was 0.791.

[0101] <Verification results using Olympus 260 images> Next, we will explain the verification using Olympus 260 images. The Olympus 260 images used for the verification were 239 in total, including 108 abnormal images and 131 normal images. The verification results are shown in Table 2 below. [Table 2] TIFF0007778334000002.tif65150

[0102] As shown in Table 2, the accuracy rate for diagnostic images (verification data) for each division number was approximately 0.58 to approximately 0.77, and the average accuracy rate was 0.695. Compared to the Olympus 290 images, the accuracy rate for each division number varied more widely, and the average accuracy rate was also lower.

[0103] <Verification results using Fuji 600 images> Next, we will explain the verification using Fuji 600 images. The Fuji 600 images used for the verification were 232 in total, including 108 abnormal images and 124 normal images. The verification results are shown in Table 3 below. [Table 3] TIFF0007778334000003.tif66150

[0104] As shown in Table 3, the accuracy rate for diagnostic images (verification data) for each division number was approximately 0.65 to approximately 0.87, with an average accuracy rate of 0.764. Although there was a greater variation in accuracy rates for each division number compared to the Olympus 290 images, the average accuracy rate was relatively high at 0.76.

[0105] To summarize the above results, the average accuracy rate of the diagnostic results using images taken with each model was highest in the order of Olympus 290 image > Fuji 600 image > Olympus 260 image. In this Example 1, the reference color tone (reference image) for color correction was selected from those with the highest average accuracy rate, and as described above, the "Olympus reference image" using the "Olympus 290 image" and the "Fuji reference image" using the "Fuji 600 image" were selected.

[0106] <Verification using Olympus 290 and Olympus 260 images> Next, we will examine the case where images taken by different models of endoscope devices are used together.

[0107] First, validation was performed using Olympus 290 images and Olympus 260 images. Stratified 5-fold cross validation was performed using a dataset that combined the Olympus 290 images and the Olympus 260 images (without color correction).

[0108] The Olympus 290 images used for the verification were 152 abnormal images and 137 normal images, totaling 289, while the Olympus 260 images were 109 abnormal images and 133 normal images, totaling 242. Therefore, adding up both images, there were 261 abnormal images and 270 normal images, totaling 631.

[0109] The validation results based on a dataset combining Olympus 290 images and Olympus 260 images (without color correction) are shown in Table 4 below. [Table 4] TIFF0007778334000004.tif52134

[0110] As shown in Table 4, the accuracy rate for the diagnostic images (verification data) for each division number ranged from about 0.74 to about 0.86, and the average accuracy rate was 0.813.

[0111] Compared to the average accuracy rate of 0.791 when testing Olympus 290 images alone, the average accuracy rate was 0.813, an improvement in accuracy. This is thought to be due to the fact that the amount of data used for machine learning was approximately doubled compared to when testing Olympus 290 images alone, improving the accuracy of the trained model. Meanwhile, the accuracy rates for diagnostic images (test data) for each division number ranged from approximately 0.74 to approximately 0.86, showing some variation.

[0112] <Verification using Olympus 290 images and color-corrected Olympus 260 images> Next, to confirm the effectiveness of color correction, we performed stratified 5-fold cross-validation using a dataset that combined an Olympus 290 image with an Olympus 260 image that had been color-corrected to match the color tone of the Olympus 290 image.

[0113] The number of Olympus 290 images and Olympus 260 images used in the verification was the same as that used in the verification based on the results in Table 4. On the other hand, in order to match the color tone of the Olympus 260 images to that of the Olympus 290 images, color correction was performed on all Olympus 260 images using the Olympus reference image as the source image.

[0114] The validation results based on a dataset combining Olympus 290 images and color-corrected Olympus 260 images are shown in Table 5 below. [Table 5] TIFF0007778334000005.tif52134

[0115] As shown in Table 5, the accuracy rate for diagnostic images (verification data) for each division number was approximately 0.81 to approximately 0.84, with little variation in accuracy rate. By adjusting the color tone through color correction, it is believed that a uniform trained model could be generated regardless of the data used for training.

[0116] In addition, the average accuracy rate was 0.827, which was an improvement over the average accuracy rate of 0.813 when Olympus 260 images were not color-corrected.

[0117] Therefore, when endoscopic images taken with different models of endoscopic devices are used together, it was confirmed that performing color correction to match the color tone of one of the models improves diagnostic accuracy compared to when color correction is not performed.

[0118] <Test using Fuji 600 images, Olympus 290 images (without color correction), and Olympus 260 images (without color correction)> Next, we conducted verification by combining endoscopic images taken with endoscopic devices from different manufacturers.

[0119] The Fuji 600 images used in the verification were 234 in total, with 109 abnormal images and 125 normal images, the Olympus 290 images were 289 in total, with 152 abnormal images and 137 normal images, and the Olympus 260 images were 242 in total, with 109 abnormal images and 133 normal images. Therefore, adding up all the images, there were 370 abnormal images and 395 normal images, for a total of 765.

[0120] A stratified 5-fold cross-validation was performed using a dataset consisting of the Fuji 600 image, the Olympus 290 image (without color correction), and the Olympus 260 image (without color correction). The validation results are shown in Table 6 below. [Table 6] TIFF0007778334000006.tif52136

[0121] As shown in Table 6, the average accuracy rate for diagnostic images (validation data) was 0.815, which is an improvement over the average accuracy rate of 0.764 when only Fuji 600 images were used. This improvement in accuracy is thought to be due to the fact that the amount of data used for machine learning increased by about three times, just like when Olympus 290 images and Olympus 260 images (without color correction) were combined, improving the accuracy of the trained model.

[0122] <Testing using Fuji 600 images, color-corrected Olympus 290 images, and color-corrected Olympus 260 images> Next, to verify the effectiveness of color correction, a dataset was created by combining the Fuji 600 image, the color-corrected Olympus 290 image, and the color-corrected Olympus 260 image, and a stratified 5-fold cross-validation was performed. The validation results are shown in Table 7 below. [Table 7] TIFF0007778334000007.tif52133

[0123] As shown in Table 7, the average accuracy rate for diagnostic images (verification data) was 0.821, which was an improvement over the average accuracy rate of 0.764 when only Fuji 600 images were used and the average accuracy rate of 0.815 when no color correction was performed.

[0124] <Verification using Fuji 600 images and color-corrected Olympus 290 images> We also performed stratified 5-fold cross-validation using a dataset that combined Fuji 600 images and Olympus 290 images that had been color-corrected based on the Fuji reference images. The validation results are shown in Table 8 below. [Table 8] TIFF0007778334000008.tif51133

[0125] As shown in Table 8, the accuracy rate for the diagnostic images (verification data) for each division number was all 0.8 or higher, and the average accuracy rate was also high at 0.843.

[0126] <Verification using Fuji 600 images and color-corrected Olympus 260 images> Furthermore, we also performed stratified 5-fold cross-validation using a dataset that combined Fuji 600 images and Olympus 260 images that had been color-corrected based on Fuji reference images. As shown in Table 9 below, the average accuracy rate was 0.845, which was a high accuracy rate. [Table 9] TIFF0007778334000009.tif51133

[0127] From the above, it has been confirmed that, according to the present invention, regardless of the manufacturer or model of the endoscopic device, by performing color correction on diagnostic images and learning images that have been accumulated to date, the color tones between the images can be smoothed, thereby improving the diagnostic accuracy of the trained model.

[0128] <Verification of trained model using only normal images> It is possible that the images that can be collected as training images for machine learning will be mostly normal images that do not contain any lesions. Therefore, we verified a trained model generated using only normal images as training data.

[0129] The images used for the verification were 371 in total: 115 normal Fuji 600 images, 131 normal Olympus 290 images, and 125 normal Olympus 260 images.

[0130] Here, we verified (1) the accuracy of the trained model when machine learning was performed on a dataset that combined Fuji 600 images, Olympus 290 images, and normal Olympus 260 images, (2) the accuracy of the trained model when machine learning was performed on a dataset that combined Fuji 600 images, color-corrected Olympus 290 images, and normal color-corrected Olympus 260 images, (3) the accuracy of the trained model when machine learning was performed on only normal Fuji 600 images, (4) the accuracy of the trained model when machine learning was performed on only normal Olympus 260 images, and (5) the accuracy of the trained model when machine learning was performed on only normal Olympus 290 images.

[0131] As indicators for evaluating the accuracy verification, we calculated the average accuracy rate, as well as the AUROC (Area Under the ROC Curve), which is one of the evaluation indicators for binary classification problems in machine learning, and the F-measure (F-measure), which is one of the evaluation indicators for binary classification problems and is the harmonic mean of precision and recall. AUROC takes a value between 0 and 1, and the closer it is to 1, the higher the model's predictive performance. F-measure also takes a value between 0 and 1, and the closer it is to 1, the better the machine learning model is balanced, achieving both precision and recall. The verification results for each are shown in Table 10 below. [Table 10] TIFF0007778334000010.tif44148

[0132] As shown in Table 10, higher values ​​were obtained for AUROC, F-value, and average correct answer rate when color correction was performed to match the Fuji reference image (2) compared to when all normal images (1) were added together. In addition, the average correct answer rate of the trained model when machine learning was performed on only normal images of the Olympus 290 image (5) was more accurate than the result when color correction (2) was performed, but in all other cases, the accuracy was higher when color correction (2) was performed.

[0133] From the above, it was confirmed that the accuracy of a trained model trained on machine learning of only normal images from endoscopic images taken with endoscopic devices of different models and manufacturers can be improved by matching the color tone with color correction. Therefore, even if the number of normal images that do not show lesions increases in the future as images that can be collected for training, it was confirmed that the accuracy of the trained model used to diagnose the presence or absence of lesions can be improved.

[0134] The endoscopic diagnostic program, endoscopic diagnostic device, endoscopic diagnostic device control method, and endoscopic diagnosis trained model generation program according to the present invention are not limited to the above-described embodiments and may be modified as appropriate. For example, the trained model generation unit 64 may further improve the diagnostic accuracy of the trained model generated by ensemble learning. Furthermore, if the image sizes of the training image and the diagnostic image are different, the color correction units 62 and 67 may perform a process to unify the images to a uniform size. [Explanation of symbols]

[0135] 1 Endoscopic diagnostic equipment 1a Endoscopic diagnostic program 2 Display means 3. Input Methods 4. Computer 5 Memory means 6. Processing means 10a Program for generating trained models for endoscopic diagnosis 20 Endoscopic device 21 Endoscopic Camera 51 Program memory section 52 Learning image memory unit 53 Reference color tone storage section 54 Trained model memory 55 Diagnostic image storage unit 61 Learning image acquisition unit 62 Color tone correction section 63 Affine transformation section 64 Trained model generation unit 65 Trained model determination unit 66 Diagnostic image acquisition unit 67 Color tone correction section 68 Lesion Presence Diagnosis Section

Claims

1. An endoscopic diagnostic program for diagnosing the presence or absence of a lesion based on an endoscopic image captured by an endoscopic device, comprising: a diagnostic image acquisition unit that acquires a diagnostic image of a diagnostic target region photographed by an endoscope device; a color correction unit that acquires a reference color tone by analyzing an image of a diagnostic target region captured in advance by an endoscope device, and corrects the color tone of the diagnostic image in accordance with the acquired reference color tone; a lesion presence / absence diagnosis unit that inputs the corrected diagnostic images into a trained model generated by machine learning a plurality of learning images of a diagnostic target region photographed by a plurality of types of endoscope devices and color-toned using the reference color tones, and diagnoses the presence or absence of a lesion from the output result; The endoscopic diagnostic program causes a computer to function as follows.

2. the color correction unit performs the color correction and image quality improvement processing on the diagnostic image; The endoscopic diagnostic program according to claim 1, wherein the lesion presence / absence diagnosis unit diagnoses the presence or absence of a lesion using the trained model generated by machine learning the training image that has been subjected to the same color correction and image quality improvement processing as the color correction unit.

3. 2. The endoscopic diagnostic program according to claim 1, wherein the lesion presence / absence diagnosis unit inputs the corrected diagnostic image into a trained model generated by machine learning a plurality of the training images that have been corrected with the reference color tone and do not reflect any lesions into an image abnormality detection algorithm, and diagnoses the presence of a lesion when an abnormality is detected in the diagnostic image as an output result.

4. 2. The endoscopic diagnostic program according to claim 1, wherein the lesion presence / absence diagnosing unit inputs the corrected diagnostic image into a trained model generated by machine learning an algorithm that detects image features using a plurality of the training images that have been corrected with the reference color tone and that do not reflect lesions, and a plurality of the training images that have been corrected with the reference color tone and that reflect lesions, and as an output result, diagnoses the presence of a lesion when the feature amount of the diagnostic image is determined to be equal to or greater than a predetermined value, and diagnoses the absence of a lesion when the feature amount of the diagnostic image is determined to be less than the predetermined value.

5. An endoscopic diagnostic device that diagnoses the presence or absence of a lesion based on an endoscopic image captured by an endoscopic device, a diagnostic image acquisition unit that acquires a diagnostic image of a diagnostic target region photographed by an endoscope device; a color correction unit that acquires a reference color tone by analyzing an image of a diagnostic target region captured in advance by an endoscope device, and corrects the color tone of the diagnostic image in accordance with the acquired reference color tone; a lesion presence / absence diagnosis unit that inputs the corrected diagnostic images into a trained model generated by machine learning the plurality of training images obtained by photographing a diagnostic target site using a plurality of types of endoscope devices and color-correcting the plurality of training images using the reference color tones, and performs a diagnostic process for determining the presence or absence of a lesion based on the output result; and

6. A control method for an endoscopic diagnostic device that diagnoses the presence or absence of a lesion based on an endoscopic image captured by the endoscopic device, comprising: a diagnostic image acquisition step of acquiring a diagnostic image of a diagnostic target region photographed by an endoscope device; a color correction step of acquiring a reference color tone obtained by analyzing an image of a diagnostic target region captured in advance by an endoscope device, and correcting the color tone of the diagnostic image in accordance with the acquired reference color tone; a lesion presence / absence diagnosis step in which the corrected diagnostic images are input to a trained model generated by machine learning a plurality of learning images of a diagnostic target site photographed by a plurality of types of endoscope devices, the plurality of learning images having color tones corrected using the reference color tones, and the corrected diagnostic images are input to a trained model, and the presence / absence of a lesion is diagnosed based on the output result; and a control method for the endoscopic diagnostic apparatus, comprising:

7. A program for generating a trained model for endoscopic diagnosis that generates a trained model used in endoscopic diagnosis for diagnosing the presence or absence of a lesion based on an endoscopic image captured by an endoscopic device, a learning image acquisition unit that acquires a plurality of learning images of a diagnostic target region photographed by a plurality of types of endoscope devices; a color correction unit that acquires a reference color tone by analyzing an image of a diagnostic target region captured in advance by an endoscope device, and performs color correction on each of the learning images in accordance with the acquired reference color tone; a trained model generation unit that performs machine learning on each of the training images that have been color-toned by the color correction unit using a machine learning algorithm to generate a plurality of trained models; a trained model determination unit that inputs a verification image, the presence or absence of which has been verified, into each of the generated trained models and determines the trained model with the highest accuracy rate as the trained model to be used for endoscopic diagnosis; The endoscopic diagnosis trained model generation program causes a computer to function as follows.

8. The program for generating a trained model for endoscopic diagnosis according to claim 7, wherein the color correction unit, when a lesion is reflected in the training image, performs color correction using the reference color tone on an area other than the area where the lesion exists.

9. causing a computer to function as an affine transformation unit that performs an arbitrary affine transformation on the learning image corrected by the color tone correction unit; The trained model generation program for endoscopic diagnosis according to claim 7, wherein the trained model generation unit performs machine learning on the training images corrected by the color correction unit and the training images transformed by the affine transformation unit to generate a trained model.

Images