Estimation device, estimation method, learning model, learning model generation method, and computer program

Abstract

To provide an estimation device, a learning model, a method for generating a learning model, and a computer program.SOLUTION: An estimation device includes: an acquisition part for acquiring an oral cavity image that can be obtained by imaging an oral cavity of a subject; an estimation part for estimating whether there is any lesion in an oral mucous membrane of the subject from the oral cavity image acquired by the acquisition part using a learning model that is configured to output information relating to a lesion in an oral mucous membrane with respect to input of the oral cavity image; and an output part for outputting an estimated result by the estimation part.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention provides an estimation device, Estimation method, The present invention relates to a learning model, a method for generating a learning model, and a computer program. [Background technology]

[0002] Oral mucosal diseases include various diseases such as oral malignant tumors and stomatitis, but diagnosis is often difficult. In particular, early-stage oral malignant tumors often present clinical findings similar to stomatitis, and are often overlooked by medical professionals who are not specialists. Research into causative genes and prognostic factors for oral malignant tumors is currently being conducted at many facilities, and the inventors have also reported on pathological prognostic predictors for oral malignant tumors (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Al-Shareef H, Hiraoka S, et al. ONCOLOGY REPORTS:2016 Summary of the Invention [Problem to be solved by the invention]

[0004] However, research in the oral cavity field is lagging behind other fields, and there is no prospect of establishing a simple diagnostic support system that can be put into practical use.

[0005] The present invention provides an estimation device capable of estimating lesions in the oral mucosa, Estimation method, The present invention aims to provide a learning model, a method for generating a learning model, and a computer program. [Means for solving the problem]

[0006] An estimation device according to one aspect of the present invention includes an acquisition unit that acquires oral images obtained by imaging the inside of the oral cavity of a subject, an estimation unit that estimates the presence or absence of lesions in the oral mucosa of the subject from the oral images acquired by the acquisition unit using a learning model configured to output information regarding lesions in the oral mucosa in response to input of the oral images, and an output unit that outputs the estimation results of the estimation unit.

[0007] A learning model according to one aspect of the present invention comprises an input layer to which oral images obtained by imaging the inside of a subject's oral cavity are input, an output layer that outputs information relating to lesions in the oral mucosa, and an intermediate layer that uses the oral images and annotations for the oral images as training data to learn the relationship between the oral images input to the input layer and the information output by the output layer. When an oral image is input to the input layer, the intermediate layer performs a calculation, causing a computer to output information relating to lesions in the oral mucosa from the output layer.

[0008] A method for generating a learning model according to one aspect of the present invention uses a computer to acquire training data including oral images obtained by capturing images of the inside of a subject's oral cavity and annotations for the oral images, and generates a learning model that outputs information regarding lesions in the oral mucosa in response to input oral images based on the acquired training data.

[0009] A computer program according to one aspect of the present invention is a computer program for causing a computer to acquire oral images obtained by capturing images of the inside of a subject's oral cavity, and to execute a process of estimating the presence or absence of lesions in the oral mucosa from the acquired oral image using a learning model configured to output information regarding lesions in the oral mucosa in response to input of the oral image, and outputting the estimation result. [Effects of the Invention]

[0010] According to the present application, lesions in the oral mucosa can be estimated. [Brief explanation of the drawings]

[0011] [Figure 1]1 is a block diagram illustrating a configuration of an estimation device according to a first embodiment. [Figure 2] FIG. 2 is a schematic diagram illustrating an example of an oral cavity image. [Figure 3] FIG. 1 is a schematic diagram illustrating an example of the configuration of a learning model. [Figure 4] 4 is a flowchart illustrating a procedure of processing executed by the estimation device according to the first embodiment. [Figure 5] FIG. 10 is a schematic diagram illustrating an example of an output from the estimation device. [Figure 6] 10 is a flowchart illustrating a procedure of processing executed by an estimation device 1 according to a second embodiment. [Figure 7] FIG. 10 is a block diagram illustrating the configuration of an estimation device according to a third embodiment. [Figure 8] FIG. 10 is a schematic diagram showing an extraction example. [Figure 9] 10 is a flowchart illustrating a procedure of processing executed by an estimation device according to a third embodiment. [Figure 10] FIG. 2 is a block diagram illustrating a configuration of a server device. [Figure 11] FIG. 1 is a conceptual diagram illustrating an example of an oral cavity image database. [Figure 12] 10 is a flowchart illustrating a procedure for generating a learning model. [Figure 13] FIG. 10 is a block diagram illustrating the configuration of an estimation device according to a fifth embodiment. [Figure 14] FIG. 10 is a schematic diagram illustrating an example of an enlarged image. [Figure 15] FIG. 1 is a schematic diagram illustrating an example of the configuration of a learning model. [Figure 16] FIG. 13 is a conceptual diagram showing an example of an oral cavity image database according to the fifth embodiment. [Figure 17] 10 is a flowchart illustrating a procedure for generating a learning model. [Figure 18] 10 is a flowchart showing an estimation procedure using a learning model. [Figure 19] FIG. 10 is a schematic diagram showing an example of presentation of an observation region. [Figure 20]FIG. 10 is a schematic diagram showing an example of an interface screen for accepting the designation of a target region. DETAILED DESCRIPTION OF THE INVENTION

[0012] The present invention will now be described in detail with reference to the drawings showing embodiments thereof. (Embodiment 1) FIG. 1 is a block diagram illustrating the configuration of an estimation device 1 according to embodiment 1. The estimation device 1 is a computer device installed in a facility such as a hospital, and estimates the presence or absence of a lesion in the oral mucosa of a subject from an oral cavity image obtained by capturing an image of the oral cavity of the subject. The estimation device 1 provides diagnostic support by presenting the estimation results to a diagnosing person such as a doctor.

[0013] The estimation device 1 includes an input unit 11, a control unit 12, a storage unit 13, an output unit 14, a communication unit 15, and an operation unit 16.

[0014] The input unit 11 includes an input interface to which image data of the oral cavity image is input. The input interface is, for example, an interface for connecting an imaging device for capturing images of the oral cavity of a subject. The imaging device is a digital camera or a digital video camera, and outputs image data in which each pixel is represented by an RGB gradation value, for example. The input unit 11 acquires image data of the oral cavity image from the imaging device connected to the input interface. The input interface may be an interface for accessing a recording medium on which captured image data is recorded. The input unit 11 acquires image data related to the oral cavity image by reading the image data recorded on the recording medium. The image data acquired by the input unit 11 is output to the control unit 12 and stored in the memory unit 13 via the control unit 12.

[0015] The control unit 12 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory). The ROM included in the control unit 12 stores a control program that controls the operation of each hardware component included in the estimation device 1. The CPU in the control unit 12 executes the control program stored in the ROM and various computer programs stored in the storage unit 13 (described later) to control the operation of each hardware component, thereby realizing the function of estimating the presence or absence of a lesion in the oral mucosa from an image of the oral cavity. The RAM included in the control unit 12 temporarily stores data used during the execution of calculations.

[0016] Although the control unit 12 is configured to include a CPU, a ROM, and a RAM, it may also be one or more arithmetic circuits including a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, volatile or non-volatile memory, etc. Furthermore, the control unit 12 may also have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when an instruction to start measurement is given until when an instruction to end measurement is given, and a counter that counts numbers.

[0017] The storage unit 13 includes a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an electronically erasable programmable read-only memory (EEPROM), etc. The storage unit 13 stores a computer program executed by the control unit 12, a learning model 130 used in the process of estimating the presence or absence of a lesion in the oral mucosa, etc.

[0018] The computer program stored in the storage unit 13 includes an estimation processing program P1 for causing the estimation device 1 to execute a process of estimating the presence or absence of a lesion in the oral mucosa from the acquired oral cavity image using the learning model 130.

[0019] The computer programs stored in the storage unit 13 may be provided by a non-transitory storage medium M1 on which the computer programs are readably recorded. The storage medium M1 is, for example, a portable memory such as a CD-ROM, a USB memory, an SD (Secure Digital) card, a micro SD card, or a CompactFlash (registered trademark). The control unit 12 reads the various programs from the storage medium M1 via, for example, the input unit 11, and installs the read various programs into the storage unit 13.

[0020] The learning model 130 is configured to output information about lesions in the oral mucosa in response to input of an oral cavity image. The learning model 130 is described by its definition information. The definition information of the learning model 130 includes structural information about the learning model 130, various parameters used in the learning model 130, such as weights and biases between nodes, and the like. In this embodiment, the learning model 130 is trained in advance by a predetermined learning algorithm using oral cavity images and annotations for these oral cavity images as training data, and is stored in the storage unit 13.

[0021] The control unit 12 executes the estimation processing program P1 stored in the memory unit 13, and provides image data of the oral cavity image to the learning model 130, thereby acquiring information about the lesion from the learning model 130. The control unit 12 estimates the presence or absence of a lesion in the oral mucosa based on the information about the lesion acquired from the learning model 130.

[0022] The output unit 14 includes an output interface for connecting an output device. An example of the output device is a display device 140 including a liquid crystal panel or an organic EL (Electro-Luminescence) panel. When outputting the estimation result, the control unit 12 generates display data to be displayed on the display device 140, and outputs the generated display data to the display device 140 via the output unit 14, thereby causing the display device 140 to display the estimation result.

[0023] The communication unit 15 includes a communication interface for transmitting and receiving various types of data. The communication interface included in the communication unit 15 is, for example, a communication interface conforming to a LAN (Local Area Network) communication standard used in WiFi (registered trademark) or Ethernet (registered trademark). When data to be transmitted is input from the control unit 12, the communication unit 15 transmits the data to be transmitted to a specified destination. Furthermore, when the communication unit 15 receives data transmitted from an external device, it outputs the received data to the control unit 12.

[0024] The operation unit 16 is equipped with an input interface such as various operation buttons, switches, a touch panel, etc., and receives various operation information and setting information. The control unit 12 performs appropriate control based on the operation information input from the operation unit 16, and stores the setting information in the storage unit 13 as necessary.

[0025] Next, the oral cavity image input to the estimation device 1 will be described. Fig. 2 is a schematic diagram showing an example of an oral cavity image. The oral cavity image in this embodiment is an image obtained by capturing an image of the inside of the oral cavity of a subject using an imaging device. The example of Fig. 2 shows an oral cavity image captured so as to include the left side of the tongue of the subject.

[0026] In this embodiment, it is sufficient that at least a portion of the oral mucosa is included in the oral cavity image. The oral mucosa includes at least a portion of the subject's tongue, upper lip, hard palate, soft palate, uvula, tonsils, buccal mucosa, floor of mouth, gums, and lower lip. The oral cavity image may also include objects other than the oral mucosa, such as the subject's teeth, the photographer's fingers, and other structures.

[0027] Next, the learning model 130 used in the estimation device 1 will be described. 3 is a schematic diagram showing an example of the configuration of the learning model 130. The learning model 130 is, for example, a learning model based on CNN (Convolutional Neural Networks), and includes an input layer 131, an intermediate layer 132, and an output layer 133. The learning model 130 is trained in advance to output information related to lesions in the oral mucosa in response to input of an oral cavity image.

[0028] Image data of an oral cavity image is input to the input layer 131. The image data of an oral cavity image input to the input layer 131 is sent to the intermediate layer 132.

[0029] The intermediate layer 132 is composed of, for example, a convolutional layer 132a, a pooling layer 132b, and a fully connected layer 132c. A plurality of convolutional layers 132a and pooling layers 132b may be provided alternately. The convolutional layer 132a and the pooling layer 132b extract features of the oral cavity image input through the input layer 131 by performing calculations using the nodes of each layer. The fully connected layer 132c combines the data from which feature portions have been extracted by the convolutional layer 132a and the pooling layer 132b into one node and outputs feature variables transformed by an activation function. The feature variables are output to the output layer 133 through the fully connected layer 132c.

[0030] The output layer 133 includes one or more nodes. Based on the feature variables input from the fully connected layer 132c of the intermediate layer 132, the output layer 133 converts the feature variables into probabilities using a softmax function, and outputs the probability that the oral image belongs to each category from each node. That is, in this embodiment, the probability that the oral image belongs to each category is output as information about the lesion. The categories for classifying the oral images can be arbitrarily set to include at least one lesion belonging to oral malignant tumors, precancerous lesions, benign tumors, traumatic ulcers, inflammatory diseases, viral diseases, fungal infections, autoimmune diseases, stomatitis, angular cheilitis, decubitus ulcers, organic changes in the tongue surface mucosa, or graft-versus-host disease. For example, the classification categories may include at least one of oral cancer and oral sarcoma belonging to oral malignant tumors, leukoplakia, erythema, and lichen planus belonging to precancerous lesions, gingivitis, periodontitis, jaw inflammation, osteomyelitis of the jaw, and drug-induced osteonecrosis of the jaw belonging to inflammatory diseases, herpes, shingles, herpangina, and hand, foot, and mouth disease belonging to viral diseases, oral candidiasis belonging to fungal infections, pemphigus, pemphigoid, and Behcet's disease belonging to autoimmune diseases, and geographic tongue, fissured tongue, black hairy tongue, and median rhomboid glossitis belonging to organic changes in the tongue surface mucosa. Furthermore, the classification categories may include pigmentation not belonging to lesions and normal states not belonging to lesions or pigmentation.

[0031] The example in Fig. 3 shows a learning model 130 in which n categories are set as categories for classifying oral cavity images. This learning model 130 is configured to output, from each node of the output layer 133, a probability X1 of oral malignancy, a probability X2 of leukoplakia, a probability X3 of lichen planus, ..., a probability Xn of normality. The number of categories (=n) set may be one or more.

[0032] The control unit 12 of the estimation device 1 acquires the probability for each of the lesions set as categories to be classified from the output layer 133 of the learning model 130, and estimates the presence or absence of a lesion in the oral mucosa based on the acquired probabilities. For example, if only the probability X1 of an oral malignant tumor exceeds a threshold (e.g., 80%), the control unit 12 can estimate that a malignant tumor has developed in the oral cavity of the subject. The same applies when any one of the probabilities X2, X3, ..., Xn-1 exceeds the threshold. On the other hand, if none of the probabilities X1, X2, ..., Xn-1 exceeds the threshold, or if the probability Xn indicating normality exceeds the threshold, the control unit 12 can estimate that no lesion is present in the oral cavity of the subject.

[0033] Although the example in FIG. 3 shows the learning model 130 based on CNN, any machine learning model can be used to construct the learning model 130. For example, instead of CNN, a learning model based on R-CNN (Region-based CNN), YOLO (You Only Look Once), SSD (Single Shot Detector), or the like may be used. Also, an algorithm such as U-Net may be used to output information about an area estimated to be a lesion.

[0034] 4 is a flowchart illustrating the procedure of processing executed by the estimation device 1 according to embodiment 1. The control unit 12 of the estimation device 1 executes the estimation processing program P1 stored in the storage unit 13 to perform the following estimation processing.

[0035] The control unit 12 acquires an oral cavity image through the input unit 11 (step S101) and provides the acquired oral cavity image to the input layer 131 of the learning model 130, thereby executing a calculation using the learning model 130 (step S102). The image data of the oral cavity image provided to the input layer 131 of the learning model 130 is sent to the intermediate layer 132. The intermediate layer 132 performs a calculation using an activation function including weights and biases between nodes. Image features are extracted in the convolutional layer 132a and pooling layer 132b of the intermediate layer 132. The data of the feature portions extracted by the convolutional layer 132a and pooling layer 132b are connected to each node constituting the fully connected layer 132c and converted into feature variables by the activation function. The converted feature variables are output to the output layer 133 via the fully connected layer 132c. The output layer 133 converts the feature variables input from the fully connected layer 132c of the intermediate layer 132 into probabilities using a softmax function, and outputs the probability of belonging to each category from each node.

[0036] The control unit 12 acquires the calculation result from the learning model 130 and estimates the presence or absence of a lesion in the oral mucosa based on the acquired calculation result (step S103). As described above, each node constituting the output layer 133 of the learning model 130 outputs a probability for each of the lesions set as a category to be classified. The control unit 12 can estimate the presence or absence of a lesion based on the probability output from each node of the output layer 133.

[0037] The control unit 12 outputs the estimation result through the output unit 14 (step S104). Specifically, the control unit 12 generates display data for displaying the estimation result on the display device 140, and outputs the generated display data to the display device 140, thereby displaying the estimation result on the display device 140. The display format of the estimation result can be set arbitrarily. For example, the control unit 12 may generate display data including characters or graphics indicating the presence or absence of a specific lesion (e.g., oral malignant tumor) and output it to the display device 140, so that the presence or absence of the specific lesion is displayed by the characters or graphics on the display device 140. The control unit 12 may also generate display data including probability values ​​corresponding to each lesion and output it to the display device 140, so that the probability values ​​corresponding to each lesion are displayed as numerical information on the display device 140.

[0038] Fig. 5 is a schematic diagram showing an example of output from the estimation device 1. The example in Fig. 5 shows a state in which a subject ID for identifying the subject, the subject's name, the oral cavity image used in the estimation process, the probability of falling into each category, and text information indicating the estimation result are displayed on the display device 140. In addition, the learning model 130 may be used to identify the position of a lesion in the oral cavity image, and information on the identified position of the lesion may also be displayed.

[0039] As described above, in this embodiment, the presence or absence of a lesion in the oral mucosa is estimated using a machine learning learning model 130 including deep learning, and the estimation result is output. Therefore, by using the estimation result for diagnostic support, the possibility of overlooking a lesion can be reduced.

[0040] In this embodiment, the estimation device 1 has been described as a computer device installed in a facility such as a hospital. However, the estimation device 1 may also be a server device accessible via communication from a computer device installed in a facility such as a hospital. In this case, the estimation device 1 acquires oral cavity images of the subject's oral cavity from the computer device of the hospital or the like via communication, and estimates the presence or absence of a lesion in the oral mucosa based on the acquired oral cavity images. The estimation device 1 transmits the estimation results to the computer device of the hospital or the like via communication. The computer device provides diagnostic support to doctors and the like by displaying the estimation results received from the estimation device 1 on a display device.

[0041] (Embodiment 2) In the second embodiment, a configuration will be described in which preprocessing is performed on an oral cavity image before a calculation using the learning model 130 is performed.

[0042] 6 is a flowchart illustrating the procedure of processing executed by the estimation device 1 according to embodiment 2. The control unit 12 of the estimation device 1 executes the estimation processing program P1 stored in the storage unit 13 to perform the following estimation processing.

[0043] The control unit 12 acquires an oral cavity image through the input unit 11 (step S201) and performs preprocessing on the acquired oral cavity image (step S202). The control unit 12 can perform, for example, gamma correction as preprocessing on the oral cavity image. That is, assuming that the luminance value of each pixel before correction is x, the luminance value of each pixel after correction is y, and the gamma value is γ, the control unit 12 calculates the luminance value of each pixel after correction y=x 1 / γThe gamma value is calculated as follows. When capturing images of the oral cavity using an imaging device, a strong light with a short emission time, such as a strobe light, may be used. In this case, the entire oral cavity image may become unnecessarily bright (whitish). Furthermore, the strobe light may be reflected by the mucous membrane in the oral cavity, causing a loss of gradation in bright areas (overexposure), making it difficult to accurately identify lesions. On the other hand, if the gamma value is set to a small value, such as less than 0.2, the entire image may become too dark, making it difficult to identify lesions. For this reason, the gamma value is preferably set to a value greater than or equal to 0.2 and less than 1.0. The control unit 12 may statistically analyze the luminance values ​​of each pixel in the uncorrected oral cavity image and set the gamma value based on, for example, the median luminance value. Alternatively, the control unit 12 may acquire model information of the imaging device, strobe light, etc. used to capture the intraoral image and set the gamma value based on the acquired model information. In the latter case, a table defining the correspondence between model information of the imaging device and gamma values ​​may be prepared in the storage unit 13, and the gamma value may be determined by referring to this table.

[0044] Furthermore, level correction may be performed as preprocessing for the oral cavity image. For example, if the oral cavity image to be corrected has a histogram peak in the range of 100 to 240 out of an input level of 0 to 255, correction can be performed to change the shadow point to within the histogram peak (e.g., 110). This preprocessing can correct the gradation values ​​of an image that is generally whitish due to the influence of strobe light, etc., and by using the corrected oral cavity image, it becomes possible to clearly identify lesions. Note that correction of highlight points is not necessary.

[0045] The control unit 12 executes a calculation using the learning model 130 by providing the preprocessed oral cavity image to the input layer 131 of the learning model 130 (step S203). That is, the control unit 12 executes a calculation using weights and biases at each node of the input layer 131, the intermediate layer 132, and the output layer 133 that constitute the learning model 130, similar to the first embodiment.

[0046] The control unit 12 acquires the calculation results from the learning model 130 and estimates the presence or absence of a lesion in the oral mucosa based on the acquired calculation results (step S204). Each node constituting the output layer 133 of the learning model 130 outputs a probability for each of the lesions set as categories to be classified. The control unit 12 estimates the presence or absence of a lesion based on the probability output from each node of the output layer 133.

[0047] The control unit 12 outputs the estimation result through the output unit 14 (step S205). Specifically, the control unit 12 generates display data for displaying the estimation result on the display device 140, and outputs the generated display data to the display device 140, thereby causing the estimation result to be displayed on the display device 140. The display manner of the estimation result is the same as in the first embodiment.

[0048] In this embodiment, the gamma correction is performed as preprocessing of the oral cavity image, but the present invention is not limited to gamma correction. Alternatively, the preprocessing may be performed according to a preset tone curve. Furthermore, the preprocessing may be performed in addition to the luminance correction, such as contrast correction or saturation correction.

[0049] (Embodiment 3) In the third embodiment, a configuration will be described in which the estimation device 1 extracts a region corresponding to the oral mucosa from an oral cavity image, and estimates the presence or absence of a lesion in the oral mucosa from the oral cavity image of the extracted region.

[0050] 7 is a block diagram illustrating the configuration of an estimation device 1 according to embodiment 3. The estimation device 1 includes an input unit 11, a control unit 12, a storage unit 13, an output unit 14, a communication unit 15, and an operation unit 16. These components are the same as those in embodiment 1, and therefore detailed description thereof will be omitted.

[0051] In addition to the learning model 130 and the estimation processing program P1, a region extraction program P2 is also stored in the storage unit 13. The region extraction program P2 is a computer program for causing the estimation device 1 to execute a process of extracting a region corresponding to the oral mucosa of the subject from an oral cavity image. A known region extraction algorithm is used in the region extraction program P2.

[0052] The GrabCut algorithm is an example of a region extraction algorithm. The GrabCut algorithm uses a Gaussian Mixture Model (GMM) to learn the distribution of pixel values ​​in the foreground and background regions, and for pixels set as unknown regions, calculates the likelihood of them being foreground or background based on the relationship between the foreground and background regions based on the statistical values ​​of the pixel values, thereby separating and extracting the foreground and background regions.

[0053] The estimation device 1 according to the present embodiment extracts a region corresponding to the oral mucosa from the entire image region of the oral cavity image as a foreground region, and extracts the entire image region excluding the region corresponding to the oral mucosa as a background region. The background region may include the subject's teeth, the photographer's fingers, various instruments, etc., but by using a region extraction algorithm such as the GrabCut algorithm, the foreground region can be separated from the background region and extracted.

[0054] In this embodiment, the configuration is such that the area corresponding to the oral mucosa (foreground area) is separated from other areas (background area), but it is also possible to extract the area corresponding to a specific oral mucosa (e.g., the tongue) as the foreground area and extract the other areas including the oral mucosa as the background area.

[0055] FIG. 8 is a schematic diagram showing an extraction example. The example in FIG. 8 shows the result of separating the entire image area of ​​an oral cavity image into an area corresponding to the oral mucosa of the subject (foreground area) and the other area (background area). In FIG. 8, the background area is shown as a hatched area. It can be seen that the background area includes an area corresponding to the teeth of the subject and an area outside the oral cavity. The estimation device 1 estimates the presence or absence of a lesion in the oral mucosa by transferring the oral cavity image of the area corresponding to the oral mucosa of the subject (foreground area) to the learning model 130.

[0056] 9 is a flowchart illustrating the procedure of processing executed by the estimation device 1 according to embodiment 3. The control unit 12 of the estimation device 1 executes an estimation processing program P1 and a region extraction program P2 stored in the storage unit 13, thereby performing the following estimation processing.

[0057] The control unit 12 acquires an oral cavity image through the input unit 11 (step S301), and extracts an area corresponding to the oral mucosa from the entire image area of ​​the acquired oral cavity image (step S302). By the processing of step S302, parts corresponding to the subject's teeth, the photographer's fingers, and various instruments are removed from the oral cavity image.

[0058] Next, the control unit 12 performs a calculation using the learning model 130 by providing the oral cavity image, from which the region corresponding to the oral cavity region has been extracted (an image from which the portion corresponding to the subject's teeth, etc. has been removed), to the input layer 131 of the learning model 130 (step S303). The oral cavity image data provided to the input layer 131 of the learning model 130 is sent to the intermediate layer 132. In the intermediate layer 132, a calculation is performed using an activation function including weights and biases between nodes. Image features are extracted in the convolutional layer 132a and pooling layer 132b of the intermediate layer 132. The data of the feature portions extracted by the convolutional layer 132a and pooling layer 132b are connected to each node constituting the fully connected layer 132c and converted into feature variables by the activation function. The converted feature variables are output to the output layer 133 via the fully connected layer 132c. The output layer 133 converts the feature variables input from the fully connected layer 132c of the intermediate layer 132 into probabilities using a softmax function, and outputs the probability of belonging to each category from each node.

[0059] The control unit 12 acquires the calculation result from the learning model 130 and estimates the presence or absence of a lesion in the oral mucosa based on the acquired calculation result (step S304). As described above, each node constituting the output layer 133 of the learning model 130 outputs a probability for each of the lesions set as categories to be classified. The control unit 12 can estimate the presence or absence of a lesion based on the probability output from each node of the output layer 133.

[0060] The control unit 12 outputs the estimation result through the output unit 14 (step S305). Specifically, the control unit 12 generates display data for displaying the estimation result on the display device 140, and outputs the generated display data to the display device 140, thereby displaying the estimation result on the display device 140. The display format of the estimation result can be set arbitrarily. For example, the control unit 12 may generate display data including characters or graphics indicating the presence or absence of a specific lesion (e.g., oral malignant tumor) and output it to the display device 140, so that the presence or absence of the specific lesion is displayed by the characters or graphics on the display device 140. The control unit 12 may also generate display data including the probability value corresponding to each lesion and output it to the display device 140, so that the probability value corresponding to each lesion is displayed as numerical information on the display device 140.

[0061] As described above, in the third embodiment, since the estimation process can be performed after removing parts unnecessary for estimating the lesion, it is possible to improve the estimation accuracy.

[0062] In the third embodiment, too, image processing such as gamma correction may be performed as preprocessing before inputting the oral cavity image into the learning model 130.

[0063] (Fourth embodiment) In the fourth embodiment, a method for generating the learning model 130 will be described.

[0064] The learning model 130 used in the estimation device 1 is generated, for example, in a server device 2 that is communicatively connected to the estimation device 1.

[0065] 10 is a block diagram illustrating the configuration of the server device 2. The server device 2 includes a control unit 21, a storage unit 22, an input unit 23, a communication unit 24, an operation unit 25, and a display unit .

[0066] The control unit 21 includes, for example, a CPU, a ROM, a RAM, etc. The ROM included in the control unit 21 stores a control program and the like for controlling the operation of each hardware unit included in the server device 2. The CPU in the control unit 21 executes the control program stored in the ROM and various programs stored in the storage unit 22, and controls the operation of each hardware unit.

[0067] The control unit 21 is not limited to the above-described configuration. The control unit 21 is not limited to a configuration including a CPU, a ROM, and a RAM. The control unit 21 may be, for example, one or more control circuits or arithmetic circuits including a GPU, an FPGA, a DSP, a volatile or non-volatile memory, etc. The control unit 21 may also have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when an instruction to start measurement is given until when an instruction to end measurement is given, and a counter that counts numbers.

[0068] The storage unit 22 includes a storage device such as a hard disk drive. The storage unit 22 stores various computer programs executed by the control unit 21, various data used by the computer programs, data acquired from the outside, and the like. One example of a computer program stored in the storage unit 22 is a model generation program P3 for generating a learning model. The storage unit 22 also includes an oral cavity image database (oral cavity image DB) 220 that stores oral cavity images and annotations of the oral cavity images in association with each other.

[0069] The input unit 23 includes an input interface for acquiring data and programs from a recording medium on which various data or programs are recorded. The various data and programs input through the input unit 23 are stored in the storage unit 22.

[0070] The communication unit 24 includes a communication interface for connecting to a communication network N. The communication network N may be the Internet, a LAN for a specific purpose, or a Wide Area Network (WAN). The communication unit 24 transmits data to be transmitted to the estimation device 1 via the communication network N. The communication unit 24 also receives data transmitted from the estimation device 1 addressed to the server device 2 via the communication network N.

[0071] The operation unit 25 is equipped with input interfaces such as a keyboard and a mouse, and receives various types of operation information and setting information. The control unit 21 performs appropriate control based on the operation information input from the operation unit 25, and stores the setting information in the storage unit 22 as necessary.

[0072] The display unit 26 includes a display device such as a liquid crystal display panel or an organic EL display panel, and displays information to be notified to the administrator of the server device 2, etc., based on a control signal output from the control unit 21.

[0073] In this embodiment, the server device 2 is configured to include an operation unit 25 and a display unit 26, but the operation unit 25 and the display unit 26 are not essential, and the server device 2 may be configured to accept operations through an externally connected computer and output information to be notified to the external computer.

[0074] FIG. 11 is a conceptual diagram showing an example of the oral cavity image database 220. The oral cavity image database 220 stores oral cavity images in association with annotations for the oral cavity images. The oral cavity images include, for example, images of oral cavities with malignant tumors, images of oral cavities with morphologies specific to oral mucosal diseases (e.g., ulcers, erosions, swellings, etc.), etc. The annotations include the doctor's diagnosis results. The diagnosis results include pathological diagnosis results or definitive diagnosis results, and in this embodiment, are used as label data indicating that the associated and stored oral cavity images are normal, or label data indicating which pathology they represent. The annotations may also include information such as the subject's ID and name.

[0075] The procedure for generating a learning model in the server device 2 will be described below. FIG. 12 is a flowchart illustrating the procedure for generating a learning model. The control unit 21 of the server device 2 accesses the oral cavity image database 220 in the storage unit 22 and acquires training data to be used for generating the learning model (step S401). The training data includes, for example, oral cavity images and annotations for the oral cavity images. In the initial stage of generating the learning model, the training data is set to data prepared by an administrator of the server device 2 or the like. Furthermore, as learning progresses, the estimation results by the learning model 130 and the oral cavity images used in the estimation process may be acquired from the estimation device 1, and the acquired data may be set as training data.

[0076] Next, the control unit 21 inputs the image data included as training data into the learning model for learning (step S402) and obtains the calculation result from the learning model (step S403). Before the learning starts, the definition information describing the learning model is assumed to have been given initial settings. The calculation by this learning model is the same as the calculation by the learning model 130 in the estimation process.

[0077] Next, the control unit 21 evaluates the calculation result obtained in step S403 (step S404) and determines whether learning is complete (step S405). Specifically, the control unit 21 can evaluate the calculation result using an error function (also referred to as an objective function, loss function, or cost function) based on the calculation result obtained in step S403 and the training data. The control unit 21 determines that learning is complete when the error function becomes equal to or smaller than a threshold (or equal to or larger than a threshold) during the process of optimizing (minimizing or maximizing) the error function using a gradient descent method such as steepest descent. Note that to avoid the problem of overfitting, techniques such as cross-validation and early termination may be employed to terminate learning at an appropriate time.

[0078] If it is determined that learning is not complete (S405: NO), the control unit 21 updates the weights and biases between the nodes of the learning model (step S406) and returns the process to step S401. The control unit 21 can update the weights and biases between each node using an error backpropagation method that sequentially updates the weights and biases between nodes from the output layer to the input layer of the learning model.

[0079] If it is determined that the learning is completed (S405: YES), the control unit 21 stores the learned model in the storage unit 22 (step S407), and ends the processing according to this flowchart.

[0080] As described above, in this embodiment, the learning model 130 used in the estimation device 1 can be generated in the server device 2. In response to a request from the estimation device 1, the server device 2 transmits the generated learning model to the estimation device 1. The estimation device 1 receives the learning model from the server device 2, stores it in the storage unit 13, and then executes the estimation processing program P1, thereby performing lesion estimation processing.

[0081] Furthermore, the server device 2 may be configured to collect new oral cavity images and annotations for the oral cavity images at an appropriate timing after the completion of learning, and re-train the learning model using these data. The oral cavity images may be oral cavity images obtained by capturing an image that includes at least a portion of the oral mucosa (see FIG. 2), or may be oral cavity images obtained by extracting a region corresponding to the oral mucosa (see FIG. 8). Furthermore, when displaying the estimation result, the estimation device 1 may accept a selection (diagnosis result) as to whether the estimation result is correct or not, and transmit the accepted diagnosis result to the server device 2 as an annotation. The re-training procedure is exactly the same as the procedure for generating the learning model. Re-training is performed by inputting oral cavity images included in training data into the learning model and evaluating the error between the calculation result obtained as the output of the learning model and the annotation included in the training data.

[0082] (Embodiment 5) In the fifth embodiment, a configuration for estimating the presence or absence of a lesion in the oral mucosa using an enlarged image of the oral cavity will be described.

[0083] FIG. 13 is a block diagram illustrating the configuration of an estimation device 1 according to embodiment 5. The estimation device 1 includes an input unit 11, a control unit 12, a storage unit 13, an output unit 14, a communication unit 15, and an operation unit 16. These components are the same as those in embodiment 1, and therefore detailed description thereof will be omitted. It shall be so decided.

[0084] In the fifth embodiment, a learning model is prepared for each part of the oral cavity, such as the tongue, upper lip, hard palate, soft palate, uvula, palatine tonsils, buccal mucosa, floor of the mouth, gums, and lower lip. The learning models 130-k (k=1, 2, ..., m; m is an integer equal to or greater than 1) for each part are stored in the storage unit 13. The learning model 130-k is configured to output information about lesions in the oral mucosa in response to input of an enlarged image, which will be described later. While the present embodiment is configured to prepare a learning model 130-k for each part of the oral cavity, a common learning model may be used for several parts. For example, a common learning model may be prepared for the hard palate and soft palate, and a common learning model may be prepared for the upper lip and lower lip. Furthermore, different learning models may be used for each divided region obtained by dividing a specific part. For example, different learning models may be prepared for the upper surface region and the lower surface region of the tongue.

[0085] FIG. 14 is a schematic diagram showing an example of a magnified image. The magnified image in this embodiment is an oral cavity image obtained by enlarging and capturing the inside of the oral cavity of a subject. Such a magnified image can be obtained by capturing an image of the inside of the oral cavity of a subject using a contact-type cell observation device such as a contact endoscope system. In this embodiment, in order to observe the shape and distribution of epithelial cell nuclei, the observation site was stained with a stain such as methylene blue, and the observation site was imaged at a magnification of about 500x with the imaging unit of the contact-type cell observation device pressed against the observation site.

[0086] The example in Figure 14 shows a magnified image of a portion of a subject's tongue, magnified 500 times. Due to space limitations, the magnified image is shown in grayscale; however, the actual magnified image obtained from the contact-type cell observation device is a color image. In the magnified image, epithelial cell nuclei stained with a staining agent are observed. The example in Figure 14 shows a large number of epithelial cell nuclei observed as small circular or oval regions with elongated circular shapes. In this embodiment, a learning model 130-k is generated by learning, for each of the aforementioned regions, characteristics of normal epithelial cell nuclei, such as shape, stainability, distribution, arrangement, and nucleus / cytoplasm ratio, and characteristics of epithelial cell nuclei in lesional regions, such as shape, stainability, distribution, arrangement, and nucleus / cytoplasm ratio. The method for generating the learning model 130-k will be described in detail later.

[0087] 15 is a schematic diagram showing an example of the configuration of learning model 130-k. The configuration of learning model 130-k is the same as the configuration of learning model 130 described in embodiment 1. That is, learning model 130-k is a learning model using CNN, and includes input layer 131, intermediate layer 132, and output layer 133. When the enlarged image described above is input to input layer 131, learning model 130 is configured to perform calculations in intermediate layer 132 and output information related to lesions in the oral mucosa from output layer 133.

[0088] The example of FIG. 15 illustrates a learning model 130-k in which n categories are set as categories for classifying enlarged images. This learning model 130-k is configured to output, from each node of the output layer 133, a probability X1 of oral malignant tumor, a probability X2 of leukoplakia, a probability X3 of lichen planus, ..., a probability Xn of normality. The number of categories (=n) set may be one or more. Furthermore, categories for classifying enlarged images may be set for each target site. For example, if the target site is the tongue, categories may be set for classifying lesions belonging to organic changes of the tongue surface mucosa, such as geographic tongue, fissured tongue, black hairy tongue, and median rhomboid glossitis, in addition to oral malignant tumor, leukoplakia, and lichen planus. Furthermore, if the target site is the gums, categories may be set for classifying gingivitis, periodontitis, and the like, in addition to oral malignant tumor, leukoplakia, and lichen planus. The same applies to other target sites; categories may be set for each target site.

[0089] 15 shows the learning model 130 based on CNN, any machine learning model can be used to construct the learning model 130-k. For example, instead of CNN, a learning model based on R-CNN, YOLO, SSD, or the like may be used. Also, an algorithm such as U-Net may be used to output information about an area estimated to be a lesion.

[0090] The learning model 130-k is generated, for example, in a server device 2 accessible from the estimation device 1. Prior to generating the learning model 130-k, the server device 2 collects data including an enlarged image, a subject ID, a subject name, and a doctor's diagnosis for each target region, and associates this information with and registers it in the oral cavity image database 220. Fig. 16 is a conceptual diagram showing an example of the oral cavity image database 220 in the fifth embodiment.

[0091] The server device 2 generates a learning model 130-k using data registered in the oral cavity image database 220 as training data. FIG. 17 is a flowchart illustrating the procedure for generating the learning model 130-k. The control unit 21 of the server device 2 receives a target region designation via, for example, the operation unit 25 (step S501), and selects data such as an enlarged image associated with the designated target region and annotations (diagnosis results) for the enlarged image stored in the oral cavity image database 220 as training data (step S502). In the initial stage of generating the learning model 130-k, the training data is set to data prepared by the operator of the server device 2, etc. Furthermore, as learning progresses, the estimation results by the learning model 130-k and the enlarged image used in the estimation process may be acquired from the estimation device 1, and the acquired data may be set as training data.

[0092] The control unit 21 inputs the image data of the enlarged image selected as training data into the learning model 130-k for learning (step S503), and acquires the calculation result from the learning model 130-k (step S504). Before learning starts, it is assumed that the definition information describing the learning model 130-k is given initial setting values. The calculation procedure using this learning model 130-k is the same as that in the fourth embodiment.

[0093] Next, the control unit 21 evaluates the calculation result obtained in step S504 (step S505) and determines whether learning is complete (step S506). Specifically, the control unit 21 can evaluate the calculation result using an error function (also referred to as an objective function, loss function, or cost function) based on the calculation result obtained in step S504 and the training data. The control unit 21 determines that learning is complete when the error function becomes equal to or smaller than a threshold (or equal to or larger than a threshold) during the process of optimizing (minimizing or maximizing) the error function using a gradient descent method such as steepest descent. Note that to avoid the problem of overfitting, techniques such as cross-validation and early termination may be employed to terminate learning at an appropriate time.

[0094] If it is determined that the learning is not completed (S506: NO), the control unit 21 updates the weights and biases between the nodes of the learning model (step S507) and returns the process to step S502. The control unit 21 can update the weights and biases between the nodes by using the backpropagation algorithm, which sequentially updates the weights and biases between the nodes from the output layer to the input layer of the learning model.

[0095] If it is determined that the learning is completed (S506: YES), the control unit 21 stores the learned learning model 130-k in the storage unit 22 (step S508), and ends the processing according to this flowchart.

[0096] The server device 2 transmits the generated learning model 130-k together with the information on the target body part to the estimation device 1. The estimation device 1 stores the learning model 130-k transmitted from the server device 2 in the storage unit 13 in association with the information on the target body part.

[0097] 18 is a flowchart showing an estimation procedure using the learning model 130-k. The control unit 12 of the estimation device 1 executes the estimation processing program P1 stored in the storage unit 13 to perform the following estimation processing.

[0098] The control unit 12 acquires a magnified image obtained by capturing an image of the oral cavity using, for example, a contact-type cell observation device (step S511). The control unit 12 may acquire the magnified image through the input unit 11 or through the communication unit 15. If an estimation result using an oral cavity image at normal magnification (e.g., an estimation result such as that shown in FIG. 5) has been obtained in advance, the control unit 12 may display the subject ID, subject name, estimated lesion name, and lesion location on the display device 140 when performing observation using the contact-type cell observation device, thereby providing the observer with information about the observation site. For example, a primary medical institution may perform a preliminary estimation by providing an oral cavity image at normal magnification to the learning model 130. When a secondary medical institution performs observation using the contact-type cell observation device, the estimation result obtained at the primary medical institution may be presented to the secondary medical institution. In this case, the estimation device 1 with the learning model 130 installed may be provided at the primary medical institution, and the estimation device 1 with the learning model 130-k installed may be provided at the secondary medical institution. Alternatively, the learning model 130 and the learning model 130-k may be installed on a server device accessible from both primary medical institutions and secondary medical institutions. Fig. 19 is a schematic diagram showing an example of a presentation of an observation site. The observer adjusts the observation position based on the information on the observation site presented on the display device 140, and then captures an image of the oral cavity using the contact-type cell observation device.

[0099] Next, the control unit 12 accepts the designation of the target region (step S512). When accepting the designation of the target region, the control unit 12 may display an interface screen on the display device 140. FIG. 20 is a schematic diagram showing an example of an interface screen 1400 for accepting the designation of the target region. The interface screen 1400 includes, for example, a pull-down menu 1401 for accepting the selection of the target region and a start button 1402 for accepting an instruction to start the estimation process. The pull-down menu 1401 is configured to display names of target regions (e.g., tongue, upper lip, hard palate, soft palate, uvula, palatine tonsils, buccal mucosa, floor of mouth, gums, lower lip, etc.) that can be designated by an operation using the operation unit 16, and to accept the designation of the region to be estimated from the displayed names of the target regions. The start button 1402 is configured to accept an instruction to start the estimation process by an operation using the operation unit 16. In the example of FIG. 20, an image of the oral cavity captured at normal magnification is shown, but an enlarged image captured using a contact-type cell observation device may be displayed on the interface screen 1400.

[0100] When the designation of the target region is accepted in step S512 and an instruction to start the estimation process is accepted, the control unit 12 selects a learning model 130-k according to the designated target region (step S513), and executes calculations using the learning model 130-k by providing the enlarged image acquired in step S511 to the selected learning model 130-k (step S514). The calculation procedure using the learning model 130-k is the same as in the first embodiment.

[0101] The control unit 12 acquires the calculation result from the learning model 130-k and estimates the presence or absence of a lesion in the oral mucosa based on the acquired calculation result (step S515). As described above, each node constituting the output layer 133 of the learning model 130-k outputs a probability for each of the lesions set as categories to be classified. The control unit 12 can estimate the presence or absence of a lesion based on the probability output from each node of the output layer 133.

[0102] The control unit 12 outputs the estimation result through the output unit 14 (step S516). Specifically, the control unit 12 generates display data for displaying the estimation result on the display device 140, and outputs the generated display data to the display device 140, thereby displaying the estimation result on the display device 140. The display format of the estimation result can be set arbitrarily. For example, the control unit 12 may generate display data including characters or graphics indicating the presence or absence of a specific lesion (e.g., oral malignant tumor) and output it to the display device 140, so that the presence or absence of the specific lesion is displayed by the characters or graphics on the display device 140. The control unit 12 may also generate display data including probability values ​​corresponding to each lesion and output it to the display device 140, so that the probability values ​​corresponding to each lesion are displayed as numerical information on the display device 140. An example of the output from the estimation device 1 is similar to that shown in FIG. 5.

[0103] As described above, in the fifth embodiment, by using the learning model 130-k, it is possible to estimate the presence or absence of a lesion in the oral mucosa based on features such as the shape, stainability, distribution, arrangement, and nucleus / cytoplasm ratio of epithelial cell nuclei observed in a magnified image. Furthermore, by estimating the presence or absence of a lesion using such a learning model 130-k, it is possible to perform a minimally invasive and accurate diagnosis, suggesting the possibility of performing a tissue diagnosis comparable to a biopsy diagnosis. Furthermore, the diagnosis results obtained by the learning model 130 may be stored in the memory unit 13 and used to clarify the risk of lesion occurrence in the oral mucosa, prognostic predictors, etc.

[0104] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0105] 1 Estimation device 2. Server device 11 Input section 12 Control Unit 13 Storage section 14 Output section 15 Communications Department 16 Control section 21 Control Unit 22 Memory section 23 Input section 24 Communications Department 25 Control section 26 Display section 130 Learning Model 130-1, 130-2, …, 130-m learning model 220 Oral Image Database P1 Estimation Processing Program P2 Region Extraction Program P3 Model Generator

Claims

1. an acquisition unit that acquires an enlarged image by imaging a specific site in the oral cavity of a subject at a magnification sufficient to observe the shape of epithelial cell nuclei; an estimation unit that estimates the presence or absence of a lesion in the specific region from the enlarged image acquired by the acquisition unit using a learning model configured to output information about a lesion in an enlarged image of the oral cavity when the enlarged image is input by learning the shape characteristics of normal epithelial cell nuclei and the shape characteristics of epithelial cell nuclei in a lesion region; an output unit that outputs the estimation result of the estimation unit; An estimation device comprising:

2. The learning model is further generated by learning the characteristics of staining, distribution, arrangement, and nucleus / cytoplasm ratio of normal epithelial cell nuclei and the characteristics of staining, distribution, arrangement, and nucleus / cytoplasm ratio of epithelial cell nuclei in the lesion area. The estimation device according to claim 1 .

3. The enlarged image is an image obtained by capturing an image using a contact-type cell observation device. The estimation device according to claim 1 or 2.

4. It is possible to use a plurality of learning models individually trained for each intraoral site, a receiving unit that receives a designation of a region captured in the enlarged image acquired by the acquiring unit; a selection unit that selects one learning model from the plurality of learning models according to the part designated through the reception unit; Equipped with The estimation unit estimates the presence or absence of a lesion in the specified region using the one learning model selected by the selection unit. The estimation device according to any one of claims 1 to 3.

5. The estimation unit estimates the presence or absence of at least one lesion belonging to oral malignant tumor, precancerous lesion, benign tumor, traumatic ulcer, inflammatory disease, viral disease, fungal infection, autoimmune disease, stomatitis, angular cheilitis, decubitus ulcer, organic change of the tongue surface mucosa, or graft-versus-host disease. The estimation device according to any one of claims 1 to 4.

6. Obtaining a magnified image by imaging a specific site in the oral cavity of the subject at a magnification large enough to observe the shape of epithelial cell nuclei; By learning the shape characteristics of normal epithelial cell nuclei and the shape characteristics of epithelial cell nuclei in a lesion area, when an enlarged image of the oral cavity is input, a learning model configured to output information about a lesion in the enlarged image is used to estimate the presence or absence of a lesion in the specific area from the acquired enlarged image, Output the estimation results A computer-implemented estimation method.

7. The enlarged image acquired by the computer is an enlarged image of a specific area identified by an estimation result including the name and location of a lesion in the oral mucosa of the subject, which is estimated by a primary medical institution using a learning model configured to output information about a lesion in the oral mucosa of the subject when an image of the oral cavity of the subject (excluding an enlarged image captured by enlarging the observation target area to such an extent that the shape of epithelial cell nuclei can be observed) is input. The estimation method according to claim 6.

8. an input layer to which an enlarged image obtained by imaging a specific site in the oral cavity of a subject at a magnification large enough to observe the shape of epithelial cell nuclei is input; an output layer that outputs information about the lesion in the specific region; and An intermediate layer that learns the relationship between the enlarged image input to the input layer and the information output by the output layer by learning the shape characteristics of normal epithelial cell nuclei and the shape characteristics of epithelial cell nuclei in diseased areas using enlarged images captured at a magnification large enough to observe the shape of epithelial cell nuclei and annotations for the enlarged images as training data. Equipped with When an enlarged image is input to the input layer, the computer is operated so as to perform calculations in the intermediate layer and output information relating to a lesion in the specific region from the output layer. Learning model.

9. By computer, Acquire training data including an enlarged image obtained by imaging a specific site in the oral cavity of a subject at a magnification large enough to observe the shape of epithelial cell nuclei and annotations for the enlarged image; Based on the acquired training data, a learning model is generated that learns the shape characteristics of normal epithelial cell nuclei and the shape characteristics of epithelial cell nuclei in the lesion area, and outputs information about the lesion in the specific area in response to the input of a magnified image. How to generate a learning model.

10. On the computer, Obtaining a magnified image by imaging a specific site in the oral cavity of the subject at a magnification large enough to observe the shape of epithelial cell nuclei; By learning the shape characteristics of normal epithelial cell nuclei and the shape characteristics of epithelial cell nuclei in a lesion area, when an enlarged image of the oral cavity is input, a learning model configured to output information about a lesion in the enlarged image is used to estimate the presence or absence of a lesion in the specific area from the acquired enlarged image, Output the estimation results A computer program for executing a process.

Images