Dental image processing system, dental image processing method, program, and program storage medium

Abstract

Provided are a dental image processing system, a dental image processing method, a program, and a program storage medium with which it is possible to reduce psychological stress for both a doctor and a patient, etc., when reporting the results of determining a lesion location and lesion information. The present invention comprises a dental image processing unit that detects the location of a lesion from a target dental image and performs lesion information determination for determining lesion information as related information including the location of the lesion, and a reporting unit that reports the determined lesion information. The reporting is performed upon adding confidence level information representing the confidence level of the lesion information determination.

Description

Dental image processing system, dental image processing method, program, and program storage medium

[0001] The present invention relates to a dental image processing system, a dental image processing method, a program, and a program storage medium, and in particular to a dental image processing system, a dental image processing method, a program, and a program storage medium for performing image processing such as detecting individual teeth (single teeth) from dental images such as dental panoramic X-ray images, dental X-ray images, intraoral photographs, and dental CT images.

[0002] In the field of dentistry, taking images such as panoramic X-ray images is widely used to assist in diagnosis and treatment.

[0003] In this case, it is necessary to detect individual teeth from the image, correctly recognize and store the tooth number and other information for each individual tooth, and use this information for diagnosis and treatment.

[0004] For example, Patent Document 1 discloses the technical concept of a dental analysis system and a dental analysis X-ray system that can detect lesions and determine the name of a disease from dental panoramic X-ray images with high accuracy and speed, and can display the probability of determining the name of a disease (equivalent to the ``correct rate'' of the determination).

[0005] Patent Document 1 employs YOLO (You Only Look Once), which realizes high-speed real-time object detection. In addition to YOLO, there are various other known methods for real-time object detection, such as SSD (Single Shot MultiBox Detector), Faster R-CNN (Region-Based Convolutional Neural Network), and RetinaNet.

[0006] Patent No. 6830082

[0007] In the above-mentioned Patent Document 1, deep learning is used to determine the location of the lesion and the name of the disease from a dental panoramic X-ray image, and the probability of determining the disease name (equivalent to the ``correct rate'' in the determination) can be displayed.

[0008] However, the ultimate responsibility for the diagnosis lies with the doctor who announces the results. When reaching a conclusion on a diagnosis or informing patients and their families (hereinafter referred to as "patients"), accuracy alone is insufficient to confidently demonstrate the basis for that conclusion, which can cause psychological stress for doctors. Furthermore, when the basis for a conclusion is not clearly stated, patients cannot listen to the conclusion with sufficient peace of mind and trust, which can also cause psychological stress for patients.

[0009] Therefore, the present invention is intended to solve the problems of the conventional technology as described above, and aims to provide a dental image processing system, dental image processing method, program, and program storage medium that can detect the location of a lesion from dental images of a target, such as panoramic X-ray images, dental X-ray images, intraoral photographs, and dental CT scans, and reduce the psychological stress of both doctors and patients when the results of the lesion information assessment are notified.

[0010] Therefore, in order to solve the above-mentioned problems, the dental image processing system according to the first aspect of the present invention comprises a dental image processing unit that detects the location of a lesion from a target dental image and performs lesion information determination to determine lesion information as related information including the location of the lesion, and a notification unit that notifies the determined lesion information, and when notifying the system, adds confidence level information that expresses the confidence level of the lesion information determination.

[0011] In addition, a dental image processing method according to a thirteenth aspect corresponding to the first aspect includes a dental image processing step of detecting the location of a lesion from a target dental image and performing a lesion information determination to determine lesion information as related information including the location of the lesion, and a notification step of notifying the determined lesion information, and when notifying, confidence level information expressing the confidence level of the lesion information determination is added.

[0012] A program according to a fourteenth aspect corresponding to the first aspect causes the dental image processing system to execute the dental image processing method.

[0013] A program storage medium according to a fifteenth aspect corresponding to the first aspect stores the above-mentioned program so as to be usable by the above-mentioned dental image processing device.

[0014] In these aspects of the dental image processing system or dental image processing method, or in these aspects of the program or program storage medium, the location of the lesion is detected from the target dental image, lesion information is determined as related information including the lesion, and confidence information of a predetermined expression is added and notified.

[0015] In this case, when reporting the determined lesion information, the degree of confidence in the lesion information assessment is added, thereby reducing the psychological stress of both the doctor and the patient at the time of reporting. For the doctor reporting, the degree of confidence can be clearly stated, for example, "I have determined that the disease name is XX, but I am yy% confident." Furthermore, the degree of confidence referred to here is the "percentage of votes for the decision option in a majority vote," and the basis for this can also be clearly reported. This eliminates the need to provide a clear basis for the assessment, thereby reducing psychological stress at the time of reporting. Furthermore, for the patient receiving the report, the basis for the report from the doctor can be understood, allowing them to receive the report with peace of mind and trust, thereby reducing psychological stress at the time of reporting.

[0016] The "pathology information" reported here may include not only the basic "disease name" but also the tooth number and tooth condition (state of restoration and prosthetics). Furthermore, the "expression of confidence" may be expressed not only numerically but also in a variety of ways, such as by changing the color when confidence is low, or by using a gauge instead of just a color. Furthermore, the basis for the confidence may be indicated by providing examples of various models and various confidence expressions, such as m types (m is multiple) from the same model or n types (n is multiple) from different models.

[0017] In addition, as a dental image processing system according to a second aspect, in the first aspect, the accuracy rate of the lesion information determination may be further added to the notification.

[0018] In this case, in addition to reporting the degree of confidence, the accuracy rate, which is the source information for the degree of confidence, can also be reported, so that the basis for the judgment can be reported in more detail, further reducing the psychological stress of both the doctor and the patient when reporting.

[0019] In addition, as a dental image processing system according to a third aspect, in the first aspect, the notification unit may have at least one of a display unit that notifies by display and a printing unit that notifies by printing.

[0020] In this case, if the device has a display unit, the doctor can notify the patient (display / print) by displaying an image, etc., or if the device has a printing unit, by printing, etc., so when notifying the determined lesion information, the degree of confidence in the lesion information determination can be added, thereby reducing the psychological stress of both the doctor and the patient at the time of notification.

[0021] In addition, as a dental image processing system according to a fourth aspect, in the first aspect, the dental image processing unit may perform deep learning for lesion information determination to execute the lesion information determination.

[0022] In this case, deep learning for lesion information determination is performed to perform lesion information determination, so the accuracy rate of lesion information determination can be improved based on the progress of learning, which can further improve confidence and further reduce the psychological stress of both doctors and patients when notified.

[0023] In addition, as a dental image processing system according to the fifth aspect, in the fourth aspect, the deep learning for determining lesion information may be performed by adopting at least one method from among a single deterministic method, a Bayesian method, an ensemble method, a test-time data augmentation method, and a dropout method.

[0024] In this case, by adopting at least one of the single deterministic method, Bayesian method, ensemble method, test-time data augmentation method, and dropout method to perform deep learning for lesion information determination, it is possible to improve the accuracy rate of lesion information determination and thereby improve confidence, and further reduce the psychological stress of both doctors and patients when notified.

[0025] Also, as a dental image processing system according to the sixth aspect, in the first aspect, the dental image processing unit includes a dental image acquisition unit that acquires the captured dental image, and a tooth detection unit that detects teeth from the dental image, and the tooth detection unit has an object detection unit and a segmentation unit for detecting teeth from the dental image, and multitasking processing may be performed by the object detection unit and the segmentation unit.

[0026] In this case, when a captured dental image is acquired and teeth are detected from the dental image, multitasking processing using object detection and segmentation is performed to detect the teeth from the dental image. In this case, simultaneous parallel processing using object detection and segmentation, i.e., multitasking processing, can solve the problem of accuracy in object detection, which may be reduced due to the emphasis on high speed, by complementing it with high-precision observation using segmentation, thereby making it possible to observe the condition of teeth with high precision and high speed from dental images such as dental panoramic X-rays, dental X-ray images, intraoral photographs, and dental CT images.

[0027] In addition, as a dental image processing system according to a seventh aspect, in the sixth aspect, the multitasking processing may be configured to perform multitasking learning for performing deep learning to detect teeth from dental images.

[0028] In this case, deep learning is further adopted in multitasking processing. Because it is multitasking processing, it ultimately becomes multitasking learning. This makes it possible to automatically detect the features necessary for data classification and analysis, which can contribute to further improving accuracy and speed. Specifically, for example, by further adding deep learning regarding the progression of periodontal disease in multitasking, it is possible to configure a periodontal disease progression assessment system that can make assessments with high accuracy and speed.

[0029] In addition, as a dental image processing system according to an eighth aspect, in the seventh aspect, a transformer may be used for the multitask learning.

[0030] In this case, adopting a transformer in multitask learning can contribute to further speedup and accuracy improvement. Here, the term "transformer" refers to a deep learning model published in 2017 in the paper "Attention Is All You Need." Since it is a well-known technology, details will be omitted. However, it is a technology that has dramatically evolved current AI, and various subsequent derivative models are also based on the transformer. Therefore, the term "transformer" here also includes various subsequent derivative models. Furthermore, further speedup can be achieved by combining these with the adoption of convolution processing, for example.

[0031] Also, as a dental image processing system according to a ninth aspect, in the sixth aspect, the object detection unit may have an object position detection unit that detects the position of the object, an object class classification unit that classifies the object, and an object detection preparation unit that prepares for at least one of the position detection and the class classification prior to that.

[0032] In this case, object position detection, which detects the position of an object, and object class classification, which classifies the object, are performed, and preparation is performed prior to one or both (at least one) of these. This preparation can contribute to higher accuracy and speed, such as narrowing the range of prediction and estimation in position detection and class classification. In this case, the preparation may be effective for both position detection and class classification, or may be effective for only one of them.

[0033] In addition, as a dental image processing system according to the tenth aspect, in the sixth aspect, single-shot object detection may be adopted as the object detection in the multitask processing, and segmentation may be performed simultaneously in parallel with the object detection.

[0034] In this case, single-shot object detection is adopted as object detection in multitask processing, and segmentation is performed simultaneously. The term "single-shot" is used almost synonymously with "one-stage." These methods focus on real-time processing and efficient inference, and analyze images in a single process to simultaneously predict the position and class of objects, eliminating the need for multiple stages or processes and enabling high-speed processing. However, since high speed is emphasized, there is a possibility of a decrease in accuracy. However, by simultaneously performing segmentation in parallel, this can be compensated for and resolved with high-precision observation. This allows the condition of teeth to be observed with high precision and speed from dental images such as dental panoramic X-rays, dental X-ray images, intraoral photographs, and dental CT images.

[0035] In addition, as a dental image processing system according to an eleventh aspect, in the sixth aspect, instance segmentation may be adopted as the segmentation in the multitask processing.

[0036] In this case, instance segmentation is adopted as the segmentation method for multitask processing. In instance segmentation, individual instance region masks are distinguished, and therefore the boundaries between objects can also be distinguished, allowing the mask of each object instance to be estimated. Therefore, even if an image of a person's teeth is captured overlapping adjacent teeth due to poor tooth alignment, it is possible to recognize the adjacent teeth individually and recognize the boundary lines between the teeth. This allows for faster processing without spending time analyzing the boundaries of the teeth, and also allows for higher accuracy due to the clarification of the boundaries, contributing to higher accuracy and speed.

[0037] In addition, as a dental image processing system according to a twelfth aspect, in the sixth aspect, an annotation unit that performs annotation to add reference information as information about the tooth portion may be further provided.

[0038] In this case, by performing annotation, reference information can be added as tooth information. For example, tooth number information can be considered as a representative example of reference information. Based on the object position information of the teeth, the tooth number of each tooth can be identified to generate tooth number information, which can be added as part of the reference information. In addition, various annotations based on the doctor's evaluation and judgment can also be added as part of the reference information. In this way, tooth data to which reference information has been added comprehensively (annotated tooth data) can be obtained. Note that "annotation" generally refers to a process performed by a human (such as a doctor), but in this case, it is defined to include processes similar or equivalent to "annotation" performed by AI, etc.

[0039] In this case, tooth data with annotation information can also be stored and used for secondary purposes. Secondary uses of annotation information include, for example, using an object detection network to reconstruct object position information and class classification information for each tooth from the rectangles of individual teeth, or reconstructing segmentation information from tooth mask images. This allows, for example, data from the same person to be compared with new tooth data with annotation information at the current time, thereby enabling the progression of periodontal disease, etc. to be understood. Even if the data is from someone else, the data can be accumulated and used for comparison, reference, analysis, etc., and can be effectively utilized for research and development in dental care. There are many other possible applications and variations of annotation information.

[0040] In addition, as a dental image processing system according to the thirteenth aspect, in the first aspect, the confidence level information may be information expressed in at least one of a digital representation indicating a discrete quantity and an analog representation indicating a continuous quantity.

[0041] In the first aspect, it was explained that various methods of expression, not just numerical expressions, can be considered as "expressions of confidence." These methods of expression can ultimately be classified as either "digital expressions showing discrete quantities" or "analog expressions showing continuous quantities." The difference between "digital" and "analog" here is the difference that can be easily imagined between a so-called "digital clock" and an "analog clock."

[0042] As mentioned above, a typical example of a "digital expression" is an expression such as "confidence level is yy%." No matter how precise the digits, it is still an "expression indicating a discrete quantity." On the other hand, as mentioned above, "analog expression" can be an expression that changes color depending on the confidence level, an expression that uses the same color but with different shades of color, or an expression based on the position within a circle that indicates 100% as in an analog clock or speedometer, or an expression based on the position within a gauge or the like that indicates 100% in a straight line of a predetermined length. Furthermore, "digital expression" and "analog expression" do not have to be used exclusively, but can be used simultaneously. An easy example to imagine is a "clock" or "stopwatch" that indicates "hours" and "minutes" with an "hour hand" and "minute hand," respectively, and only "seconds" with a digital number rather than a "second hand."

[0043] According to each aspect of the present invention, the condition of the teeth is observed with high accuracy and speed from dental images of the target, such as dental panoramic X-rays, dental X-ray images, intraoral photographs, and dental CT images, the location of the lesion is detected from the dental images of the target, and when reporting the results of the lesion information assessment, the confidence level of the lesion information assessment is added, thereby reducing the psychological stress of both the doctor and the patient when reporting the results.

[0044] FIG. 1 is an explanatory diagram of functional blocks of a dental image processing system according to an embodiment; FIG. 2 is an explanatory diagram of functional blocks focusing on the hardware configuration of a dental image processing device according to an embodiment; FIG. 3 is an explanatory diagram of functional blocks focusing on the image processing algorithm in a dental image processing device according to an embodiment; FIG. 4 is an explanatory diagram showing the flow of an algorithm for an annotation function; FIG. 5 is an explanatory diagram showing an example of an accuracy rate by vote rate by bagging; and FIG. 6 is an explanatory diagram of the principle of the bagging algorithm.

[0045] The inventive concepts are described more fully below with reference to the accompanying drawings, in which specific embodiments of the inventive concepts are shown. However, the inventive concepts may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided as examples so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concepts to those skilled in the art. Like numbers refer to like elements throughout the description.

[0046] Hereinafter, the first embodiment of the present invention will be described in order of [1. System configuration example], [2. Key points for solving problems], and [3. Summary].

[0047] 1 is an explanatory diagram of the functional blocks of a dental image processing system 1 according to an embodiment. As shown in the figure, the dental image processing system 1 includes a dental image processing function (device / unit / step) 100, which is the core of image processing, an imaging function (device / unit / step) 10 for capturing a dental image of a target, and a display function (device / unit / step) 20 for displaying the dental image of the target and / or detected teeth.

[0048] Here, the dental image processing system 1 has, for example, a "dental image processing function." From a hardware perspective, the function is executed by a part of the hardware configuration within the dental image processing system 1, i.e., it can be called a "dental image processing device" or a "dental image processing unit." On the other hand, from an algorithm (software) perspective, the function is executed by a part of a process within the overall algorithm, i.e., it can be called a "dental image processing process." For this reason, in the illustrations, each part is expressed (described) as "~function (device / part / process)." However, in the following explanation, for simplicity, we will stick to the expression (description) of "~function." In other words, the "~function (device / part / process)" is realized by the cooperation of the (above-mentioned) hardware and (above-mentioned) software (the same applies below).

[0049] 2 is an explanatory diagram of functional blocks focusing on the hardware configuration (device) of a dental image processing function (device / unit / process) 100 according to an embodiment. As shown in the figure, the dental image processing device 100 includes a central processing unit (CPU; which may optionally include a GPU (Graphics Processing Unit) for image rendering such as 3D graphics) 41 that executes programs for various processes, a memory (M) 42 that stores control / processing programs and data, an input / output device 44 such as an operating device (keyboard) and a display device, an input / output interface (I / O) 43 that communicates with the outside via the input / output device 44, an external storage device (DK) 46 such as a disk that stores various information (which may include captured images and dental images), and an interface (IF) 45.

[0050] This IF 45 is for transmitting and receiving information between other terminals and a management server (not shown, but can be installed) via various communication networks, for example, and may also be provided with an interface for connecting to a local network, etc. Although M42 is shown as one block, it may be configured with multiple memories, and may be configured with a memory RAM that the CPU 41 reads and writes, a read-only memory ROM, etc.

[0051] 3 is an explanatory diagram of functional blocks focusing on the image processing algorithm in the dental image processing function 100 of the embodiment. As shown in the figure, the dental image processing function 100 includes a dental image acquisition function (unit / step) 110, a tooth detection function (unit / step) 120, an annotation function (unit / step) 130, etc.

[0052] In addition, an algorithm is generally governed by a program processed by, for example, CPU 41 in this case, i.e., so-called "software." However, by devising a logic configuration, it is also possible to configure it using so-called "data flow" type hardware, in which case further speed increases can be expected.

[0053] Referring to Figure 3, dental image acquisition function 110 acquires dental images (dental image data) to be processed in the following steps. The dental image data acquired by dental image acquisition function 110 is input to tooth detection function 120, which then detects teeth based on the dental image data and acquires tooth data with that information (tooth data with tooth detection information).

[0054] Then, in the subsequent annotation function 130, the tooth data acquired by the dental detection function 120 is input, and based on this tooth data, a tooth number is assigned to each tooth in the tooth data (or tooth numbers are assigned), tooth number information is assigned (or tooth number information is assigned), and other reference information such as a doctor's evaluation / judgment and annotations based on past data is assigned (or assigned), thereby obtaining tooth data to which reference information has been comprehensively assigned (tooth data with annotation information). Note that "annotation" generally refers to a process performed by a human (such as a doctor), but here it is also intended to include processes similar or equivalent to "annotation" performed by AI, etc.

[0055] In the above explanation, drawings, and the following description, it may be understood that the next function is executed after data processing for one function is completed, for example, "data acquired by inputting data acquired by inputting data, and then data acquired based on this data." Even in this case, the objective of the present invention, "detecting a single tooth with high accuracy and high speed," can be achieved.

[0056] However, in this embodiment, each piece of data (not the whole piece of data) processed in the previous stage is passed to the subsequent stage function that requires that data as soon as it is acquired, without waiting for the previous stage function to finish processing all the data. In other words, each piece of data is passed to the next function in a data flow manner, so that at any given moment, processing in each function can be executed simultaneously in parallel. In the following explanation, it is assumed that each piece of data from the previous stage function is passed to the subsequent stage function in a data flow manner.

[0057] Furthermore, the above-mentioned "concurrent execution" can be considered as multitasking in terms of algorithm, but in this embodiment, deep learning, for example, is further adopted in this multitasking (since it is multitasking, it ultimately becomes multitasking learning). This makes it possible to automatically detect the features necessary for data classification and analysis, contributing to further improvements in accuracy and speed. Specifically, for example, it is possible to configure a periodontal disease progression assessment system that can assess the progression of periodontal disease with high accuracy and high speed.

[0058] In these cases, it is preferable to employ a transformer for multitask learning. The employment of such a transformer can contribute to further speedup. Here, the transformer is a deep learning model published in 2017 in a paper titled "Attention Is All You Need." Since it is a well-known technology, details will be omitted. However, it is a technology that has dramatically evolved current AI, and various subsequent derived models, such as BERT, GPT, DALL-E, and Whisper, are also based on the transformer. Therefore, the term "transformer" here also includes various subsequent derived models.

[0059] In these cases, it is preferable to employ single-shot object detection as object detection in multitask processing and simultaneously perform segmentation. The term "single-shot" is used almost synonymously with "one-stage." These techniques focus on real-time processing and efficient inference, and analyze images in a single process to simultaneously predict the object's position and class, eliminating the need for multiple stages or processes and enabling high-speed processing. However, the emphasis on high speed may result in a decrease in accuracy. By simultaneously performing segmentation in parallel, this can be compensated for and resolved with high-precision observation. This allows the condition of teeth to be observed with high precision and speed from dental images such as dental panoramic X-rays, dental X-ray images, intraoral photographs, and dental CT images.

[0060] Returning to the explanation of Figure 3, when dental image data acquired by the dental image acquisition function 110 is input, the dental detection function 120 executes the object detection function 121 and the segmentation function 122 simultaneously in parallel as shown in the figure to detect teeth (acquire tooth data).

[0061] The object detection function 121 performs position detection and class classification for detected objects (specifically, individual teeth, gums, etc.) based on dental image data. As shown in the figure, the object detection preparation function 1210 is first executed, and then, based on the output data, the object position detection function 1211 and the object class classification function 1212 are executed simultaneously in parallel.

[0062] The object detection preparation function 1210 executes a preparation function for performing the subsequent object position detection function 1211 and object class classification function 1212 with high accuracy and high speed. Specifically, for example, it determines whether dental image data acquired by imaging is child data or adult data, and sets this as additional data (preparation information data) of the dental image data, thereby acquiring dental image data with the preparation information.

[0063] This preparation can contribute to higher accuracy and speed by narrowing the range of prediction and estimation in position detection and class classification by the object position detection function 1211 and the object class classification function 1212. In this case, the preparation may be effective for both the object position detection function 1211 and the object class classification function 1212, or may be effective for only one of them.

[0064] In the above example, as shown in the figure, "the object detection function 121 and the segmentation function 122 are executed simultaneously in parallel," but similar to the relationship between the object position detection function 1211 and the object class classification function 1212 described above, the segmentation function 122 may also wait for the results of the object detection preparation function 1210 if it is advantageous in terms of accuracy and speed to use the results of the object detection preparation function 1210.

[0065] Conversely, as is common to the object position detection function 1211, object class classification function 1212, and segmentation function 122, even if the results of the object detection preparation function 1210 are used, it is also possible to proceed with the processing that can be carried out even if the results are not yet fully available, and receive the results as soon as they are available. It is preferable to be able to appropriately select whether to proceed with the processing in advance or wait for the results depending on the situation at the time, the object to be verified, etc.

[0066] The object position detection function 1211 in Fig. 3 detects the rectangle of the tooth having the crown and root, and detects the position of each tooth that can contribute to the identification of the tooth number in the subsequent stage, thereby obtaining tooth data with object position information. Also, the object classifying function 1212 in Fig. 3 classifies the classes of each detected object into the crown and root, as well as gums, etc., and obtains tooth data with class classification information.

[0067] In this case, the multitasking process defines a rectangle for each tooth, making it easier to detect each tooth and identify the position and class of each tooth, which can contribute to higher accuracy and faster processing. Furthermore, while a tooth rectangle normally has a crown and a root, in this case, if only one of the crown and root is present, a rectangle is defined for the portion where the crown or root is present. This makes it possible to identify the presence of an abnormal tooth, and, like other normal teeth, makes it easier to detect each tooth and identify the position and class of each tooth, which can contribute to higher accuracy and faster processing.

[0068] Segmentation functions, on the other hand, typically identify object regions for each pixel in an image, and generate a mask (region) that indicates whether each pixel belongs to the object or the background.

[0069] Using this function, the segmentation function in this embodiment, i.e., segmentation function 122 in Figure 3, generates a mask (area) that indicates whether each pixel belongs to the ``tooth'' as an object or to other parts such as the gums as a background.

[0070] Specifically, image data (segmentation image data) that can clearly show the shape of the tooth (shape of the outer periphery) is created, and tooth data with segmentation information is obtained.

[0071] In this case, it is preferable to employ instance segmentation as the segmentation method for multitask processing. In instance segmentation, individual instance region masks are distinguished, and therefore the boundaries between objects can be distinguished and the masks of each object instance can be estimated.

[0072] Therefore, even if an image of a person's teeth overlaps with adjacent teeth due to poor alignment, the adjacent teeth can be individually recognized, and the boundaries of the teeth can be recognized. This can increase speed without taking time to analyze the boundaries of the teeth, and can also increase accuracy by clarifying the boundaries, contributing to higher accuracy and speed.

[0073] Returning to the explanation of FIG. 3, the teeth detection function 120 acquires image data including object position information, class classification information, and segmentation information (tooth data with tooth detection information) using the above-mentioned functions.

[0074] Then, in the subsequent annotation function 130, as described above, the tooth data (tooth data with tooth detection information) acquired by the dental detection function 120 is input, and based on this tooth data, a tooth number is assigned to each tooth in the tooth data, tooth number information is assigned, and other reference information such as the doctor's evaluation / judgment and annotations based on past data is assigned, thereby acquiring tooth data (tooth data with annotation information) to which comprehensive reference information has been assigned.

[0075] 4 is an explanatory diagram showing the algorithm flow of the annotation function. As shown in the figure, when the annotation function (S1) starts, first, "extraction of a rectangle containing the teeth" (S2) is performed, followed by "detection of the object position of the teeth" (S3).

[0076] However, in reality, the object position detection information of the tooth data with object position information is acquired as part of the tooth detection information of the tooth data with tooth detection information by the object position detection function 1211 described above in Figure 3, so here, processing begins from that stage.

[0077] 4, the tooth number of each tooth is then identified based on the object position information of the tooth (S4), tooth number information is generated based on the tooth number information, and the tooth number information data is assigned to the tooth data as part of the reference information (S5), and then other reference information data indicating other reference information is assigned to the tooth data (S6), and the annotation function processing is terminated (S7). In this way, tooth data to which the reference information has been assigned comprehensively (annotated tooth data) is obtained.

[0078] [2. Key Points for Problem Solving] Here, we will consider "certifying the validity of the conclusion of the diagnostic result." That is, when there are three possible conclusions, for example, "Yes," "No," and "Unknown," and the result is determined to be "Yes," we will consider how to confirm the validity of that determination.

[0079] For example, in so-called "paternity testing," there is a diagnostic result such as "~%" probability of parentage. For example, if the test result is 95%, and based on this, the "parent-child" is judged as "Yes," the reason for the judgment is based on the grounds that "the probability is high," and it is unclear whether the judgment is valid or not. This is the same even if the result is, for example, 99%. In other words, it is impossible to eliminate the suspicion that "the judgment is arbitrary and lacks validity."

[0080] In other words, the percentage (hereinafter referred to as the "boundary percentage") that forms the boundary between "Yes" and "Unknown" and the value of the boundary percentage between "Unknown" and "No" are unclear, and even if these boundary percentages are determined, the validity of those values ​​cannot be confirmed.

[0081] Of course, if this were 70% or 50%, then "Yes" would not be appropriate (a conclusion of "unknown" or "No" would be appropriate), so although the correlation with the so-called accuracy rate of the judgment (here, parent-child possibility) is positive, one has to say that there is little evidence to support whether the conclusion is appropriate or not.

[0082] In other words, although the accuracy rate can be used as a reference value, it must be said that it is difficult to use as a basis for determining the validity of a judgment. To summarize the judgment results, a judgment result such as "The accuracy rate is 95%, so I will judge it as 'Yes'" is difficult to recognize as "valid" whether the judge is the judge himself or someone else. Of course, it goes without saying that if the accuracy rate is 70%, 50%, etc.

[0083] In contrast, if there are ten people (parties, related parties, hereafter referred to generally as "monitors") who received the same appraisal result with a 95% accuracy rate, and there is a report (record, etc.) stating that nine of the ten people concluded "Yes" and the remaining one concluded "Unclear," then it can be said that there is sufficient basis for recognizing the judgment result, "The accuracy rate is 95%, and nine out of ten people would judge it to be 'Yes,' so I agree with 'Yes,'" as "valid," whether the judge is yourself or someone else. At the very least, there is a basis for not calling it a "dogmatic judgment."

[0084] In the case of this latter "majority vote," if you are the judge, you can present (report / inform) to others with confidence, and if you are the one listening (others), you can listen with peace of mind.

[0085] (The numbers here have no basis and are therefore purely fictitious examples) For example, if the result of the above paternity test was 75%, and 5 out of 10 monitors judged it to be "Yes" (parent-child relationship exists), 3 judged it to be "Unknown" (parent-child relationship unknown: cannot say for sure whether it is "Yes" or "No"), and 2 judged it to be "No" (no parent-child relationship), the result of the majority vote would be "Yes" from "5 out of 10", so the judgement of the parent-child relationship would be "Yes" which means there is a possibility, but because the basis for the judgement (conclusion) by majority vote is 5 out of 10, the confidence level would be around 50% (50 / 50 chance).

[0086] In other words, the confidence level of a judgment is the percentage of votes for the "majority decision option" (here, "Yes"), and by being able to present this confidence level along with the accuracy rate, in the above example, a conclusion can be reached (presented / reported / announced) while clearly indicating the basis (validity) of the judgment (conclusion), such as a 90% confidence level for a 95% accuracy rate, or a 50% confidence level for a 75% accuracy rate.

[0087] Now, let us move on to a discussion of the problem at hand. In the aforementioned Patent Document 1, deep learning is used to determine the location of a lesion and the name of a disease from dental images such as dental panoramic X-rays, dental X-ray images, intraoral photographs, and dental CT images, and the probability of determining the name of the disease (corresponding to the "correct rate" of the determination) can be displayed.

[0088] However, the ultimate responsibility for the judgment lies with the doctor who announces (notifies) the results of the judgment. However, when reaching a conclusion on the judgment (diagnosis) result or announcing that conclusion to the patient or their family (hereinafter referred to as "patient"), accuracy rate alone is not enough to confidently demonstrate the basis for the conclusion, which can cause psychological stress. Furthermore, in situations where the basis for the conclusion is not clearly stated, patients cannot hear the conclusion with sufficient peace of mind or trust, which can also cause psychological stress for them.

[0089] Therefore, in order to solve the above-mentioned problems of the conventional technology, the present invention adopts the following measures.

[0090] First, as shown in FIG. 1, the dental image processing system 1 of the embodiment includes an imaging function 10 and a display function 20 in addition to a dental image processing function 100 that is the core of image processing.

[0091] The components of this dental image processing system 1 that are relevant to the problem are a dental image processing function 100, shown as the first aspect, that detects lesion locations from the target dental image and performs lesion information determination, and a notification unit that notifies the lesion location identified by the dental image processing function 100 and the determined disease name.This notification unit is equipped with a display function 20 that notifies by display, and when notifying (displaying), the notification (display) is accompanied by the degree of confidence in the lesion information determination.

[0092] In this case, by reporting the lesion location and lesion information determination result together with the confidence level, psychological stress for both the doctor and the patient at the time of reporting can be reduced. For the doctor reporting, the confidence level can be clearly stated, such as "I have determined that the disease name is XX, but my confidence level is yy%." Furthermore, the confidence level referred to here is the "percentage of votes for the decision option in a majority vote," and the basis for this can be clearly reported. This eliminates the need to provide a clear basis for the determination, thereby reducing psychological stress at the time of reporting. Furthermore, for the patient receiving the report, the basis for the report from the doctor can be understood, allowing them to receive the report with peace of mind and trust, thereby reducing psychological stress at the time of reporting.

[0093] Next, as a second embodiment, when reporting in the above-described first embodiment, the accuracy rate of lesion information determination is further added and reported.

[0094] In the case of this second embodiment, in addition to reporting the degree of confidence, the accuracy rate, which is the source information for the degree of confidence, can also be reported, so that the basis for the judgment can be reported in more detail, further reducing the psychological stress of both the doctor and the patient when reporting.

[0095] Next, in the case of the third embodiment, the notification unit in the first embodiment has at least one of a display unit that notifies by display and a printing unit that notifies by printing. The configuration in Fig. 1 is an example having a display function (display unit) 20, but as another example, the notification unit may have a printing unit that notifies by printing.

[0096] In the case of this third embodiment, if a display unit is provided, the doctor can notify the patient, etc. (display / print) by image display, etc., or if a printing unit is provided, by printing, etc., so by notifying the location of the lesion and the lesion information determination result together with the degree of confidence, it is possible to reduce the psychological stress of both the doctor and the patient, etc., when notifying them.

[0097] Next, in the case of the fourth embodiment, in the first embodiment, deep learning for lesion information determination is performed in the dental image processing function 100. This deep learning for lesion information determination can improve the accuracy rate of lesion information determination based on the progress of learning, thereby further improving confidence and further reducing psychological stress on both the doctor and the patient when a notification is made.

[0098] Next, in the case of the fifth embodiment, by adopting at least one method from among the single deterministic method, the Bayesian method, the ensemble method, the test-time data augmentation method, and the dropout method in the fourth embodiment to perform deep learning for lesion information determination, it is possible to improve the accuracy rate of lesion information determination and thereby improve the confidence level, and further reduce the psychological stress of both doctors and patients when notified.

[0099] Here, the inferences made by the above-mentioned "deep learning for lesion information determination" are summarized below.

[0100] {Definition of "confidence" and "accuracy rate" in "inference"} "Confidence Score" Confidence represents the degree of confidence that a machine learning model has in a particular inference result. For example, in classification problems, it is often expressed as a probability value when a model classifies a data point into a particular class. The higher the confidence, the more confident the model is in its inference.

[0101] "Accuracy" indicates the percentage of data points that a machine learning model correctly predicts and classifies. By calculating the percentage of correct predictions among all data points, it serves as an index to evaluate the overall performance of a model. Accuracy is widely used to measure the reliability and effectiveness of a model.

[0102] {Methods for quantifying the uncertainty of inference confidence} (1) Single deterministic methods <Characteristics>: A method for inference using a single model. It is simple and efficient, but it is difficult to estimate uncertainty. <Relationship between confidence and accuracy rate>: With this method, confidence can be obtained directly from the model output, but it is difficult to perform a detailed analysis of uncertainty.

[0103] (2) Bayesian methods <Characteristics>: Derives posterior probability from prior knowledge and data. Excellent at quantifying uncertainty, allowing for probabilistic interpretation. <Relationship between confidence level and accuracy rate>: Confidence level is interpreted as a posterior probability, allowing for the balance between accuracy rate and uncertainty to be taken into account.

[0104] (3) Ensemble methods <Characteristics>: Inference is performed by combining multiple models. The uncertainty of each individual model is averaged out, enabling robust inference. <Relationship between confidence and accuracy rate>: By aggregating the confidence of each individual model, more reliable inference is possible. The accuracy rate also generally improves.

[0105] (4) Test-time data augmentation methods <Characteristics>: Data is augmented during inference, and inference is performed using multiple transformations. Inferences from different perspectives are integrated to reduce uncertainty. <Relationship with confidence / accuracy rate>: Multiple predictions based on augmented data are averaged to improve confidence and accuracy rate.

[0106] (5) Other methods: Example: Dropout as a Bayesian Approximation: Using dropout within the network to estimate uncertainty in inference. Random feature deletion reflects the uncertainty of the model.

[0107] In the present embodiment, as a fifth embodiment, "(3) Ensemble methods" among the above is adopted, and in particular, "Bagging (Bootstrap Aggregating)", which is a type of ensemble method, is adopted.

[0108] FIG. 5 is an explanatory diagram showing an example of the accuracy rate for each vote rate obtained by bagging, and FIG. 6 is an explanatory diagram showing the principle of the bagging algorithm.

[0109] As described above in relation to the first to fifth embodiments, the confidence level of a judgment is the percentage of votes received by the "majority decision option," and by being able to present this confidence level along with the accuracy rate, it becomes possible to reach (present / report / announce) a conclusion while clearly indicating the basis (validity) of the judgment (conclusion).

[0110] The example shown in FIG. 5 shows the results of inference on test data, with the "vote rate of inference results from 10 classification models" used as the confidence level, along with the corresponding "accuracy rate."

[0111] In addition, in the algorithm applied to the inference in Figure 5, as shown in Figure 6, "a model in which learning data is randomly assigned to training and validation" is used as one learning model, and when this is repeated 10 times, 10 learning models (classification models) are created because of the random assignment.

[0112] Although the original test data is the same, 10 different learning models are generated, and inference is performed for each model, as shown in Figure 6, and the final inference is determined by majority vote for each model. The percentage of votes for the decision option in each model is used as the confidence (uncertainty).

[0113] In the example of Figure 5, the accuracy rate for the entire test data was 89%, but the number of test data with a confidence level of 100% exceeded 70% of the total, and the accuracy rate corresponding to this confidence level was 98%. Conversely, the accuracy rate for test data with a low confidence level was far below 89%.

[0114] [3. Summary] As mentioned above, by being able to present the "confidence level," which is the percentage of votes for the "majority decision option," along with the "correct answer rate," it becomes possible to reach a conclusion that clearly states the basis (validity) of the judgment (conclusion).

[0115] How the inference results shown in Figure 5 are handled in the future will depend on the individual dentist who makes the final decision (final assessment), but by actively utilizing them and repeating inference using deep learning, we can expect to see an improvement in the accuracy rate and therefore in confidence. This is something that can be considered in the future.

[0116] The above-mentioned bagging (Bootstrap Aggregating) is a method of ensemble learning, in which multiple models (usually the same type of models) are trained and their predictions are combined to make an overall prediction. Ensemble methods and bagging have a common goal of combining predictions from multiple models to achieve higher prediction accuracy and robustness than a single model.

[0117] In addition to bagging, ensemble methods may incorporate other approaches, such as boosting and stacking, which employ different strategies from bagging and each have their own advantages and disadvantages.

[0118] For this reason, although bagging was adopted in the examples described above in FIGS. 5 and 6, there is room for consideration of adopting other approaches that fall under the same ensemble method.

[0119] In addition to the ensemble method, other inference methods include the single deterministic method, the Bayesian method, the test-time data augmentation method, and the dropout method. By adopting at least one of these methods to perform deep learning for lesion information determination, it is possible to improve the accuracy rate of lesion information determination and thereby improve confidence, and it is also expected that the psychological stress of both doctors and patients when notified can be further reduced.

[0120] Furthermore, regardless of the method adopted, by performing deep learning for lesion information determination, it is possible to improve the accuracy rate and confidence of lesion information determination, and therefore, by utilizing proactive inference in the future, it is expected that the psychological stress of both doctors and patients when notified may be further reduced.

[0121] As described above, according to each aspect and embodiment of the present invention, the condition of the teeth can be observed with high accuracy and speed from dental images such as dental panoramic X-rays, and by reporting the lesion location and lesion information determination results together with the confidence level, and further by reporting them together with the accuracy rate, the psychological stress of both the doctor and the patient at the time of reporting can be further reduced.

[0122] In the above embodiment, the "disease name" is used as an example of the "lesion information" to be determined and reported. However, the "lesion information" to be reported here may include not only the basic "disease name" but also tooth number and tooth condition (restoration status and prosthetics). Furthermore, the "expression of confidence" may be expressed not only numerically but also in various ways, such as by changing the color when confidence is low, or by using a gauge instead of just a color. Furthermore, the basis for the confidence may be indicated by providing various models and various confidence expressions, such as m types (m is multiple) from the same model or n types (n is multiple) from different models.

[0123] Furthermore, I explained that there are various ways to express confidence, not just numerically. When we organize these ways of expression, they can ultimately fall into either a "digital expression showing a discrete quantity" or an "analog expression showing a continuous quantity." The difference between "digital" and "analog" here is the same as the difference between a so-called "digital clock" and an "analog clock."

[0124] As mentioned above, a typical example of a "digital expression" is an expression such as "confidence level is yy%." No matter how precise the digits, it is still an "expression indicating a discrete quantity." On the other hand, as mentioned above, "analog expression" can be an expression that changes color depending on the confidence level, an expression that uses the same color but with different shades of color, or an expression based on the position within a circle that indicates 100% as in an analog clock or speedometer, or an expression based on the position within a gauge or the like that indicates 100% in a straight line of a predetermined length. Furthermore, "digital expression" and "analog expression" do not have to be used exclusively, but can be used simultaneously. An easy example to imagine is a "clock" or "stopwatch" that indicates "hours" and "minutes" with an "hour hand" and "minute hand," respectively, and only "seconds" with a digital number rather than a "second hand."

[0125] The inventive concept has been described primarily with reference to certain embodiments, however, as will be readily understood by those skilled in the art, embodiments other than those described above are equally possible within the scope of the inventive concept as defined by the appended claims.

[0126] The dental image processing system, dental image processing method, program, and program storage medium according to each aspect of the present invention are effective for use in, for example, dental practice, in image management software for patient explanations (e.g., VisualMax, a product related to this invention), for use in visually explaining or recording dental conditions. In particular, they can be used to construct, for example, a periodontal disease progression assessment system that can accurately and quickly assess the progression of periodontal disease. They can also be used to create objects for treatment simulations. Treatments include implants, tooth transplants, orthodontics, (difficult) tooth extractions, surgical orthodontics, fracture repair, and reconstruction after jaw resection surgery due to tumors, etc. These systems are related to medical database management, and in this sense, they utilize the laws of nature. They have industrial value in that they operate hardware under program control, and have broad industrial applicability in these technical fields.

[0127] 1 Dental image processing system 10 Imaging function (device / unit / process) 20 Display function (device / unit / process) 100 Dental image processing function (device / unit / process) 110 Dental image acquisition function (unit / process) 120 Tooth detection function (unit / process) 121 Object detection function (unit / process) 122 Segmentation function (unit / process) 130 Annotation function (unit / process) 1210 Object detection preparation function (unit / process) 1211 Object position detection function (unit / process) 1212 Object class classification function (unit / process)

Claims

1. A dental image processing system comprising: a dental image processing unit that detects the location of a lesion from a target dental image and executes lesion information determination to determine lesion information as related information including the location of the lesion; and a notification unit that notifies the determined lesion information, wherein, when notifying, the system adds confidence level information that expresses the confidence level of the lesion information determination.

2. The dental image processing system according to claim 1, wherein the accuracy rate of the lesion information determination is further added to the notification.

3. The dental image processing system according to claim 1, wherein the notification unit has at least one of a display unit that notifies by display and a printing unit that notifies by printing.

4. The dental image processing system according to claim 1, wherein the dental image processing unit performs deep learning for lesion information determination to execute the lesion information determination.

5. The dental image processing system according to claim 4, wherein the deep learning for determining lesion information is performed by employing at least one of a single deterministic method, a Bayesian method, an ensemble method, a test-time data augmentation method, and a dropout method.

6. The dental image processing system of claim 1, wherein the dental image processing unit comprises: a dental image acquisition unit that acquires the captured dental image; and a tooth detection unit that detects teeth from the dental image; the tooth detection unit has an object detection unit and a segmentation unit for detecting teeth from the dental image; and multitasking processing is performed by the object detection unit and the segmentation unit.

7. The dental image processing system according to claim 6, wherein the multitask processing performs multitask learning for performing deep learning to detect teeth from dental images.

8. The dental image processing system according to claim 7, wherein a transformer is used for the multitask learning.

9. A dental image processing system as described in claim 6, wherein the object detection unit has: an object position detection unit that detects the position of the object; an object class classification unit that classifies the object; and an object detection preparation unit that makes preparations prior to at least one of the position detection and the class classification.

10. The dental image processing device according to claim 6, wherein single-shot object detection is adopted as object detection in the multitask processing, and segmentation is performed simultaneously in parallel with the object detection.

11. The dental image processing system according to claim 6, wherein instance segmentation is employed as the segmentation in the multitask processing.

12. The dental image processing system according to claim 6, further comprising an annotation unit that performs annotation to add reference information as information about the tooth portion.

13. The dental image processing system according to claim 1, wherein the confidence level information is information expressed in at least one of a digital expression indicating a discrete quantity and an analog expression indicating a continuous quantity.

14. A dental image processing method comprising: a dental image processing step of detecting the location of a lesion from a target dental image and executing a lesion information determination to determine lesion information as related information including the location of the lesion; and a notification step of notifying the determined lesion information, wherein, when notifying, confidence level information expressing the degree of confidence in the lesion information determination is added.

15. A program for causing the dental image processing system according to claim 1 to execute the dental image processing method according to claim 14.

16. A program storage medium storing the program according to claim 15 so as to be usable by the dental image processing system according to the same claim.

Images