Image processing apparatus and its control method, program

Abstract

To deal with a case where it is sometimes difficult to determine the state of the inside of a mouth radiographically.SOLUTION: An image processing device of the present invention includes: first reading means for reading a first image to read a visible light image of the occlusion surface in a mouth; second reading means for reading a second image to read the visible light image when teeth are being in an occlusion state; determination means for determining the state of tooth rows and teeth on the basis of the first image, the second image, and a learning model; and output means for outputting the result of determination by the determination means.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device for estimating the condition of a patient's oral cavity. [Background technology]

[0002] In recent years, medical image diagnostic support devices have been proposed that determine tooth frames and identify missing teeth from dental images, and display both the tooth frame and the dental image (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2010-51349 Public Relations [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] X-ray images, such as those disclosed in Patent Document 1, have the problem of making it difficult to determine, for example, the condition of the gums. Therefore, the present invention aims to make it easier to determine the condition of the oral cavity, which is difficult to determine with X-ray images. [Means for solving the problem]

[0005] The image processing apparatus of the present invention includes a first reading means for reading a first image of a visible light image of the occlusal surface in the oral cavity, a second reading means for reading a second image of a visible light image of teeth in a biting position, a determination means for determining the state of the dentition and each tooth based on the first image, the second image, and a learning model, and an output means for outputting the determination result from the determination means. , a complementation means that complements the dentition determination result for the second image with the dentition determination result for the first image. It is characterized by having the following features. [Effects of the Invention]

[0006] This invention makes it easier to determine the condition of the oral cavity, which is difficult to determine using X-ray images. [Brief explanation of the drawing]

[0007] [Figure 1] It is a diagram for explaining a dental system in the first embodiment. [Figure 2] It is a processing sequence diagram of the dental system in the first embodiment. [Figure 3] (A) It is a hardware block diagram of an image processing device in the first embodiment. (B) It is a software configuration diagram of the image processing device in the first embodiment. [Figure 4] It is a flowchart of a dental formula inference process executed by the image processing device in the first embodiment. [Figure 5] It is a diagram for explaining an inference process list in the first embodiment. [Figure 6] It is a flowchart of a comprehensive determination process in the first embodiment. [Figure 7A] It is a diagram for explaining a determination list in the first embodiment. [Figure 7B] It is a diagram for explaining a determination list in the first embodiment. [Figure 7C] It is a diagram for explaining a determination list in the first embodiment. [Figure 8] It is a diagram showing an example of a UI of a dental electronic medical record display terminal when associating a captured image with an occlusal surface in the first embodiment. [Figure 9] (A) to (B) It is a diagram showing an example of a UI of a dental electronic medical record display terminal when confirming and correcting an inference result in the first embodiment. [Figure 10] It is a flowchart showing a dental formula inference process in the second embodiment. [Figure 11] It is a flowchart showing a dental formula inference process in the third embodiment. [Figure 12] (A) to (E) It is a diagram for explaining an example when an inference model in the fourth embodiment fails to correctly infer the position and number of teeth. [Figure 13]This is a flowchart for determining whether the inference result in the fourth embodiment is correct as a dental arch. [Figure 14] This flowchart shows the error detection and correction process for teeth detected outside the dental arch in the fourth embodiment. [Figure 15A] This flowchart shows the process for determining and correcting errors in the left / right orientation of dental formula numbers in the fourth embodiment. [Figure 15B] This flowchart shows the process for calculating the X-coordinate of the midline in the fourth embodiment. [Figure 15C] This is a flowchart showing the process of determining the midline of the front view in the fourth embodiment. [Figure 15D] This flowchart shows the process for determining the midline of the occlusal view in the fourth embodiment. [Figure 15E] This flowchart shows the process of determining the midline using oral tissue in the fourth embodiment. [Figure 15F] This flowchart shows the process of determining the midline of the occlusal view using the midline of the frontal view in the fourth embodiment. [Figure 16] (A) This flowchart shows the error detection and correction process when multiple teeth with the same number are detected in the fourth embodiment. (B) This flowchart shows the error correction process in the fourth embodiment. [Figure 17] (A) This flowchart shows the error detection and correction process when multiple tooth formula numbers are detected in the same tooth region in the fourth embodiment. (B) This flowchart shows the error correction process in the fourth embodiment. [Figure 18] (A) This is a flowchart showing the error detection and correction process for the order of dental formula numbers in the fourth embodiment. (B) This is a flowchart showing the error correction process in the fourth embodiment. [Figure 19] This diagram illustrates the UI for manually correcting errors in dental formula numbers in the fourth embodiment. [Figure 20]This diagram illustrates the UI for manually correcting errors in dental formula numbers in the fourth embodiment. [Figure 21] This is a flowchart of the error correction process for dental formula numbers in the fourth embodiment. [Figure 22] This is a diagram illustrating the site where the dental electronic medical record display terminal is located in the fifth embodiment. [Figure 23] This figure shows an example of a management data structure in the fifth embodiment. [Figure 24] This figure shows an example of a UI for selecting the dental electronic medical record display terminal to be targeted by the imaging device in the fifth embodiment. [Figure 25] This is a sequence diagram illustrating the procedure by which the imaging device in the fifth embodiment retains patient information. [Figure 26] (A) to (B) are diagrams illustrating an example of displaying a patient ID in the fifth embodiment. [Figure 27] This is a sequence diagram illustrating the procedure for identifying a patient ID and storing patient information in the fifth embodiment. [Figure 28] (A) to (C) are diagrams illustrating examples of information that identifies the dental electronic medical record display terminal 103 in the fifth embodiment. [Figure 29] This is a sequence diagram illustrating the procedure for linking image data and patient ID in the fifth embodiment. [Figure 30] This is a sequence diagram illustrating the procedure for linking image data and patient ID in the fifth embodiment. [Figure 31] This is a flowchart showing the training data generation process in the sixth embodiment. [Figure 32] This figure shows an example of dental information in the sixth embodiment. [Figure 33] This figure shows an example of training data in the sixth embodiment. [Figure 34] This figure shows the inference result data format output by the inference process in the first embodiment. [Figure 35] (A) A diagram showing the midline in a frontal view image in the fourth embodiment. (B) A diagram illustrating the process of determining the midline from the palatal folds, lingual frenulum, etc. in the fourth embodiment. (C) A diagram illustrating the process of determining the midline of the occlusal view using the midline of the frontal view in the fourth embodiment. [Figure 36] This figure shows an example of displaying intraoral images of the same patient during a consultation on the same day in the first embodiment, and the results of inferences made based on those intraoral images. [Figure 37] This is a flowchart showing the operation of the image processing device in the dental formula inference process in the seventh embodiment. [Figure 38] This is a flowchart illustrating the process for making an overall judgment in the seventh embodiment. [Figure 39] This is a flowchart illustrating the process for image alignment in the seventh embodiment. [Figure 40] This figure illustrates the process of changing the size of an intraoral image in the seventh embodiment, and the process of changing the inference result rectangular coordinates that accompany this change. [Figure 41] This is a perspective projection of the front view image in the seventh embodiment onto a stereoscopic V. [Figure 42] This is a flowchart showing the dental formula number correction process using information from multiple images in the seventh embodiment. [Figure 43] This figure illustrates a method for linking the rectangles representing the source and target of correction in the seventh embodiment based on positional information. [Figure 44] This is a flowchart illustrating the state determination process using information from multiple imaging surfaces in the seventh embodiment. [Figure 45] This is a matrix used to derive an overall judgment from the combination of each imaging plane and the inference result in the seventh embodiment. [Figure 46] This figure shows the UI of a dental electronic medical record display terminal for confirming and correcting the inference results in the seventh embodiment. [Figure 47]This is a flowchart illustrating the process for correcting the inference results in the seventh embodiment. [Figure 48] This figure illustrates an example of a display screen in the seventh embodiment. [Figure 49] This figure illustrates an example of a display screen in the seventh embodiment. [Figure 50] This is a flowchart illustrating the process when the presence or absence of teeth is changed in the seventh embodiment. [Figure 51] This is a flowchart showing the process when teeth are added in the seventh embodiment. [Modes for carrying out the invention]

[0008] The embodiments for carrying out the present invention will be described in detail below with reference to the attached drawings.

[0009] The embodiments described below are merely examples of means for realizing the present invention, and may be modified or changed as appropriate depending on the configuration of the apparatus to which the present invention is applied and various conditions. Furthermore, the embodiments can be combined as appropriate.

[0010] [First Embodiment] This embodiment describes an image processing device that infers the state of teeth from intraoral images of multiple imaging surfaces and comprehensively determines the state of each tooth based on the results. In this embodiment, a machine learning algorithm such as deep learning is used as the inference method. Of course, inference may also be performed using template matching or the like, based on the color and shape of the teeth.

[0011] <System Configuration> Figure 1 is a diagram showing the configuration of the dental system in this embodiment. The dental system in this embodiment consists of devices that perform the following processes.

[0012] The image processing device 101 of the present invention determines the tooth number and the condition of each tooth. Here, the tooth number in this embodiment is a symbol used to represent the position of a tooth in a dental formula, and is assigned in ascending order from the front to the back for both the upper and lower, left and right teeth. For permanent teeth, the central incisor is 1 and the third molar is 8. For deciduous teeth, the deciduous central incisor is A and the second deciduous molar is E. In this invention, the tooth number refers to a name such as "upper right 6". In contrast, the type of tooth is a designation based on the shape of the tooth. For both the upper and lower, left and right teeth, from front to back, they are the central incisor, lateral incisor, canine, (first and second) premolar, and (first, second, and third) molar. For deciduous teeth, they are the deciduous central incisor, deciduous lateral incisor, deciduous canine, and (first and second) deciduous molar. The tooth number identifies a single tooth and is described separately from the type of tooth. The condition of the tooth in this embodiment indicates the health status of the tooth and whether or not dental treatment has been performed. For each tooth, at least one is selected from healthy teeth, caries, fillings, crowns, implants, etc. In this embodiment, dental information refers to information that associates the dental formula number with the condition and image data of each tooth for each patient. Although not explained in this embodiment, the process of adding information about the imaging surface to the oral image performed on the dental electronic medical record display terminal 103, which will be described later, may be performed in the image processing device 101 using a method to determine the imaging surface of the oral image by determining the dental formula number and the condition of the tooth.

[0013] The imaging device 108 captures images of the patient's oral cavity 109, which will be described later, and generates visible light image data. In the following description, the images captured by the imaging device 108 will be described as visible light image data. Patient information, such as a patient ID associated with the patient, is added to the captured image data and transmitted to the image processing device 101. The captured images are also deleted at the request of the image processing device 101. Although not shown in the diagram, the imaging device 108 is assumed to be equipped with input / output devices for confirming captured images and selecting patient information, similar to commercially available digital cameras.

[0014] The dental electronic medical record system 104 receives a request from the dental electronic medical record display terminal 103 (described later) and communicates with the image processing device 101, the in-hospital system 106 (described later), and the dental information DB 105 to send and receive data related to the creation of dental electronic medical records.

[0015] The dental electronic medical record display terminal 103 communicates with the dental electronic medical record system 104 to display and accept input during the creation of dental electronic medical records, and to perform processing to add imaging surface information to oral images.

[0016] The dental information DB 105 communicates with the dental electronic medical record system 104 and stores and transmits patient dental information, such as patient IDs, in a manner that is linked to patient information.

[0017] The in-hospital system 106 communicates with the patient information DB 107 (described later) to perform patient information registration and retrieval processing, and also communicates with the dental electronic medical record system 104 to perform patient information transmission processing.

[0018] The patient information database 107 communicates with the in-hospital system 106 to store, send, and receive patient information.

[0019] The patient's oral cavity 109 indicates the object of examination in the dental system.

[0020] The image processing device 101, imaging device 108, electronic medical record system 104, and in-hospital system 106 communicate via network 102. Communication between the dental electronic medical record display terminal 103 and the dental electronic medical record system 104 is performed using an HDMI® cable or similar. Communication between the dental information database 105 and the dental electronic medical record system 104, and between the in-hospital system 106 and the patient information database 107, is performed using a USB cable or similar. While network, HDMI, and USB are given as examples of communication methods, the system is not limited to these.

[0021] <Procedure for inferring a patient's condition using a dental system> Figure 2 is a sequence diagram showing the procedure for inferring the patient's condition using the dental system in this embodiment.

[0022] First, steps S201 to S202 show the sequence of patient information registration related to the generation of patient information, which will be described later.

[0023] In step S201, the hospital system 106 registers patient information such as patient name, gender, and date of birth, which are associated with the patient ID.

[0024] In step 202, the hospital system 106 transmits and saves patient information to the patient information database 107.

[0025] Steps S203 to S211 show the sequence in which dental information, described later, is obtained or generated from patient information and displayed.

[0026] In step 203, the dental electronic medical record display terminal 103 requests the dental electronic medical record system 104 to display patient information in response to user operations.

[0027] In step 204, the dental electronic medical record display terminal 103 sends a request to the in-house system 106 to retrieve patient information.

[0028] In step 205, the hospital system 106 retrieves patient information stored in the patient information database 107.

[0029] In step 206, the patient information database 107 transmits the patient information requested in step 205 to the in-hospital system 106.

[0030] In step 207, the in-hospital system 106 transmits patient information to the dental electronic medical record system 104.

[0031] In step 208, the dental electronic medical record system 104 uses the patient ID in the patient information to request dental information associated with the patient ID from the dental information database 105.

[0032] In step 209, the dental information DB 105 transmits dental information to the dental electronic medical record system 104.

[0033] In step 210, the dental electronic medical record system 104 transmits to the dental electronic medical record display terminal 103.

[0034] In step 211, the dental electronic medical record display terminal 103 displays the information. If there is no dental information associated with the patient ID, the dental information DB 105 saves it as new information.

[0035] Steps S212 to S215 show the sequence of how the image processing device 101 and the imaging device 108 are connected.

[0036] In step 212, the image processing device 101 repeatedly polls the network 102 to confirm whether the imaging device 108 is connected to the network 102.

[0037] In step 213, it is detected that the imaging device 108 is connected to the network 102.

[0038] In step 214, the image processing device 101 sends a connection request to the imaging device 108, and in step 215, the imaging device 108 connects to the image processing device 101. For example, the connection between the two is established by HTTP (Hypertext Transport Protocol) communication. The method of HTTP (Hypertext Transport Protocol) communication is not limited to this.

[0039] Steps S216 to S221 show a sequence for acquiring patient information in order to add patient information to the captured images taken by the imaging device 108.

[0040] In step S216, the imaging device 108 requests the image processing device 101 to acquire patient information.

[0041] In step S217, the image processing device 101 requests the dental electronic medical record system 104 to acquire patient information.

[0042] In step S219, the dental electronic medical record system 104 transmits patient information acquired in advance to the image processing device 101.

[0043] In step S220, the image processing device 101 transfers the patient information received in step S219 to the imaging device 108.

[0044] In step S221, the imaging device 108 displays the acquired patient information. The user can then confirm and select the information on the imaging device 108.

[0045] Steps S222 to S231 show the sequence in which dental images are captured and saved as dental information.

[0046] In step S222, when the user determines that it is necessary to capture an image as dental information, they perform an imaging operation on the imaging device 108 connected to the image processing device 101. The imaging device 108 captures an image according to the imaging operation, adds patient ID information to the image, and saves it.

[0047] In step S223, the imaging device 108 transmits the image to the image processing device 101.

[0048] In step S224, the image processing device 101 sends the image to the dental electronic medical record system 104.

[0049] In step S225, the dental electronic medical record system 104 sends the image to the dental information database 105.

[0050] In step S226, the dental information DB105 saves the image based on the patient ID information.

[0051] In step S227, the dental information DB 105 notifies the dental electronic medical record system 104 that saving is complete.

[0052] In step S229, the dental electronic medical record system 104 notifies the image processing device 101 that saving is complete.

[0053] In step S230, the image processing device 101 notifies the imaging device 108 that saving is complete.

[0054] In step S231, the imaging device 108 receives the image, does not retain it, and deletes it. This sequence shows the deletion of the captured image from the imaging device 108, but it is possible to choose not to delete it depending on the settings.

[0055] Steps S232 to S235 show the sequence in which imaging surface information is added to the images acquired and saved as described above, and then saved to the dental information DB105.

[0056] In step S232, the dental electronic medical record system 104 transfers the stored images to the dental electronic medical record display terminal 103 and displays them.

[0057] In step S233, the dental electronic medical record display terminal 103 adds imaging surface information such as occlusal view and surface view to each image.

[0058] In step S234, the dental electronic medical record display terminal 103 requests the dental electronic medical record system 104 to register the imaging plane information for each image.

[0059] In step S235, the dental electronic medical record system 104 adds imaging surface information to each image and registers it in the dental information DB 105.

[0060] Steps S236 to S244 show the sequence for inferring the dental formula number and tooth condition, which constitute dental information.

[0061] In step S236, the dental electronic medical record display terminal 103 requests the dental electronic medical record system 104 to infer the tooth number and tooth condition for the captured image for which imaging surface information has been registered.

[0062] In step S237, the dental electronic medical record system 104 requests the captured images from the dental information DB 105.

[0063] In step S238, the dental information DB 105 transmits the requested image to the dental electronic medical record system 104.

[0064] In step S239, the dental electronic medical record system 104 requests the image processing device 101 to transmit an image and perform inference processing.

[0065] In step S240, the image processing device 101 performs a dental formula inference process for each image of the received imaging surface, inferring the dental formula number and the state of the teeth for each imaging surface.

[0066] In step S241, the image processing device 101 performs an overall determination of the tooth condition for each tooth number based on the inference results for each imaging surface.

[0067] In step S242, the image processing device 101 transmits the result of the overall assessment to the dental electronic medical record system 104.

[0068] In step S243, the dental electronic medical record system 104 transmits the result of the overall assessment to the dental electronic medical record display terminal 103.

[0069] In step S244, the dental electronic medical record display terminal 103 displays the result of the overall assessment.

[0070] Further details regarding the inference process and the overall decision process in this invention will be described later with reference to Figures 4 and 6.

[0071] Steps S245 to S250 show the sequence in which the user confirms and corrects the overall judgment result, and the result is saved as the inference result.

[0072] In step S245, the user checks the result of the overall inference judgment from the dental electronic medical record display terminal 103, makes corrections as necessary, and approves it. The dental electronic medical record display terminal 103 changes its display according to the user's correction and approval operations.

[0073] In step S246, the dental electronic medical record display terminal 103 sends the inference result or the result after editing the inference result, along with an approval notification, to the dental electronic medical record system 104.

[0074] In step S247, the dental electronic medical record system 104 requests the dental information DB 105 to save the results from step S246.

[0075] In step S248, the dental information DB 105 saves the results from step S246 and notifies the dental electronic medical record system 104 that the saving is complete.

[0076] In step S249, the dental electronic medical record system 104 notifies the dental electronic medical record display terminal 103 that saving is complete.

[0077] In step S250, the dental electronic medical record display terminal 103 displays that saving is complete.

[0078] The above describes the procedure for inferring the patient's condition using the dental system in this embodiment.

[0079] <Device Configuration> Next, we will explain the configuration of the devices used in the dental system.

[0080] Figure 3(A) shows a hardware configuration diagram of the image processing device. The hardware configuration 300 of the image processing device 101 of the present invention is a diagram showing an example of the hardware configuration according to this embodiment. The image processing device 101 has a processor CPU 301 that processes programs, and a memory ROM (Read Only Memory) 302 for storing programs. Furthermore, it has a memory RAM (Random Access Memory) 303 for loading data necessary when the program is executed. Furthermore, it has a hard disk drive (HDD) 305 for storing inference data, inference results, and data for generating inference data, such as training data. Furthermore, it includes an input device 306 and a display 304 used for input and display confirmation when registering setting information for the program, an interface 307 for communication with an external system, and a bus 308.

[0081] Each function in the image processing device 101 is realized by loading a predetermined program onto hardware such as the processor CPU 301 and memory ROM 302, which then performs calculations on the processor CPU 301. Furthermore, communication with an external system via the I / F 307 and control of data reading and writing in the memory RAM 303 and hard disk HDD 305 are also implemented.

[0082] Figure 3(B) shows the software configuration diagram of the image processing device. The software configuration 350 of the image processing device 101 of the present invention is composed of processing modules that can be divided into a communication unit 351, a data storage / acquisition unit 352, a learning unit 353, an inference unit 354, and a result determination unit 35. The communication unit 351 receives requests from the other processing modules and performs data transmission and reception with an external system. The data storage / acquisition unit 352 stores and acquires data necessary for processing, such as inference image data acquired via communication, inference results, patient information, determination results, and learning data. The learning unit 353 uses learning data consisting of information on the type and condition of teeth and images to infer the type and condition of teeth and performs data generation processing for inference to make a determination. The inference unit 354 acquires an inference image from the inference processing list 500, which will be described later, determines the type and condition of the teeth, and records the determination result in the determination list 700 in Figure 7A, the determination list 750 in Figure 7B, or the determination list 770 in Figure 7C, which will be described later. The result determination unit 355 performs the process of creating and recording an overall determination from the determination results of determination list 700, determination list 750, or determination list 770, which will be described later. The method for generating data to be used for inference using the training data in the learning unit 353 will be explained in detail in the sixth embodiment described later.

[0083] In this embodiment, for the sake of simplicity, we will use the example of a CPU being installed as the main control unit of the image processing device, but this is not the only option. For example, in addition to the CPU, a GPU may also be installed, and the CPU and GPU may work together to perform processing. Since GPUs can perform calculations efficiently by processing more data in parallel, it is effective to use a GPU when performing learning multiple times using a learning model such as deep learning. Therefore, the learning unit may use a GPU in addition to the CPU for processing. Specifically, when executing a learning program that includes a learning model, the CPU and GPU work together to perform calculations to perform learning. Note that the processing of the learning unit may be performed by the CPU or GPU alone. Furthermore, the processing of the estimation unit may also be performed using a GPU, similar to the processing of the learning unit.

[0084] <Inference Processing> Figure 4 is a flowchart showing the operation of the image processing device 101 in the dental formula inference processing step S240 of Figure 2. The processing shown in this flowchart is achieved by the CPU 301 of the image processing device 101 controlling each part of the image processing device 101 according to input signals and programs. Unless otherwise specified, the same applies to other flowcharts showing the processing of the image processing device 101. Figures 7A, 7B, or 7C are examples of judgment lists containing the inference processing results. The process until the judgment lists shown in Figures 7A, 7B, or 7C are output will be explained using Figure 4.

[0085] In step S401, the CPU 301 reads an oral cavity image as the input image. In this embodiment, information about the imaging surface is transmitted along with the image from the dental electronic medical record system 104. If information about the imaging surface cannot be obtained, inference may be made using a model created by machine learning.

[0086] In step S402, the CPU301 loads a model that infers the tooth position and dental formula number corresponding to the information on the imaging surface.

[0087] In step S403, the CPU 301 uses the model loaded in step S402 to perform inference of tooth position and dental formula number. As a result, the region of each tooth is detected in the input image, and the dental formula number inference result is obtained for each detection result. The detected tooth regions are numbered DN1 to DN N It is called that.

[0088] In step S404, the CPU301 loads a model that infers the position and state of the teeth, corresponding to the information on the imaging surface.

[0089] In step S405, the CPU 301 uses the model loaded in step S404 to perform inference on the position and state of the teeth. As a result, the region of each tooth is detected in the input image, and the inference result of the tooth state is obtained for each detection result. The tooth regions detected here are then used for state detection results DS1~DS M It is called that.

[0090] In step S406, the CPU 301 associates the position of each tooth and the state of each tooth from the number detection results DN1 to DN N and the state detection results DS1 to DS M Specifically, for each of the number detection results DN1 to DN N select the one with the smallest distance between the center coordinates among the state detection results DS1 to DS M and associate and record these. For example, if DS5 is in the closest position to DN3, and DN3 is inferred to be "the third upper right tooth" and DS5 is inferred to be "a healthy tooth", an association of "the third upper right tooth = healthy tooth" is recorded. Also, part of the state may be associated and recorded with multiple teeth. For example, if DS8 is inferred to be "gingivitis" covering a wide range, associate "gingivitis" with all of those among DN1 to DN N whose overlapping area with DS8 is a certain amount or more. Further, for each of the number detection results DN1 to DN N perform area division as in 955 of FIG. 9(B) and 957 of FIG. 9(B) described later, and associate the area information to which the distance between the center coordinates belongs as the position information of the state detection results DS1 to DS M in each area. For example, for tooth numbers 4 to 8, associate with "occlusal surface", "distal adjacent surface", "mesial adjacent surface", "buccal surface", "lingual surface", and for tooth numbers 1 to 3, associate with "incisal edge", "distal adjacent surface", "mesial adjacent surface", "labial surface", "lingual surface" and record as the position of the state. When there are multiple states, record them in the order of the position of state 1 and the position of state 2 respectively.

[0091] In step S407, the CPU 301 outputs all the combinations of tooth numbers and states obtained in step S406 as judgment list 700, judgment list 750, or judgment list 770. Note that this is the stage before the comprehensive judgment processing described later. Therefore, comprehensive judgment 708, comprehensive judgment 766, comprehensive judgment: position 1 767, and comprehensive judgment: position 2 768 are not registered. Also, comprehensive judgment 794, comprehensive judgment: occlusal surface 795, comprehensive judgment: labial (buccal) surface 796, comprehensive judgment: palatal (lingual) surface 797, comprehensive judgment: mesial adjacent surface 798, and comprehensive judgment: distal adjacent surface 799 are output without being registered.

[0092] Here, we will explain the data format of the output inference results.

[0093] Figure 34 shows the data format of the inference results output by the inference process.

[0094] The inference result list 3400 is a list of inference results obtained for a single image file when the inference shown in Figure 4 is performed. In this embodiment, it is recorded in XML format. Image identification information 3401 is information for identifying the image file that was the target of the inference, and in this embodiment, the file name is recorded. Image size information 3402 is information about the resolution of the entire image, and in this embodiment, the number of pixels in the width and height is recorded. Inference result 3403 is information about the number or state detected from the image, and consists of rectangle position information, a label, and a confirmation flag. In this embodiment, the rectangle position information is the coordinate x of the left edge of the rectangle in the image. min , coordinate x at the rightmost end max , upper end coordinate y min , coordinate y of the lower end maxThe following is recorded. For number inference results, the label is recorded in the label column, such as the string "uR3" representing "upper right 3" or the string "caries" representing "dental caries" if the result is a state inference result. The confirmation flag is a flag used to consider the existence confirmed when the dentist corrects the inference result in the process described later, and either true or false is recorded in the fixed column. The default is false. Inference result 3403 is recorded for each object detected from the image.

[0095] Figure 36 shows an example of displaying on a screen the intraoral images of the same patient taken during a consultation on the same day, along with the inference data from Figure 34, which is the result of the inferences made based on those intraoral images.

[0096] Figure 36(A) shows maxillary image 3601 and frontal image 3602 of the same patient during a consultation on the same day.

[0097] Figure 36(B) shows the results of inference using the machine learning model described above for the maxillary image 3601 and frontal image 3602. 3603 represents the inference result for the maxillary image 3601, and 3604 represents the inference result for the frontal image 3602. The inference result images in 3603 and 3604 display the tooth type and its rectangular information as inference results. In this way, the inference result data output by the inference process can also be displayed.

[0098] <Inference Processing List> Figure 5 shows the inference processing list used by the image processing device 101 for inference processing. The inference processing list 500 is generated by the image processing device 101 in the order in which it receives the images to be inferred transmitted from the dental electronic medical record system 104. However, it is not limited to this. For example, the inference processing list 500 may be received in advance by the dental electronic medical record system 104 through a separate communication from the images to be inferred.

[0099] The inference processing list 500 is an information list used to manage the processing target and the progress of the inference process when performing the inference shown in Figure 4. The processing list 501 is a number used to manage the order of processing. The image file name 502 is a string used to identify the image file to be processed. The imaging surface registration information 503 is information about the imaging surface used when registering the inference results in the judgment list 700. The inference execution status 504 is information that describes whether or not processing has been performed. Here, "True" indicates that inference has been completed regarding the type and state of teeth for the target image, and the judgment results have been recorded in the judgment list 700, judgment list 750, or judgment list 770. "False" means that the inference is not yet complete.

[0100] <Overall Judgment Processing> Next, the overall judgment process in this embodiment will be described.

[0101] Figure 6 is a flowchart showing the process for making an overall judgment. This flowchart corresponds to the process in step S241 of Figure 2. In this flowchart, the overall judgment 708 is determined based on the occlusal view 704, frontal view 705, left lateral view 706, and right lateral view 707 from list numbers 1 to 32 of the judgment processing list 701, which is the result of the inference process in Figure 4. This will be explained in detail below.

[0102] First, in step S601, the CPU 301 initializes the list number i of the judgment processing list 701 to 1.

[0103] In step S602, the CPU 301 obtains the inference result for the occlusal view of list number i in the judgment list 701.

[0104] Similarly, in steps S603, S604, and S605, the CPU 301 obtains the inference result for list number i related to the front view, left side view, and right side view, respectively.

[0105] In step S606, the CPU 301 performs an OR addition of the inference judgment results obtained in steps S602 to S605 and records the result in the overall judgment 708 of list number i in the judgment list 701.

[0106] For example, in List 1, the inference result is that the occlusal view is "healthy," the frontal and left lateral views are "carious / WSD," and the right lateral view is "undeterminable (recorded as "-")." Therefore, when ORed, the result is "healthy / carious / WSD / undeterminable." However, after this information is recorded, a process is performed to correct it so that the judgment result is easier to understand.

[0107] Here, we will explain the process of correcting the overall judgment. First, if the result of the OR addition is "Undeterminable only (indicated as "-" in judgment lists 700 and 750)" or "Healthy only", the value of the overall judgment 708, which has already been recorded, will not be changed, and the overall judgment for list number i in judgment list 701 will be considered complete. Furthermore, if the result of the OR addition is "Undeterminable" and "Healthy", "Undeterminable" will be deleted and the result will be "Healthy". Then the overall judgment will be considered complete. In all other cases, "Undeterminable" and "Healthy" will be deleted from the value recorded in the overall judgment 708 of list number i in judgment list 701 (the result of the OR addition), and the value will be recorded as the overall judgment value. The result of the OR addition for list 1 falls into this case, so "Undeterminable" and "Healthy" will be deleted from "Healthy, Cavity, WSD, Undeterminable", and "Cavity, WSD" will be recorded in the overall judgment 708.

[0108] Returning to Figure 6, the above correction process will be explained.

[0109] If the CPU 301 determines that the overall determination made in step S606 is only "cannot be determined" (Yes in step S607), the CPU 301 determines that the overall determination for list number i in determination list 701 is complete and proceeds to step S613. On the other hand, if the CPU 301 determines that it is not only "cannot be determined" (indicated as "-" in Figure 7A) (No in step S607), the CPU 301 proceeds to the process of determining whether "cannot be determined" is included in the overall determination (step S608).

[0110] If CPU 301 determines in step S608 that the overall judgment includes an "undeterminable" result (Yes), CPU 301 removes the "undeterminable" result from the overall judgment calculated by OR in step S606 (step S609). On the other hand, if CPU 301 determines in step S608 that the overall judgment does not include an "undeterminable" result (No), the process proceeds to determine whether the overall judgment consists only of "healthy" results (step S610).

[0111] If the CPU 301 determines in step S610 that the overall judgment is only healthy (Yes), the CPU 301 determines that the overall judgment for list number i in judgment list 701 is complete and proceeds to step S613. On the other hand, if the CPU 301 determines in S610 that the overall judgment is not only healthy (No), the CPU 301 proceeds to the process of determining whether healthy is included in the overall judgment (step S611).

[0112] If CPU 301 determines in step S611 that "healthy" is included in the overall assessment (Yes), CPU 301 removes "healthy" from the overall assessment (Step S612) and proceeds to step S613. On the other hand, if CPU 301 determines in step S611 that "healthy" is not included in the overall assessment (No), proceeds to step S613.

[0113] In step S613, which follows the aforementioned processing, the CPU 301 determines whether the list number i in the judgment processing list 701 is the maximum value. If the list number i in the judgment processing list 701 is the maximum value (Yes in step S613), the CPU 301 terminates the overall judgment flow. On the other hand, if the list number i in the judgment processing list 701 is less than the maximum value (No in step S613), the CPU 301 adds 1 to the list number i in the judgment processing list 701 (step S614) and returns to step S602. In other words, the processing from step S602 onwards is executed for the next list number in the judgment processing list 701.

[0114] The above is an explanation of the overall judgment process.

[0115] Furthermore, for positions 767 and 768 of each state in the overall judgment shown in judgment list 750, the positions corresponding to each state in the inference judgment results of judgment processing list 701, such as occlusal view 704, frontal view 705, left lateral view 706, and right lateral view 707, are added together using OR. For example, in list number 1 of judgment processing list 751, there are multiple states in the overall judgment, such as "tooth decay" and "WSD". In this case, the positions of each state in the overall judgment are recorded as the positions corresponding to each state, such as "tooth decay" and "WSD" in the overall judgment mentioned above. Specifically, if the overall judgment "tooth decay" in list number 1 of judgment processing list 751 is taken as state 1, then position 767 of overall judgment: state 1 is taken as "distal interproximal surface". Similarly, if the overall judgment "WSD" is taken as state 2, then position 768 of overall judgment: state 2 is taken as "labial surface". Here, the aforementioned states 1 and 2 are not limited to those, and the number of states can be adjusted to 1, 2, 3, etc., depending on the total number of states in the overall assessment.

[0116] Furthermore, as shown in judgment list 770, the status information for each tooth may also be presented as table information indicating which status was detected at each position of a single tooth (labial surface, mesial interproximal surface, etc.). For example, in the case of tooth number 1 in judgment processing list 771, the status of the occlusal surface 774, labial (or buccal) surface 775, palatal (or lingual) surface 776, mesial interproximal surface 777, and distal interproximal surface 778 is recorded. Similarly, record the state of the occlusal surface 779, labial (or buccal) surface 780, palatal (or lingual) surface 781, mesial adjacent surface 782, distal adjacent surface 783 in a frontal view; the occlusal surface 784, labial (or buccal) surface 785, palatal (or lingual) surface 786, mesial adjacent surface 787, distal adjacent surface 788 in a left lateral view; and the occlusal surface 789, labial (or buccal) surface 790, palatal (or lingual) surface 791, mesial adjacent surface 792, and distal adjacent surface 793 in a right lateral view.

[0117] In this case, the overall judgment for list number 1 of judgment processing list 771 can be processed using the objects processed in S602 to S605 of the flowchart in Figure 6 as the occlusal surface, labial surface (or buccal surface), palatal surface (or lingual surface), mesial adjacent surface, and distal adjacent surface. As a result, the overall judgments for list number 1 of judgment processing list 771 become 794, 795, 796, 797, 798, and 799.

[0118] This method is applicable to list numbers 1 to 32 (maximum value) of the determination processing list 771.

[0119] <Evaluation List> Figure 7A shows a judgment list that lists the judgments of the inference process and the overall judgment results.

[0120] The judgment list 700 is an information list for obtaining judgment information, including the processing target, the progress of inference, and the judgment process, when performing the comprehensive judgment process shown in Figure 6. The judgment processing list 701 is a sequential number for managing the order of processing. The maxillary / mandibular list 702 is a list containing information for determining whether it is the maxillary or mandibular. The dental formula number 703 is a list showing the dental formula number of the teeth in the maxillary or mandibular. The occlusal view 704, frontal view 705, left lateral view 706, and right lateral view 707 are condition description lists corresponding to the 5-image imaging method in dental diagnosis. The condition description is recorded in the corresponding location in Figure 4 based on the inference results. The comprehensive judgment 708 is a list showing the comprehensive judgment of the condition of the tooth type described in Figure 6.

[0121] Figure 7B shows a judgment list in which the positional information of the tooth state is recorded in accordance with the state when the positional information of the tooth state is output along with the state for the judgment of the inference process. Judgment list 750 is not shown here, but it is an information list for obtaining judgment information, the processing target, the progress of inference, and the judgment target when performing a comprehensive judgment process similar to that in Figure 6. Judgment processing list 751 is a sequential number for managing the order of processing. Maxillary / mandibular 752 is a list that records information for determining whether it is the maxillary or mandibular. Dental formula number 753 is a list that shows the dental formula number of the tooth in the maxillary or mandibular. Occlusal view: state 754, frontal view: state 757, left lateral view: state 760, right lateral view: state 763 are list records of the state corresponding to the 5-image method in dental diagnosis. The occlusal view: position 755 for state 1, the occlusal view: position 756 for state 2, the frontal view: position 758 for state 1, the frontal view: position 759 for state 2, and the left lateral view: position 761 for state 1 each show a list of positional information for each state. Similarly, the left lateral view: position 762 for state 2, the right lateral view: position 764 for state 1, and the right lateral view: position 765 for state 2 each show a list of positional information for each state. If there are multiple states in the state description list, the positional information for each state is described so that the positional information is clear. Here, we use the positions of state 1 and state 2, but if the number of states increases, the positional information for each state may be added to the state list. Alternatively, a specific description rule may be set to separate the positional information for each state, for example, "position of state 1 · position of state 2" with a "·". The description of the states and their positional information is written in the corresponding locations in Figure 4 based on the inference results.

[0122] Overall rating 766, Overall rating: Position 1 767, and Overall rating: Position 2 768 are lists showing the overall rating of the condition for each tooth type as described in Figure 6.

[0123] Figure 7C is a judgment list that describes the inferred state for each tooth, and is a list that shows which state was detected at each position of a single tooth (labial surface, mesial interproximal surface, etc.). The overall judgment 794 of list number 1 in judgment processing list 771 records the result of processing the state detected in list number 1 of judgment processing list 771 using the flowchart in Figure 6. Furthermore, the overall judgment: occlusal surface 795 records the result of processing the occlusal view: occlusal surface detection result 774, frontal view: occlusal surface detection result 779, left lateral view: occlusal surface detection result 784, and right lateral view: occlusal surface detection result 789 using the flowchart in Figure 6. The overall judgment: labial (buccal) surface 796, overall judgment: palatal (lingual) surface 797, overall judgment: mesial interproximal surface 798, and overall judgment: distal interproximal surface 799 are processed and recorded in the same manner. Note that in Figure 7C, the values ​​for list numbers 2 to 32 in List 771 are not shown, but it is assumed that the values ​​will be determined and recorded in the same way as the processing for list number 1 in the aforementioned determination processing List 771.

[0124] <UI of the display terminal for electronic medical records> Figure 8 shows the UI of a dental electronic medical record display terminal that associates the captured image with the occlusal surface. This figure shows the user interface operations when adding imaging surface information to the captured image on the dental electronic medical record display terminal 103 before requesting the image processing device 101, as described in step S236 of Figure 2, to infer the type and condition of the tooth. This UI corresponds to, for example, the one displayed in S232.

[0125] The user operates the operation cursor 812 while checking the preview area 808 from the image list 806 of the dental electronic medical record 801 before adding imaging surface information, and drags and drops the target image 809 to be added to the unassigned imaging surface area 811. This adds imaging surface information to the target image 809. The method for adding imaging surface information may be a program that checks for the existence of a file in a predetermined user interface and adds the imaging surface information as an EXIF ​​(Exchangeable Image File Format) tag to the image. Alternatively, the user may select an imaging surface information tag indicating which surface to image via the GUI of the imaging device before imaging, and then image is taken, so that the imaging surface information is automatically added as an EXIF ​​tag to the image obtained. In this case, the image processing device 101 will acquire an image with imaging surface information already added, so the work of adding imaging surface information is unnecessary. Alternatively, the image of each imaging surface may be used as input, and a trained model that has learned the imaging surface information to be added as training data may be used to infer imaging surface information using the captured image as input, and the imaging surface information output as the inference result may be added to the captured image. In this case, for example, a deep learning algorithm can be used for the learning model. Alternatively, the input interface can be simply manipulated to change the image file name on the dental electronic medical record display terminal 103; this is just one example and does not limit the method. Following these steps, processing corresponding to S23 to S235 is performed.

[0126] After adding imaging plane information to the image to be inferred, pressing the request button 813 with the operation cursor 812 saves the image and sends a request to the image processing device 101 to perform inference in step S236.

[0127] Figure 9(A) shows the UI of a dental electronic medical record display terminal for reviewing and correcting inference results. This UI is displayed, for example, in response to the execution of S244. The operations performed using this UI are those in which a dentist reviews, corrects, and approves the inference results on the dental electronic medical record display terminal 103, as described in step S245 of Figure 2.

[0128] The inference result 903 in the lower right dental diagram shows the inference result information in accordance with the dental diagram, indicating each tooth number and condition. In the inference result 903 of the lower right dental diagram, the numerical value indicates the type of tooth, and the text indicates the condition of the tooth. 905 and 906 show the correct tooth conditions, being classified as healthy and prosthetic, respectively, while 910 shows that the tooth is healthy but has been incorrectly classified as prosthetic. To confirm and correct the judgment, select with the operation cursor 907, and simultaneously enter the correct tooth condition 911 in the input field displayed at the top, overwriting the previous entry. Similarly, for incorrectly judged tooth types 908, enter the correct type in 909. Pressing the approval button 912 sends an approval notification to the dental electronic medical record system 104.

[0129] Figure 9(B) shows the UI of a dental electronic medical record display terminal that reflects the overall judgment status and the positional information of the status, which are the inference results, in the dental diagram.

[0130] In the inference result 952 of the lower right dental diagram, the inference result information is shown in accordance with the dental diagram, indicating each tooth number and state according to the position information of the state. The user interface reflects the overall judgment 766 for 25-32 in the judgment processing list 751 of the judgment list 750, the position of overall judgment: state 1 767, and the position of overall judgment: state 2 768. 953 is the display unit for the state string and the icon indicating the state. 955 and 957 are display units that represent the position information corresponding to the state for tooth numbers 4-8 and tooth numbers 1-3, respectively. Although not shown in the diagram, the information in judgment list 750 is associated with the information in 953, 955, and 957. For display on the dental electronic medical record display terminal 103, for example, a program that can display the tooth state at each position of each dental diagram in advance is used to switch the display by providing state information to the tooth position from the information in judgment list 750. The state information is, for example, 1 for white prosthesis, 2 for metal prosthesis, 3 for missing tooth, and 4 for WSD. Furthermore, even in cases where the tooth position information is not relevant, such as the absence of tooth 962, the information can be displayed using only the condition information. 958 shows that when the overall assessment for both the lower right 1st and lower right 2nd teeth is WSD and the position of the overall assessment: condition 1 is both on the labial surface, the icon representing WSD is reflected on the labial surface of the lower right 1st and lower right 2nd teeth in the dental diagram, as shown in position information 957. Similarly, in 959-962, the inference result of the overall assessment and the position information of the condition are reflected in the dental diagram.

[0131] As described above, with the image processing device of this embodiment, symptoms such as cavities occurring between the front teeth and WSD occurring at the base of the teeth can be detected from the front view, left lateral view, and right lateral view, while cavities occurring in the recesses of the molars and prosthetics can be detected from the occlusal view. Furthermore, in a comprehensive judgment, the results can be output by taking both into account. As a result, the image processing device of this embodiment can detect the condition of the teeth with greater accuracy. In addition, since all images use visible light image data, oral conditions that are difficult to judge with X-ray images, such as the condition of the gums, can also be appropriately judged.

[0132] [Second Embodiment] In the first embodiment, an example was described in which one model for inferring the state of teeth is prepared for each type of imaging surface. However, for example, incisors and molars have significantly different shapes, so even teeth in the same state may have different visual characteristics. If the position and number of the teeth can be inferred in the stage prior to inferring the state of teeth, it is possible to infer the state of teeth with higher accuracy by using a model suitable for that dental formula number. In this embodiment, an example is described in which the model used when inferring the state of teeth is switched according to the inference result of the dental formula number.

[0133] In this embodiment, descriptions of parts similar to those in the above-described embodiment will be omitted, and the description will mainly focus on the configurations unique to this embodiment.

[0134] The dental formula inference processing step S240, which operates on the image processing device 101, will be explained using Figure 10. The flowchart in Figure 10 starts in accordance with the execution of the dental formula inference processing step S240.

[0135] Steps S1001 to S1003 perform the same processing as steps S401 to S403 in Figure 4.

[0136] In step S1004, the CPU 301 extracts one of the tooth regions detected in step S1003 from the input image whose tooth state has not yet been inferred.

[0137] In step S1005, the CPU 301 selects and loads a tooth condition classification model based on the inference result of the tooth formula number obtained in step S1003, which corresponds to the tooth region extracted in step S1004. The method of selecting the tooth condition classification model is one of the following (1) to (3): (1) One tooth condition classification model is provided for each tooth formula number for both the left, right, upper, and lower jaws. For example, for the tooth region inferred to be "upper right 6th tooth" in step S1003, the tooth condition classification model for "upper right 6th tooth" is selected. (2) One tooth condition classification model is provided for each tooth formula number, common to both the left and right jaws. For example, for the tooth region inferred to be "upper right 6th tooth" in step S1003, the tooth condition classification model for the upper 6th tooth is selected. (3) One tooth condition classification model is provided for each type of tooth. For example, for the tooth region inferred to be "upper right 6th tooth" in step S1003, since the 6th tooth is a molar, the tooth condition classification model for molars is selected.

[0138] In step S1006, the CPU 301 uses the tooth condition classification model loaded in step S1005 to perform inference about the tooth condition. As a result, an inference result for the tooth condition is obtained for one tooth region.

[0139] In step S1007, the CPU 301 records the inference results from steps S1003 and S1005 in association with each other.

[0140] In step S1008, if the CPU 301 has completed the tooth condition determination for all tooth regions, it proceeds to step S1009; otherwise, it proceeds to step S1004.

[0141] In step S1009, the CPU 301 outputs the number and status of each tooth.

[0142] Thus, according to the device of this embodiment, the model used to infer the state of the teeth is switched according to the inference result of the dental formula number. This makes it possible to infer the state of the teeth with higher accuracy.

[0143] [Third Embodiment] In the second embodiment, an example was described in which, in the dental formula inference processing step S240, the model used to infer the state of the teeth is switched according to the inference result of the dental formula number. However, if there is an error in the inference result of the dental formula number, it is not possible to select the correct tooth state classification model, and the correct result cannot be obtained in the inference of the state of the teeth. Therefore, in this embodiment, an example is described in which a process is performed to correct the result that is likely to be erroneous using the accuracy of the inference.

[0144] In this embodiment, descriptions of parts similar to those in the above-described embodiment will be omitted, and the description will mainly focus on the configurations unique to this embodiment.

[0145] The dental formula inference processing step S240, which operates on the image processing device 101, will be explained using Figure 11. The flowchart in Figure 11 starts in accordance with the execution of the dental formula inference processing step S240.

[0146] Steps S1101 to S1104 perform the same processing as steps S1001 to S1004 in Figure 10.

[0147] In step S1105, CPU301 assigns 1 to integer N.

[0148] In step S1106, the CPU 301 selects and loads a tooth condition classification model based on the Nth-ranked result in the inference of tooth formula numbers obtained in step S1103, which corresponds to the tooth region extracted in step S1104. The method for selecting the tooth condition classification model is the same as the process described in step S1005 in Figure 10.

[0149] Step S1107 performs the same process as step S1006 in Figure 10.

[0150] In step S1108, if the accuracy of the most accurate inference result in step S1107 is above a threshold, the CPU 301 proceeds to step S1110; otherwise, it proceeds to step S1109.

[0151] In step S1109, CPU 301 adds 1 to the integer N and proceeds to step S1106.

[0152] Steps S1110 to S1112 perform the same processing as steps S1007 to S1009 in Figure 10.

[0153] Alternatively, instead of branching the process in step S1108 based on the accuracy of the inference result for the tooth condition, steps S1106 and S1107 may be executed a predetermined number of times, and the result with the highest accuracy in inferring the tooth condition may be adopted.

[0154] Thus, according to the device of this embodiment, if the accuracy of the tooth condition inference result is low, the tooth formula number inference result is corrected retrospectively, and the tooth condition is inferred again. This makes it possible to correct the tooth formula number inference result.

[0155] [Fourth Embodiment] In the embodiments described above, a technique for inferring tooth position and number using a model created by machine learning was explained in the dental formula inference processing step S240. However, in reality, the model may not be able to correctly infer the dental formula number. In the third embodiment, an example of correcting the dental formula number inference result when the accuracy of the tooth state inference result is low was explained, but if teeth with similar shapes, such as premolars and molars, are mistakenly inferred, a clear decrease in accuracy may not be observed even in the tooth state inference.

[0156] Figures 12(A) to (E) show examples where the inference model failed to correctly infer the position and number of teeth. In Figure 12(A), when a mirror was inserted into the oral cavity and an image of the upper jaw teeth was taken, tooth 1201, which was outside the reflection of the mirror, was incorrectly detected as "upper left incisor 1". In this image, the purpose is to detect upper jaw teeth, so it is desirable that lower jaw teeth not be detected. In addition, parts of the instruments used to take oral images, or non-tooth tissues in the oral cavity, may be incorrectly inferred as teeth. Figure 12(B) is an example where the right central incisor 1202 (upper right incisor 1) was incorrectly inferred as the left central incisor (upper left incisor 1). In Figure 12(C), the second premolar 1203 (upper right fifth incisor) and the first molar 1204 (upper right sixth incisor) were both inferred as third premolars (upper right sixth incisor). Figure 12(D) shows an example where two detection results overlapped at the position of the second premolar 1207 (upper right 5th tooth), and were inferred as the first premolar (upper right 4th tooth) and the second premolar (upper right 5th tooth), respectively. Figure 12(E) shows an example where the second premolar 1210 (upper left 5th tooth) and the first molar 1211 (upper left 6th tooth) were swapped in the inference. These are errors related to the occlusal view image, but inference errors can also occur in the frontal and lateral view images.

[0157] In this embodiment, we will describe an image processing device that determines whether the inference result is correct as a dentition even if the dental formula number cannot be correctly inferred in the dental formula inference processing step S240, and corrects it if it is incorrect. For the sake of simplicity, the images used in this embodiment will consist only of maxillary and mandibular images, unless otherwise specifically required.

[0158] In this embodiment, descriptions of parts similar to those in the above-described embodiment will be omitted, and the description will mainly focus on the configurations unique to this embodiment.

[0159] Using Figure 13, we will explain the process of determining whether the inference result is correct as a dental arch after inferring the position and number of the teeth in step S403 of Figure 4, and correcting it if it is incorrect. The flowchart in Figure 13 is executed before step S404, depending on the completion of step S403 of Figure 4.

[0160] In step S1301, the CPU 301 performs error detection and correction processing for teeth detected to be outside the dental arch. This process allows for the detection and correction of errors like those shown in Figure 12(A). Detailed processing will be described later using Figure 14.

[0161] In step S1302, the CPU 301 performs error detection and correction processing for left / right errors in the dental formula number. This process can detect and correct errors like those shown in Figure 12(B). Detailed processing will be described later using Figure 15.

[0162] In step S1303, the CPU 301 performs error detection and correction processing when multiple teeth with the same number are detected. This process allows for the detection and correction of errors like those shown in Figure 12(C). Detailed processing will be described later using Figure 16.

[0163] In step S1304, the CPU 301 performs error detection and correction processing when multiple dental formula numbers are detected in the same tooth region. This process allows for the detection and correction of errors like those shown in Figure 12(D). The detailed process will be described later using Figure 17.

[0164] In step S1305, the CPU 301 performs error detection and correction processing for the order of the dental formula numbers. This process can detect and correct errors like those shown in Figure 12(E). Detailed processing will be described later using Figure 18.

[0165] Next, using flowchart 14, we will explain the error detection and correction process for teeth detected as being outside the dental arch in step S1301 of Figure 13 in the occlusal view image.

[0166] In step S1401, the CPU 301 calculates the center coordinates of all tooth regions detected in step S403 in Figure 4.

[0167] In step S1402, the CPU 301 calculates a curve representing the dental arch. In this embodiment, an n-th degree polynomial approximating the dental arch is calculated from the center coordinates of each tooth region using the least squares method.

[0168] Steps S1403 to S1406 are a loop in which the CPU 301 processes each tooth region sequentially. The following processing is performed on each tooth region.

[0169] In step S1404, the CPU 301 calculates the distance between the center of the tooth region and the dental arch calculated in step S1402.

[0170] In step S1405, if the distance calculated in step S1404 is greater than or equal to a preset threshold, the CPU 301 considers that tooth region to be an error and proceeds to step S1406. Otherwise, it proceeds to step S1407 and executes the next loop. Here, for example, since lateral incisors are often located further from the dental arch than other teeth, the threshold may be made change for each dental formula number.

[0171] In step S1406, the CPU 301 performs error correction processing on the dental formula number detected at a position outside the dental arch. In this embodiment, the error correction processing is performed by the CPU 301 to automatically delete the inference result of the dental formula number detected at a position outside the dental arch from the detection result in step S403 of Figure 4.

[0172] Next, using flowchart 15A, we will explain the error detection and correction process for left / right tooth formula numbers in step S1302 of Figure 13. This process is performed on frontal and lateral view images.

[0173] In step S1501, the CPU 301 calculates the X-coordinate of the midline of the input image. The midline is the area between the two central incisors, and the teeth on the left and right sides are distributed around the midline. The detailed process will be described later using Figure 15B.

[0174] Steps S1502 to S1506 are a loop in which the CPU 301 processes each tooth region sequentially. The following processing is performed for each tooth region.

[0175] In step S1503, the CPU 301 calculates the central coordinates of the tooth region.

[0176] In step S1504, the CPU 301 compares whether the inferred dental formula number of the tooth region is on the left or right side with whether the X-coordinate of the center of the tooth region is on the left or right side of the midline X-coordinate calculated in step S1501. If the comparison results in a match, the process moves to step S1506 to execute the next loop; otherwise, the tooth region is considered incorrect and the process moves to step S1505.

[0177] In step S1505, the CPU 301 performs error correction processing for dental formula numbers that have been determined to be incorrect in terms of left and right. In this embodiment, the error correction processing is performed by the CPU 301 to automatically change the left and right sides of the inference result for dental formula numbers that have been determined to be incorrect in terms of left and right.

[0178] Next, using Figure 15B, we will explain the process of calculating the X-coordinate of the midline in step S1501 of Figure 15A.

[0179] In step S1507, the CPU 301 proceeds to step S1508 if the input image is a front view; otherwise, it proceeds to step S1509. In step S1508, the CPU 301 starts the process of finding the midline of the front view image. Details of this process will be described later using Figure 15C.

[0180] In step S15082, the CPU 301 records the calculated midline to memory or a file.

[0181] In step S1509, if the input image is an occlusal view, the CPU 301 proceeds to step S1510. Otherwise, since the input image is a lateral view, the process of determining the midline is not performed and the process terminates.

[0182] In step S1510, the CPU 301 starts the process of determining the midline of the side view image. Details of this process will be described later using Figure 15D.

[0183] Next, using Figure 15C, we will explain the process of calculating the midline X-coordinate from the front view image in step S1508 of Figure 15B.

[0184] In step S1511, the CPU 301 uses the tooth type inferred in step S403 to determine whether there are four upper incisors near the center of the image. For example, Figure 35(A) shows how four upper incisors, 3502-05, are inferred in the frontal view image 3501. In this way, in a typical frontal view image, the upper incisors are usually closer to the viewer than the lower incisors and are therefore easier to infer correctly, so they are used to determine the midline with priority over the lower incisors. If they are present, the process proceeds to step S1512. Otherwise, the process proceeds to step S1513.

[0185] In step S1512, the CPU 301 determines the center of the tooth regions of the two central teeth among the four upper incisors, and sets the midline 3506 as the center of their X coordinates. Then the process ends.

[0186] In step S1513, the CPU 301 uses the tooth type inferred in step S403 to determine whether there are four lower incisors near the center of the image. If there are, the process proceeds to step S1514. Otherwise, the process proceeds to step S1515.

[0187] In step S1514, the CPU 301 determines the center of the tooth regions of the two central teeth among the four lower incisors, and sets the midline at the center of their X coordinates. Then the process ends.

[0188] In step S1515, the CPU 301 determines whether there are pairs of teeth, such as canines, central incisors, and lateral incisors, for each of the upper and lower teeth shown in the image. If they exist, the process proceeds to step S1516. Otherwise, the process proceeds to step S1517.

[0189] In step S1516, the CPU 301 finds the center of the tooth region of a pair of teeth (for example, the upper left and right canines) and sets the midline at the center of their X coordinates. Then it terminates the process.

[0190] In step S1517, CPU301 sets the X-coordinate of the center of the image to the midline. This is because capturing a frontal view image is easier than capturing occlusal or lateral view images, and a mirror is not required, making it relatively more likely that the center of the image will be the midline. The process is now complete.

[0191] Next, using Figure 15D, we will explain the process of calculating the midline X-coordinate from the occlusal view image in step S1510 of Figure 15B.

[0192] In step S1518, CPU 301 determines whether the four incisors inferred in step S403 are located near the apex of the curve approximating the dental arch calculated in step S1402. If they are located there, the process proceeds to step S1519. Otherwise, the process proceeds to step S1520.

[0193] In step S1519, the CPU 301 determines the center of the tooth regions of the two central teeth out of the four incisors, and sets the midline at the center of their X coordinates. Then it terminates the process.

[0194] In step S1520, the CPU 301 determines whether the image shows the palatine fold, palatine raphe, or lingual frenulum. The palatine fold is the wavy tissue 3509 shown in the maxillary image 3507 of Figure 35(B), and the palatine raphe is the linear tissue 3510. The lingual frenulum is the linear tissue 3511 shown in the mandibular image 3508 of Figure 35(B). Inferring these tissues in parallel with the teeth in step S403 is desirable from a processing time perspective. If these tissues are visible, the process proceeds to step S1521. Otherwise, the process proceeds to step S1522.

[0195] In step S1521, the CPU 301 determines the midline based on these structures. This process will be described later using Figure 15E.

[0196] In step S1522, the CPU 301 determines whether a specific tooth condition has been inferred from the occlusal view image and the frontal view image currently being processed. Here, a specific tooth condition refers to a condition that can be determined from either the frontal or occlusal view, such as a full metal crown (a metal prosthesis that covers the entire tooth). It is desirable that the inference of the frontal view image be completed at this time, and that the inference of the frontal view image be performed before the inference of the occlusal view image when executing the tooth formula inference processing step S240. This makes it easier to determine the midline in the frontal view image, which will be necessary in the following step S1523. Alternatively, if the inference of the frontal view image or the determination of the midline in the frontal view image has not been completed at this point, the CPU 301 may interrupt and perform those processes. If a specific tooth condition has been inferred, the process proceeds to step S1523. Otherwise, the process proceeds to step S1524.

[0197] In step S1523, the CPU 301 determines the midline based on the midline in the frontal view image and the state of a specific tooth in the frontal and occlusal view images. This process will be described later using Figure 15F.

[0198] In step S1524, the CPU 301 searches for the tooth closest to the apex of the dental arch in both the left and right directions, and sets the center of the X coordinate of the center of those tooth regions as the first midline candidate.

[0199] In step S1525, the CPU 301 calculates the two tooth regions whose Y coordinates are closest to the apex of the dental arch from among the inferred tooth regions. More specifically, if it is an image of the maxilla, it calculates the two tooth regions with the smallest Y coordinates at the upper end of the tooth region, and if it is an image of the mandible, it calculates the two tooth regions with the largest Y coordinates at the lower end of the tooth region. The center of the X coordinates of the centers of these tooth regions is then designated as the second midline candidate.

[0200] In step S1526, the CPU 301 calculates the pairs of teeth, such as canines, central incisors, and lateral incisors, as in step S1515, and sets the center of the X coordinate of the center of the tooth region as the third midline candidate. If no pairs of teeth exist, there is no third midline candidate.

[0201] In step S1527, the CPU 301 determines the center from the first to third center candidates by majority vote or weighted majority vote. Note that to improve processing speed, one or two of the processes in steps S1524 to S1526 may be omitted.

[0202] Of the above processes, the midline determination processes shown in steps S1519, S1521, and S1523 are highly accurate, but their applicability is limited. For example, step S1519 cannot be applied to patients who do not have all four incisors, and step S1521 cannot be applied if the lingual frenulum is hidden by the tongue during imaging. Therefore, when these processes cannot be applied, a processing method with slightly lower accuracy but a wider range of applicability, as shown in S1524 to S1526, is used. This allows for high-precision determination of the midline in images where high-precision processing is applicable, and for images where it is not applicable, the midline can be determined with a reasonable degree of accuracy.

[0203] Next, using Figure 15E, we will explain the process of determining the midline using the palatine fold, palatine raphe, and lingual frenulum in step S1521 of Figure 15D.

[0204] In step S1528, the CPU 301 detects the region where the palatine folds or palatine raphe are located if the image is of the maxilla, or the region where the lingual frenulum is located if the image is of the mandible. This process may use the inference results used in the determination in step S1520. Alternatively, since the palatine raphe of the maxilla is more difficult to detect than the palatine folds, the CPU may first attempt to detect the palatine raphe, and if that fails, then attempt to detect the palatine folds. Figure 35(B) shows an example of the region of the palatine folds as shown in 3512, and an example of the region of the lingual frenulum as shown in 3513.

[0205] In step S1529, the CPU 301 calculates the inclination of the palate at the time of imaging. Capturing images of the occlusal view is difficult, and images taken by inexperienced photographers may be rotated, so this process is to correct that. To calculate the inclination of the palate, a curve approximating the dental arch is calculated, and the curve is rotated so that it is symmetrical on both sides when viewed from the vertex. Alternatively, if tissues extending vertically, such as the palatine raphe or lingual frenulum, are detected, edges within the region can be extracted by image processing, edges that extend long vertically can be extracted and approximated with straight lines, and the average of the slopes of these lines can be taken as the inclination of the palate. Alternatively, if palatine folds are detected, edges within the region can be extracted by image processing, edges that extend long horizontally can be extracted and approximated with straight lines, and the region can be rotated so that the slopes of these lines are symmetrical on both sides of the region, and the rotation angle can be taken as the inclination of the palate.

[0206] In step S1529, the CPU 301 rotates the region by the calculated inclination of the palate, as shown in 3514 and 3515 of Figure 35(B).

[0207] In step S1529, the CPU 301 extends the center line of the region as shown in 3516 and 3517 in Figure 35(B), calculates the intersection points 3518 and 3519 with the dental arch, and sets their X coordinates as the midline.

[0208] Next, using Figure 15F, we will explain the process of determining the midline of the occlusal view using the midline of the frontal view in step S1523 of Figure 15D.

[0209] In step S1532, the CPU 301 selects the teeth to be used when determining the midline. First, if the occlusal view image for which the midline is to be determined is of the maxilla, it extracts the upper teeth from the frontal view image. On the other hand, if the occlusal view image is of the mandible, it extracts the lower teeth from the frontal view image. Next, from among these, it extracts the teeth that have the "tooth condition" described in step S1522.

[0210] Finally, from among these, we select the tooth whose X-coordinate of the tooth region is closest to the center of the image. For example, in Figure 35(C), the tooth fitted with a full metal crown appears as 3522 in the frontal view image 3520 and as 3522 in the occlusal view image 3521. This tooth satisfies all the aforementioned conditions and is therefore selected as the tooth to be used when determining the midline.

[0211] In step S1533, the CPU 301 calculates the number of teeth between the selected tooth and the midline in the frontal view image. For example, in Figure 35(C), there are 3 teeth between the midline 3524 and tooth 3522.

[0212] In step S1534, the CPU 301 determines the midline using the number of teeth selected in the occlusal view and the number of teeth between the selected tooth and the midline in the frontal view. For example, in Figure 35(C), the midline is defined as tooth 3525, which is the point between the 3rd and 4th tooth from tooth 3523 toward the center of the image in the X-axis direction.

[0213] Through the above process, the midline can be determined from the frontal and occlusal view images.

[0214] Next, using Figure 16(A), we will explain the error detection and correction process in step S1303 of Figure 13 when multiple teeth with the same number are detected.

[0215] Steps S1601 to S1603 are a loop in which the CPU 301 processes each tooth region sequentially. The following processing is performed on each tooth region.

[0216] In step S1602, if there are no other tooth regions inferred to have the same dental formula number, the CPU 301 proceeds to step S1604 and executes the next loop; otherwise, it considers that tooth region to be incorrect and proceeds to step S1603.

[0217] In step S1603, the CPU 301 performs error correction processing on the dental formula number that was determined to be incorrect.

[0218] Here, the error correction process in step S1603 of Figure 16(A) will be explained using flowchart 16(B).

[0219] In step S1605, the CPU 301 obtains the multiple inferred dental formula numbers and the center coordinates of each tooth region on the same side.

[0220] In step S1606, the CPU 301 finds tooth numbers that have been inferred multiple times, are on the same side, and have not been inferred in that image. For example, if the inference result is as shown in Figure 12(C), the image processing device 101 extracts "upper left 5th tooth" and "upper left 8th tooth" as tooth numbers that have not been inferred.

[0221] Steps S1607 to S1610 are a loop that processes the regions of the multiple inferred dental formula numbers sequentially, moving distally. For example, if the inference result is Figure 12(C), the regions of tooth 1203 and tooth 1204 are processed in that order.

[0222] In step S1608, the CPU 301 determines the formula number of the tooth region adjacent to the mesial side of the tooth region being processed. For example, if the tooth region of 1203 is being processed, the tooth region adjacent to the mesial side is the tooth region of 1205, so the formula number of the tooth region adjacent to the mesial side is "upper left 4th tooth".

[0223] In step S1609, the CPU 301 assigns a formula number to the tooth region being processed based on the formula number of the tooth region adjacent to the mesial side. The formula number to be assigned is selected from formula numbers that have been assigned multiple times and formula numbers that have not been inferred in step S1606. For example, if the inference result is Figure 12(C), the formula number of the tooth region adjacent to the mesial side is "upper left 4th tooth," and the candidate numbers to be assigned are "upper left 6th tooth," "upper left 5th tooth," and "upper left 8th tooth," so "upper left 5th tooth," which is closest to "upper left 4th tooth," is selected.

[0224] Next, using flowchart 17(A), we will explain the error detection and correction process in step S1304 of Figure 13 when multiple dental formula numbers are detected in the same tooth region.

[0225] Steps S1701 to S1707 are a loop in which the CPU 301 processes each tooth region sequentially. Let the tooth region of interest in this loop be region A. The following processing is performed on each tooth region.

[0226] Steps S1702 to S1706 are a loop in which the CPU 301 sequentially processes each tooth region other than region A. The tooth region of interest in this loop is called region B. The following processing is performed on each tooth region.

[0227] In step S1703, CPU301 calculates the area of ​​the overlapping region A and B.

[0228] In step S1704, CPU 301 calculates the percentage of the overlapping area calculated in step S1703 that occupies regions A and B. If neither is above a threshold, the process moves to step S1706 to execute the next loop. Otherwise, it is determined that either region A or B is incorrect, and the process moves to step S1705.

[0229] In step S1705, the CPU 301 performs error correction processing on the dental formula number that was determined to be incorrect. The detailed processing will be described later using Figure 17(B).

[0230] Next, the error correction process in step S1705 of Figure 17(A) will be explained using flowchart 17(B).

[0231] In step S1708, the CPU 301 obtains the center coordinates of each tooth region, which are on the same side as the tooth regions inferred from multiple dental formula numbers.

[0232] In step S1709, the CPU 301 determines, based on the central coordinates of each tooth region, whether adjacent tooth regions exist in both the mesial and distal directions. If it determines that such regions exist, it proceeds to step S1710; otherwise, it proceeds to step S1712.

[0233] In step S1710, the CPU 301 calculates the tooth region closest to the center of the multiple tooth regions inferred to have tooth formula numbers among the tooth regions adjacent to the mesial and distal sides, by calculating the distance between the center coordinates, and obtains the tooth formula number of that tooth region. For example, if the inference result is Figure 12(D), the image processing device 101 obtains the tooth formula number "upper left 4" of the tooth region of tooth 1208, which is closer to the center coordinates of tooth 1209, among the tooth regions of tooth 1208 and tooth 1209 adjacent to the tooth region of tooth 1207.

[0234] In step S1711, the CPU 301 uses the acquired dental formula number to determine the dental formula number of the tooth region inferred from multiple dental formula numbers. For example, if the inference result is Figure 12(D), the closer tooth region is the tooth region of tooth step S1208, and its dental formula number is "upper left 4th tooth". Therefore, the CPU 301 determines the next number, "upper left 5th tooth", as the dental formula number of tooth region 1207 inferred from multiple dental formula numbers. Then the process ends. The reason for acquiring the dental formula number of the region that is closer in distance in S0810 is to use more reliable information when the patient has missing teeth.

[0235] In step S1712, if there is an area of a tooth adjacent to the mesial side, the CPU 301 transfers the process to step S1713; otherwise, the CPU 301 transfers the process to step S1715.

[0236] In step S1713, the CPU 301 obtains the tooth formula number of the tooth area adjacent to the mesial side.

[0237] In step S1714, the CPU 301 determines the tooth formula number of the tooth area inferred from the plurality of tooth formula numbers using the obtained tooth formula number.

[0238] In step S1715, the CPU 301 obtains the tooth formula number of the area adjacent to the distal side.

[0239] In step S1716, the CPU 301 determines the tooth formula number of the tooth area inferred from the plurality of tooth formula numbers using the obtained tooth formula number.

[0240] Next, with reference to the flowchart in FIG. 18(A), the error determination and correction process of the order of the tooth formula numbers in step S1305 of FIG. 13 will be described.

[0241] In step S1801, the CPU 301 sorts each tooth area on the left side based on the central Y coordinate.

[0242] In step S1802, if the tooth formula numbers of the sorted tooth areas are arranged in order, the CPU 301 transfers the process to step S1803 assuming that the order is incorrect; otherwise, the CPU 301 transfers the process to step S1804.

[0243] In step S1803, the CPU 301 performs an error correction process for the tooth formula number determined to be incorrect. The detailed process will be described later with reference to FIG. 18(B).

[0244] In step S1804, the CPU 301 sorts each tooth area on the right side based on the central Y coordinate.

[0245] In step S1805, CPU 301 determines that the order is incorrect if the dental formula numbers of each sorted tooth region are in sequential order, and proceeds to step S1806; otherwise, it terminates the process.

[0246] In step S1806, the CPU 301 performs error correction processing on the dental formula numbers that were determined to be incorrect. The detailed processing will be described later using Figure 18(B).

[0247] Next, the error correction process in steps S1803 and S1806 of Figure 18(A) will be explained using flowchart 18(B).

[0248] In step S1807, the CPU 301 calculates the range of tooth regions where the tooth formula numbers are not in sequential order. For example, if the inference result is Figure h12(E), the range where the tooth formula numbers are not in sequential order is "from tooth region 1210 to tooth region 0111".

[0249] In step S1808, CPU301 renumbers the tooth formula numbers in that range so that they are in order. If there are multiple ranges where the tooth formula numbers are not in order, the renumbering is performed for each range.

[0250] There are several methods for correcting errors in dental formula numbers.

[0251] The first method, as described in this embodiment, is for the image processing device 101 to automatically correct the error. Specifically, the CPU 301 deletes the information of the tooth region that was determined to be erroneous from the detection result in step S403 of Figure 4, or changes it to correct content.

[0252] The second method involves the image processing device 101 highlighting the tooth area that has been determined to be incorrect on the display 304 to prompt the user to make corrections, and the user then manually correcting the area based on the highlighting.

[0253] The third method involves the image processing device 101 transmitting erroneous information to the dental electronic medical record system 104, which then displays the erroneous information on the dental electronic medical record display terminal 103 to prompt the user to correct it, and the user then manually correcting it.

[0254] The image processing device 101 may implement any of the first to third methods described above, or it may implement multiple methods and use a different method depending on the type of error. Furthermore, the user may be able to switch between any method through user settings or other means.

[0255] Using Figure 19, an example of the UI to be displayed on display 304 for implementing the second method will be explained.

[0256] 1901 is the input image, and 1902 is the dental diagram. In this example, the tooth region 1904, which includes the lower jaw tooth 1903 captured in the maxillary image 1901, is incorrectly inferred to be the "upper left incisor". Since the distance of tooth region 1904 from the dental arch is greater than the threshold, it is determined to be incorrect in step S1405. The CPU 301 highlights tooth region 1904 with a thick border, and also highlights tooth 1905, which corresponds to the "upper left incisor", with a thick border in the dental diagram 1902, prompting the user to make a correction.

[0257] Next, using Figure 20, we will explain an example of a UI to be displayed on the dental electronic medical record display terminal 103 to implement the third method.

[0258] In this example, the tooth region 2003, which includes the lower tooth 2001 captured in the maxillary image 904, is mistakenly identified as the "upper left incisor." Since the tooth region 2003 is at a distance greater than a threshold from the dental arch, it is determined to be an error in step S1405. Based on the received error information, the dental electronic medical record system 104 highlights the tooth region 2003 in the oral image 904 with a thick border, and also highlights the tooth 2004 corresponding to the "upper left incisor" with a thick border in the dental diagram 903, prompting the user to make a correction.

[0259] Next, using FIG. 21, a second method for correcting an error in the tooth formula number will be described.

[0260] In step S2101, the CPU 301 displays a UI as shown in FIG. 19 on the display 304.

[0261] In step S2102, the CPU 301 highlights the tooth area 1904 determined to be an error.

[0262] In step S2103, the CPU 301 highlights the tooth 1905 in the tooth formula diagram corresponding to the tooth formula number inferred for the tooth area 1904.

[0263] In step S2104, the CPU 301 receives a correction instruction from the user via the input device 306.

[0264] In step S2105, if there is a correction instruction from the user, the CPU 301 transfers the process to step S2106; otherwise, it transfers the process to step S2107. Examples of correction instructions from the user include when the user clicks on 1905, a UI for correction is displayed, and the user performs an operation to input the correction content. A detailed description of the UI for correction is omitted.

[0265] In step S2106, the CPU 301 corrects the data based on the correction instruction received from the user.

[0266] [[ID=二十九]]In step S2107, if there is an end instruction from the user, the CPU 301 ends the process; otherwise, it transfers the process to step S2105.

[0267] In the third method, the processes performed by the CPU 301 are executed by the dental electronic medical record system 104. Further, the processes performed by the display 304 are executed by the dental electronic medical record display terminal 103. Further, the processes performed by the input device 306 are executed by an input device connected to the dental electronic medical record system 104.

[0268] As described above, in this embodiment, the inference results from the model are used to determine whether the resulting dentition is correct based on rules using tables and predetermined conditions. This allows for the appropriate correction of the dental formula number even if the model fails to correctly infer it.

[0269] [Fifth Embodiment] In the above-described embodiment, Figure 2 illustrates an embodiment of the process by which the imaging device 108 acquires patient information in steps S216 to S221. In addition, Figure 2 illustrates an embodiment of the process of associating the image data captured by the imaging device 108 with the previously acquired patient ID in steps S222 to S231. Hereafter, this association will be referred to as linking.

[0270] This embodiment describes an embodiment of the process of associating a patient ID with image data captured by the imaging device 108 using means other than those in Embodiment 1.

[0271] Note that, in this embodiment, the processing other than the process of associating the captured image data with the patient ID is the same as in Embodiment 1, so the explanation will be omitted.

[0272] The dental electronic medical record display terminal 10) in Figure 1 is connected to the dental electronic medical record system 104 and displays information about the examined patient. In actual hospitals, multiple dental electronic medical record display terminals are often connected to the dental electronic medical record system 104.

[0273] Figure 22 shows an example configuration in which five dental electronic medical record display terminals are connected to a dental electronic medical record system 104. One dental electronic medical record display terminal 2203a1, 2203a2, 2203b1~2203b3 is installed in each examination space (examination space A1, examination space A2, examination spaces B1~B3). Similarly, one chair 2201a1, 2201a2, 2201b1~2201b3 is installed in each examination space (examination space A1, examination space A2, examination spaces B1~B3).

[0274] In the system configuration described above, when taking photographs of a patient's oral cavity during examination, the imaging device used may be one unit installed in each examination space, or one imaging device may be used in multiple examination spaces. Therefore, the image processing device 101 maintains management data that controls which examination space, chair, and dental electronic medical record display terminal 103 the imaging device 108 is used in.

[0275] Figure 23 shows an example of the aforementioned management data structure held by the image processing device 101. This management data structure 2301 holds, for example, the space ID 2302 and the chair ID (2303). Furthermore, it holds the dental electronic medical record display terminal ID 2304, the dental electronic medical record display terminal IP address 2305, the imaging device ID 2306 used by the dental electronic medical record display terminal, and the aforementioned imaging device IP address 2307. Figure 23 shows an example in which imaging device Camera-101 is used in examination spaces A1 and A2, and imaging device Camera-201 is used in examination spaces B1, B2, and B3.

[0276] The means for setting such management data may be implemented in a program implemented in the image processing device 101, or it may be a program on a device that can communicate with the image processing device 101.

[0277] Figure 24 shows an example of a UI for selecting a dental electronic medical record display terminal to be used by the imaging device 108 on the rear LCD screen 2401. In the example in Figure 23 described above, the records of P-A01 (2403) and P-A02 (2404), which are dental electronic medical record display terminals registered for use by the imaging device Camera-101, are displayed in the selection list. The user selects record 2404 from the displayed selection list, which contains the dental electronic medical record display terminal set up in the examination space where the patient to be photographed will be examined. The means by which the user can select a record may be the touch panel on the rear LCD screen 2401 of the imaging device 108, or a physical selection device such as a button provided on the imaging device 108. After selecting the desired record as described above, the user presses the setting button 2405 if they want to retain the selection state on the imaging device 108. Conversely, if they do not want to retain the selection state on the imaging device 108, they press the cancel button 2406.

[0278] As a result, it is possible to maintain the state in which the target dental electronic medical record display terminal 103 is set on a specific imaging device 108.

[0279] Using Figure 25, we will explain the process by which the imaging device 108 identifies the patient ID based on the information captured by the imaging device 108 and how the imaging device 108 stores the patient information.

[0280] In step S2501, the imaging device 108 captures patient ID information. As mentioned above, the patient ID information to be captured only needs to be uniquely identifiable as a patient ID, and in this embodiment, a patient ID barcode will be used as an example. This patient ID barcode may be displayed on a dental electronic medical record display terminal 103 in a specific examination space when a patient is examined, as shown in Figure 26 (2602 in Figure 26(A)). Alternatively, it may be printed on something distributed to the patient by the hospital when they register for their examination, such as a patient ID wristband (2604 in Figure 26(B)) (2605 in Figure 26(B)).

[0281] In step S2502, the imaging device 108 captures the aforementioned patient ID barcode and transmits it to the image processing device 101 as a patient information acquisition request.

[0282] In step S2503, the image processing device 101 performs barcode decoding on the image data received from the imaging device 108 along with the patient information acquisition request. Subsequently, the image processing device 101 stores the resulting patient ID in the image processing device 101.

[0283] In step S2504, the image processing device 101 processes a patient information acquisition request to the dental electronic medical record system 104, along with the patient ID it has previously stored. In step S2505, the dental electronic medical record system 104 transmits patient information to the image processing device 101 in response to the patient information acquisition request it received from the image processing device 101.

[0284] In step S2506, the image processing device 101 transmits the patient information received from the dental electronic medical record system 104 to the imaging device 108 as a reply to the patient information acquisition request received from the imaging device 108.

[0285] In step S2507, the imaging device 108 displays patient information received from the image processing device 101, such as the patient's name and gender, on the display module of the imaging device 108, such as the rear LCD screen 2401.

[0286] In step S2508, the user confirms the patient information displayed on the display module of the imaging device 108. If the imaging device 108 confirms that the patient is the same as the one being photographed, the user performs a confirmation completion operation. Upon receiving the confirmation completion operation, the imaging device 108 proceeds to step S2509.

[0287] In step S2509, the imaging device 108 stores the patient information as the patient information to be photographed. 25 On the other hand, if the patient information displayed on the display module of the imaging device 108 is not the same as the patient to be photographed, the process returns to step S2501 and restarts from the process of photographing the barcode of the correct patient ID to be photographed.

[0288] On the other hand, this section describes a method in which the imaging device 108 acquires patient information by identifying the patient ID using the information displayed on the dental electronic medical record display terminal 103 in the examination space during the examination, which is that of the patient being examined.

[0289] Using Figure 27, we will explain the process of identifying a patient ID and storing patient information based on the information captured by the imaging device 108.

[0290] In step S2701, the imaging device 108 captures information that identifies the dental electronic medical record display terminal 103. The information to be captured that identifies the dental electronic medical record display terminal 103 only needs to uniquely identify the dental electronic medical record display terminal 103. As shown in Figure 28, this could be a barcode 2801 attached to the dental electronic medical record display terminal 103, or it could be something that identifies the chair 2201a located in the examination space where the dental electronic medical record display terminal 103 is located. As a specific example, it could be a barcode 2802 attached to the chair, or a number 2803 that identifies the chair. However, the process by which the image processing device 101 decodes the aforementioned information based on the image data of the information that identifies the dental electronic medical record display terminal 103 captured by the imaging device 108 described later will be a decoding process corresponding to the aforementioned information type. In this embodiment, a barcode will be used as an example to explain the information that identifies the dental electronic medical record display terminal 103.

[0291] In step S2702, the imaging device 108 captures a barcode that identifies the dental electronic medical record display terminal 103 and transmits it to the image processing device 101 as a patient information acquisition request.

[0292] In step S2703, the image processing device 101 performs barcode decoding on the image data received from the imaging device 108 along with the patient information acquisition request. Subsequently, the image processing device 101 stores an ID that identifies the dental electronic medical record display terminal 103, which is the result of the decoding.

[0293] In step S2704, the image processing device 101 performs a patient information acquisition request process to the dental electronic medical record system 104, along with the ID that identifies the previously held dental electronic medical record display terminal 103.

[0294] In step S2705, the dental electronic medical record system 104 obtains the patient ID currently displayed on the identified dental electronic medical record display terminal 103 based on the ID that identifies the dental electronic medical record display terminal 103 received from the image processing device 101.

[0295] In step S2706, the dental electronic medical record system 104 retrieves patient information from the patient ID.

[0296] In step S2707, the dental electronic medical record system 104 transmits the patient information acquired above to the image processing device 101.

[0297] In step S2708, the image processing device 101 transmits the received patient information to the imaging device 108.

[0298] In step S2709, the imaging device 108 displays patient information received from the image processing device 101, such as the patient's name and gender, on its own display module, such as the rear LCD screen 2401.

[0299] In step S2710, the user confirms the patient information displayed on the display module of the imaging device 108. If the imaging device 108 confirms that the patient is the same as the one to be photographed, the user performs a confirmation completion operation. Upon receiving the confirmation completion operation, the imaging device 108 proceeds to step S2711.

[0300] In step S2711, the imaging device 108 stores the patient information as the patient information to be imaged.

[0301] On the other hand, if the patient information displayed on the aforementioned imaging device 108 is not the same as that of the patient to be photographed, the process returns to step S2701, and the dental electronic medical record display terminal 103, imaging device 108, etc., are checked and the process is restarted from barcode scanning.

[0302] Furthermore, as explained in Figure 24, patient information can also be acquired by performing steps S2704 to S2711, following the process of selecting the target dental electronic medical record display terminal on the rear LCD screen 2401 of the imaging device 108.

[0303] Using Figure 29, we will explain the process by which the imaging device 108 associates and saves the patient information, particularly the patient ID, acquired as described above, with the image data captured by the imaging device 108.

[0304] As described above, the imaging device 108 confirmed that the patient to be photographed and the patient information acquired were the same (steps S2508, S2710).

[0305] In step S2901, the imaging device 108 takes an image of the patient's oral cavity at the user's discretion.

[0306] In step S2902, the imaging device 108 associates the previously held patient information with the image data output by the aforementioned acquisition and stores it.

[0307] In step S2903, the imaging device 108 sends a request to the image processing device 101 to save the captured image.

[0308] In step S2904, the image processing device 101 similarly transmits the image storage request received from the imaging device 108 to the dental electronic medical record system 104.

[0309] In step S2905, the dental electronic medical record system 104, upon receiving a request from the image processing device 101 to save the captured image data received along with the request, sends a save request to the dental information DB 105.

[0310] In step S2906, the dental information DB 105 stores the image storage request and image data received from the dental electronic medical record system 104.

[0311] In step S2907, a notification of completion of saving the aforementioned captured image data is sent from the dental information DB 105 to the dental electronic medical record system 104.

[0312] In step S2908, a notification of completion of saving the aforementioned captured image data is sent from the dental electronic medical record system 104 to the image processing device 101.

[0313] In step S2909, a notification that the aforementioned captured image data has been saved is sent to the image processing device 101 and the imaging device 108.

[0314] In step S2910, the imaging device 108 deletes the image data from the imaging device 108 upon receiving the aforementioned notification from the image processing device that the image data has been saved.

[0315] As a result, the image data of a patient captured by the imaging device 108 can be stored in association with the patient's information.

[0316] Using Figure 30, we will explain the process by which the image processing device 101 associates and saves the patient information, particularly the patient ID, acquired as described above, with the image data captured by the imaging device 108.

[0317] In step S3001, the imaging device 108 takes an image of the patient's oral cavity at the user's discretion.

[0318] In step S3002, the imaging device 108 sends a request to the image processing device 101 to save the captured image.

[0319] In step S3003, the image processing device 101 associates the image data received from the imaging device 108 with the patient information sent along with the image data and stores it there.

[0320] In step S3004, the image processing device 101 sends a request to save the captured image to the dental electronic medical record system 104 along with the data that has undergone the aforementioned linking process.

[0321] In step S3005, the dental electronic medical record system 104, upon receiving a request from the image processing device 101 to save the captured image data received along with the request, sends a save request to the dental information DB 105.

[0322] In step S3006, the dental information DB 105 stores the image storage request and image data received from the dental electronic medical record system 104.

[0323] In step S3007, a notification that the aforementioned captured image data has been saved is sent from the dental information DB 105 to the dental electronic medical record system 104.

[0324] In step S3008, the image is transmitted from the dental electronic medical record system 104 to the image processing device 101.

[0325] In step S3009, the image is transmitted from the image processing device 101 to the imaging device 108.

[0326] In step S3010, the imaging device 108 deletes the image data from the imaging device 108 upon receiving the aforementioned notification from the image processing device that the image data has been saved.

[0327] As a result, the patient image data captured by the imaging device 108 can be stored linked to the patient's information immediately after acquisition, or before being transmitted to the dental electronic medical record after acquisition.

[0328] Regarding the aforementioned process of linking captured image data with patient information, patient information may be embedded in the actual image data file, or a separate file containing patient information may be generated and linked to the actual image data file. Furthermore, in the dental information DB105, the image data and patient information data may be linked in the database during the image saving process performed in response to an image saving request.

[0329] [Sixth Embodiment] In the embodiments described above, it is necessary to create a machine learning model for determining the condition of teeth. To create a tooth condition determination model, diagnostic result data created by a physician is compared with oral images, and training data is created by labeling each tooth in the oral images with the condition of the tooth (hereinafter referred to as annotation). Then, a tooth condition learning process is performed to create a tooth condition determination model.

[0330] However, training a highly accurate model requires a large amount of training data, and manually matching and annotating diagnostic results with images to prepare such a large amount of training data requires a tremendous amount of effort.

[0331] This embodiment describes an image processing device for generating a highly accurate tooth condition determination model with minimal effort by automatically creating such a large amount of training data.

[0332] Figure 31 is a flowchart illustrating the training data generation process in this embodiment. Note that the image processing device in this embodiment may be the same as the image processing device 300, or it may be a different device equipped with a CPU, etc. In this embodiment, it is assumed that the processing is executed on the image processing device 300.

[0333] In step S3101, the CPU 301 loads images for training. The images to be loaded may be oral images of patients acquired directly from the imaging device 108, or they may be loaded and acquired from oral images stored in the storage of the image processing device 306.

[0334] In step S3102, the CPU 301 loads a model for inferring tooth position and dental formula number. The model loaded at this time is the same as the model loaded in step S402 in Figure 4.

[0335] In step S3103, the CPU 301 uses the model read in step S3102 to perform inference of tooth position and dental formula number for the oral image acquired in step S3101. As a result, the region of each tooth is detected in the image, and the dental formula number inference result for each detection result is obtained. At this time, there is a possibility of errors in the inference of the dental formula number, so error correction processing as described in Embodiment 4 may be performed.

[0336] In step S3104, the CPU 301 accesses the dental information database 105, where the results of the dentist's patient diagnosis are stored. Then, it retrieves the patient's dental information corresponding to the tooth image data read in step S3101 via the patient's dental electronic medical record system 104. The retrieved dental information is shown in Figure 32. It includes the dental formula number for each of the patient's teeth and the condition of the teeth, such as cavities and prosthetics.

[0337] In step S3105, the CPU 301 acquires one of the tooth regions detected in step S3103 that has not yet been associated with a tooth condition.

[0338] In step S3106, the CPU 301 associates each tooth region acquired in step S3105 with the dental information acquired in step S3105 based on the tooth formula number and generates training data. An example of the associated training data is shown in Figure 33. The tooth determined to be "lower left 7th tooth" in step S3103 is assigned the label information "C2" because the tooth condition is "C2" based on the dental information acquired in step S3104.

[0339] In step S3107, if the CPU 301 has completed associating the tooth state with all tooth regions, it proceeds to step S3108; otherwise, it proceeds to step S3105.

[0340] In step S3108, the CPU 301 takes the tooth state learning data created in steps S3105 to S3107 as input and executes the learning process in the tooth state learning processing unit. The machine learning algorithm is assumed to be a commonly used one, so its explanation is omitted.

[0341] In step S3109, CPU 301 saves the tooth condition determination model created in the learning process of step S3108 onto HDD 306 and uses it for tooth condition determination in embodiment 1 and other embodiments.

[0342] Thus, according to the present invention, oral images and corresponding dental diagnosis results are acquired and compared from a dental electronic medical record system. This allows for the automatic creation of a large amount of training data, making it possible to create a highly accurate tooth condition assessment model with minimal effort.

[0343] [Seventh Embodiment] In the embodiments described above, a technique for inferring tooth position and number using a model created by machine learning was explained in the dental formula inference processing step S240. However, the model may not be able to correctly infer the dental formula number or state from a single image. In the fourth embodiment, a method for determining whether the number inference result is correct as a dental arch and automatically correcting errors was described, but errors cannot be corrected if, for example, tooth detection itself fails. Alternatively, in the first embodiment, a method for comprehensively determining the state of each tooth from the inference results of images of the occlusal surface, front view, left lateral view, and right lateral view was described. However, if any of the state inference results in each image are incorrect and they cannot coexist in the same tooth, there is a problem that the determination result will be inconsistent. Furthermore, in the first embodiment, a method for correcting the inference results by operation by a dentist was described. However, for example, if one inference result is incorrect and the effect extends to multiple teeth in the process of associating dental formula numbers with states, the dentist has to correct the errors of all teeth themselves, which is inefficient.

[0344] In this embodiment, we describe an image processing device that, even if the dental formula number and state cannot be correctly inferred in the dental formula inference processing step S240 as described above, corrects the error by using the inference results of other images, and further automatically re-associates the dental formula number and state in accordance with the corrections made by the dentist.

[0345] In this embodiment, descriptions of parts similar to those in the above-described embodiment will be omitted, and the description will mainly focus on the configurations unique to this embodiment.

[0346] Figure 37 is a flowchart showing the operation of the image processing device 101 in this embodiment during the dental formula inference processing step S240 in Figure 2. The process up to the output of the inference result list shown in Figure 34 will be explained using Figure 37.

[0347] Steps S3701 to S3705 perform the same processing as steps S401 to S405 in Figure 4.

[0348] In step S3706, the CPU 301 records the inference result of the tooth formula number from step S3703 and the inference result of the state from step S3705 as an inference result list 3400 as shown in Figure 34.

[0349] Next, the comprehensive judgment process in this embodiment will be explained using Figures 38 to 45.

[0350] <Overall Judgment Processing> Figure 38 shows a flowchart illustrating the process of making an overall judgment based on the inference result list containing the inference processing results. The process in this flowchart corresponds to step S241 in Figure 2.

[0351] In step S3801, the CPU 301 obtains the inference results for the tooth formula number and state of each tooth on the imaging plane from the inference result list 3400.

[0352] In step S3802, the CPU 301 performs image alignment processing. This process modifies the inference result list 3400 so that the lateral position and size of the same tooth captured on each imaging plane match. The detailed processing will be described later using Figure 39.

[0353] In step S3803, the CPU 301 performs tooth number correction processing using information from multiple imaging surfaces. This process corrects any missing or falsely detected tooth numbers for each imaging surface in the inference result list 3400. The detailed process will be described later with reference to Figure 42.

[0354] In step S3804, the CPU 301 performs the same processing as in step S406 in Figure 4, associating the position and state of each tooth from the dental formula number detection result and the state detection result, and records it in the judgment list 700.

[0355] In step S3805, the CPU 301 performs a state determination process using information from multiple imaging surfaces. This process determines the overall determination 708 of the determination list 700 so that there are no discrepancies in the state of the same tooth as captured on each imaging surface. The detailed process will be described later with reference to Figure 44.

[0356] <Image alignment process> Next, using flowchart 39, the image alignment process in step S3802 of Figure 38 will be explained. This process is performed to align the images and inference results, as it is desirable that the position coordinates (X coordinates) of each tooth be the same in the multiple intraoral images used in the dental formula number correction process using information from multiple imaging planes, which will be described later.

[0357] In step S3901, the CPU 301 performs distortion correction processing on the frontal view image and its inference results. Since the frontal view image is a perspective projection of the dental arch, distortion occurs. Teeth further back in the dental arch appear smaller in the image due to their greater distance from the lens, resulting in a discrepancy in the tooth positional relationship between the occlusal view and the frontal view. This processing corrects this discrepancy. Specifically, the processing is performed as follows: It is assumed that the dental arch can be approximated as a three-dimensional object V obtained by dividing an elliptical cylinder in half along its minor axis. The major axis radius is assumed to be L, the minor axis radius is assumed to be S, the major axis coincides with the optical axis of the lens, and the focal point is near the incisors, which is at the end of the major axis. L and S are set using the average size of the dental arch of a normal adult occlusal patient. Additionally, the focal length f of the lens, the distance Z from the lens to the subject (focal point), and the width and height S of the sensor are obtained from the image's EXIF ​​data. w ×S h Obtain the information. Assume that the area of ​​the 3D V occupies 90% of the image width.

[0358] At this point, the front view image can be considered as a perspective projection of the stereoscopic V, as shown in Figure 41. Therefore, based on the information obtained in step S3901, the perspective projection is calculated inversely for each pixel, and then parallel projection is performed to transform the image into one without distortion. At this time, the coordinates of each rectangle in the inference result list are also recalculated using the same calculation as for the image. Similar processing may be performed for the left and right side views, but this may be omitted because the variation in the distance between each tooth and the lens is relatively smaller than that of the front view. Alternatively, instead of this step, a step that considers the positional relationship shift may be added in the processing shown in Figure 42(A), which will be described later. Details will be described later.

[0359] In step S3902, the CPU 301 performs position and size correction processing on the image using the inference results of teeth with good detection accuracy. Specifically, the processing is performed as follows. In this embodiment, the X coordinate position of the rectangle of the inference results (4001, 4002) for the upper right first tooth is adjusted.

[0360] First, calculate the length of the x-side of the inference result 4001 for the upper right incisor tooth in the maxillary image, and the length of the x-side of the inference result 4001 for the upper right incisor tooth in the frontal image. The former is the lower right X coordinate (x) of the inference result rectangle for the upper right incisor tooth in the maxillary image. max_ur1 ) and the lower left X coordinate (x min_ur1 The difference is that the lower right X coordinate (x) of the inference result rectangle of the upper right first tooth in the frontal image. max_fr1 ) and the lower left X coordinate (x min_fr1 ) is the difference. The former is L ur1 The latter is L fr1 Yes. In other words, L ur1 =(x max_ur1 )―(x min_ur1 ). L fr1 =(x max_fr1 )―(x min_fr1 ), and next, the calculated L ur1 and L fr1 Compare them. Here, (L ur1 )<(L fr1 ) Therefore, (L fr1 / L ur1 ) Maxillary image magnification R r1 Let's assume that.

[0361] The magnification R calculated in this way r1 The result of applying this to the maxillary image 4003 is the enlarged maxillary image 4005 in Figure 40(B). As a result, the inferred rectangular coordinates of each tooth are close in the maxillary image 4005 and the frontal image 4006 in Figure 40(B). Note that the change in the coordinates of the inferred rectangle due to image enlargement (reduction) is related to the aforementioned magnification ratio R r1 The coordinate information of the inference result rectangle is determined by recalculating it using [a specific method / tool].

[0362] Specifically, for the inference result data output in Figure 34 above, the coordinates x of the left edge of the rectangle exist for each detected object. min , coordinate x at the rightmost end max , upper end coordinate y min , coordinate y of the lower end max The aforementioned magnification R r1 The data will be updated based on this. Additionally, the image may be moved so that the X-coordinate of the midline of the frontal view image, obtained using the same method as in step S1508 in Figure 15, matches the X-coordinate of the midline of the maxillary image, obtained using the same method as in step S1510. Furthermore, the CPU 301 may calculate the tilt of the maxillary image using the same method as in step S1529 and rotate the maxillary image according to the calculated tilt so that the maxillary image is in the correct orientation.

[0363] <Correction process for dental formula number> Next, using flowcharts 42(A) and (B), the tooth number correction process using information from multiple images in step S3803 of Figure 38 will be explained. The tooth number correction process may be performed using both Figures 42(A) and (B), or either one of them may be performed.

[0364] Figure 42(A) illustrates a process that corrects missed detection of tooth formula numbers on imaging surfaces that do not have a high recognition rate, based on the knowledge that there are differences in tooth recognition rates depending on the imaging surface. However, if the fixed flag of the inference result to be corrected in the inference result list 3400 is set to true, it is considered that the correction has already been made by a dentist and no correction is necessary, and no correction is performed. In this embodiment, an example is described in which the inference result of the tooth formula number of the target is corrected, with either the upper or lower occlusal view as the source for correction and one of the frontal view, left lateral view, or right lateral view as the target for correction.

[0365] In step S4201, the CPU 301 obtains the inference results for the tooth formula numbers in the source and target teeth from the inference result list 3400.

[0366] In step S4202, the CPU 301 links the rectangles of the source and target correction inference results based on their positional information. The specific method is explained using Figures 43(A) and (B). Figure 43(A) shows an example of the linking method when the source correction is an occlusal view of the maxilla and the target correction is a frontal view. Assume that the source image has inference results 4301 to 4304, and the target image has inference results 4305, 4307, and 4308. Only some of the inference results are shown for simplicity. Among these, the rectangles with the closest position in the X-axis direction are linked.

[0367] This association can be performed by selecting the rectangle with the closest X-coordinate of its center, as in 4301 and 4305, or by selecting the rectangle with the largest overlap between the left and right edges. In either case, if the difference in the X-coordinates of the centers is greater than or equal to a threshold, or if the overlapping area is less than or equal to a threshold, it is determined that there is no matching object.

[0368] Furthermore, to prevent incorrectly linking the lower jaw teeth, rectangles whose centers are below the center of the image can be excluded from the linking process, or only rectangles that are inferred to be upper jaw teeth in the correction process can be included in the linking process. Also, the further back a tooth is in the dental arch, the larger the area hidden by the teeth in front in the frontal view, which can lead to a larger shift in the center's X-coordinate and potentially result in incorrect linking. In this example, 4304 and 4308 are the correct link, but they are incorrectly linked to 4307, which has the closest center's X-coordinate.

[0369] Therefore, for example, as in 4310, the center coordinates can be determined using only the areas that extend beyond the left and right edges of rectangle 4309, which is located above itself, and used for linking (linking 4310 and 4311). Figure 43(B) shows an example of a linking method when the source image is an occlusal view of the maxilla and the target image is a left lateral view. Assume that the source image has inference results 4312-4314 and the target image has inference results 4315-4317. Only some of the inference results are shown for simplicity of explanation.

[0370] First, the image of the occlusal view from the source of correction and the inference result are rotated by a certain angle so that the top of the image is at the position of the camera that captured the left lateral view. The rotation angle is generally considered to be the angle at which the left lateral view is captured, and in this embodiment, it is set to 45° counterclockwise. The subsequent linking of rectangles is the same as the method described in Figure 43(A).

[0371] Alternatively, instead of performing the lens correction process in step S3902 of Figure 39, this step may be used to perform processing that takes into account the positional shift caused by distortion due to perspective projection. That is, when comparing positions in the X-axis direction, instead of comparing the values ​​as they are, the coordinates are recalculated taking the shift into account and compared with the recalculated coordinates. An example of this is shown in Figure 43(C). From the focal length f of the lens obtained from the EXIF ​​of the image to be corrected, the distance Z from the lens to the subject (focus), and the size l of the subject per image pixel obtained from the EXIF ​​of the image to be corrected, the pixel-converted values ​​f'=f / l and Z'=Z / l are obtained. If the center coordinates of each rectangle are (x, y) with respect to the intersection of the top edge of the topmost rectangle in the image to which the image is divided horizontally and the origin, then the recalculated coordinates (x', y') are x'=x×(Z'-y') / (Z'-y) and y'=Z'-f', so finally x'=x×f' / (Z'-y). When linking rectangles, compare the X-coordinate of the center of the target rectangle with the value calculated above for the X-coordinate of the center of the source rectangle.

[0372] Steps S4203 to S4209 are a loop in which the CPU 301 sequentially processes the inference results of the original correction. Let the original inference result be rectangle A.

[0373] In step S4204, the CPU 301 proceeds to step S4205 if there is an inference result for the correction target associated in step S4202 for rectangle A; otherwise, it proceeds to step S4207. The associated inference result is designated as rectangle B.

[0374] In step S4205, if the labels of rectangle A and rectangle B are the same, the CPU 301 proceeds to step S4206; otherwise, it proceeds to step S4209.

[0375] In step S4206, the CPU 301 overwrites the label of rectangle B with the label of rectangle A and updates the inference result list 3400. However, if the fixed flag for rectangle B is true, no change is made.

[0376] In step S4207, the CPU 301 sets a candidate rectangle B' to be associated with the correction target. If the average brightness of the pixels within rectangle B' is greater than or equal to a threshold, the process proceeds to step S4208; otherwise, the process proceeds to step S4209. The left and right ends of rectangle B' are at the same positions as rectangle A, and the top and bottom ends are the average of the two rectangles closest to the left and right ends, respectively, from the inference results of the correction target. The candidate rectangles associated with 4302 in Figure 43(A) are shown as dashed lines in 4306.

[0377] In step S4208, the CPU 301 considers rectangle B', which was set in step S4207, to be undetected, sets the label to be corrected, adds it to the corrected inference result, and updates the inference result list 3400.

[0378] Steps S4210 to S4213 are a loop in which the CPU 301 sequentially processes the corrected inference result. The corrected inference result is denoted as rectangle C.

[0379] In step S4211, the CPU 301 proceeds to step S4213 if there is an inference result for the correction source associated in step S4202 with rectangle C, and to step S4212 otherwise.

[0380] In step S4212, the CPU 301 considers rectangle C to be a false detection, removes it from the corrected inference results, and updates the inference result list 3400. However, if the fixed flag for rectangle C is true, it is not removed.

[0381] <Correction process for inference results> Figure 42(B) illustrates the process of correcting the inference result of the dental formula number for the same image based on the state inference result for the same image. In this embodiment, if the inference result includes a state that can be detected with a high recognition rate, the missing dental formula number is corrected based on the knowledge that a tooth is present at the same location with a high probability.

[0382] In step S4214, the CPU 301 obtains the inference result for the dental formula number.

[0383] In step S4215, the CPU 301 obtains the state inference results that can be used for correction. State inference results that can be used for correction are those that have labels whose recognition rate by the learning model is above a threshold.

[0384] Steps S4216 to S4219 are a loop in which the CPU 301 sequentially processes the state inference results obtained in step S4215. Let the state inference result be rectangle A.

[0385] In step S4217, if the CPU 301 finds a rectangle among the dental formula number inference results obtained in step S4214 whose overlap with rectangle A is greater than or equal to a threshold, it proceeds to step S4219; otherwise, it proceeds to step S4218.

[0386] In step S4218, the CPU 301 sets a new rectangle at the same position as rectangle A, sets a label to make it sequential with the rectangles in the other dental number inference results that are in close proximity, and adds it to the dental number inference results.

[0387] <State determination processing based on multiple images> Next, using flowchart 44, we will explain the state determination process using information from multiple imaging planes in step S3805 of Figure 38.

[0388] Steps S4401 through S4408 are a loop that processes all dental formula numbers in order.

[0389] In step S4402, the CPU 301 obtains the inference results for all imaging planes, corresponding to the target dental formula number in step S3804.

[0390] In step S4403, if the CPU 301 finds any incompatible combinations among the state inference results obtained in step S4402, it proceeds to step S4404; otherwise, it proceeds to step S4407. Incompatible combinations refer to combinations where both cannot exist on the same tooth, such as a veneer crown covering the entire tooth and a bridge. Conversely, examples of compatible combinations include gingivitis and a crown. The list of incompatible combinations may be written in the program beforehand, or it may be stored on a storage medium and read from there for easier modification later.

[0391] In step S4404, the CPU 301 proceeds to step S4405 if the incompatible combination extracted in step S4403 exists in the overall judgment model, and proceeds to step S4406 otherwise. The overall judgment model takes the inference results of the state from each imaging surface as input and estimates the state of the tooth based on prior knowledge regarding differences in appearance from different viewpoints, such as the shape and surface color of the object being inferred. In this embodiment, as shown in Figure 45, it is assumed to be a matrix that derives the overall judgment from the combination of each imaging surface and the inference results. For example, if it is inferred to be a resin-veneered crown in the occlusal view and a tooth-colored crown in the frontal view, the overall judgment will be a resin-veneered crown. Also, if it is inferred to be a metal bridge in the occlusal view and a tooth-colored pontic in the frontal view, the overall judgment will be a resin-veneered pontic.

[0392] In step S4405, the CPU 301 replaces the incompatible combinations extracted in step S4403 with the judgment results of the comprehensive judgment model.

[0393] In step S4406, the CPU 301 considers that one of the incompatible combinations extracted in step S4403 was incorrectly inferred, and removes the one with the lower recognition rate by the learning model. Alternatively, the one with the lower accuracy during the inference run by the learning model may also be removed.

[0394] In step S4407, the CPU 301 records all remaining inference results in the overall judgment of the judgment list 700.

[0395] <Regarding corrections to inference results> Using Figure 46, we will explain the process for correcting inference results in a dental electronic medical record display terminal UI by manipulating a rectangle superimposed on an image.

[0396] Figure 46(A) shows the dental electronic medical record display terminal UI for checking and correcting inference results in this embodiment. Unless otherwise specified, the UI is the same as in Figure 9. 4601 is a UI that displays the inference results of the dental formula numbers recorded in the inference result list 3400, superimposed on the image. 4602, 4605, and 4606 show the inferred tooth positions and numbers. 4612 is the inference result for the prosthesis. The user can move the rectangle by clicking and dragging and dropping it on 4601. The user can also enlarge or reduce the rectangle in the direction of selection by dragging and dropping it after selecting one of its edges. At this time, the CPU 301 updates the position information of the manipulated rectangle in the inference result list 3400 and sets the fixed flag to true. 4604 is a UI that displays the state of each tooth recorded in the overall judgment 708 of the judgment list 700, aligned with the dental formula diagram. 4607 and 4608 show the tooth states as text. The inference results can also be corrected by user operations on 4604. Detailed processing will be described later using Figure 47. 4603 is a recalculation button that the user can press. When the recalculation button 4603 is pressed, the CPU 301 executes the comprehensive judgment processing step S241 in Figure 2 and reflects the result in 4604.

[0397] As an example, let's explain the process when the user moves the rectangle 4606 to the position of 4609 and presses the recalculation button 4603. Before processing, the position of tooth L6 was incorrectly inferred, resulting in 4606 being displayed in a position overlapping with tooth L5. Therefore, although the prosthesis 4612 should be associated with L6 by step S3804 in Figure 38, it is associated with L7. Correctly, 4606 is expected to be displayed superimposed on L6, not L5, so this needs to be corrected. Therefore, the user moves the rectangle 4606 to the position of 4609, as shown in Figure 46(B). After that, the user presses the recalculation button 4603. As a result, according to the result of the overall judgment, as shown in Figure 46(B), the CPU 301 changes the status string displayed in 4607 from "Prosthetic" to "Healthy" and the status string displayed in 4608 from "Healthy" to "Prosthetic".

[0398] Next, using Figures 47-51, we will explain the process for correcting inference results in a dental electronic medical record display terminal UI by having the user operate the UI on the dental diagram.

[0399] Using flowchart 47, we will explain the process of correcting inference results through user operations on 4604. This flowchart describes the process from when the dental electronic medical record display terminal UI is displayed until the user presses the recalculation button.

[0400] In step S4701, the CPU 301 reads the tooth number inference results from the inference result list 3400 into the RAM 303 and displays them on the dental electronic medical record display terminal UI.

[0401] In step S4702, the CPU 301 determines whether or not a user operation has occurred. User operation refers to an operation performed on 4604 using a pointing device such as a mouse. If the CPU 301 determines that a user operation has occurred, it proceeds to step S4718; otherwise, it proceeds to step S4702.

[0402] In step S4718, if the CPU 301 determines that the user has pressed the recalculation button 4603, it proceeds to step S4714; otherwise, it proceeds to step S4716.

[0403] In step S4714, the CPU 301 uses the tooth number and position information stored in RAM 303 to perform the process of correcting the tooth position and number as explained in Figure 13. This corrects the tooth numbers that have become inconsistent due to user-initiated corrections, which will be described later.

[0404] In step S4715, the CPU 301 executes the comprehensive judgment process step S241 shown in Figure 2. As a result, the tooth with the closest center coordinates to the inferred tooth position and state is associated with it.

[0405] In step S4719, CPU301 updates the display of the dental electronic medical record display terminal UI with the corrected inference results. This allows the user to visually confirm the presence and condition of teeth.

[0406] In step S4716, if the CPU 301 determines that the user has changed the state of a tooth from missing to present, the process proceeds to step S4717; otherwise, the process proceeds to step S4703. An example of this operation is illustrated in Figure 49. Unless otherwise specified, the process is the same as in Figure 46. 4901 corresponds to 4601, and 4904 corresponds to 4604. 4905 indicates that the L2 tooth should have been detected, but it was not detected. Therefore, the user wants to correct the situation so that a tooth exists as L2 at the position of 4905. The procedure for doing this is for the user to right-click on L2 at 4904, which causes the CPU 301 to display a state change list like 4907. The state change list 4907 is a UI that displays a list of operations that can be performed on a tooth, depending on the state of the tooth that was right-clicked. Right-clicking is described as an example of an operation to display the state change list 4907, but it can also be displayed by other operations such as mouseover. The status change list 4907 displays "Change to present" if the target tooth is missing, and "Change to missing" otherwise. Furthermore, if the number of non-missing teeth among those displayed in 4904 is seven or less, it also displays "Split" and "Add". By displaying the necessary items according to the status, the user can select only the operations they need. In this example, since the tooth's status is missing, "Change to present" is displayed. The user performs the operation by pressing "Change to present". Additionally, the CPU 301 may save the operations selected by the user in the status change list 4907 for each tooth as a history in RAM 303, and allow the order of operations displayed in the status change list 4907 to be changed to the order in which they are most frequently saved in the history. This makes it easier for the user to find the operations they are most likely to perform.

[0407] In step S4717, CPU 301 executes the process for when the user changes the presence or absence of teeth from "absent" to "present". A detailed explanation of the process will be given later using Figure 50. After this, the recalculation button is pressed, and as a result of CPU 301 executing steps S4714 and S4715, the state of 4909 is changed from missing to healthy.

[0408] In step S4703, CPU301 proceeds to step S4704 if the user has performed an operation to add a tooth; otherwise, it proceeds to step S4706. An example of this operation is illustrated in Figure 49. 4906 indicates that tooth L6 should have been detected, but it has been missed. Additionally, tooth L7 is detected as L6, and tooth L8 is detected as L7. Therefore, the user wants to correct 4906 so that tooth L6 is present. To do this, the user right-clicks on L6 in 4904, causing CPU301 to display a state change list like 4908. State change list 4908 has the same function as 4907, and in this example, "Add" is displayed. The user performs the operation by clicking "Add".

[0409] In step S4704, CPU301 performs the process of adding a tooth and determines the number of tooth L6 and its position. A detailed explanation of the process will be described later using Figure 51. After this, the recalculation button is pressed, and CPU301 performs steps S4714 and S4715. As a result, the status of 4910 is changed from prosthetic to healthy, the status of 4911 is changed from prosthetic to prosthetic, and the status of 4912 is changed from missing to prosthetic.

[0410] In step S4706, CPU301 proceeds to step S4707 if a tooth has been deleted, and to step S4708 otherwise. An example of this operation is illustrated using Figure 48. Unless otherwise specified, the process is the same as in Figure 46. 4801 corresponds to 4601, and 4804 corresponds to 4604. 4805 indicates that if an object other than a tooth, 4812, was captured at the position of L1, that object was mistakenly inferred as the tooth at L1. Therefore, the user wants to correct L1 to be missing. The procedure for this is to drag and drop the symbol or state indicating missing tooth, 4814, from the state list shown in 4813 to L1 in 4816. At this time, CPU301 displays the missing tooth symbol in 4816. Note that the state list 4813 shows prostheses and missing teeth as an example, but other states such as caries and WSD may also be displayed.

[0411] In step S4707, the CPU 301 erases the information of the teeth deleted by the user. This is done by the CPU 301 deleting the tooth numbers and position information of the teeth specified by the user for deletion from the occlusal view, frontal view, left lateral view, and right lateral view tooth numbers and position information that the CPU 301 read into the RAM 303 in step S4701, and saving this information back to the RAM 303. After this, the recalculation button is pressed, and as a result of the CPU 301 executing steps S4714 and S4715, the status of L1 is changed from healthy to missing in step 4817.

[0412] In step S4708, CPU 301 proceeds to step S4709 if an adjacent missing tooth has been moved, and to step S4711 otherwise. This operation is explained using Figure 48. 4806 shows that a tooth that should have been inferred as L2 is inferred as L3, and L3 is missing. Therefore, the user wants to correct this so that L2 exists and L3 is missing. To do this, the user drags and drops the missing tooth symbol 4809, which is displayed on the L2 tooth in 4804, onto the L3 tooth.

[0413] In step S4709, CPU301 modifies the tooth numbers and position information of the occlusal view, frontal view, left lateral view, and right lateral view that were loaded into RAM303 in step S4701, and saves the changes made by the user to RAM303. Specifically, it changes the tooth numbers of teeth that have been changed from present to missing to teeth that have been changed from missing to present. If the operation described in step S4708 is performed, it changes L3 to L2 among the tooth numbers loaded in step S4701. After this, the recalculation button is pressed, and as a result of CPU301 executing steps S4714 and S4715, the status of L3 is changed from healthy to missing at 4811, and the status of missing L2 is changed to healthy at 4810.

[0414] In step S4711, CPU301 proceeds to step S4712 if tooth splitting has occurred, and to step S4702 otherwise. This operation is explained using Figure 48. 4807 shows a state where teeth L5 and L6 have been incorrectly inferred as a single L5. Therefore, the user wants to split the inferred result into two so that both L5 and L6 exist. To do this, the user right-clicks on L5 in 4804, causing CPU301 to display a state change list like 4818. State change list 4818 has the same function as 4907, and in this example, "Split" is displayed. The user performs the operation by clicking "Split".

[0415] In step S4712, the CPU 301 performs a division on the tooth specified by the user. The method involves the CPU 301 dividing the tooth number and position, which it had loaded into RAM 303 in step S4701, into two parts for the tooth specified by the user for division. The numbers of each part after division are the same as the tooth number specified by the user for division. The coordinates of the resulting rectangle are determined as follows: If the tooth number specified by the user for division is 1 to 3, the X coordinate is divided at the midpoint of the X coordinate, and the Y coordinate remains the same. If the tooth number specified by the user for division is 4 to 8, the Y coordinate is divided at the midpoint of the Y coordinate, and the X coordinate remains the same. In this way, the tooth can be divided without the user specifying coordinates. After this, the recalculation button is pressed, and the CPU 301 executes steps S4714 and S4715. As a result, at 4819, the status of L5 is changed from prosthetic to healthy, and at 4820, the status of L6 is changed from missing to prosthetic.

[0416] In step S4713, the CPU 301 overwrites the tooth numbers and position information related to the occlusal view, which was loaded into RAM 303 in step S4701, with the tooth numbers and position information divided in step S4712, and saves it to RAM 303.

[0417] Next, using flowchart 50, we will explain the process in step S4717 of Figure 47 when the presence or absence of teeth changes from "absent" to "present".

[0418] In step S5001, CPU301 proceeds to step S5002 if only teeth with numbers smaller than the tooth number that the user changed from none to present exist; otherwise, it proceeds to step S5004.

[0419] In step S5004, CPU301 proceeds to step S5005 if only teeth with numbers greater than the tooth number that the user changed from none to present exist; otherwise, it proceeds to step S5007.

[0420] In step S5007, CPU301 calculates the tooth number and position information of the tooth that the user changed from "none" to "present". The tooth number to be calculated is the tooth number that the user changed from "none" to "present".

[0421] Let A(Axmin, Axmax, Aymin, Aymax) be the coordinates of the tooth that the user changed from "absent" to "present," B(Bxmin, Bxmax, Bymin, Bymax) be the coordinates of the tooth with a number one less than the tooth the user changed from "absent" to "present," and C(Cxmin, Cxmax, Cymin, Cymax) be the coordinates of the tooth with a number one greater than the tooth the user changed from "absent" to "present." Equation 1 shows the coordinates of the upper left of A (Axmin, Aymin). Equation 2 shows the coordinates of the lower right (Axmax, Aymax). Note that Equations 1 and 2 below are for the modification of the left side of the mandible. When modifying the left side of the maxilla, replace Cymax with Bymax and Bymin with Cymin. When modifying the right side of the mandible, replace Bxmin with Cxmin and Cxmax with Bxmax. When correcting the left side of the upper jaw, replace Bxmin with Cxmin, Cxmax with Bxmax, Cymax with Bymax, and Bymin with Cxmax. Axmin = Bxmin Aymin = Cymax...Equation 1 Ax max = Cx max Aymax = Bymin...Equation 2

[0422] Subsequently, CPU301 proceeds to step S5008. In step S5008, CPU301 adds the tooth number and position information calculated in step S5007 to the tooth number and position information that was read into RAM303 in step S4701.

[0423] In step S5002, the CPU 301 calculates the tooth number and position information of the tooth that the user changed from "none" to "present". The tooth number to be calculated is the tooth number that the user changed from "none" to "present". When the coordinates of the tooth that the user changed from "none" to "present" are A(Axmin, Axmax, Aymin, Aymax), and the coordinates of the tooth with a number one less than the tooth that the user changed from "none" to "present" are B(Bxmin, Bxmax, Bymin, Bymax), the coordinates of the upper left of A (Axmin, Aymin) are shown in Equation 3. The coordinates of the lower right (Axmax, Aymax) are shown in Equation 4. Note that Equations 3 and 4 below are for the correction of the left side of the mandible. When correcting the maxilla, Aymin = Bymax and Aymax = Bymax + (Bymax - Bymin). Axmin = Bxmin Aymin=Bymin-(Bymax-Bymin)...Equation 3 Axmax = Bxmax Aymax = Bymin...Equation 4

[0424] Subsequently, CPU301 proceeds to step S5003. In step S5003, CPU301 adds the tooth number and position information calculated in step S5002 to the tooth number and position information that was read into RAM303 in step S4701.

[0425] In step S5005, CPU301 calculates the tooth number and position information of the tooth that the user changed from "none" to "present". The tooth number to be calculated is the tooth number that the user changed from "none" to "present".

[0426] Let A(Axmin, Axmax, Aymin, Aymax) be the coordinates of the tooth that the user changed from "none" to "present," and let B(Bxmin, Bxmax, Bymin, Bymax) be the coordinates of the tooth with a number one greater than the tooth the user changed from "none" to "present." The coordinates of the upper left of A (Axmin, Aymin) are shown in Equation 5. The coordinates of the lower right (Axmax, Aymax) are shown in Equation 6. Note that Equations 5 and 6 below are for the modification of the left side of the mandible. When modifying the maxilla, Aymin = Bymin - (Bymax - Bymin) and Aymax = Bymin. Axmin = Bxmin Aymin = Bymax...Equation 5 Axmax = Bxmax Aymax=Bymax+(Bymax-Bymin)...Equation 6

[0427] Subsequently, CPU301 proceeds to step S5006. In step S5006, CPU301 adds the tooth number and position information calculated in step S5005 to the tooth number and position information that was loaded into RAM303 in step S4701.

[0428] Next, using flowchart 51, we will explain the process in step S4704 of Figure 47 when a tooth is added.

[0429] In step S5101, if there is a tooth with a number smaller than the number of the added tooth, CPU 301 proceeds to step S5102; otherwise, it proceeds to step S5104.

[0430] In step S5102, the CPU 301 calculates the position and number of the tooth added by the user. The tooth number to be calculated is the number of the tooth added by the user. When the coordinates of the tooth added by the user are A(Axmin, Axmax, Aymin, Aymax), the coordinates of the tooth with a number one less than the tooth added by the user are B(Bxmin, Bxmax, Bymin, Bymax), and the coordinates of the tooth inferred from the number of the tooth added by the user are C(Cxmin, Cxmax, Cymin, Cymax), the coordinates of the upper left of A (Axmin, Aymin) are shown in Equation 7. The coordinates of the lower right (Axmax, Aymax) are shown in Equation 8. Note that Equations 7 and 8 below are for the correction of the left side of the mandible. When correcting the left side of the maxilla, replace Cymax with Bymax and Bymin with Cymin. When correcting the right side of the mandible, replace Bxmin with Cxmin and Cxmax with Bxmax. When correcting the left side of the upper jaw, replace Bxmin with Cxmin, Cxmax with Bxmax, Cymax with Bymax, and Bymin with Cxmax. Axmin = Bxmin Aymin = Cymax...Equation 7 Ax max = Cx max Aymax = Bymin... Equation 8

[0431] Subsequently, CPU301 proceeds to step S5103. In step S5103, CPU301 adds the tooth number and position information calculated in step S5102 to the tooth number and position information that was loaded into RAM303 in step S4701.

[0432] In step S5104, the CPU 301 calculates the position and number of the tooth added by the user. The tooth number to be calculated is the number of the tooth added by the user. When the coordinates of the tooth added by the user are A(Axmin, Axmax, Aymin, Aymax) and the coordinates of the tooth inferred from the tooth number added by the user are B(Bxmin, Bxmax, Bymin, Bymax), the coordinates of the upper left of A (Axmin, Aymin) are shown in Equation 9. The coordinates of the lower right (Axmax, Aymax) are shown in Equation 10. Note that Equations 9 and 10 below are for the correction of the left side of the mandible. When correcting the maxilla, Aymin = Bymin - (Bymax - Bymin) and Aymax = Bymin. Axmin = Bxmin Aymin = Bymax... Equation 9 Axmax = Bymax Aymax=Bymax+(Bymax-Bymin)...Equation 10

[0433] Subsequently, CPU301 proceeds to step S5105. In step S5105, CPU301 adds the tooth number and position information calculated in step S5104 to the tooth number and position information that was loaded into RAM303 in step S4701.

[0434] As described above, in this embodiment, the inference results from the model are corrected using inference results from different perspectives and models, as well as a knowledge-based decision model. This allows for appropriate correction of the dental formula number and condition even if they could not be correctly inferred from a single image. Furthermore, by automatically matching the dental formula number and condition of the teeth after the position of the teeth has been corrected by the dentist, consistent results can be obtained without the dentist having to correct all the inference results.

[0435] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

Claims

1. A first reading means for reading a first image which reads a visible light image of the occlusal surface inside the oral cavity, A second reading means reads a second image which is a visible light image of the teeth in a biting position, A determination means for determining the dentition and the state of each tooth based on the first image, the second image, and a learning model, An output means for outputting the determination result by the aforementioned determination means, An image processing apparatus characterized by having a complementation means for complementing the dentition determination result for the second image with the dentition determination result for the first image.

2. The image processing apparatus according to claim 1, characterized in that the model used to determine the alignment of teeth and the model used to determine the condition of teeth are different.

3. The image processing apparatus according to claim 1, characterized in that the model used to determine the condition of the teeth differs for each type of tooth.

4. The image processing apparatus according to claim 3, characterized in that, after determining the dentition, the condition of the teeth is determined using models prepared individually for each type of tooth.

5. The image processing apparatus according to claim 1, characterized in that the determination of the tooth condition is based on whether the determination is made for the first image or for the second image.

6. The image processing apparatus according to claim 1, characterized in that if the judgment result is modified by user operation, the learning model is updated based on the content of the modification.

7. The first reading process involves reading a first image, which is a visible light image of the occlusal surface inside the oral cavity, A second reading process involves reading a second image, which is a visible light image of the teeth in a biting position. A judgment process to determine the dentition and the state of each tooth based on the first image, the second image, and the learning model, An output step that outputs the result of the above determination step, A control method for an image processing apparatus, characterized by having a interpolation step of supplementing the dentition determination result for the second image with the dentition determination result for the first image.

8. A computer-readable program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 6.

Images