Image processing apparatus and method, program, and storage medium

Abstract

To provide an image processing apparatus that can more correctly determine a row of teeth having permanent teeth and milk teeth mixed therein.SOLUTION: An image processing apparatus comprises: an acquisition unit that acquires an image of the inside of the oral cavity; an estimation unit that estimates a row of teeth from an image acquired by the acquisition unit by using a learning model that has learned the relationship between the image of the inside of the oral cavity and the row of teeth; and a correction unit that makes a correction on a result of the estimated row of teeth on the basis of a rule related to the order of losing the milk teeth and getting the permanent teeth.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device that estimates the state of the oral cavity of a patient. [Background technology]

[0002] In recent years, a medical image diagnosis support device has been proposed that determines a tooth frame and determines missing teeth from a dentist's image, and displays the tooth frame and the dentist's image (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-78940 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 uses a machine learning model to infer tooth positions and dental formula numbers, and determines whether or not a tooth with each dental formula number is present. However, depending on the patient's age, there may be cases where not all teeth have erupted, some teeth are in the process of erupting (starting to erupt), or there is a mixture of baby and permanent teeth in the mouth. When using an object detection model to detect each tooth, it is more difficult to correctly detect a dentition where permanent and baby teeth are mixed compared to a patient with a complete set of permanent teeth.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image processing device that can more accurately determine a dental row where permanent teeth and baby teeth are mixed. [Means for solving the problem]

[0006] The image processing device of the present invention is characterized by comprising an acquisition means for acquiring an image of the oral cavity, an estimation means for estimating the tooth row from the image acquired by the acquisition means using a learning model that has learned the relationship between the image of the oral cavity and the tooth row, and a correction means for correcting the estimated tooth row based on rules regarding the order in which baby teeth are replaced by permanent teeth. [Effects of the Invention]

[0007] According to the present invention, it is possible to more accurately determine a tooth row in which permanent teeth and baby teeth are mixed. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the system configuration of an image processing apparatus according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing an example of the screen configuration of a user interface of an application according to the first embodiment. [Figure 3] 5 is a flowchart showing a processing procedure of an application in the first embodiment. [Figure 4] 4 is a flowchart showing the procedure of a tooth formula determination process according to the first embodiment. [Figure 5] FIG. 10 is a diagram showing a result of a tooth formula determination process in the first embodiment. [Figure 6] FIG. 3 is a diagram showing the display content of a tooth formula display area in the first embodiment. [Figure 7] 10 is a flowchart showing the procedure of a tooth formula determination process according to the second embodiment. [Figure 8] FIG. 10 is a diagram showing a result of a tooth formula determination process in the second embodiment. [Figure 9] FIG. 10 is a diagram showing the display content of a tooth formula display area in the second embodiment. [Figure 10] 10 is a flowchart showing a processing procedure of an application according to the third embodiment. [Figure 11] 10 is a flowchart showing the procedure of a tooth formula determination process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] (First embodiment) In this embodiment, we will describe an image processing device that can more accurately determine tooth rows where permanent and baby teeth are mixed by inferring tooth numbers and positions from intraoral images and supplementing the results with knowledge of the order in which baby teeth are replaced by permanent teeth.

[0011] Here, determining the tooth alignment means determining which teeth are baby teeth and which teeth are permanent teeth, which teeth are erupting, and which teeth according to dental number are in a state where they have grown, which are difficult to determine immediately by visual inspection.

[0012] In addition, the order in which baby teeth are replaced by permanent teeth is generally from the central incisors (front teeth) to the molars (back teeth), so this is called the knowledge (rule) regarding the order in which baby teeth are replaced by permanent teeth. Also, since only permanent teeth exist from the first to third molars, judgments are made based on this.

[0013] In addition, in this embodiment, a machine learning algorithm such as deep learning is used as the inference method. More specifically, an image of the oral cavity is input, and the corresponding information on the dentition is used as training data to learn a learning model (which has learned the relationship between the image of the oral cavity and the dentition), and the dentition is estimated from the image of the oral cavity. Of course, inference (estimation) may also be performed using template matching or the like, using the color and shape of the teeth as clues.

[0014] FIG. 1 is a block diagram showing the system configuration of an image processing apparatus 100 according to this embodiment.

[0015] The CPU 101 is a central processing unit that processes programs and can control the entire image processing apparatus 100.

[0016] The RAM 102 is a random access memory that temporarily stores programs and data supplied from the outside, and is also used as a temporary storage area for data output as the programs are executed.

[0017] The ROM 103 is a read-only memory that stores programs and parameters. In this embodiment, the ROM 103 stores program code of software executed by the CPU 101, such as the application 200, and parameters required for the operation of the application 200 (described later). The ROM 103 is configured so that the CPU 101 executes this program code. Note that the ROM 103 in this embodiment is a flash ROM, and is configured so that the control program can be rewritten.

[0018] The recording medium 104 is a recording medium that can be read and written by the image processing device 100. For example, this refers to a medium capable of recording electronic data, such as an internal memory of a computer, a memory card detachably connected to a computer, a HDD, a CD-ROM, an MO disk, an optical disk, or a magneto-optical disk. Digital data such as image data is stored as a file in the recording medium 104.

[0019] The operation unit 105 is composed of a keyboard, a mouse, etc. Instructions input via the operation unit 105 make it possible to specify input / output data, operate the screen, execute or stop processing, and so on.

[0020] The display unit 106 is made up of a liquid crystal display or the like incorporated in the image processing device 100, and displays, for example, a Graphical User Interface (GUI) screen of the application 200, processing results, and the like.

[0021] The internal bus 107 functions as a transmission path for control signals and data signals between the various elements in the image processing device 100.

[0022] In this embodiment, for ease of understanding, an example in which a CPU is installed as the main control unit of the image processing device will be described, but this is not limited thereto. For example, in addition to the CPU, a graphics processing unit (GPU) may also be installed, and the CPU and GPU may work together to perform processing. Since a GPU can perform efficient calculations by processing a larger amount of data in parallel, it is effective to use a GPU for processing when performing learning multiple times using a learning model such as deep learning. Specifically, when executing a learning program including a learning model, learning is performed by the CPU and GPU working together to perform calculations. Note that the processing of the learning unit may be performed by calculations performed only by the CPU or the GPU. Furthermore, the processing of the estimation unit may also be performed using a GPU, similar to the processing of the learning unit.

[0023] FIG. 2 is a diagram showing an example of the screen configuration of a UI that the application 200 displays on the display unit 106. As shown in FIG.

[0024] Reference numeral 201 denotes a save destination selection button. Reference numeral 202 denotes a thumbnail list display area. In the thumbnail list display area 202, thumbnails of image files present in a folder selected by the user through an operation described below are displayed in a list together with the file name and the date and time of shooting.

[0025] An image display area 203 displays an image selected by the user.

[0026] Reference numeral 204 denotes an imaging plane selection button. When the user clicks on the imaging plane selection button 204, candidate imaging planes are displayed in a pull-down menu, and the user can select one of them. In this embodiment, it is assumed that the selected image has been captured using the five-photo method, and the candidate imaging planes are "occlusal view (upper jaw)," "occlusal view (lower jaw)," "front view," "right lateral view," and "left lateral view."

[0027] Reference numeral 205 denotes a dental formula display area, which displays tooth information for each dental formula number. Symbols are defined in the dental formula to represent the position of the teeth, and are assigned in ascending order from the front to the back, for each of the top, bottom, left, and right directions. For permanent teeth, the central incisor is numbered 1, and the third molar is numbered 8. For deciduous teeth, the deciduous central incisor is numbered A, and the second deciduous molar is numbered E. In this embodiment, the dental formula number refers to a name such as "upper right no. 6." In this embodiment, the dental formula display area 205 is in the form of a table. From the top, each column represents the maxillary deciduous teeth (upper right no. E to upper left no. E), the mandibular deciduous teeth (lower right no. E to lower left no. E), the maxillary permanent teeth (upper right no. 8 to upper left no. 8), and the mandibular permanent teeth (lower right no. 8 to lower left no. 8), with tooth information displayed in each cell.

[0028] 3 is a flowchart showing the processing procedure of the application 200. This processing is realized by the CPU 101 executing the application (software) 200.

[0029] In step S301, CPU 101 determines whether or not a user operation has been performed. If a user operation has been performed, CPU 101 proceeds to step S302, and if not, CPU 101 repeats step S301.

[0030] In step S302, CPU 101 determines whether or not the user has performed an end operation. If the end operation has been performed, CPU 101 ends application 200. If not, CPU 101 proceeds to step S303. The end operation is performed by the user closing the UI of application 200 or selecting "end" from the menu.

[0031] In step S303, CPU 101 determines whether or not the user has pressed save destination selection button 201. If save destination selection button 201 has been pressed, CPU 101 proceeds to step S304, and if not, proceeds to step S309.

[0032] In step S304, the CPU 101 displays a folder selection screen on the display unit 106. On the folder selection screen, the user can select a folder in which images are saved from the recording medium 104 or other devices accessible from the image processing device 100.

[0033] In step S305, CPU 101 determines whether or not a folder has been selected by the user. If a folder has been selected, CPU 101 proceeds to step S306; if not, CPU 101 proceeds to step S307.

[0034] In step S306, the CPU 101 acquires a list of image files contained in the folder selected in step S305, and updates the display contents of the thumbnail list display area 202. In this embodiment, an image file is defined as a file with an extension of either "jpg," "png," or "gif."

[0035] In step S307, CPU 101 determines whether or not a cancel operation has been performed by the user. If a cancel operation has been performed, CPU 101 proceeds to step S308; otherwise, CPU 101 returns to step S305. A cancel operation is performed by closing the folder selection screen or by pressing the cancel button.

[0036] In step S308, CPU 101 closes the folder selection screen.

[0037] In step S309, CPU 101 determines whether or not an image selection operation has been performed by the user. If an image selection operation has been performed, CPU 101 proceeds to step S310; if not, CPU 101 proceeds to step S311. In this embodiment, an image selection operation refers to a mouse operation of double-clicking one of the thumbnails displayed in thumbnail list display area 202.

[0038] In step S310, in response to the image selection operation in step S309, CPU 101 updates the display in image display area 203. Specifically, for example, when the thumbnail "IMG_0001.JPG" in thumbnail list display area 202 is double-clicked, "IMG_0001.JPG" is read and the contents of the image are displayed.

[0039] In this embodiment, the user can select one image, but multiple images may also be selected. For example, the user may click a thumbnail while pressing the shift key or control key to select it, and then use a context menu or button to display all of the selected images side by side in the image display area 203. Alternatively, the user may add images to be selected by dragging and dropping any of the thumbnails displayed in the thumbnail list display area 202 into the image display area 203.

[0040] In step S311, CPU 101 determines whether or not an imaging plane selection operation has been performed by the user. If an imaging plane selection operation has been performed, CPU 101 proceeds to step S312; otherwise, CPU 101 returns to step S301. In this embodiment, the imaging plane selection operation refers to an operation of clicking imaging plane selection button 204 and selecting one of the imaging planes from a displayed pull-down menu. Note that the imaging plane may be configured to be automatically determined by reading a model that infers on which imaging plane the input image was captured and executing inference when an image is selected in step S309.

[0041] In step S312, the CPU 101 executes a tooth formula determination process on the image displayed in step S310. The tooth formula determination process will be described in detail later.

[0042] In step S313, the CPU 101 displays the results of the tooth formula determination process executed in step S312 in the tooth formula display area 205.

[0043] Fig. 4 is a flowchart showing the procedure of the tooth formula determination process executed in step S312 of Fig. 3. This process is realized by the CPU 101 loading a control program stored in the ROM 103 into the RAM 102 and executing it. This also applies to the flowcharts described below.

[0044] In step S401, the CPU 101 reads a model for inferring the dental formula numbers and their positions for the specified image, according to the specified imaging plane, and performs inference on the specified image. In this embodiment, teeth are detected using a learning model created using a machine learning algorithm such as deep learning. The learning model is also created for each imaging plane. As a result, the area of ​​each tooth is detected in the input image, and the dental formula number of each tooth (e.g., number 1 in the upper right, number 8 in the lower left, etc.) and coordinates indicating the area (X coordinates of the left and right ends and Y coordinates of the top and bottom ends) are obtained. The detected tooth areas are called number detection results DN1 to DNN. If detection fails, the position of each tooth in an average five-photon image may be used as a default value.

[0045] In step S402, CPU 101 deletes all but the rectangle with the highest reliability in inference from the combinations of rectangles whose overlapping areas are equal to or greater than a threshold value for the number detection results DN1 to DNN. Note that in this embodiment, the overlap of rectangles is measured using IoU (Intersection over Union), which is a numerical value obtained by dividing the area of ​​the overlapping portion of the rectangles by the total area of ​​the combined figure. In this embodiment, the threshold value for IoU is set to 0.75.

[0046] In step S403, CPU 101 determines whether or not there is a direction for which the following processing has not yet been performed among the four directions of "upper right," "lower right," "upper left," and "lower left." If there is a direction for which the processing has not yet been performed, CPU 101 proceeds to step S404; otherwise, CPU 101 ends the processing.

[0047] In step S404, CPU 101 selects one of the four directions, "upper right", "lower right", "upper left" and "lower left", for which subsequent processing has not yet been performed.

[0048] In step S405, the CPU 101 determines whether or not there is a rectangle in the number detection result whose dental formula number is a permanent tooth number 1 to 5 in the direction selected in step S404 and for which subsequent processing has not yet been performed. If there is a rectangle for which processing has not yet been performed, the CPU 101 proceeds to step S406; otherwise, the CPU 101 returns to step S403.

[0049] In step S406, the CPU 101 selects one tooth on the most mesial side from the rectangles in the number detection results whose dental formula numbers are 1 to 5 in the direction selected in step S404 and for which no subsequent processing has been performed yet. The selected number detection result is designated as rectangle N.

[0050] In step S407, CPU 101 determines whether or not a deciduous tooth other than C (other than a deciduous canine) is present on the mesial side of rectangle N. If a deciduous tooth other than C is present, CPU 101 proceeds to step S408; otherwise, CPU 101 returns to step S405.

[0051] In step S408, CPU 101 determines whether or not a rectangle deleted from the estimation result in step S402 exists at a position overlapping with rectangle N. If a deleted rectangle exists, CPU 101 proceeds to step S409; otherwise, CPU 101 returns to step S405. Note that step S409 may be executed even if no such rectangle exists.

[0052] In step S409, if the dental formula number of rectangle N is a permanent tooth, the CPU 101 changes it to the number of the primary tooth at the same position. For example, number 4 is changed to number D, and number 5 is changed to number E.

[0053] FIG. 5 is a diagram showing an example of a number detection result in the tooth formula determination process of FIG.

[0054] Figure 5(a) shows the number detection results of step S401. The dashed rectangles are the baby teeth from the number detection results, with 501 to 503, 505, and 507 being numbers A to C, E, and D at the bottom left, respectively. The solid rectangles are the permanent teeth from the number detection results, with 504 and 506 being numbers 4 and 6 at the bottom left, respectively. Furthermore, the reliability of number 4 504 and number D 507 at the bottom left is 0.7 and 0.6, respectively, and the IoU is 0.8. For the sake of clarity, explanations of the other rectangles will be omitted.

[0055] 5(b) is a diagram showing the number detection result after the processing of step S402 is completed. As a result of the processing of step S402, bottom left number D 507 is deleted because the IoU with bottom left number 4 504 is equal to or greater than the threshold and its reliability is lower than that of bottom left number 4 504.

[0056] 5(c) shows the number detection result after the processing of steps S403 to S409 has been completed. When lower left tooth number 4 504 is selected in step S406, lower left tooth number A 501 and lower left tooth number B 502, which are baby teeth, are present on the mesial side in step S407. Furthermore, in step S408, it is determined that lower left tooth number D 507, which is also a baby tooth, is present within the rectangle deleted in step S402. Therefore, as a result of the processing of step S409, the dental formula number of 504 is changed to lower left tooth number D, which is also a baby tooth.

[0057] FIG. 6 is a diagram showing an example of the display contents of the dental formula display area 205 displayed in step S313 of FIG. 3 when "Occlusal view (mandibular)" is selected with the imaging plane selection button 204 and the inference result is as shown in FIG. 5.

[0058] As a result of the dental formula determination process in Figure 4, cells with dental formula numbers that exist in the number detection results are marked with "○", and cells with dental formula numbers that do not exist are marked with "missing". Because the selected imaging surface is "Occlusal view (mandibular)", cells for the maxillary jaw are marked with "-", indicating that they are not subject to detection.

[0059] As described above, according to this embodiment, tooth numbers and positions are inferred from intraoral images, and the results are supplemented based on knowledge of the order in which baby teeth are replaced by permanent teeth, making it possible to more accurately determine tooth rows where permanent and baby teeth are mixed.

[0060] (Second embodiment) In the first embodiment, an example of processing was described in which tooth numbers and positions are inferred from intraoral images and the results are supplemented based on knowledge about the order in which baby teeth are replaced by permanent teeth, thereby more accurately determining whether a tooth row contains a mixture of permanent and baby teeth. In this embodiment, tooth numbers and positions are inferred from intraoral images, and teeth that are erupting (in the middle of erupting) are detected by inference. Then, based on knowledge about the order in which baby teeth are replaced by permanent teeth, it is determined whether the tooth is a permanent tooth or a baby tooth. This makes it possible to more accurately determine whether a tooth row contains a mixture of permanent teeth, baby teeth, and erupting teeth.

[0061] In this embodiment, the system configuration, an example of the screen configuration of the UI that the application 200 displays on the display unit 106, and a flowchart showing the processing procedure of the application 200 are the same as those in Figures 1 to 3 of the first embodiment, and therefore their explanations will be omitted.

[0062] Fig. 7 is a flowchart showing the procedure of the tooth formula determination process executed in step S312 in Fig. 3. In Fig. 7, steps that perform the same processes as in Fig. 4 are assigned the same step numbers as in Fig. 4.

[0063] In step S401, similarly to the case of FIG. 4, the CPU 101 reads a model for inferring the dental formula numbers of the teeth and their positions according to the designated imaging plane, and executes inference on the designated image.

[0064] In step S702, CPU 101 detects partially erupted teeth and their positions. In this embodiment, a partially erupted tooth is a tooth that has not fully erupted from the cervical region, with part of the crown still visible. The detected areas are referred to as partially erupted tooth detection results DI1 to DIM. Furthermore, for each of the number detection results detected in step S401, a rectangle that includes any of the partially erupted tooth detection results at a rate equal to or greater than a threshold is associated with that partially erupted tooth detection result, and information indicating that it is a partially erupted tooth is added.

[0065] In step S403, CPU 101 determines whether or not there is a direction for which the following processing has not yet been performed among the four directions of "upper right," "lower right," "upper left," and "lower left." If there is a direction for which the processing has not yet been performed, CPU 101 proceeds to step S404; otherwise, CPU 101 ends the processing.

[0066] In step S404, CPU 101 selects one of the four directions, "upper right", "lower right", "upper left" and "lower left", for which subsequent processing has not yet been performed.

[0067] In step S705, CPU 101 determines whether there are any rectangles in the partially erupted tooth detection results that have not yet been processed. If there are any rectangles that have not yet been processed, CPU 101 proceeds to step S706; otherwise, CPU 101 returns to step S403.

[0068] In step S706, the CPU 101 selects one tooth on the most mesial side from rectangles that have not yet been subjected to the subsequent processes among the results of the detection of partially erupted teeth. The selected result of the detection of partially erupted teeth is designated as rectangle M.

[0069] In step S707, CPU 101 determines whether or not there are any number detection results that are not partially erupted teeth other than those associated with rectangle M. If there are any number detection results that are not partially erupted teeth, CPU 101 proceeds to step S708; otherwise, CPU 101 proceeds to step S711.

[0070] In step S708, CPU 101 determines whether or not there are any number detection results for permanent teeth that are not partially erupted teeth other than those associated with rectangle M. If there are any number detection results for permanent teeth, CPU 101 proceeds to step S712; otherwise, CPU 101 proceeds to step S709.

[0071] In step S709, CPU 101 determines whether or not there is a number detection result for a baby tooth that is not a semi-erupted tooth on the mesial side of rectangle M among the number detection results. If there is a number detection result for a baby tooth, CPU 101 proceeds to step S710; otherwise, CPU 101 proceeds to step S712.

[0072] In step S710, CPU 101 determines whether rectangle M is located more distally than number 5 or number E in the same direction. If rectangle M is located more distally than number 5 or number E, CPU 101 proceeds to step S712; otherwise, CPU 101 proceeds to step S711.

[0073] In step S711, if there is a number detection result associated with rectangle M and the dental formula number is a permanent tooth, CPU 101 changes it to the number of the baby tooth in the same position. For example, number 4 is changed to number D, and number 5 is changed to number E. If there is no number detection result associated with rectangle M, CPU 101 adds a rectangle in the same position as rectangle M and with a dental formula number of a baby tooth. The dental formula number of the rectangle to be added is selected as follows.

[0074] First, from the number detection results, rectangle A is found to be closest to rectangle M, followed by rectangle B, which is the next closest. If rectangle M is located between rectangles A and B, and the tooth formula numbers of rectangles A and B are not adjacent teeth, the tooth formula number between rectangles A and B is selected. For example, if rectangle A is number D on the upper right and rectangle B is number 6 on the upper right, the tooth formula number of the added rectangle will be number E on the upper right. If there are two or more empty tooth formula numbers between rectangles A and B, and there are multiple candidates, the coordinates of the points that divide the line connecting the centers of rectangles A and B equally into the number of candidates are found, and the tooth formula numbers of the candidates are assigned, and the tooth formula number of the point closest to the center of rectangle M is adopted. For example, if rectangle A is number 4 on the upper right and rectangle B is number 7 on the upper right, the points that divide the line connecting them into thirds are number 5 on the upper right and number 6 on the upper right, from the side closest to rectangle A, and of these two, the tooth formula number closest to the center of rectangle M is adopted. Whether rectangle M is located between rectangles A and B is determined by whether the X coordinate of the center of rectangle M is within the range of the minimum and maximum X coordinates of the left and right ends of rectangles A and B, and whether the Y coordinate of the center of rectangle M is within the range of the minimum and maximum Y coordinates of the top and bottom ends of rectangles A and B.

[0075] If rectangle M is not located between rectangles A and B, rectangle A is located on the mesial side of rectangle M, and there is an empty dental formula number on the distal side of rectangle A, select the dental formula number one location distal to rectangle A. For example, if rectangle A is number D on the upper right, the dental formula number of the added rectangle will be number E on the upper right. Note that depending on the distance between rectangles M and A, two or more dental formula numbers of the added rectangle may be set to distal numbers. Note that whether rectangle A is located on the mesial side of rectangle M is determined by comparing the coordinates of the centers of rectangles M and A. If the specified imaging plane is "occlusal view (upper jaw)," the Y coordinate closest to the bottom of the image is determined to be the mesial side, and if it is "occlusal view (lower jaw)," the Y coordinate closest to the top of the image is determined to be the mesial side. Additionally, in the case of "front view," "right lateral view," or "left lateral view," if the dental formula number of rectangle C is on the right side, the X coordinate closest to the right edge of the image is determined to be the mesial side, and in the case of "front view," "right lateral view," or "left lateral view," if the dental formula number of rectangle C is on the left side, the X coordinate closest to the left edge of the image is determined to be the mesial side.

[0076] If there are no other number detection results, the tooth formula number of the closest position is estimated by comparing with the tooth position in the average five-photon image.

[0077] In step S712, if there is a number detection result associated with rectangle M and the dental formula number is a baby tooth, CPU 101 changes it to the number of a permanent tooth in the same position. If there is no number detection result associated with rectangle M, CPU 101 adds a rectangle in the same position as rectangle M and with a dental formula number of a permanent tooth. The dental formula number of the rectangle to be added is selected from the permanent teeth in the same way as in step S711.

[0078] Figure 8 shows an example of the number detection result in step S401 and the partially erupted tooth detection result in step S702 in the dental formula determination process of Figure 7. The solid line rectangles indicate the number detection results, and the dashed line rectangles indicate the partially erupted tooth detection results. Note that some explanations will be omitted for clarity.

[0079] In the case of Figure 8(a), teeth 801 and 802 are partially erupted. Teeth 803 is the result of detecting the number A at the bottom left. As determined in step S707, there are no other teeth that are not partially erupted, so teeth 801 are determined to be primary teeth.

[0080] In the case of Figure 8(b), 804 is a partially erupted tooth. 805 is the result of detecting the number 6 on the bottom left. As a result of the determination in steps S707 and S708, there are other permanent teeth that are not partially erupted teeth, so 804 is determined to be a permanent tooth.

[0081] In the case of Figure 8(c), teeth 806 and 807 are partially erupted. Teeth 808 to 810 are the detection results for the bottom left teeth C to E, respectively. As a result of the determinations in steps S707 to S709, there are no other permanent teeth that are not partially erupted, and there are no teeth on the mesial side that are not partially erupted, so 806 is determined to be a permanent tooth.

[0082] In the case of Figure 8(d), 811 is a partially erupted tooth. 812 is the result of detecting the number E on the lower left. As a result of the determination in steps S707 to S710, there are no other permanent teeth that are not partially erupted, and there is a tooth on the mesial side that is not a partially erupted tooth, and it is located more distally than E, so 811 is determined to be a permanent tooth.

[0083] In the case of Figure 8(e), tooth 813 is a partially erupted tooth. 814 is the result of detecting tooth number D on the lower left. As a result of the determination in steps S707 to S710, there are no other permanent teeth that are not partially erupted, and there is a primary tooth that is not partially erupted mesially, and it is more mesial than tooth E, so tooth 813 is determined to be a primary tooth.

[0084] 9(a) to 9(e) are diagrams showing examples of the display contents of the dental formula display area 205 displayed in step S313 of FIG. 3 when "Occlusal view (mandibular)" is selected with the imaging plane selection button 204 and the inference results are as shown in FIG. 8(a) to 8(e), respectively.

[0085] As a result of the dental formula determination process in Figure 7, the cells of the dental formula numbers of partially erupted teeth that exist in the number detection results are marked with "Half", the cells of the dental formula numbers of teeth other than partially erupted teeth are marked with "○", and the cells of non-existent dental formula numbers are marked with "Missing". Because the selected imaging plane is "Occlusal view (mandible)", the cells of the maxilla are marked with "-" to indicate that they are not subject to detection.

[0086] As described above, in this embodiment, the tooth numbers and positions are inferred from intraoral images, and erupting teeth are detected by inference, and whether the tooth is a permanent tooth or a baby tooth is determined based on knowledge of the order in which baby teeth are replaced by permanent teeth. This allows for more accurate determination of tooth rows that contain a mixture of permanent teeth, baby teeth, and erupting teeth.

[0087] (Third embodiment) In the first embodiment, a process for more accurately determining a tooth row where permanent and baby teeth are mixed was described, by inferring tooth numbers and positions from an intraoral image and complementing the results based on knowledge about the order in which baby teeth are replaced by permanent teeth. In the second embodiment, a process for more accurately determining a tooth row where permanent teeth, baby teeth, and erupting teeth are mixed was described, by detecting erupting teeth through inference and determining whether the tooth is a permanent tooth or a baby tooth based on knowledge about the order in which baby teeth are replaced by permanent teeth. In this embodiment, a process for determining a tooth row that combines the complementation methods of the first and second embodiments is described. In this embodiment, a process for determining whether to complement the inference results based on patient information is described, preventing the tooth row from being incorrectly determined due to complementation.

[0088] In this embodiment, the system configuration and an example of the screen configuration of the UI that the application 200 displays on the display unit 106 are the same as those in the first embodiment shown in FIGS. 1 and 2, respectively, and therefore will not be described again.

[0089] Fig. 10 is a flowchart showing the processing procedure of the application 200. In Fig. 10, steps in which the same processing as in Fig. 3 of the first embodiment is performed are assigned the same step numbers, and descriptions thereof will be omitted.

[0090] In step S1004, the CPU 101 displays a patient selection screen. Using the patient selection screen, the user can view a list of patients acquired from the recording medium 104 or from an external system such as an image server accessible from other image processing apparatus 100, and select a patient.

[0091] In step S1005, CPU 101 determines whether or not a patient has been selected by the user. If a patient has been selected, CPU 101 proceeds to step S1006; if not, CPU 101 proceeds to step S1008.

[0092] In step S1006, the CPU 101 acquires information about the patient. In this embodiment, the acquired information about the patient includes at least one of age, medical history, and tooth extraction history.

[0093] In step S1007, the CPU 101 acquires a list of image files stored in association with the patient selected in step S1005, and updates the display contents of the thumbnail list display area 202. In this embodiment, an image file is defined as a file with an extension of "jpg," "png," or "gif."

[0094] In this embodiment, a patient is selected from a patient list in steps S1003 to S1005, but the list of image files may be acquired by other methods. For example, image files may be acquired by specifying conditions such as a patient ID or date.

[0095] In step S1008, CPU 101 determines whether or not a cancel operation has been performed by the user. If a cancel operation has been performed, CPU 101 proceeds to step S1009; otherwise, CPU 101 returns to step S1005. The cancel operation is performed by closing the patient selection screen or by pressing the cancel button.

[0096] In step S1009, the CPU 101 closes the patient selection screen.

[0097] Fig. 11 is a flowchart showing the procedure of the tooth formula determination process executed in step S312 of Fig. 10. In Fig. 11, the steps that perform the same processes as those in Fig. 4 of the first embodiment and Fig. 7 of the second embodiment are given the same step numbers, and the explanations thereof will be omitted.

[0098] In step S1101, the CPU 101 determines whether the patient is a target for dental chart correction based on the patient information acquired in step S1006 of Fig. 10. If it is determined that the patient is a target for dental chart correction, the CPU 101 proceeds to step S702; otherwise, it ends the process. In this embodiment, a patient who meets the predetermined conditions to be a target for dental chart correction is a patient who meets any or all of the following: age is below a threshold; medical history does not include a diagnosis of abnormalities in tooth replacement such as delayed eruption or impacted teeth; and the patient has not lost teeth due to causes other than tooth replacement, such as accidents or tooth extractions.

[0099] As described above, in this embodiment, tooth numbers and positions, as well as erupting teeth, are detected by inference from intraoral images, and each tooth is determined to be a permanent tooth or a baby tooth based on knowledge of the order in which baby teeth are replaced by permanent teeth. This allows for more accurate determination of a tooth row in which permanent teeth, baby teeth, and erupting teeth are mixed. Furthermore, by determining whether to complement the inference results based on patient information, it is possible to prevent the tooth row from being incorrectly determined due to complementation.

[0100] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

[0101] The disclosure of this specification includes the following image processing device, method, program, and storage medium.

[0102] (Item 1) an acquisition means for acquiring an image of the oral cavity; an estimation means for estimating the dentition from the image acquired by the acquisition means using a learning model that has learned the relationship between the intraoral image and the dentition; a correcting means for correcting the estimated dentition based on a rule regarding the order in which baby teeth are replaced by permanent teeth; An image processing device comprising:

[0103] (Item 2) The image processing device described in item 1 is characterized in that, for each tooth determined to be a permanent tooth No. 1 to No. 5 in the tooth alignment estimated by the estimation means, if a baby tooth other than the baby canine No. C, which is the baby tooth, exists on the mesial side, the correction means changes the determination of the tooth determined to be a permanent tooth to a determination that it is a baby tooth.

[0104] (Item 3) Item 3. The image processing device according to item 2, wherein the correction means, when changing the result estimated by the estimation means, performs the change only if there is an estimated result of the tooth formula number after the change in the same position.

[0105] (Item 4) The image processing device according to any one of items 1 to 3, wherein the estimation means further estimates the position of a tooth in the middle of eruption, and the correction means determines that the partially erupted tooth is a primary tooth when there are no teeth that are not partially erupted other than the partially erupted tooth in the estimation result of the estimation means.

[0106] (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the estimation means further estimates the position of a tooth in the middle of eruption, and the correction means determines that a permanent tooth that is not a partially erupted tooth exists in addition to a partially erupted tooth in the results of the estimation means.

[0107] (Item 6) The image processing device according to any one of items 1 to 5, wherein the estimation means further estimates the position of a tooth in the middle of eruption, and the correction means determines that a tooth determined to be a partially erupted tooth is a permanent tooth when, in the result of estimation by the estimation means, only a deciduous tooth is not a partially erupted tooth and there is no deciduous tooth on the mesial side of the tooth determined to be a partially erupted tooth.

[0108] (Item 7) The image processing device according to any one of items 1 to 6, wherein the estimation means further estimates the position of teeth in the middle of eruption, and the correction means determines that the tooth determined to be a partially erupted tooth is a permanent tooth if, in the result of estimation by the estimation means, only a deciduous tooth is not a partially erupted tooth and, for a tooth determined to be a partially erupted tooth, there is a deciduous tooth that is not a partially erupted tooth that is located distal to and mesial to the deciduous tooth E or the permanent tooth 5.

[0109] (Item 8) The image processing device according to any one of items 1 to 7, wherein the estimation means further estimates the position of teeth in the middle of eruption, and the correction means determines that the tooth determined to be a partially erupted tooth is a primary tooth if, in the result of estimation by the estimation means, the only tooth that is not a partially erupted tooth is a primary tooth and, for a tooth determined to be a partially erupted tooth, the tooth is a permanent tooth number 5 or more proximal to it and there is a primary tooth that is not a partially erupted tooth on the mesial side.

[0110] (Item 9) 9. The image processing device according to any one of items 1 to 8, further comprising a second acquisition means for acquiring patient information, wherein the correction means, when the patient information satisfies a predetermined condition, corrects the result of the dentition estimated by the estimation means based on a rule regarding the order in which baby teeth are replaced by permanent teeth.

[0111] (Item 10) 10. The image processing device according to item 9, wherein the predetermined condition is that the patient's age is equal to or less than a threshold value.

[0112] (Item 11) 11. The image processing device according to item 9 or 10, wherein the predetermined condition is that the patient has not lost teeth due to factors other than tooth replacement in the medical history of the patient.

[0113] (Item 12) 12. The image processing device according to claim 9, wherein the predetermined condition is that the patient has not been diagnosed with any abnormality in tooth replacement in the patient's medical history.

[0114] (Item 13) an acquisition step of acquiring an image of the oral cavity; an estimation step of estimating the dentition from the image acquired in the acquisition step using a learning model that has learned the relationship between the intraoral image and the dentition; a correction step of correcting the estimated dentition based on rules regarding the order in which baby teeth are replaced by permanent teeth; An image processing method comprising:

[0115] (Item 14) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 11.

[0116] (Item 15) 12. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

[0117] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0118] 101: CPU, 102: RAM, 103: ROM, 104: recording medium, 105: operation unit, 106: display unit, 107: system bus

Claims

1. an acquisition means for acquiring an image of the oral cavity; an estimation means for estimating the dentition from the image acquired by the acquisition means using a learning model that has learned the relationship between the intraoral image and the dentition; a correcting means for correcting the estimated dentition based on a rule regarding the order in which baby teeth are replaced by permanent teeth; An image processing device comprising:

2. The image processing device described in claim 1, characterized in that, for each tooth determined to be a permanent tooth 1 to 5 in the tooth alignment estimated by the estimation means, if there is a baby tooth other than the baby canine tooth 1, which is the baby tooth C, on the mesial side, the correction means changes the determination of the tooth determined to be a permanent tooth to a determination that it is a baby tooth.

3. 3. The image processing device according to claim 2, wherein the correction means, when changing the result estimated by the estimation means, executes the change only when an estimated result of the tooth formula number after the change exists in the same position.

4. 2. The image processing device according to claim 1, wherein the estimation means further estimates the position of a tooth in the middle of eruption, and the correction means determines that the partially erupted tooth is a baby tooth when, in the result of the estimation by the estimation means, there are no teeth that are not partially erupted other than the partially erupted tooth.

5. 2. The image processing device according to claim 1, wherein the estimation means further estimates the position of teeth in the middle of eruption, and the correction means determines that a permanent tooth that is not a partially erupted tooth exists in addition to a partially erupted tooth as a permanent tooth.

6. 2. The image processing device according to claim 1, wherein the estimation means further estimates the position of teeth in the middle of eruption, and the correction means determines that a tooth determined to be a partially erupted tooth is a permanent tooth if, in the result of the estimation by the estimation means, only a deciduous tooth is not a partially erupted tooth and there is no deciduous tooth on the mesial side of the tooth determined to be a partially erupted tooth.

7. The image processing device described in claim 1, characterized in that the estimation means further estimates the position of teeth in the middle of eruption, and the correction means determines that the tooth determined to be a partially erupted tooth is a permanent tooth if, in the result of the estimation by the estimation means, the only tooth that is not a partially erupted tooth is a baby tooth, and for the tooth determined to be a partially erupted tooth, there is a baby tooth that is not a partially erupted tooth that is located distal to and mesial to the baby tooth number E or the permanent tooth number 5.

8. The image processing device described in claim 1, characterized in that the estimation means further estimates the position of teeth that are in the middle of eruption, and the correction means determines that the tooth determined to be a partially erupted tooth is a primary tooth if, in the results of the estimation by the estimation means, the only tooth that is not a partially erupted tooth is a primary tooth, and if the tooth determined to be a partially erupted tooth is a permanent tooth number 5 or more proximal to it and there is a primary tooth that is not a partially erupted tooth mesial to it.

9. 2. The image processing device according to claim 1, further comprising a second acquisition means for acquiring patient information, wherein the correction means, when the patient information satisfies predetermined conditions, corrects the tooth alignment estimated by the estimation means based on rules regarding the order in which baby teeth are replaced by permanent teeth.

10. 10. The image processing apparatus according to claim 9, wherein the predetermined condition is that the patient's age is equal to or less than a threshold value.

11. 10. The image processing device according to claim 9, wherein the predetermined condition is that the patient has not lost any teeth due to causes other than tooth replacement in the medical history of the patient.

12. 10. The image processing apparatus according to claim 9, wherein the predetermined condition is that the patient has not been diagnosed with any abnormality in tooth replacement in the patient's medical history.

13. an acquisition step of acquiring an image of the oral cavity; an estimation step of estimating the dentition from the image acquired in the acquisition step using a learning model that has learned the relationship between the intraoral image and the dentition; a correction step of correcting the estimated dentition based on rules regarding the order in which baby teeth are replaced by permanent teeth; An image processing method comprising:

14. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

15. 12. A computer-readable storage medium storing a program for causing a computer to function as each of the means of the image processing apparatus according to claim 1.

Images