PROGRAM, COMPUTER AND INFORMATION PROCESSING METHOD
The program and method enable remote assessment of orthodontic treatment progress by analyzing dentition images with AI, reducing the need for frequent dentist visits by determining mouthpiece replacement or completion based on floating amount calculations.
Patent Information
- Application Number
- JP2024029659
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2024-02-29
- Publication Date
- 2024-12-04
- Estimated Expiration
- 2044-02-29
AI Technical Summary
Users undergoing orthodontic treatment with multiple mouthpieces must regularly visit a dentist to determine whether to replace or complete treatment, which is burdensome.
A program and information processing method that uses a computer to analyze images of a user's dentition with a transparent mouthpiece, calculating the floating amount of the mouthpiece for each tooth and determining whether to replace it or complete treatment based on machine learning models, reducing the need for frequent dentist visits.
Reduces the burden on users by allowing them to determine whether to replace or complete orthodontic treatment remotely, using AI to efficiently assess the alignment progress without regular dentist visits.
Smart Images

Figure 0007594333000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a program, a computer, and an information processing method. [Background technology]
[0002] In recent years, a technique for orthodontic treatment by having a user wear a mouthpiece on the teeth in daily life has become known. Patent Document 1 discloses an orthodontic treatment support system that enables a user to order orthodontic appliances and undergo orthodontic treatment safely and at low cost. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent Publication No. 2021-90669 Summary of the Invention [Problem to be solved by the invention]
[0004] When a user orthodontizes his / her teeth by wearing a mouthpiece on his / her teeth in daily life, it is common for multiple mouthpieces with different shapes to be prepared for each user. Then, after the user starts orthodontic treatment, a certain mouthpiece is worn and a certain time has passed. When the dentist judges it is OK to proceed to the next stage by directly visually checking the user's teeth or by looking at an image of the user's teeth with the mouthpiece worn on the teeth, the mouthpiece is replaced with a different shape, and the user wears the replaced mouthpiece on the teeth. By repeating such operations, the user's teeth are orthodontized to a predetermined shape.
[0005] However, if a dentist determines whether or not to change the mouthpiece or complete the orthodontic treatment, the user must visit the dentist regularly, which is a heavy burden on the user.
[0006] The present disclosure has been made in consideration of these points, and aims to provide a program, a computer, and an information processing method that can reduce the burden on a user when orthodontic treatment. [Means for solving the problem]
[0007] The program of the present disclosure is A program that causes a computer to function as a receiving unit, a calculating unit, and a determining unit, The receiving means receives, from a user terminal, an image of a row of teeth of a user with a transparent or semi-transparent mouthpiece attached to the teeth, The calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The judgment means judges whether or not to replace the mouthpiece or whether or not to complete the orthodontic treatment based on the amount of floating calculated by the calculation means.
[0008] In the program of the present disclosure, The calculation means may use a first trained model generated by machine learning using teacher data including an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and the amount of floating of the mouthpiece for each tooth to calculate the amount of floating of the mouthpiece for each tooth from the image of the dentition accepted by the acceptance means.
[0009] In the program of the present disclosure, The calculation means may calculate the amount of floating of the mouthpiece from each tooth based on the number of pixels in an area of the mouthpiece that protrudes from each tooth in the image received by the reception means.
[0010] The program of the present disclosure is The computer further functions as a first trimming means, The first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth accepted by the accepting means, The calculation means may calculate an amount of floating of the mouthpiece for each tooth based on the image of the row of teeth trimmed by the first trimming means.
[0011] In the program of the present disclosure, The first trimming means may use a second trained model generated by machine learning by using training data including an image of the user's dentition with a transparent or translucent mouthpiece attached to the teeth and the trimmed image of the dentition, to trim an image surrounded by a first bounding box including the dentition from the image of the dentition accepted by the accepting means.
[0012] In the program of the present disclosure, The second trained model may be generated using a YOLO model.
[0013] The program of the present disclosure is The computer further functions as a second trimming means, the second trimming means trims an image surrounded by a second bounding box including the upper dentition and an image surrounded by a third bounding box including the lower dentition from the image of the dentition trimmed by the first trimming means, The calculation means may calculate an amount of floating of the mouthpiece for each tooth based on the image of the upper row of teeth and the image of the lower row of teeth trimmed by the second trimming means.
[0014] In the program of the present disclosure, The receiving means also receives information regarding the wearing time of the mouthpiece from the user terminal, The determination means may also refer to information regarding the length of time the mouthpiece has been worn, received by the reception means, to determine whether or not to replace the mouthpiece or whether or not to complete dental correction.
[0015] The program of the present disclosure is The computer is further caused to function as a display instruction means, The display instruction means may transmit a display instruction signal to the user terminal to cause the user terminal to display a result of the determination made by the determination means.
[0016] The program of the present disclosure is The computer further functions as a color image generating means, The colored image generating means colors the areas of the mouthpiece that stand out from each tooth in the image received by the receiving means, The display instruction means may transmit a display instruction signal to the user terminal to cause the user terminal to display the image of the row of teeth colored by the colored image generation means.
[0017] In the program of the present disclosure, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks at the front, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the left are respectively acquired in the user terminal; The receiving means receives, from the user terminal, the three images acquired by the user terminal; the calculation means calculates the amount of floating of the mouthpiece for each tooth for each of the three images of the row of teeth received by the reception means, The judgment means may judge whether or not to change the mouthpiece or whether or not to complete the dental correction based on the amount of floating calculated by the calculation means for the three images.
[0018] In the program of the present disclosure, In the user terminal, when an image of the dentition is captured by an imaging unit of the user terminal, a frame of the dentition is displayed on a display unit of the user terminal, and an image of the dentition may be captured when the user's dentition falls within the frame of the dentition.
[0019] In the program of the present disclosure, in the user terminal, when the imaging unit captures an image of the row of teeth, a frame of the row of teeth when the user looks at the front, a frame of the row of teeth when the user looks to the right, and a frame of the row of teeth when the user looks to the left are displayed in sequence on the display unit, By capturing an image of the user's teeth when they fit within each of the tooth frame, an image of the teeth with the mouthpiece attached to the teeth when the user is looking straight ahead, an image of the teeth with the mouthpiece attached to the teeth when the user turns to the right, and an image of the teeth with the mouthpiece attached to the teeth when the user turns to the left may be obtained.
[0020] In addition, the program of the present disclosure is A program that causes a user terminal to function as a display unit, an image capture unit, and a transmission unit, The display means displays a frame of the row of teeth on the display unit of the user terminal when an image of the row of teeth is captured by the imaging unit of the user terminal, The imaging means captures an image of the row of teeth when the row of teeth of the user is included in the frame of the row of teeth, The transmitting means transmits the captured image of the row of teeth to an external device.
[0021] In the program of the present disclosure, the display means causes the display unit to sequentially display a frame of the row of teeth when the user looks at the front, a frame of the row of teeth when the user turns to the right, and a frame of the row of teeth when the user turns to the left when the imaging unit captures an image of the row of teeth; The imaging means may capture images of the user's teeth when the user's teeth are within the respective tooth row frames, thereby obtaining an image of the user's teeth when viewed from the front, an image of the teeth when the user faces to the right, and an image of the teeth when the user faces to the left.
[0022] The computer of the present disclosure includes: A computer that functions as a reception unit, a calculation unit, and a determination unit by executing a program, The receiving means receives, from a user terminal, an image of a row of teeth of a user with a transparent or semi-transparent mouthpiece attached to the teeth, The calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The judgment means judges whether or not to replace the mouthpiece or whether or not to complete the orthodontic treatment based on the amount of floating calculated by the calculation means.
[0023] The information processing method of the present disclosure includes: An information processing method executed by a computer having a control unit, The control unit receives an image of a user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from a user terminal; A step of calculating the amount of floating of the mouthpiece for each tooth based on the received image of the row of teeth by the control unit; A step in which the control unit determines whether or not to replace the mouthpiece or complete the orthodontic treatment based on the calculated amount of floating; The present invention is characterized by comprising: Effect of the Invention
[0024] According to the program, computer, and information processing method disclosed herein, it is possible to reduce the burden on a user when orthodontic treatment for a tooth alignment. [Brief description of the drawings]
[0025] [Figure 1] 1 is a diagram illustrating a schematic diagram of an information processing system according to an embodiment of the present disclosure. [Diagram 2] FIG. 2 is a diagram showing an image captured and processed in an information processing system according to an embodiment of the present disclosure. [Diagram 3] 1 is a diagram showing a tooth row and a frame for the tooth row displayed on a user terminal in an information processing system according to an embodiment of the present disclosure. FIG. [Figure 4]FIG. 1 is a diagram illustrating an exemplary process in an information processing system according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a diagram showing an exemplary information processing flow of an information processing method according to an embodiment of the present disclosure. [Figure 6] FIG. 2 is a schematic process flow diagram of the orthodontic method according to the present disclosure from a user's perspective. [Figure 7] FIG. 11 is a diagram showing another exemplary flow of information processing of the information processing method according to an embodiment of the present disclosure. [Figure 8] FIG. 11 is a diagram showing another exemplary flow of information processing of the information processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] As described above, when orthodontic treatment is performed in daily life by attaching a plurality of pre-prepared mouthpieces with different shapes to the teeth, the user must visit the dentist regularly to know whether to replace the mouthpiece or complete the orthodontic treatment, which is a heavy burden. As a result of intensive research into this problem, the present inventors have found that if it is possible to determine whether to replace the mouthpiece or complete the orthodontic treatment from the amount of floating of the mouthpiece for each tooth calculated based on an image of the user's teeth in which a transparent or translucent mouthpiece is attached, which is received from a user terminal, the burden on the user when orthodontic treatment can be reduced, and have come up with the invention of the present disclosure.
[0027] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. Fig. 1 to Fig. 8 are diagrams illustrating an information processing system 1 and an information processing method according to the present disclosure.
[0028] [Information Processing System 1] The information processing system 1 according to this embodiment shown in Fig. 1 includes a server 10 and a user terminal 32. The server 10 is a management server (computer) capable of transmitting and receiving information to and from a plurality of user terminals 32 through a communication network 50 such as an Internet line. Note that information (images of dentition, judgment results, etc.) can be transmitted and received between the server 10 and the user terminal 32 via a website and / or a so-called app (application software).
[0029] In this specification, unless otherwise specified, the term "image of dentition" refers to an image of a state in which a transparent or semi-transparent mouthpiece is attached to the dentition of a user who operates the user terminal 32 (for example, FIG. 2). In addition, the term "mouthpiece" is not particularly limited as long as it is used for orthodontic treatment of dentition. "Transparent or semi-transparent" refers to the characteristics of the mouthpiece that allow the calculation of the floating amount of the mouthpiece relative to the teeth in the image, as described later. In other words, the degree of transparency or translucency is not limited as long as the mouthpiece is distinguished from the teeth to which it is attached in the image and the dentition can be seen through the mouthpiece. In addition, as an image of the dentition, images of the user's face (dentition) viewed from the front and horizontal directions are exemplified in FIG. 2 and FIG. 3, but images of the user, i.e., the dentition viewed from below and above, may be used for calculations, judgments, etc. according to the present disclosure.
[0030] Moreover, the "amount of floating" of the mouthpiece relative to the teeth refers to the gap between the teeth and the mouthpiece (particularly in FIG. 2(C), it is highlighted by coloring), and serves as a criterion for determining whether or not to replace the mouthpiece or to complete orthodontic treatment. Any threshold value can be set as this criterion. For example, the threshold value can be set to a value within a range of 0 to 10%, preferably 2 to 6%, and more preferably 3 to 5%. When the ratio of the amount of floating (gap) to the area of the entire image (for example, FIG. 2(A)) is equal to or less than the above threshold, it is determined that the mouthpiece should be replaced or orthodontic treatment should be completed.
[0031] The number of teeth in the dentition for which the amount of floating is calculated can be appropriately set and changed according to the purpose of orthodontics, and is not particularly limited. The amount of floating may be calculated for all teeth on which the mouthpiece is attached, or when only the front teeth are orthodontic, the amount of floating may be calculated for 2 to 6 teeth. In addition, the amount of floating may be calculated for both the upper and lower dentitions (for example, 4 to 12 teeth).
[0032] <Server (Computer) 10> The server 10 may be an industrial computer, a personal computer, a tablet terminal, or the like, or may be a virtual server in a physical server accessible via the Internet. In addition, there may be a plurality of these hardware and virtual servers, and any combination may be used. The server 10 shown in FIG. 1 includes a control unit 12, a storage unit 28, and a communication unit 30.
[0033] (Control unit 12) The control unit 12 is composed of a CPU (Central Processing Unit), a GPU (Image Processing Unit), etc., and controls the operation of the server 10. Specifically, the control unit 12 executes a program stored in a storage unit 28 described later, thereby functioning as a reception means 14, a first trimming means 16, a second trimming means 18, a calculation means 20, a determination means 22, a colored image generating means 24, a display instruction means 26, etc. Note that these functions may be achieved by executing one or more independent programs or applications. Furthermore, these programs and applications may be provided in a single terminal (the server 10) or may be provided in a distributed manner in multiple terminals (including the server 10), and in the latter case, may be connected to each other via a wired cable or a communication network 50. Each means will be described later.
[0034] (Storage unit 28) The storage unit 28 is composed of, for example, a hard disk drive (HDD), a random access memory (RAM), a read only memory (ROM), a solid state drive (SSD), etc. The storage unit 28 is not limited to being built into the server 10, and may be a storage medium (for example, a USB memory) that is detachably attached to the server 10. Instead of providing the storage unit 28, various information may be stored in other storage means (such as a cloud server).
[0035] The memory unit 28 is capable of storing an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth (e.g., Figure 3) and the amount of floating of the mouthpiece for each tooth (e.g., highlighted in Figure 2(C)) received from multiple user terminals 32 as a first trained model 320 and its teacher data 310, an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and a cropped image of the dentition (e.g., Figures 2(A) and (B)) as a second trained model 220 and its teacher data 210, and judgment criteria such as thresholds.
[0036] (Communication unit 30) The communication unit 30 connects the control unit 12 to an external device (eg, a user terminal 32) wirelessly or via a wired connection so as to be able to communicate with the external device. The control unit 12 can transmit and receive information to and from the external device via the communication unit 30.
[0037] (others) In addition, the server 10 may be provided with an operation unit (input unit) such as a keyboard that enables an administrator of the server 10 to give various commands to the control unit 12, and a display unit (output unit) such as a monitor that displays various screens upon receiving a display command signal from the control unit 12. In one embodiment, a display operation unit such as a touch panel in which the operation unit and display unit are integrated may be used.
[0038] (Details of control unit 12) (Means of reception 14) The receiving means 14 receives an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from the user terminal 32 ("100" in FIG. 4).
[0039] Furthermore, the receiving means 14 may also receive information on the wearing time of the mouthpiece from the user terminal 32. This information can be reflected in the judgment process by the judging means 22 described later. For example, if there are fewer than a predetermined number of days (four days) in a week when the wearing time of the mouthpiece is a predetermined time or more (e.g., 20 hours), the orthodontic effect of the mouthpiece in this case is considered to be small. Therefore, in such a case, it is considered that there is little need to verify the orthodontic effect, and the judgment process does not necessarily have to be performed.
[0040] (First trimming means 16) The first trimming means 16 trims an image surrounded by a first bounding box 16a including the row of teeth from the image of the row of teeth received by the receiving means 14 (FIG. 2(A), "110" in FIG. 4). FIG. 2(A) shows an exemplary image after trimming, and 16b represents the margin of the image after trimming. In this way, the calculation means 20 may calculate the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth trimmed by the first trimming means 16.
[0041] The first trimming means 16 may trim an image surrounded by a first bounding box 16a including the dentition from the image of the dentition accepted by the accepting means 14, using a second trained model 220 generated by machine learning by using teacher data 210 including an image of the dentition of a user in which a transparent or translucent mouthpiece is attached to the teeth and the trimmed image of the dentition. Specifically, the first trimming means 16 can "input" the image accepted by the accepting means 14 to the second trained model 220, and "output" an image of only the inside of the mouth part from which the outside of the mouth part has been trimmed from the second trained model 220. As a result, the calculation means 20 can calculate the floating amount of the mouthpiece for each tooth based on the image acquired by the first trimming means 16. There is no particular limitation on "machine learning" as long as it can perform the above-mentioned input and output, and various methods such as deep learning and supervised learning can be used.
[0042] Here, the second trained model 220 can generally be generated as follows. First, unprocessed images captured by multiple users are collected as teacher data 210. Next, annotation is performed on each image using a rectangle circumscribing the dentition, and annotation data (cropped image of the dentition) is obtained as another teacher data 210. Next, the second trained model 220 is generated by learning using these teacher data 210 (at this time, parameters such as weights are updated and deployed). Although R-CNN, SSD, YOLO, etc. can be used as a learning model that is the basis of the second trained model 220, it is preferable that the second trained model 220 is generated using the YOLO model in terms of accuracy and speed.
[0043] (Second trimming means 18) The second trimming means 18 trims an image surrounded by a second bounding box 18a including the upper teeth row and an image surrounded by a third bounding box 18b including the lower teeth row from the image of the teeth trimmed by the first trimming means 16 (FIG. 2(B), "120" in FIG. 4). Note that FIG. 2(B) shows the image after trimming. In this way, the calculation means 20 can also calculate the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth row and the image of the lower teeth row (FIG. 2(B)) trimmed by the second trimming means 18.
[0044] The trimming by the second trimming means 18 can also use machine learning and a trained model as described above (not shown). This trained model can be generated, for example, as follows. First, images of the trimmed dentition are collected as training data. Next, annotation is performed separately for the upper and lower dentition in each image, and annotation data (images of the trimmed upper and lower dentition) are obtained as separate training data. Next, a trained model is generated by learning using these training data (at this time, parameters such as weights are updated and deployed). To generate this trained model, it is preferable to use the YOLO model in terms of accuracy and speed.
[0045] (Calculation means 20) The calculation means 20 calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means 14 (FIG. 2(C)). As described above, the amount of floating may be calculated for some or all of the teeth included in the image.
[0046] The calculation means 20 may use the first trained model 320 generated by machine learning using teacher data 310 including an image of a row of teeth of a user wearing a transparent or translucent mouthpiece on the teeth and the amount of floating of the mouthpiece for each tooth to calculate the amount of floating of the mouthpiece for each tooth from the image of the row of teeth accepted by the acceptance means 14. Specifically, the calculation means 20 may, for example, "input" the image accepted by the acceptance means 14 to the first trained model 320, and "output" the space between the teeth and the mouthpiece from the first trained model 320 as the amount of floating. This gap can be obtained as contour information by performing segmentation on the image ("130" in FIG. 4). The first trained model 320 can be generated as follows. First, unprocessed images taken by multiple users are collected as the teacher data 310 (or images of the rows of teeth cropped separately for the top and bottom may be used). Next, in each image, annotation is performed along the contour of the gap between the mouthpiece and the teeth to obtain annotation data (contour information, amount of lift) as another teacher data 310. Next, a first trained model 320 is generated by learning using these teacher data 310 (at this time, parameters such as weights are updated and deployed). It is preferable to use YOLO as this segmentation model in terms of accuracy and speed.
[0047] The calculation means 20 can also calculate the amount of floating of the mouthpiece from each tooth based on the number of pixels of the area of the mouthpiece protruding from each tooth in the image received by the reception means 14. Here, if the image sent from the user terminal 32 is set to have a predetermined resolution, it is preferable because the amount of floating can be easily calculated based on the area (number of pixels) of the image.
[0048] (Judgment means 22) The determination means 22 determines whether or not to replace the mouthpiece or whether or not to complete the orthodontic treatment based on the amount of floating calculated by the calculation means 20 ("150" in FIG. 4). Note that in the present disclosure, it is assumed that a plurality of shapes of mouthpieces are used for orthodontic treatment, and if there is no mouthpiece to replace, the determination is made to complete the orthodontic treatment. In relation to this, in FIG. 6, which is a schematic process flow diagram of the orthodontic treatment method according to the present disclosure from the user's perspective, after determining whether the amount of floating is equal to or less than a threshold value, it is also determined whether or not the "last mouthpiece" is being used (step S65).
[0049] As described above, the determination means 22 may perform a rule-based determination based on whether the amount of floating is equal to or less than a predetermined threshold value. Also, when the calculated amount of floating differs for each tooth, the determination may be made based on whether the largest amount of floating is equal to or less than a threshold value, or based on whether the average amount of floating exceeds a threshold value.
[0050] When the reception means 14 also receives information on the duration of time the mouthpiece was worn from the user terminal 32, the determination means 22 may refer to the information on the duration of time the mouthpiece was worn received by the reception means 14 to determine whether or not to replace the mouthpiece or to complete orthodontic treatment, and may change the threshold for the amount of floating. In the example shown in Fig. 6, if the number of days in a week on which the mouthpiece was worn for a predetermined time or more is less than a predetermined number of days, the user is allowed to "extend" the wearing of the mouthpiece for a predetermined period of time without transmitting an image. This predetermined period of time can be set according to the number of days or time the user did not wear the mouthpiece.
[0051] (Colored image generation means 24) The colored image generating means 24 colors the area (amount of floating) of the mouthpiece protruding from each tooth in the image received by the receiving means 14 (FIG. 2(C), “140” in FIG. 4). There is no particular limitation to such a coloring method, and a person skilled in the art can adopt a desired coloring method or means.
[0052] (Display instruction means 26) The display instruction means 26 transmits a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the determination result by the determination means 22 ("160" in Fig. 4). The display instruction means 26 can also transmit a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the image of the row of teeth colored by the colored image generation means 24.
[0053] (First model generation means 300, second model generation means 200) The first model generating means 300 and the second model generating means 200 generate the first trained model 320 and the second trained model 220 by machine learning using the teacher data 310 and 210, respectively (FIG. 4). More specifically, the first model generating means 300 trains the first trained model 320 to "output" the space between the teeth and the mouthpiece as the amount of float when the image accepted by the accepting means 14 is "input". The second model generating means 200 trains the second trained model 220 to "output" an image of the row of teeth with the part outside the mouth trimmed when the image accepted by the accepting means 14 is "input". In this way, the first trained model 320 and the second trained model 220 are not limited to those generated by an external device, and may be generated in the server 10. Note that the server 10 may be provided with a means for generating a trained model used in trimming by the second trimming means 18.
[0054] The first trained model 320 and / or the second trained model 220 may function by the control unit 12 executing a program stored in the memory unit 28, or may function by a device other than the server 10 executing a predetermined program. In the latter case, the first trained model 320 and / or the second trained model 220 generated by the other device is transmitted to the server 10 and stored in the memory unit 28 of the server 10.
[0055] In addition, in the case where the user terminal 32 is configured to acquire an image of the row of teeth with the mouthpiece attached to the teeth when the user looks forward, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the left, the reception means 14 may accept the three images acquired by the user terminal 32 from the user terminal 32, the calculation means 20 may calculate the floating amount of the mouthpiece for each tooth for each of the three images of the row of teeth accepted by the reception means 14, and the judgment means 22 may judge whether or not to replace the mouthpiece or whether or not to complete the orthodontic treatment based on the floating amount calculated by the calculation means 20 for the three images. In this way, when multiple images are accepted simultaneously from the same user, the judgment may be made based on whether the floating amount is exceeded in all images, or based on whether the average value of the floating amount of all images exceeds a threshold. The judgment criteria can be arbitrarily determined according to the amount of information (number of images, imaging angle, etc.), and the orthodontic effect can be judged comprehensively. The number of images used for such a comprehensive assessment is not limited to three, and can be increased or decreased as appropriate depending on the purpose of correction.
[0056] <User terminal 32> The user terminal 32 may be composed of an industrial computer, a personal computer, a tablet terminal, a mobile terminal, etc., and is not particularly limited. In this specification, a "user" is a person who operates the user terminal 32 to transmit and receive information such as images to the server 10 during orthodontic treatment using a mouthpiece. The user terminals 32 do not need to be the same terminal for transmitting and receiving information to and from the server 10, and the same user may transmit and receive information using different user terminals 32 at different times as long as they are identified by the same registration information (identification ID, etc.). The user terminal 32 shown in FIG. 1 includes a control unit 34, an imaging unit 42, an (operation) display unit 44, a storage unit 46, and a communication unit 48.
[0057] (Control unit 34) The control unit 34 is composed of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the operation of the user terminal 32. Specifically, the control unit 34 functions as a display means 36, an imaging means 38, a transmission means 40, etc., by executing a program stored in the storage unit 28 described later. These functions may be realized by executing one or more independent programs or applications. Furthermore, these programs and applications may be provided in a single terminal (user terminal 32) or may be provided in a distributed manner in multiple terminals (including the user terminal 32), and in the latter case, may be connected to each other via a wired cable or a communication network 50.
[0058] The display means 36 displays a frame of the row of teeth on the display unit 44 of the user terminal 32 when the imaging unit 42 of the user terminal 32 captures an image of the row of teeth (see FIG. 3). The display means 36 may display a frame of the row of teeth when the user looks at the front (FIG. 3(A)), a frame of the row of teeth when the user turns to the right (FIG. 3(B)), and a frame of the row of teeth when the user turns to the left (FIG. 3(C)) on the display unit 44 when the imaging unit 42 captures an image of the row of teeth. For example, each frame may be displayed sequentially at a predetermined time interval, or each frame may be displayed sequentially each time an image is captured. The display means 36 may also display the result of the determination transmitted from the server 10 and an image of the row of teeth that has been colored.
[0059] The imaging means 38 captures an image of the row of teeth when the row of teeth of the user is included in the row of teeth frame. By capturing an image automatically in this manner, it is possible to unify the imaging distance, prevent blurring caused when the user presses the shutter button, and capture a more standardized photograph. When a plurality of rows of teeth frames are displayed as described above, the imaging means 38 may capture an image of the row of teeth when the row of teeth of the user is included in each row of teeth frame, thereby acquiring an image of the row of teeth when the user looks at the front, an image of the row of teeth when the user looks to the right, and an image of the row of teeth when the user looks to the left.
[0060] The transmission means 40 transmits (uploads) the captured image of the row of teeth to an external device (such as the server 10).
[0061] (Imaging unit 42) The imaging unit 42 captures an image of the dentition. The imaging unit 42 is not particularly limited, but is preferably an AI camera app that detects objects and can automatically capture an image of the dentition when the user's dentition falls within a frame of the dentition.
[0062] (Display section 44) The display unit 44 displays various screens including the frame of the dentition by receiving a display command signal from the control unit 34. The display unit 44 is typically a monitor, a display, or the like, but is not particularly limited thereto. The user terminal 32 may be provided with an operation unit (input unit) such as a keyboard that allows the user to give various commands to the control unit 34, and when the user terminal 32 is a mobile phone such as a smartphone, an operation display unit 44 such as a touch panel in which an operation unit and a display unit are integrated may be used (FIG. 1).
[0063] (Storage unit 46) The storage unit 46 is composed of, for example, a hard disk drive (HDD), a random access memory (RAM), a read only memory (ROM), a solid state drive (SSD), etc. Furthermore, the storage unit 46 is not limited to being built into the user terminal 32, and may be a storage medium (for example, a USB memory) that is detachably attached to the user terminal 32. Furthermore, instead of providing the storage unit 46, various information may be stored in other storage means (such as a cloud server).
[0064] (Communications Department 48) The communication unit 48 connects the control unit 34 to an external device (such as the server 10) wirelessly or via wired communication so that the control unit 34 can communicate with the external device. The control unit 34 can transmit and receive information to and from the external device via the communication unit 48.
[0065] [Information processing method 1] Next, the information processing method 1 in the information processing system 1 described above will be described with reference to Fig. 5 and Fig. 6. Here, the server 10 performs a process of determining whether or not to replace the mouthpiece or to complete the orthodontic treatment based on the amount of floating of the mouthpiece for each tooth calculated based on the image received from the user terminal 32. In the following description, components with the same reference numerals are the same as the components described above, and duplicated descriptions will be omitted as appropriate. Note that the process described below is performed by executing a program stored in the storage unit 28, but the information processing method according to the present disclosure is not limited to this.
[0066] First, the control unit 12 (accepting means 14) accepts an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from the user terminal 32 (FIG. 5, step S10).
[0067] Next, the control unit 12 (first trimming means 16) trims the received image of the row of teeth to an image surrounded by a first bounding box 16a including the row of teeth (FIG. 5, step S20; FIG. 2(A)).
[0068] Next, the control unit 12 (second cropping means 18) crops from the cropped image of the teeth an image surrounded by a second bounding box 18a including the upper teeth row and an image surrounded by a third bounding box 18b including the lower teeth row (Figure 5, step S30, Figure 2(B)).
[0069] Next, the control unit 12 (the calculation means 20) calculates the floating amount of the mouthpiece for each tooth from the trimmed image of the upper dentition and the image of the lower dentition (FIG. 5, step S40). As described above, the floating amount may be calculated based on the number of pixels, or the first trained model 320 generated by machine learning using the image of the dentition of the user in which a transparent or translucent mouthpiece is attached to the teeth and the teacher data 310 including the floating amount of the mouthpiece for each tooth may be used to calculate the floating amount.
[0070] Next, the control unit 12 (colored image generating means 24) colors the areas of the mouthpiece that stand out from each tooth in the image accepted by the control unit 12 (accepting means 14) (FIG. 5, step S50; FIG. 2(C)).
[0071] Next, the control unit 12 (determination means 22) determines whether or not to replace the mouthpiece or to complete the orthodontic treatment based on the calculated amount of floating (FIG. 5, step S60). As described above, the determination may be made based on whether the amount of floating is equal to or less than a threshold value.
[0072] Next, the control unit 12 (display instruction means 26) transmits a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the determination result and the image of the colored dentition (FIG. 5, step S70).
[0073] As shown in FIG. 6, if it is determined in step S60 that the mouthpiece should be replaced or that the orthodontic treatment should be completed (if the amount of floating is equal to or less than the threshold, "YES"), it is also determined whether the mouthpiece worn by the user at that time is the last mouthpiece (step S65), and if it is the last mouthpiece ("YES"), the process ends. This determination result is sent to the user terminal 32 (FIG. 5, step S70). If it is determined in step S65 that the mouthpiece worn by the user is not the last mouthpiece ("NO"), the user will start wearing a new mouthpiece as the next treatment step. This determination result is also sent to the user terminal 32 (FIG. 5, step S70). On the other hand, if the amount of floating exceeds the threshold ("NO"), the user will "extend" wearing the mouthpiece for a predetermined period of time, and this determination result (and the image of the colored dentition) is also sent to the user terminal 32 (FIG. 5, step S70). FIG. 6 is a schematic process flow diagram of the orthodontic method according to the present disclosure from the user's perspective. It is preferable to send a notification from the server 10 every day (at a specified time) before each step (action) performed by the user, prompting the user to perform that step, as this can prevent the user from forgetting to wear the mouthpiece, etc.
[0074] [Information processing method 2] Next, an information processing method 2 in the information processing system 1 will be described with reference to Fig. 7. The information processing method 2 is an example in which a program for acquiring an image to be determined according to the present disclosure is executed in the user terminal 32.
[0075] First, the control unit 34 (display means 36) causes the display unit 44 of the user terminal 32 to display a frame of the row of teeth when the imaging unit 42 of the user terminal 32 captures an image of the row of teeth (step S110).
[0076] Next, when the user's dentition, with a transparent or semi-transparent mouthpiece attached to the teeth, is placed in the dentition frame, the control unit 34 (imaging means 38) images the dentition (step S120).
[0077] Next, the control unit 34 (transmission means 40) transmits the captured image of the row of teeth to an external device (step S130). Note that the transmission means 40 may also transmit information regarding the wearing time of the mouthpiece according to an input by the user.
[0078] [Information processing method 3] Next, an information processing method 3 in the information processing system 1 will be described with reference to Fig. 8. The information processing method 3 is an example in which, in addition to the processing in the information processing method 2, a program executed by the server 10 described above is executed by the user terminal 32. That is, a program corresponding to the calculation means 20, the determination means 22, etc. other than the reception means 14 and the display instruction means 26 is executed by the control unit 34 of the user terminal 32. Steps S210 and S220 in Fig. 8 correspond to steps S110 and S120 shown in Fig. 7 of the information processing method 2, and therefore description thereof will be omitted.
[0079] After step S220, the control unit 34 (the calculation means 20) calculates the amount of floating of the mouthpiece for each tooth based on the captured image of the row of teeth (step S230).
[0080] Next, as in step S60 of FIG. 5, the control unit 34 (determination means 22) determines whether or not to replace the mouthpiece or to complete the dental correction based on the calculated amount of floating (step S240).
[0081] Next, the control unit 34 (display means 36) causes the display unit 44 to display the determination result (step S250).
[0082] The information processing method as described above is an example, and the flow of the process is not limited to the above. For example, in the information processing method 1, the coloring step (step S50) of coloring the area of the mouthpiece protruding from each tooth in the image may be performed after the judgment step (step S60) of judging whether or not to replace the mouthpiece or to complete the orthodontic treatment. Similarly, in the information processing method 3, after the judgment step (step S240), the control unit 34 (colored image generating means 24) may color the area of the mouthpiece protruding from each tooth in the image as in step S50 of FIG. 5, and the colored image of the row of teeth may be displayed on the display unit 44 in step S250.
[0083] In the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, which are configured as described above, a reception means 14, a calculation means 20, and a judgment means 22 are provided. The reception means 14 receives an image of a user's dentition in which a transparent or semi-transparent mouthpiece is attached to the teeth from the user terminal 32, the calculation means 20 calculates the amount of floating of the mouthpiece for each tooth based on the image of the dentition received by the reception means 14, and the judgment means 22 judges whether or not to replace the mouthpiece or to complete the orthodontic treatment based on the amount of floating calculated by the calculation means 20. In one embodiment according to the present disclosure, the user terminal 32 is provided with a display means 36, an imaging means 38, and a transmission means 40. The display means 36 displays a frame of the tooth row on the display unit 44 of the user terminal 32 when the imaging unit 42 of the user terminal 32 images the tooth row, the imaging means 38 images the tooth row when the user's tooth row enters the tooth row frame, and the transmission means 40 transmits the image of the captured tooth row to an external device.
[0084] Conventionally, when orthodontic treatment is performed in daily life by attaching a plurality of pre-prepared mouthpieces of different shapes to the teeth, the user must visit the dentist regularly, which is a heavy burden. In response to this problem, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure can calculate the floating amount of the mouthpiece for each tooth based on an image of the user's teeth, and determine whether or not to replace the mouthpiece or complete the orthodontic treatment based on the floating amount, thereby reducing the burden on the user when orthodontic treatment is performed. As described above, the display unit 44 of the user terminal 32 displays the frame of the teeth, and an image of the teeth is taken when the teeth are inserted into the frame of the teeth, thereby efficiently acquiring an image of the user's teeth.
[0085] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the calculation means 20 may use a first trained model 320 generated by machine learning using teacher data 310 including an image of a user's dentition in which a transparent or translucent mouthpiece is attached to the teeth and the amount of floating of the mouthpiece for each tooth, to calculate the amount of floating of the mouthpiece for each tooth from the image of the dentition accepted by the acceptance means 14. In this way, it is possible to automatically and efficiently calculate the amount of floating to be determined using AI.
[0086] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the calculation means 20 may calculate the amount of floating of the mouthpiece from each tooth based on the number of pixels of the area of the mouthpiece protruding from each tooth in the image accepted by the acceptance means 14. In this way, the amount of floating can also be calculated based on the number of pixels (image area).
[0087] Furthermore, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure may further include a first cropping means 16 for cropping an image surrounded by a first bounding box 16a including the row of teeth from the image of the row of teeth accepted by the accepting means 14, in which case the calculating means 20 calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth cropped by the first cropping means 16. In this way, by omitting extraoral image information (feature amounts) that is not related to the calculation of the amount of floating, the subsequent calculation process and judgment process (including segmentation) can be stabilized.
[0088] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the first trimming means 16 may trim an image surrounded by a first bounding box 16a including the dentition from the image of the dentition accepted by the accepting means 14, using a second trained model 220 generated by machine learning by using teacher data 210 including an image of the dentition of a user with a transparent or semi-transparent mouthpiece attached to the teeth and the trimmed image of the dentition. In this way, it is possible to automatically and efficiently calculate the amount of floating to be determined using AI.
[0089] In addition, in the information processing system 1, the computer (the server 10, the user terminal 32), the program, and the information processing method according to the present disclosure, the second trained model 220 may be generated using the YOLO model. It is preferable to generate the second trained model 220 using the YOLO model in terms of accuracy and speed.
[0090] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, a second trimming means 18 may be further provided for trimming an image surrounded by a second bounding box 18a including the upper dentition and an image surrounded by a third bounding box 18b including the lower dentition from the image of the dentition trimmed by the first trimming means 16, in which case the calculation means 20 calculates the amount of floating of the mouthpiece for each tooth based on the image of the upper dentition and the image of the lower dentition trimmed by the second trimming means 18. In this way, by omitting image information (feature amount) of the mouth that is not related to the calculation of the amount of floating, the subsequent calculation process and judgment process (including segmentation) can be stabilized.
[0091] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the reception means 14 may also receive information relating to the time the mouthpiece has been worn from the user terminal 32, and the determination means 22 may also refer to the information relating to the time the mouthpiece has been worn received by the reception means 14 to determine whether or not to replace the mouthpiece or whether or not to complete orthodontic treatment. Since orthodontic treatment may not be effective if the mouthpiece is not worn for a specified period of time, it is not necessary to perform calculation processing or determination processing in the first place (FIG. 6).
[0092] Furthermore, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure may further include a display instruction means 26 that transmits a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the result of the determination by the determination means 22. In this way, an explanation can be provided, similar to a diagnosis by a dentist, as to whether the same or a different mouthpiece should be used as a future treatment step, or whether orthodontic treatment will be completed, without the need to visit a clinic.
[0093] Furthermore, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure may further include a colored image generating means 24 that colors the area of the mouthpiece protruding from each tooth in the image received by the receiving means 14. In this case, the display instruction means 26 transmits a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the image of the row of teeth colored by the colored image generating means 24. In this way, the amount of protrusion can be presented to the user without the need for a dental visit, and a persuasive explanation of future treatment steps can be provided, similar to a diagnosis by a dentist.
[0094] In the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, when the user terminal 32 is configured to acquire an image of the row of teeth with the mouthpiece attached to the teeth when the user looks at the front, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the left, the reception means 14 may receive the three images acquired by the user terminal 32 from the user terminal 32, the calculation means 20 may calculate the amount of floating of the mouthpiece for each tooth for each of the three images of the row of teeth received by the reception means 14, and the judgment means 22 may judge whether or not to replace the mouthpiece or whether or not to complete the orthodontic treatment based on the amount of floating calculated by the calculation means 20 for the three images. In this way, the judgment criteria can be determined according to the purpose of orthodontic treatment (number of teeth, position, etc.) and the amount of information (number of images, imaging angle, etc.), and the orthodontic effect can be judged comprehensively based on each image.
[0095] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the display means 36 may, when imaging the row of teeth with the imaging unit 42, sequentially display on the display unit 44 a frame of the row of teeth when the user looks from the front, a frame of the row of teeth when the user turns to the right, and a frame of the row of teeth when the user turns to the left, and the imaging means 38 may capture images of the row of teeth when the user's row of teeth falls within each of the frames of the row of teeth, thereby obtaining an image of the row of teeth when the user looks from the front, an image of the row of teeth when the user turns to the right, and an image of the row of teeth when the user turns to the left. In this way, the above-mentioned comprehensive judgment can be efficiently performed.
[0096] The information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure are not limited to the above-mentioned aspects and combinations, and various modifications can be made. [Explanation of symbols]
[0097] 1. Information Processing Systems 10 Server (computer) 12 Control section 14. Method of reception 16 First trimming means 16a First bounding box 16b Margin 18 Second trimming means 18a Second bounding box 18b Third bounding box 20 Calculation means 22 Judgment means 24 Colored image generation means 26 Display instruction means 28 Memory section 30 Communications Department 32 User terminal 34 Control section 36 Display means 38 Imaging means 40 Transmission Method 42 Imaging unit 44 (Operation) Display section 46 Memory section 48 Communications Department 50 Communication Network 200 second model generation means 210 Teacher Data 220 Second trained model 300 First model generation means 310 Teacher Data 320 First trained model
Claims
1. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the user's row of teeth is within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, The computer functions as a receiving unit, a computing unit, and a determining unit by executing a first program; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The system has a judgment means for determining whether or not to replace the mouthpiece or complete orthodontic treatment based on the amount of floating calculated by the calculation means, and if the amount of floating calculated for each tooth differs, the judgment is made based on whether the largest amount of floating is below a predetermined threshold.
2. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the user's row of teeth is within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, The computer functions as a receiving unit, a computing unit, and a determining unit by executing a first program; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The system is characterized in that the judgment means judges whether or not to replace the mouthpiece or whether or not to complete dental orthodontic treatment based on the amount of floating calculated by the calculation means, and makes the judgment depending on whether the average amount of floating exceeds a predetermined threshold value.
3. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the user's row of teeth is within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, The computer functions as a receiving unit, a computing unit, and a determining unit by executing a first program; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The determination means determines whether or not to replace the mouthpiece or to complete the orthodontic treatment based on whether the amount of floating calculated by the calculation means is equal to or less than a predetermined threshold value, The system, wherein the predetermined threshold is a predetermined ratio of the amount of floating to the total area of the image accepted by the accepting means.
4. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the user's row of teeth is within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, The computer functions as a receiving unit, a computing unit, and a determining unit by executing a first program; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the teeth received by the reception means, using a first trained model generated by machine learning by using teacher data including an image of the user's dentition in which a transparent or translucent mouthpiece is attached to the teeth and an amount of floating of the mouthpiece for each tooth, and The system wherein the judgment means judges whether or not to replace the mouthpiece or whether or not to complete dental orthodontics based on the amount of floating calculated by the calculation means.
5. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the row of teeth in a front view of the user falls within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, the computer executes a first program to function as a receiving unit, a first trimming unit, a second trimming unit, a calculation unit, and a determination unit; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth accepted by the accepting means, the second trimming means trims an image surrounded by a second bounding box including the upper dentition and an image surrounded by a third bounding box including the lower dentition from the image of the dentition trimmed by the first trimming means, the calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth and the image of the lower teeth trimmed by the second trimming means, The system wherein the judgment means judges whether or not to replace the mouthpiece or whether or not to complete dental orthodontics based on the amount of floating calculated by the calculation means.
6. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the user's row of teeth is within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, The computer functions as a receiving unit, a computing unit, and a determining unit by executing a first program; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the teeth received by the reception means, using a first trained model generated by machine learning by using teacher data including an image of the user's dentition in which a transparent or translucent mouthpiece is attached to the teeth and an amount of floating of the mouthpiece for each tooth, and The system has a judgment means for determining whether or not to replace the mouthpiece or complete orthodontic treatment based on the amount of floating calculated by the calculation means, and if the amount of floating calculated for each tooth differs, the judgment is made based on whether the largest amount of floating is below a predetermined threshold.
7. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the row of teeth in a front view of the user falls within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, the computer executes a first program to function as a receiving unit, a first trimming unit, a second trimming unit, a calculation unit, and a determination unit; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth accepted by the accepting means, the second trimming means trims an image surrounded by a second bounding box including the upper dentition and an image surrounded by a third bounding box including the lower dentition from the image of the dentition trimmed by the first trimming means, the calculation means uses a first trained model generated by machine learning using teacher data including an image of a front view of a user's teeth with a transparent or translucent mouthpiece attached to the teeth and an amount of floating of the mouthpiece for each tooth, to calculate the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth and the image of the lower teeth trimmed by the second trimming means; The system has a judgment means for determining whether or not to replace the mouthpiece or complete orthodontic treatment based on the amount of floating calculated by the calculation means, and if the amount of floating calculated for each tooth differs, the judgment is made based on whether the largest amount of floating is below a predetermined threshold.
8. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the row of teeth in a front view of the user falls within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, the computer executes a first program to function as a receiving unit, a first trimming unit, a second trimming unit, a calculation unit, and a determination unit; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth accepted by the accepting means, the second trimming means trims an image surrounded by a second bounding box including the upper dentition and an image surrounded by a third bounding box including the lower dentition from the image of the dentition trimmed by the first trimming means, the calculation means uses a first trained model generated by machine learning using teacher data including an image of a front view of a user's teeth with a transparent or translucent mouthpiece attached to the teeth and an amount of floating of the mouthpiece for each tooth, to calculate the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth and the image of the lower teeth trimmed by the second trimming means; The determination means determines whether or not to replace the mouthpiece or complete the orthodontic treatment based on the amount of floating calculated by the calculation means, and when the amount of floating calculated for each tooth is different, determines whether the largest amount of floating is equal to or smaller than a predetermined threshold value. The system, wherein the predetermined threshold is a predetermined ratio of the amount of floating to the total area of the image accepted by the accepting means.
9. A system including a computer and a user terminal, The user terminal has a control unit, a display unit, and an imaging unit, The control unit functions as a display means, an imaging means, and a transmission means by executing a second program, the display means causes the display unit to display a frame of the row of teeth when the imaging unit images the row of teeth, the imaging means automatically images the row of teeth when the row of teeth in a front view of the user falls within the frame of the row of teeth, and the transmission means transmits the image of the row of teeth that has been captured to the computer, the computer executes a first program to function as a receiving unit, a first trimming unit, a second trimming unit, a calculation unit, and a determination unit; The receiving means receives, from the user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, The first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth accepted by the accepting means, the second trimming means trims an image surrounded by a second bounding box including the upper dentition and an image surrounded by a third bounding box including the lower dentition from the image of the dentition trimmed by the first trimming means, the calculation means uses a first trained model generated by machine learning using teacher data including an image of a front view of a user's teeth with a transparent or translucent mouthpiece attached to the teeth and an amount of floating of the mouthpiece for each tooth, to calculate the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth and the image of the lower teeth trimmed by the second trimming means; The determination means determines whether or not to replace the mouthpiece or complete the orthodontic treatment based on the amount of floating calculated by the calculation means, and when the amount of floating calculated for each tooth is different, determines whether the largest amount of floating is equal to or smaller than a predetermined threshold value. the predetermined threshold is a predetermined ratio of the amount of floating to the area of the entire image accepted by the accepting unit, The system performs the calculation of the amount of lift for two to six teeth in each of the upper and lower rows of teeth of the user.
10. The system according to any one of claims 1 to 9, wherein the calculation means calculates the amount of floating of the mouthpiece from each tooth based on the number of pixels of the area of the mouthpiece that protrudes from each tooth in the image accepted by the acceptance means.
11. The system according to any one of claims 5, 7 to 9, wherein the first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth accepted by the accepting means, using a second learned model generated by machine learning by using teacher data including an image of the row of teeth viewed from the front of a user with a transparent or translucent mouthpiece worn on the teeth and the trimmed image of the row of teeth.
12. The system of claim 11 , wherein the second trained model is generated using a YOLO model.
13. The receiving means also receives information regarding the wearing time of the mouthpiece from the user terminal, The system according to any one of claims 1 to 9, wherein the determination means also refers to information regarding the length of time the mouthpiece has been worn that has been accepted by the acceptance means to determine whether or not to replace the mouthpiece or to complete dental orthodontic treatment.
14. The computer is further caused to function as a display instruction means, The system according to any one of claims 1 to 9, wherein the display instruction means transmits a display instruction signal to the user terminal to cause the user terminal to display the determination result by the determination means.
15. The computer further functions as a color image generating means, The colored image generating means colors the areas of the mouthpiece that stand out from each tooth in the image received by the receiving means, The system according to claim 14 , wherein the display instruction means transmits a display instruction signal to the user terminal to cause the user terminal to display the image of the dentition colored by the colored image generation means.
16. an image of the row of teeth with the mouthpiece attached to the teeth when the user looks at the front, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the left are respectively acquired in the user terminal; the receiving means receives, from the user terminal, the three images acquired by the user terminal; the calculation means calculates the amount of floating of the mouthpiece for each tooth for each of the three images of the row of teeth received by the reception means, The system described in any one of claims 1 to 9, wherein the judgment means judges whether or not to replace the mouthpiece or whether or not to complete dental orthodontics based on the amount of floating calculated by the calculation means for the three images.
17. the display means causes the display unit to sequentially display a frame of the row of teeth when the user looks at the front, a frame of the row of teeth when the user turns to the right, and a frame of the row of teeth when the user turns to the left when the imaging unit captures an image of the row of teeth; The system according to any one of claims 1 to 9, wherein the imaging means images the user's teeth when they fall within the tooth row frame, thereby obtaining an image of the user's teeth when viewed from the front, an image of the teeth when the user turns to the right, and an image of the teeth when the user turns to the left.
18. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the dentition when the user's dentition is included in the dentition frame; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from the user terminal; A step of calculating an amount of floating of the mouthpiece for each tooth by the first control unit based on the received image of the row of teeth; The first control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated floating amount, and when the calculated floating amount differs for each tooth, determines whether the largest floating amount is equal to or less than a predetermined threshold value; An information processing method comprising:
19. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the dentition when the user's dentition is included in the dentition frame; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition in which a transparent or semi-transparent mouthpiece is attached to the teeth from the user terminal; A step of calculating an amount of floating of the mouthpiece for each tooth by the first control unit based on the received image of the row of teeth; The first control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated amount of floating, and determines whether the average value of the amount of floating exceeds a predetermined threshold value. An information processing method comprising:
20. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the dentition when the user's dentition is included in the dentition frame; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition in which a transparent or semi-transparent mouthpiece is attached to the teeth from the user terminal; A step of calculating an amount of floating of the mouthpiece for each tooth by the first control unit based on the received image of the row of teeth; A step in which the first control unit determines whether or not to replace the mouthpiece or to complete orthodontic treatment based on whether the calculated floating amount is equal to or less than a predetermined threshold value; Equipped with The information processing method, wherein the predetermined threshold is a predetermined ratio of the amount of floating to the entire area of the received image.
21. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the dentition when the user's dentition is included in the dentition frame; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from the user terminal; The first control unit calculates the amount of floating of the mouthpiece for each tooth based on the received image of the dentition using a first trained model generated by machine learning by using teacher data including an image of the dentition of a user in which a transparent or semi-transparent mouthpiece is attached to the teeth and the amount of floating of the mouthpiece for each tooth; A step in which the first control unit determines whether or not to replace the mouthpiece or complete orthodontic treatment based on the calculated amount of floating; An information processing method comprising:
22. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the row of teeth when the row of teeth in a front view of the user is included in the frame of the row of teeth; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from the user terminal; A step of the first control unit trimming an image surrounded by a first bounding box including the row of teeth from the received image of the row of teeth; A step in which the first control unit crops an image surrounded by a second bounding box including an upper dentition and an image surrounded by a third bounding box including a lower dentition from the cropped image of the dentition; A step in which the first control unit calculates the amount of floating of the mouthpiece for each tooth based on the trimmed image of the upper teeth and the trimmed image of the lower teeth; A step in which the first control unit determines whether or not to replace the mouthpiece or complete orthodontic treatment based on the calculated amount of floating; An information processing method comprising:
23. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the dentition when the user's dentition is included in the dentition frame; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition in which a transparent or semi-transparent mouthpiece is attached to the teeth from the user terminal; The first control unit calculates the amount of floating of the mouthpiece for each tooth based on the received image of the dentition using a first trained model generated by machine learning by using teacher data including an image of the dentition of a user in which a transparent or translucent mouthpiece is attached to the teeth and the amount of floating of the mouthpiece for each tooth; The first control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated floating amount, and when the calculated floating amount differs for each tooth, determines whether the largest floating amount is equal to or less than a predetermined threshold value; An information processing method comprising:
24. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the row of teeth when the row of teeth in a front view of the user is included in the frame of the row of teeth; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition in which a transparent or semi-transparent mouthpiece is attached to the teeth from the user terminal; A step of the first control unit trimming an image surrounded by a first bounding box including the row of teeth from the received image of the row of teeth; A step in which the first control unit crops an image surrounded by a second bounding box including an upper dentition and an image surrounded by a third bounding box including a lower dentition from the cropped image of the dentition; A step in which the first control unit calculates the amount of floating of the mouthpiece for each tooth based on the trimmed image of the upper teeth and the trimmed image of the lower teeth, using a first trained model generated by machine learning using teacher data including an image of a front view of a user's teeth with a transparent or translucent mouthpiece attached to the teeth and the amount of floating of the mouthpiece for each tooth; The first control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated floating amount, and when the calculated floating amount differs for each tooth, determines whether the largest floating amount is equal to or less than a predetermined threshold value; An information processing method comprising:
25. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the row of teeth when the row of teeth in a front view of the user is included in the frame of the row of teeth; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from the user terminal; A step of the first control unit trimming an image surrounded by a first bounding box including the row of teeth from the received image of the row of teeth; A step in which the first control unit crops an image surrounded by a second bounding box including an upper dentition and an image surrounded by a third bounding box including a lower dentition from the cropped image of the dentition; A step in which the first control unit calculates the amount of floating of the mouthpiece for each tooth based on the trimmed image of the upper teeth and the trimmed image of the lower teeth, using a first trained model generated by machine learning using teacher data including an image of a front view of a user's teeth with a transparent or translucent mouthpiece attached to the teeth and the amount of floating of the mouthpiece for each tooth; The first control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated floating amount, and when the calculated floating amount differs for each tooth, determines whether the largest floating amount is equal to or less than a predetermined threshold value; Equipped with The information processing method, wherein the predetermined threshold is a predetermined ratio of the amount of floating to the entire area of the received image.
26. An information processing method executed by a computer having a first control unit and a user terminal having a second control unit, A step in which the second control unit displays a frame of the row of teeth on a display unit of the user terminal when an image of the row of teeth is captured by an imaging unit of the user terminal; The second control unit automatically captures an image of the row of teeth when the row of teeth in a front view of the user is included in the frame of the row of teeth; The second control unit transmits the captured image of the dentition to the computer; The first control unit receives an image of a user's dentition in which a transparent or semi-transparent mouthpiece is attached to the teeth from the user terminal; A step of the first control unit trimming an image surrounded by a first bounding box including the row of teeth from the received image of the row of teeth; A step in which the first control unit crops an image surrounded by a second bounding box including an upper dentition and an image surrounded by a third bounding box including a lower dentition from the cropped image of the dentition; A step in which the first control unit calculates the amount of floating of the mouthpiece for each tooth based on the trimmed image of the upper teeth and the trimmed image of the lower teeth, using a first trained model generated by machine learning using teacher data including an image of a front view of a user's teeth with a transparent or translucent mouthpiece attached to the teeth and the amount of floating of the mouthpiece for each tooth; The first control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated floating amount, and when the calculated floating amount differs for each tooth, determines whether the largest floating amount is equal to or less than a predetermined threshold value; Equipped with the predetermined threshold is a predetermined ratio of the amount of floating to the entire area of the received image, The information processing method includes calculating the amount of floating for two to six teeth in each of the upper and lower dentition of the user.
Citation Information
Patent Citations
Orthodontic support system
JP2021090669A
Method for generating a model of a dental arch - Patent application
JP2022533423A
JPP7202729B
Methods and apparatuses for dental images
US20180125610A1
Method of evaluating an orthodontic aligner
US20180353263A1
Cited By
Programs, computers, and information processing methods
JP2026166378A
Programs, computers, and information processing methods
JP2026167383A
Program, computer and information processing method
JP7742681B1
