Program, computer, and information processing method

Abstract

To provide a program, a computer, and an information processing method capable of reducing a user's burden when performing correction of a tooth row.SOLUTION: There are provided reception means 14, calculation means 20, and determination means 22. The reception means 14 receives an image of the user's tooth row in which a transparent or semitransparent mouthpiece is fitted to teeth from a user terminal 32, the calculation means 20 calculates the amount of floating of the mouthpiece relative to each tooth on the basis of an image of a tooth row received by the reception means 14, and the determination means 22 determines whether a mouthpiece is exchanged or not, or correction of teeth is completed or not on the basis of the amount of floating calculated by the calculation means 20.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a program, a computer, and an information processing method. [Background technology]

[0002] In recent years, a technique for orthodontic treatment by having a user wear a mouthpiece on their teeth in their daily life has become known. Patent Document 1 discloses an orthodontic treatment support system that enables a user to order orthodontic appliances and undergo orthodontic treatment safely and at low cost. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-90669 Summary of the Invention [Problem to be solved by the invention]

[0004] When a user orthodontizes their teeth by wearing mouthpieces on their teeth in their daily lives, it is common for multiple mouthpieces of different shapes to be prepared for each user. After the user begins orthodontic treatment, a certain mouthpiece is worn and a predetermined time has passed. The dentist directly visually checks the user's teeth or views an image of the user's teeth with the mouthpieces worn on them. If the dentist determines that it is OK to proceed to the next stage, the mouthpiece is replaced with a different shape, and the user wears the new mouthpiece on their teeth. By repeating this process, the user's teeth are orthodontized to the desired shape.

[0005] However, if a dentist were to decide whether to replace the mouthpiece or complete the orthodontic treatment, the user would have to visit the dentist regularly, which would be a significant burden.

[0006] The present disclosure has been made in consideration of these points, and aims to provide a program, a computer, and an information processing method that can reduce the burden on a user when orthodontic treatment. [Means for solving the problem]

[0007] The program of the present disclosure is A program that causes a computer to function as a receiving means, a calculating means, and a determining means, The receiving means receives, from a user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, the calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The determining means determines whether or not to change the mouthpiece or complete the orthodontic treatment based on the amount of floating calculated by the calculating means.

[0008] In the program of the present disclosure, The calculation means may use a first trained model generated by machine learning using training data including an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and the amount of mouthpiece floating for each tooth, to calculate the amount of floating of the mouthpiece for each tooth from the image of the dentition accepted by the acceptance means.

[0009] In the program of the present disclosure, The calculation means may calculate the amount of floating of the mouthpiece relative to each tooth based on the number of pixels in an area of ​​the mouthpiece that is floating from each tooth in the image received by the reception means.

[0010] The program of the present disclosure is causing the computer to further function as a first trimming means; the first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth received by the receiving means; The calculation means may calculate the amount of floating of the mouthpiece relative to each tooth based on the image of the row of teeth trimmed by the first trimming means.

[0011] In the program of the present disclosure, The first trimming means may use a second trained model generated by machine learning using training data including an image of the user's dentition with a transparent or translucent mouthpiece attached to the teeth and the trimmed image of the dentition, to trim an image surrounded by a first bounding box including the dentition from the image of the dentition accepted by the accepting means.

[0012] In the program of the present disclosure, The second trained model may be generated using a YOLO model.

[0013] The program of the present disclosure is causing the computer to further function as a second trimming means; the second trimming means trims an image surrounded by a second bounding box including the upper tooth row and an image surrounded by a third bounding box including the lower tooth row from the image of the tooth row trimmed by the first trimming means; The calculation means may calculate the amount of floating of the mouthpiece for each tooth based on the image of the upper row of teeth and the image of the lower row of teeth trimmed by the second trimming means.

[0014] In the program of the present disclosure, The receiving means also receives information regarding the wearing time of the mouthpiece from the user terminal, The determination means may also refer to information regarding the length of time the mouthpiece has been worn, received by the reception means, to determine whether or not to replace the mouthpiece or whether or not to complete orthodontic treatment.

[0015] The program of the present disclosure is causing the computer to further function as a display instruction means; The display instruction means may transmit a display instruction signal to the user terminal to cause the user terminal to display the determination result by the determination means.

[0016] The program of the present disclosure is The computer further functions as a color image generating means, The colored image generating means colors the area of ​​the mouthpiece that stands out from each tooth in the image received by the receiving means, The display instruction means may transmit a display instruction signal to the user terminal to cause the user terminal to display the image of the row of teeth colored by the colored image generation means.

[0017] In the program of the present disclosure, the user terminal is configured to acquire an image of the row of teeth with the mouthpiece attached to the teeth when the user looks from the front, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the left, the receiving means receives, from the user terminal, the three images acquired by the user terminal; the calculation means calculates the amount of floating of the mouthpiece for each tooth for each of the three images of the row of teeth received by the reception means, The determining means may determine whether or not to change the mouthpiece or whether or not to complete the dental correction based on the amount of floating calculated by the calculating means for the three images.

[0018] In the program of the present disclosure, In the user terminal, when an image of the dentition is captured by the imaging unit of the user terminal, a frame of the dentition is displayed on the display unit of the user terminal, and an image of the dentition may be captured when the user's dentition falls within the frame of the dentition.

[0019] In the program of the present disclosure, In the user terminal, when the imaging unit captures an image of the row of teeth, a frame of the row of teeth when the user looks straight ahead, a frame of the row of teeth when the user looks to the right, and a frame of the row of teeth when the user looks to the left are displayed on the display unit in sequence, By capturing an image of the user's teeth when they fit into each of the tooth frame images may be obtained, such that an image of the teeth with the mouthpiece attached to the teeth when the user is looking straight ahead, an image of the teeth with the mouthpiece attached to the teeth when the user is facing to the right, and an image of the teeth with the mouthpiece attached to the teeth when the user is facing to the left are obtained.

[0020] In addition, the program of the present disclosure A program that causes a user terminal to function as a display means, an imaging means, and a transmission means, the display means displays a frame of the row of teeth on the display unit of the user terminal when an image of the row of teeth is captured by the imaging unit of the user terminal, the imaging means captures an image of the row of teeth when the row of teeth of the user is within the frame of the row of teeth; The transmitting means transmits the captured image of the row of teeth to an external device.

[0021] In the program of the present disclosure, the display means, when capturing an image of the row of teeth with the imaging unit, sequentially displays on the display unit a frame of the row of teeth when the user is looking straight ahead, a frame of the row of teeth when the user is looking to the right, and a frame of the row of teeth when the user is looking to the left; The imaging means may capture images of the user's teeth when they fit within the respective tooth row frames, thereby obtaining an image of the user's teeth when viewed from the front, an image of the teeth when the user faces to the right, and an image of the teeth when the user faces to the left.

[0022] The computer of the present disclosure includes: A computer that functions as a receiving means, a calculation means, and a determination means by executing a program, The receiving means receives, from a user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, the calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The determining means determines whether or not to change the mouthpiece or complete the orthodontic treatment based on the amount of floating calculated by the calculating means.

[0023] The information processing method of the present disclosure includes: An information processing method executed by a computer having a control unit, The control unit receives, from a user terminal, an image of the user's dentition with a transparent or translucent mouthpiece attached to the teeth; a step in which the control unit calculates the amount of floating of the mouthpiece for each tooth based on the received image of the row of teeth; a step in which the control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated amount of floating; The present invention is characterized by the following features. [Effects of the Invention]

[0024] According to the program, computer, and information processing method of the present disclosure, it is possible to reduce the burden on the user when orthodontic treatment of teeth. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a diagram schematically illustrating an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an image captured and processed in an information processing system according to an embodiment of the present disclosure. [Figure 3] FIG. 10 is a diagram showing a row of teeth and a frame of the row of teeth displayed on a user terminal in an information processing system according to an embodiment of the present disclosure. [Figure 4]FIG. 1 is a diagram illustrating exemplary processing in an information processing system according to an embodiment of the present disclosure. [Figure 5] FIG. 1 is a diagram illustrating an exemplary information processing flow of an information processing method according to an embodiment of the present disclosure. [Figure 6] FIG. 1 is a schematic process flow diagram of an orthodontic method according to the present disclosure from a user's perspective. [Figure 7] FIG. 10 is a diagram showing another exemplary flow of information processing of the information processing method according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a diagram showing another exemplary flow of information processing of the information processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0026] As described above, when orthodontic treatment is performed in daily life by attaching a plurality of prepared mouthpieces of different shapes to the teeth, the user must regularly visit a dentist to find out whether to replace the mouthpiece or complete the orthodontic treatment, which is a significant burden. As a result of intensive research into this problem, the inventors have found that if it is possible to determine whether to replace the mouthpiece or complete the orthodontic treatment from the amount of mouthpiece floating for each tooth calculated based on an image of the user's teeth in which transparent or translucent mouthpieces are attached, the burden on the user when orthodontic treatment can be reduced, and have conceived the invention of the present disclosure.

[0027] 1 to 8 are diagrams illustrating an information processing system 1 and an information processing method according to the present disclosure.

[0028] [Information Processing System 1] The information processing system 1 according to this embodiment shown in Fig. 1 includes a server 10 and a user terminal 32. The server 10 is a management server (computer) capable of transmitting and receiving information to and from a plurality of user terminals 32 via a communication network 50 such as the Internet. Note that information (images of dentition, assessment results, etc.) can be transmitted and received between the server 10 and the user terminal 32 via a website and / or so-called app (application software).

[0029] In this specification, unless otherwise specified, the term "image of dentition" refers to an image of a user operating the user terminal 32 wearing a transparent or translucent mouthpiece on the user's dentition (e.g., FIG. 2). The term "mouthpiece" is not particularly limited as long as it is used for orthodontic treatment. "Transparent or translucent" refers to the characteristics of the mouthpiece that enable calculation of the amount of floating of the mouthpiece relative to the teeth in the image, as described below. In other words, the degree of transparency or translucency is not limited as long as the mouthpiece is distinguishable from the teeth to which it is attached and the dentition can be seen through the mouthpiece in the image. Furthermore, although images of the user's face (dentition) viewed from the front and horizontal directions are shown as examples of images of the dentition in FIGS. 2 and 3, images of the user, i.e., the dentition, viewed from below and above may also be used for calculations, determinations, and the like according to the present disclosure.

[0030] Furthermore, the "amount of floating" of the mouthpiece relative to the teeth refers to the gap between the teeth and the mouthpiece (particularly in FIG. 2(C) , this is highlighted by coloring), and serves as a criterion for determining whether to change the mouthpiece or complete orthodontic treatment. Any threshold value can be set as this criterion. For example, the threshold value can be set within a range of 0 to 10%, preferably 2 to 6%, and more preferably 3 to 5%. If the ratio of the amount of floating (gap) to the area of ​​the entire image (e.g., FIG. 2(A)) is equal to or less than the threshold, it is determined that the mouthpiece should be changed or orthodontic treatment should be completed.

[0031] The number of teeth in the dentition for which the amount of floating is calculated can be appropriately set and changed depending on the purpose of orthodontics, and is not particularly limited. The amount of floating may be calculated for all teeth to which the mouthpiece is attached, or if only the front teeth are being corrected, the amount of floating may be calculated for 2 to 6 teeth. The amount of floating may also be calculated for both the upper and lower dentition (for example, 4 to 12 teeth).

[0032] <Server (Computer) 10> The server 10 can be configured as an industrial computer, a personal computer, a tablet terminal, or the like, or can be a virtual server within a physical server accessible via the Internet. There may be multiple pieces of hardware or virtual servers, or any combination of these may be used. The server 10 shown in FIG. 1 includes a control unit 12, a storage unit 28, and a communication unit 30.

[0033] (Control unit 12) The control unit 12 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and the like, and controls the operation of the server 10. Specifically, the control unit 12 executes programs stored in a storage unit 28 (described later) to function as a reception unit 14, a first trimming unit 16, a second trimming unit 18, a calculation unit 20, a determination unit 22, a colored image generation unit 24, a display instruction unit 26, and the like. These functions may be realized by executing one or more independent programs or applications. Furthermore, these programs and applications may be provided in a single terminal (the server 10) or may be distributed among multiple terminals (including the server 10), and in the latter case, may be connected to each other via a wired cable or a communication network 50. Each unit will be described later.

[0034] (Storage unit 28) The storage unit 28 is configured, for example, with an HDD (Hard Disk Drive), RAM (Random Access Memory), ROM (Read Only Memory), SSD (Solid State Drive), etc. Furthermore, the storage unit 28 is not limited to being built into the server 10, but may be a storage medium (for example, a USB memory) that can be detachably attached to the server 10. Furthermore, instead of providing the storage unit 28, various pieces of information may be stored in other storage means (such as a cloud server).

[0035] The memory unit 28 is capable of storing an image of a user's teeth with a transparent or translucent mouthpiece attached to the teeth (e.g., Figure 3) and the amount of floating of the mouthpiece for each tooth (e.g., highlighted in Figure 2(C)) received from multiple user terminals 32 as the first trained model 320 and its teacher data 310, an image of a user's teeth with a transparent or translucent mouthpiece attached to the teeth and a cropped image of the teeth (e.g., Figures 2(A) and (B)) as the second trained model 220 and its teacher data 210, and judgment criteria such as thresholds.

[0036] (Communication unit 30) The communication unit 30 connects the control unit 12 to an external device (such as a user terminal 32) wirelessly or via a wired connection so that the control unit 12 can communicate with the external device.

[0037] (others) In addition, the server 10 may be provided with an operation unit (input unit) such as a keyboard that enables an administrator of the server 10 to give various commands to the control unit 12, and a display unit (output unit) such as a monitor that displays various screens in response to a display command signal from the control unit 12. In one embodiment, a display operation unit such as a touch panel that integrates the operation unit and display unit may be used.

[0038] (Details of control unit 12) (Method of reception 14) The receiving means 14 receives an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth from the user terminal 32 ("100" in FIG. 4).

[0039] Furthermore, the receiving means 14 may also receive information regarding the length of time the mouthpiece is worn from the user terminal 32. This information can be reflected in the determination process by the determining means 22, which will be described later. For example, if there are fewer than a predetermined number of days (four days) in a week when the mouthpiece is worn for a predetermined time or longer (e.g., 20 hours), the corrective effect of the mouthpiece is considered to be small. Therefore, in such cases, it is considered that there is little need to verify the corrective effect, and the determination process does not necessarily have to be performed.

[0040] (First trimming means 16) The first trimming means 16 trims an image enclosed by a first bounding box 16a including the row of teeth from the image of the row of teeth received by the receiving means 14 (FIG. 2(A) and "110" in FIG. 4). FIG. 2(A) shows an exemplary image after trimming, and 16b represents the margin of the image after trimming. In this way, the calculation means 20 may calculate the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth trimmed by the first trimming means 16.

[0041] The first trimming means 16 may use a second trained model 220 generated by machine learning using training data 210 including an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and the trimmed image of the dentition to trim an image surrounded by a first bounding box 16a including the dentition from the image of the dentition accepted by the accepting means 14. Specifically, the first trimming means 16 "inputs" the image accepted by the accepting means 14 into the second trained model 220 and "outputs" an image of only the inside of the mouth, with the outside of the mouth trimmed, from the second trained model 220. As a result, the calculation means 20 can calculate the amount of floating of the mouthpiece for each tooth based on the image acquired by the first trimming means 16. The "machine learning" is not particularly limited as long as it can perform the above-described input and output, and various techniques such as deep learning and supervised learning can be used.

[0042] Here, the second trained model 220 can generally be generated as follows. First, unprocessed images captured by multiple users are collected as training data 210. Next, annotation is performed on each image using a rectangle circumscribing the dentition, and annotation data (cropped images of the dentition) is obtained as another training data 210. Next, the second trained model 220 is generated by learning using this training data 210 (at this time, parameters such as weights are updated and deployed). Although R-CNN, SSD, YOLO, etc. can be used as the training model on which the second trained model 220 is based, it is preferable that the second trained model 220 be generated using the YOLO model in terms of accuracy and speed.

[0043] (Second trimming means 18) The second trimming means 18 trims an image enclosed by a second bounding box 18a including the upper teeth and an image enclosed by a third bounding box 18b including the lower teeth from the image of the teeth trimmed by the first trimming means 16 (FIG. 2(B) and "120" in FIG. 4). Note that FIG. 2(B) shows the image after trimming. In this way, the calculation means 20 can also calculate the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth and the image of the lower teeth (FIG. 2(B)) trimmed by the second trimming means 18.

[0044] The trimming by the second trimming means 18 can also use machine learning and a trained model as described above (not shown). This trained model can be generated, for example, as follows: First, images of trimmed teeth are collected as training data. Next, annotation is performed separately for the upper and lower teeth in each image, and annotation data (images of trimmed upper and lower teeth) is obtained as separate training data. Next, a trained model is generated by learning using this training data (at this time, parameters such as weights are updated and deployed). To generate this trained model, it is preferable to use the YOLO model in terms of accuracy and speed.

[0045] (Calculation means 20) The calculation means 20 calculates the amount of floating of the mouthpiece for each tooth (FIG. 2(C)) based on the image of the row of teeth received by the reception means 14. Here, as described above, the amount of floating may be calculated for some or all of the teeth included in the image.

[0046] The calculation means 20 may use a first trained model 320 generated by machine learning using teacher data 310 including an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and the amount of mouthpiece lift for each tooth to calculate the amount of mouthpiece lift for each tooth from the image of the dentition accepted by the acceptance means 14. Specifically, the calculation means 20 may, for example, "input" the image accepted by the acceptance means 14 into the first trained model 320 and "output" the space between the teeth and the mouthpiece from the first trained model 320 as the amount of lift. This gap can be obtained as contour information by segmenting the image ("130" in FIG. 4). The first trained model 320 can be generated as follows: First, unprocessed images taken by multiple users are collected as the teacher data 310 (or separately cropped images of the upper and lower teeth may be used). Next, annotation is performed on each image along the contour of the gap between the mouthpiece and the teeth to obtain annotation data (contour information, amount of lift) as separate training data 310. Next, a first trained model 320 is generated by learning using this training data 310 (at this time, parameters such as weights are updated and deployed). Using YOLO as this segmentation model is preferable in terms of accuracy and speed.

[0047] The calculation means 20 can also calculate the amount of floating of the mouthpiece for each tooth based on the number of pixels in the area of ​​the mouthpiece that is floating from each tooth in the image received by the reception means 14. Here, it is preferable if the image sent from the user terminal 32 is set to a predetermined resolution, because this makes it easy to calculate the amount of floating based on the area (number of pixels) of the image.

[0048] (Determination means 22) The determination means 22 determines whether to replace the mouthpiece or whether to complete the orthodontic treatment based on the amount of floating calculated by the calculation means 20 ("150" in FIG. 4). Note that the present disclosure is premised on the use of mouthpieces of multiple shapes for orthodontic treatment, and if there is no mouthpiece to replace, the determination is made to complete the orthodontic treatment. In relation to this, in FIG. 6, which is a schematic process flow diagram of the orthodontic treatment method according to the present disclosure from the user's perspective, after determining whether the amount of floating is equal to or less than a threshold, it is also determined whether the "last mouthpiece" is being used (step S65).

[0049] As described above, the determining means 22 may make a rule-based determination based on whether the amount of floating is equal to or less than a predetermined threshold. Also, if the calculated amount of floating differs for each tooth, the determining means 22 may determine whether the largest amount of floating is equal to or less than a threshold, or whether the average amount of floating exceeds a threshold.

[0050] If the reception means 14 also receives information regarding the length of time the mouthpiece has been worn from the user terminal 32, the determination means 22 may refer to the information regarding the length of time the mouthpiece has been worn received by the reception means 14 to determine whether to replace the mouthpiece or complete orthodontic treatment, or may change the threshold for the amount of floating. In the example shown in FIG. 6, if the number of days in a week on which the mouthpiece has been worn for a predetermined time or more is less than a predetermined number, the user is asked to "extend" wearing the mouthpiece for a predetermined period of time without transmitting an image. This predetermined period can be set according to the number of days or time the user has not worn the mouthpiece.

[0051] (Colored image generation means 24) The colored image generating means 24 colors the areas of the mouthpiece that protrude from each tooth (the amount of protrusion) in the image received by the receiving means 14 (FIG. 2(C), "140" in FIG. 4). There are no particular limitations on the coloring method, and a person skilled in the art can adopt any desired coloring method or means.

[0052] (Display instruction means 26) The display instruction means 26 transmits a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the determination result by the determination means 22 on the user terminal 32 ("160" in Fig. 4). The display instruction means 26 can also transmit a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the image of the row of teeth colored by the colored image generation means 24 on the user terminal 32.

[0053] (First model generation means 300, second model generation means 200) The first model generation means 300 and the second model generation means 200 generate the first trained model 320 and the second trained model 220 through machine learning using the training data 310 and 210, respectively (FIG. 4). More specifically, the first model generation means 300 trains the first trained model 320 so that, when an image received by the receiving means 14 is "input," the first trained model 320 "outputs" the space between the teeth and the mouthpiece as the amount of lift. The second model generation means 200 trains the second trained model 220 so that, when an image received by the receiving means 14 is "input," the second trained model 220 "outputs" an image of the row of teeth with the portion outside the mouth trimmed. In this way, the first trained model 320 and the second trained model 220 are not limited to those generated by an external device and may be generated by the server 10. Note that the server 10 may be provided with a means for generating a trained model used in trimming by the second trimming means 18.

[0054] The first trained model 320 and / or the second trained model 220 may function by the control unit 12 executing a program stored in the memory unit 28, or may function by a device other than the server 10 executing a predetermined program. In the latter case, the first trained model 320 and / or the second trained model 220 generated by the other device is transmitted to the server 10 and stored in the memory unit 28 of the server 10.

[0055] In addition, if the user terminal 32 is configured to acquire an image of the row of teeth with the mouthpiece attached to the teeth when the user looks straight ahead, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the left, the accepting unit 14 may accept the three images acquired by the user terminal 32 from the user terminal 32, the calculation unit 20 may calculate the amount of mouthpiece lift for each tooth for each of the three images of the row of teeth accepted by the accepting unit 14, and the judgment unit 22 may judge whether to replace the mouthpiece or complete the orthodontic treatment based on the amount of lift calculated by the calculation unit 20 for the three images. In this way, when multiple images are accepted simultaneously from the same user, the judgment may be made based on whether the amount of lift exceeds a threshold for all images, or based on whether the average amount of lift for all images exceeds a threshold. The judgment criteria can be determined arbitrarily depending on the amount of information (number of images, imaging angle, etc.), allowing for a comprehensive judgment of the orthodontic effect. The number of images used for such a comprehensive assessment is not limited to three, and can be increased or decreased as appropriate depending on the purpose of correction.

[0056] <User terminal 32> The user terminal 32 may be composed of an industrial computer, a personal computer, a tablet terminal, a mobile terminal, etc., and is not particularly limited. In this specification, a "user" is a person who operates the user terminal 32 to send and receive information such as images to the server 10 while undergoing orthodontic treatment using a mouthpiece. The user terminals 32 do not need to be the same terminal for sending and receiving information with the server 10; as long as they are identified by the same registration information (such as an identification ID), the same user may send and receive information using different user terminals 32 at different times. The user terminal 32 shown in FIG. 1 includes a control unit 34, an imaging unit 42, an (operation) display unit 44, a storage unit 46, and a communication unit 48.

[0057] (Control unit 34) The control unit 34 is configured with a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and controls the operation of the user terminal 32. Specifically, the control unit 34 functions as a display unit 36, an image capture unit 38, a transmission unit 40, etc., by executing programs stored in the storage unit 28, which will be described later. Note that these functions may be realized by executing one or more independent programs or applications. Furthermore, these programs and applications may be provided in a single terminal (the user terminal 32) or may be distributed among multiple terminals (including the user terminal 32), and in the latter case, may be connected to each other via a wired cable or a communication network 50.

[0058] The display means 36 displays a frame of the row of teeth on the display unit 44 of the user terminal 32 when the imaging unit 42 of the user terminal 32 captures an image of the row of teeth (see FIG. 3). When the imaging unit 42 captures an image of the row of teeth, the display means 36 may sequentially display on the display unit 44 a frame of the row of teeth when the user is looking straight ahead (FIG. 3(A)), a frame of the row of teeth when the user is facing right (FIG. 3(B)), and a frame of the row of teeth when the user is facing left (FIG. 3(C)). For example, the frames may be displayed sequentially at predetermined intervals, or each frame may be displayed sequentially each time an image is captured. The display means 36 may also display the determination results and colored images of the row of teeth transmitted from the server 10.

[0059] The imaging means 38 captures an image of the user's teeth when the user's teeth are placed in the tooth row frame. This automatic imaging makes it possible to standardize the imaging distance, avoid blurring caused by the user pressing the shutter button, and capture a more standardized photograph. When multiple tooth row frames are displayed as described above, the imaging means 38 may capture images of the user's teeth when the user's teeth are placed in each tooth row frame, thereby obtaining an image of the user's teeth when looking straight ahead, an image of the user's teeth when the user faces right, and an image of the user's teeth when the user faces left.

[0060] The transmission means 40 transmits (uploads) the captured image of the row of teeth to an external device (such as the server 10).

[0061] (Image capture unit 42) The imaging unit 42 captures an image of the row of teeth. There are no particular limitations on the imaging unit 42, but it is preferable that the imaging unit 42 is an AI camera app that performs object detection and can automatically capture an image of the row of teeth when the user's row of teeth falls within the frame of the row of teeth.

[0062] (Display section 44) The display unit 44 displays various screens including the frame of the dentition upon receiving a display command signal from the control unit 34. The display unit 44 is typically a monitor, a display, or the like, but is not particularly limited to such. The user terminal 32 may be provided with an operation unit (input unit) such as a keyboard that allows the user to give various commands to the control unit 34, and if the user terminal 32 is a mobile phone such as a smartphone, an operation display unit 44 such as a touch panel in which the operation unit and display unit are integrated may be used ( FIG. 1 ).

[0063] (Storage unit 46) The storage unit 46 is configured, for example, with an HDD (Hard Disk Drive), RAM (Random Access Memory), ROM (Read Only Memory), SSD (Solid State Drive), etc. Furthermore, the storage unit 46 is not limited to being built into the user terminal 32, but may be a storage medium (for example, a USB memory) that is detachably attached to the user terminal 32. Furthermore, instead of providing the storage unit 46, various pieces of information may be stored in other storage means (such as a cloud server).

[0064] (Communications Department 48) The communication unit 48 connects the control unit 34 to an external device (such as the server 10) wirelessly or via a wire so that the control unit 34 can communicate with the external device. The control unit 34 can send and receive information to and from the external device via the communication unit 48.

[0065] [Information processing method 1] Next, information processing method 1 in the information processing system 1 described above will be described with reference to Figures 5 and 6. Here, server 10 performs a process of determining whether to replace the mouthpiece or complete orthodontic treatment based on the amount of floating of the mouthpiece for each tooth calculated based on an image received from user terminal 32. In the following description, components with the same reference numerals are the same as those described above, and redundant description will be omitted as appropriate. Note that the process described below is performed by executing a program stored in storage unit 28, but the information processing method according to the present disclosure is not limited to this.

[0066] First, the control unit 12 (accepting means 14) accepts from the user terminal 32 an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth (FIG. 5, step S10).

[0067] Next, the control unit 12 (first trimming means 16) trims the image enclosed by the first bounding box 16a including the row of teeth from the received image of the row of teeth (FIG. 5, step S20; FIG. 2(A)).

[0068] Next, the control unit 12 (second trimming means 18) trims from the trimmed image of the teeth an image surrounded by a second bounding box 18a including the upper teeth and an image surrounded by a third bounding box 18b including the lower teeth (Figure 5, step S30; Figure 2(B)).

[0069] Next, the control unit 12 (calculation means 20) calculates the amount of floating of the mouthpiece for each tooth from the cropped image of the upper and lower teeth (FIG. 5, step S40). As described above, the amount of floating may be calculated based on the number of pixels, or the first trained model 320 generated by machine learning using training data 310 including images of the user's teeth with a transparent or translucent mouthpiece attached to the teeth and the amount of floating of the mouthpiece for each tooth may be used to calculate the amount of floating.

[0070] Next, the control unit 12 (colored image generating means 24) colors the areas of the mouthpiece that stand out from each tooth in the image received by the control unit 12 (receiving means 14) (FIG. 5, step S50; FIG. 2(C)).

[0071] Next, the control unit 12 (determination means 22) determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated amount of floating (FIG. 5, step S60). As described above, the determination may be made based on whether the amount of floating is equal to or less than a threshold value.

[0072] Next, the control unit 12 (display instruction means 26) transmits a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the determination result and the image of the colored row of teeth (FIG. 5, step S70).

[0073] As shown in FIG. 6, if the result of step S60 is that the user should change the mouthpiece or complete orthodontic treatment (if the amount of floating is below the threshold, "YES"), it is also determined whether the mouthpiece the user is currently wearing is the last mouthpiece (step S65). If it is the last mouthpiece ("YES"), the process ends. This determination result is sent to the user terminal 32 (FIG. 5, step S70). If it is determined in step S65 that the mouthpiece the user is wearing is not the last mouthpiece ("NO"), the user will begin wearing a new mouthpiece as the next treatment step. This determination result is also sent to the user terminal 32 (FIG. 5, step S70). On the other hand, if the result of step S60 is that the amount of floating exceeds the threshold ("NO"), the user will "extend" wearing the mouthpiece for a predetermined period of time, and this determination result (and an image of the colored dentition) is also sent to the user terminal 32 (FIG. 5, step S70). FIG. 6 is a schematic process flow diagram of the orthodontic method according to the present disclosure from the user's perspective. It is preferable to have the server 10 send a notification every day (at a specified time) before each step (action) performed by the user to prompt the user to perform that step, as this can prevent the user from forgetting to wear the mouthpiece, for example.

[0074] [Information processing method 2] Next, an information processing method 2 in the information processing system 1 will be described with reference to Fig. 7. The information processing method 2 is an example in which a program for acquiring an image to be determined according to the present disclosure is executed in the user terminal 32.

[0075] First, the control unit 34 (display means 36) causes the display unit 44 of the user terminal 32 to display a frame of the row of teeth when the imaging unit 42 of the user terminal 32 captures an image of the row of teeth (step S110).

[0076] Next, when the user's dentition, with a transparent or semi-transparent mouthpiece attached to the teeth, is placed in the dentition frame, the control unit 34 (imaging means 38) takes an image of the dentition (step S120).

[0077] Next, the control unit 34 (transmission means 40) transmits the captured image of the row of teeth to an external device (step S130). Note that the transmission means 40 may also transmit information regarding the wearing time of the mouthpiece, based on input by the user.

[0078] [Information processing method 3] Next, information processing method 3 in the information processing system 1 will be described with reference to Fig. 8. Information processing method 3 is an example in which, in addition to the processing in information processing method 2, a program executed by the server 10 described above is executed by the user terminal 32. That is, a program corresponding to the calculation means 20, determination means 22, etc. other than the reception means 14 and display instruction means 26 is executed by the control unit 34 of the user terminal 32. Steps S210 and S220 in Fig. 8 correspond to steps S110 and S120 shown in Fig. 7 of information processing method 2, and therefore description thereof will be omitted.

[0079] After step S220, the control unit 34 (calculation means 20) calculates the amount of floating of the mouthpiece for each tooth based on the captured image of the row of teeth (step S230).

[0080] Next, as in step S60 of Figure 5, the control unit 34 (determination means 22) determines whether or not to replace the mouthpiece or whether or not to complete dental correction based on the calculated amount of floating (step S240).

[0081] Next, the control unit 34 (display means 36) causes the display unit 44 to display the determination result (step S250).

[0082] The information processing method described above is merely an example, and the processing flow is not limited to the above. For example, in information processing method 1, the coloring step (step S50) of coloring the area of ​​the mouthpiece protruding from each tooth in the image may be performed after the determination step (step S60) of determining whether to replace the mouthpiece or complete the orthodontic treatment. Similarly, in information processing method 3, after the determination step (step S240), the control unit 34 (colored image generating means 24) may color the area of ​​the mouthpiece protruding from each tooth in the image, as in step S50 of FIG. 5, and the colored image of the row of teeth may be displayed on the display unit 44 in step S250.

[0083] The information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, configured as described above, are provided with a receiving means 14, a calculating means 20, and a determining means 22. The receiving means 14 receives an image of a user's dentition with a transparent or translucent mouthpiece attached from the user terminal 32, the calculating means 20 calculates the amount of mouthpiece floating for each tooth based on the image of the dentition received by the receiving means 14, and the determining means 22 determines whether to replace the mouthpiece or complete orthodontic treatment based on the amount of floating calculated by the calculating means 20. In one embodiment according to the present disclosure, the user terminal 32 is provided with a display means 36, an imaging means 38, and a transmitting means 40. When the imaging unit 42 of the user terminal 32 images the dentition, the display means 36 displays a frame of the dentition on the display unit 44 of the user terminal 32, the imaging means 38 images the dentition when the user's dentition falls within the frame of the dentition, and the transmission means 40 transmits the image of the captured dentition to an external device.

[0084] Conventionally, orthodontic treatment in daily life by fitting multiple pre-prepared mouthpieces of different shapes to the user's teeth required the user to visit the dentist regularly, which was a significant burden. To address this issue, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method disclosed herein calculate the amount of mouthpiece floating for each tooth based on an image of the user's teeth, and determine whether to replace the mouthpiece or complete the orthodontic treatment based on the amount of floating, thereby reducing the burden on the user when orthodontic treatment is performed. Furthermore, as described above, by displaying a tooth frame on the display unit 44 of the user terminal 32 and capturing an image of the teeth when the teeth are placed within the tooth frame, an image of the user's teeth can be efficiently obtained.

[0085] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the calculation means 20 may use a first trained model 320 generated by machine learning using training data 310 including an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and the amount of mouthpiece lift for each tooth, to calculate the amount of mouthpiece lift for each tooth from the image of the dentition accepted by the acceptance means 14. In this way, it is possible to automatically and efficiently calculate the amount of lift to be determined using AI.

[0086] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the calculation means 20 may calculate the amount of floating of the mouthpiece relative to each tooth based on the number of pixels in the area of ​​the mouthpiece that is floating from each tooth in the image received by the reception means 14. In this way, the amount of floating can also be calculated based on the number of pixels (image area).

[0087] Furthermore, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure may further include first cropping means 16 that crops an image surrounded by a first bounding box 16a that includes the row of teeth from the image of the row of teeth received by the receiving means 14. In this case, the calculation means 20 calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth cropped by the first cropping means 16. In this way, by omitting extraoral image information (features) that is not related to the calculation of the amount of floating, subsequent calculation processes and determination processes (including segmentation) can be stabilized.

[0088] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the first trimming means 16 may use a second trained model 220 generated by machine learning using training data 210 including an image of the user's dentition with a transparent or translucent mouthpiece attached to the teeth and the trimmed image of the dentition to trim an image enclosed by a first bounding box 16a including the dentition from the image of the dentition accepted by the accepting means 14. In this way, the amount of floating to be determined can be calculated automatically and efficiently using AI.

[0089] In addition, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the second trained model 220 may be generated using the YOLO model. Generating the second trained model 220 using the YOLO model is preferable in terms of accuracy and speed.

[0090] Furthermore, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure may further include second cropping means 18 that crops an image surrounded by a second bounding box 18a including the upper teeth and an image surrounded by a third bounding box 18b including the lower teeth from the image of the teeth cropped by the first cropping means 16. In this case, the calculation means 20 calculates the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth and the image of the lower teeth cropped by the second cropping means 18. In this way, by omitting image information (features) of the mouth that is not related to the calculation of the amount of floating, subsequent calculation processes and determination processes (including segmentation) can be stabilized.

[0091] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, the reception means 14 may also receive information relating to the length of time the mouthpiece has been worn from the user terminal 32, and the determination means 22 may determine whether to replace the mouthpiece or whether to complete orthodontic treatment by referring to the information relating to the length of time the mouthpiece has been worn received by the reception means 14. Since orthodontic treatment may not be effective if the mouthpiece has not been worn for a predetermined period of time, it is not necessary to perform calculation processing or determination processing in the first place (FIG. 6).

[0092] Furthermore, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure may further include display instruction means 26 that transmits a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the determination result by the determination means 22. In this way, an explanation can be provided, similar to a diagnosis by a dentist, as to whether the same mouthpiece or a different mouthpiece should be used as a future treatment step, or whether orthodontic treatment will be completed, without the need for a dental visit.

[0093] Furthermore, the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure may further include colored image generation means 24 that colors the area of ​​the mouthpiece that protrudes from each tooth in the image received by the reception means 14. In this case, the display instruction means 26 sends a display instruction signal to the user terminal 32 to cause the user terminal 32 to display the image of the row of teeth that has been colored by the colored image generation means 24. In this way, the amount of protrusion can be presented to the user without the need for a dental visit, and a convincing explanation of future treatment steps can be provided, similar to a diagnosis by a dentist.

[0094] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, if the user terminal 32 is configured to acquire an image of the row of teeth with a mouthpiece attached to the teeth when the user looks straight ahead, an image of the row of teeth with a mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with a mouthpiece attached to the teeth when the user looks to the left, the accepting means 14 may accept the three images acquired by the user terminal 32 from the user terminal 32, the calculation means 20 may calculate the amount of mouthpiece floating for each tooth for each of the three images of the row of teeth accepted by the accepting means 14, and the determining means 22 may determine whether to replace the mouthpiece or complete the orthodontic treatment based on the amount of floating calculated by the calculation means 20 for the three images. In this way, the criteria for judgment can be determined according to the purpose of orthodontics (number of teeth, position, etc.) and the amount of information (number of images, imaging angle, etc.), and the orthodontic effect can be comprehensively determined based on each image.

[0095] Furthermore, in the information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure, when the imaging unit 42 captures an image of the row of teeth, the display unit 36 ​​may cause the display unit 44 to sequentially display a frame of the row of teeth when the user is looking straight ahead, a frame of the row of teeth when the user is facing right, and a frame of the row of teeth when the user is facing left, and the imaging unit 38 may capture an image of the row of teeth when the user is looking straight ahead, an image of the row of teeth when the user is facing right, and an image of the row of teeth when the user is facing left by capturing an image of the row of teeth when the user is looking straight ahead, an image of the row of teeth when the user is facing right, and an image of the row of teeth when the user is facing left. In this way, the comprehensive assessment as described above can be performed efficiently.

[0096] The information processing system 1, computer (server 10, user terminal 32), program, and information processing method according to the present disclosure are not limited to the above-described aspects and combinations, and various modifications can be made. [Explanation of symbols]

[0097] 1. Information Processing Systems 10 Servers (computers) 12 Control Unit 14. Reception methods 16 First trimming means 16a First bounding box 16b Margin 18 Second trimming means 18a Second bounding box 18b Third bounding box 20 Calculation means 22 Judgment means 24 Colored image generation means 26 Display instruction means 28 Memory section 30 Communications Department 32 User terminals 34 Control Unit 36 Display means 38 Imaging means 40 Transmission Method 42 Imaging unit 44 (Operation) Display section 46 Memory section 48 Communications Department 50 Communication Network 200 Second model generation means 210 Teacher Data 220 Second trained model 300 first model generation means 310 Teacher Data 320 First trained model

Claims

1. A program that causes a computer to function as a receiving means, a calculating means, and a determining means, The receiving means receives, from a user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, the calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The determination means determines whether or not to replace the mouthpiece or complete the orthodontic treatment based on the amount of floating calculated by the calculation means.

2. The program of claim 1, wherein the calculation means uses a first trained model generated by machine learning using training data including an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and the amount of mouthpiece floating for each tooth to calculate the amount of floating for each tooth from the image of the dentition accepted by the acceptance means.

3. 3. The program according to claim 1, wherein the calculation means calculates the amount of floating of the mouthpiece relative to each tooth based on the number of pixels in the area of ​​the mouthpiece that is floating from each tooth in the image received by the reception means.

4. causing the computer to further function as a first trimming means; the first trimming means trims an image surrounded by a first bounding box including the row of teeth from the image of the row of teeth received by the receiving means; 2. The program according to claim 1, wherein the calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth trimmed by the first trimming means.

5. The program of claim 4, wherein the first trimming means uses a second trained model generated by machine learning using training data including an image of a user's dentition with a transparent or translucent mouthpiece attached to the teeth and the trimmed image of the dentition to trim an image surrounded by a first bounding box including the dentition from the image of the dentition accepted by the accepting means.

6. The program according to claim 5 , wherein the second trained model is generated using a YOLO model.

7. causing the computer to further function as a second trimming means; the second trimming means trims an image surrounded by a second bounding box including the upper tooth row and an image surrounded by a third bounding box including the lower tooth row from the image of the tooth row trimmed by the first trimming means, 5. The program according to claim 4, wherein the calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the upper teeth and the image of the lower teeth trimmed by the second trimming means.

8. The receiving means also receives information regarding the wearing time of the mouthpiece from the user terminal, 2. The program according to claim 1, wherein the determination means also refers to information regarding the length of time the mouthpiece has been worn that has been received by the reception means, and determines whether or not to replace the mouthpiece or whether or not to complete orthodontic treatment.

9. causing the computer to further function as a display instruction means; 2. The program according to claim 1, wherein the display instruction means transmits a display instruction signal to the user terminal to cause the user terminal to display the determination result made by the determination means.

10. The computer further functions as a color image generating means, The colored image generating means colors the area of ​​the mouthpiece that stands out from each tooth in the image received by the receiving means, 10. The program according to claim 9, wherein the display instruction means transmits a display instruction signal to the user terminal to cause the user terminal to display the image of the row of teeth colored by the colored image generation means.

11. the user terminal is configured to acquire an image of the row of teeth with the mouthpiece attached to the teeth when the user looks from the front, an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the right, and an image of the row of teeth with the mouthpiece attached to the teeth when the user looks to the left, the receiving means receives, from the user terminal, the three images acquired by the user terminal; the calculation means calculates the amount of floating of the mouthpiece for each tooth for each of the three images of the row of teeth received by the reception means, 2. The program according to claim 1, wherein the judgment means judges whether to change the mouthpiece or complete the orthodontic treatment based on the amount of floating calculated by the calculation means for the three images.

12. The program of claim 1, wherein when the imaging unit of the user terminal captures an image of the dentition, a frame of the dentition is displayed on the display unit of the user terminal, and when the user's dentition falls within the frame of the dentition, the image of the dentition is captured.

13. In the user terminal, when the imaging unit captures an image of the row of teeth, a frame of the row of teeth when the user looks straight ahead, a frame of the row of teeth when the user looks to the right, and a frame of the row of teeth when the user looks to the left are displayed on the display unit in sequence, The program of claim 12, wherein an image of the user's teeth is captured when the user's teeth are placed within each of the tooth row frames, thereby obtaining an image of the teeth with the mouthpiece attached to the teeth when the user is looking from the front, an image of the teeth with the mouthpiece attached to the teeth when the user is facing to the right, and an image of the teeth with the mouthpiece attached to the teeth when the user is facing to the left.

14. A program that causes a user terminal to function as a display means, an imaging means, and a transmission means, the display means displays a frame of the row of teeth on the display unit of the user terminal when an image of the row of teeth is captured by the imaging unit of the user terminal, the imaging means captures an image of the row of teeth when the row of teeth of the user is within the frame of the row of teeth; The transmitting means transmits the captured image of the row of teeth to an external device.

15. the display means, when capturing an image of the row of teeth with the imaging unit, sequentially displays on the display unit a frame of the row of teeth when the user is looking straight ahead, a frame of the row of teeth when the user is looking to the right, and a frame of the row of teeth when the user is looking to the left; The program according to claim 14, wherein the imaging means captures images of the user's teeth when they are within the respective tooth row frames, thereby obtaining an image of the user's teeth when viewed from the front, an image of the teeth when the user is facing right, and an image of the teeth when the user is facing left.

16. A computer that functions as a receiving means, a calculation means, and a determination means by executing a program, The receiving means receives, from a user terminal, an image of the user's dentition with a transparent or semi-transparent mouthpiece attached to the teeth, the calculation means calculates the amount of floating of the mouthpiece for each tooth based on the image of the row of teeth received by the reception means, The judgment means is a computer that judges whether or not to replace the mouthpiece or whether or not to complete dental correction based on the amount of floating calculated by the calculation means.

17. An information processing method executed by a computer having a control unit, The control unit receives, from a user terminal, an image of the user's dentition with a transparent or translucent mouthpiece attached to the teeth; a step in which the control unit calculates the amount of floating of the mouthpiece for each tooth based on the received image of the row of teeth; a step in which the control unit determines whether to replace the mouthpiece or complete the orthodontic treatment based on the calculated amount of floating; An information processing method comprising:

Images