Augmented reality display for dental instruments

Abstract

The dental tool system includes a dental tool having a camera, a UV light, and a work implement, and associated software for receiving video of the user's teeth captured by the dental tool camera, analyzing the video to determine the amount of plaque on the teeth, overlaying icons on unhygienic locations so that the icons are visible to the user in a substantially real-time display of the user's teeth, and removing the icons after the user has sufficiently brushed the unhygienic locations.

Description

[Technical Field]

[0001] This application claims the benefit of priority to U.S. Application No. 63 / 404,451, filed September 7, 2022, which is incorporated herein by reference in its entirety. [Background technology]

[0002] This disclosure relates to augmented reality techniques for images captured by medical tools, such as dental instruments. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of an exemplary imaging dental instrument system. [Figure 2] FIG. 2 is a schematic diagram of a portion of the exemplary system of FIG. 1 in use. [Figure 3] FIG. 3 is a schematic diagram of an overlay display that can be used in the system of FIG. [Figure 4] FIG. 4 illustrates an exemplary graphical user interface that can be used in conjunction with the system of FIG. [Figure 5] FIG. 5 is a perspective view of an imaging toothbrush used in the system of FIG. [Figure 6] FIG. 6 is a perspective view of the toothbrush of FIG. 5 with a portion of the case cut away to show the internal components. [Figure 7] FIG. 7 is a perspective view of a portion of the system of FIG. 1, including an exemplary video toothbrush. [Figure 8] FIG. 8 is a side view of the toothbrush of FIG. 7, with a portion of the case shown as transparent to show the internal components. [Figure 9] FIG. 9 is a schematic diagram of a light pipe that can be used to transmit ultraviolet light in an imaging toothbrush. [Figure 10] FIG. 10 is a flow chart illustrating an exemplary method for enhancing video from a dental instrument. [Figure 11] FIG. 11 is a flow chart illustrating an exemplary method for enhancing video from a dental instrument. [Figure 12] FIG. 12 is a flow chart illustrating an exemplary method for operating ultraviolet light in the system of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0004] The inventors believe that it may be beneficial to enhance the display of images from a medical tool to provide information about the patient's teeth to the patient and / or healthcare provider. For example, images acquired by a dental tool can be enhanced in accordance with the present disclosure. Such an imaging dental instrument is shown or disclosed in U.S. Application No. 14 / 645,145, which is incorporated herein by reference and which is published as U.S. Publication No. 2015 / 0257636. The remainder of this disclosure will refer to an imaging toothbrush for ease of explanation, but is not limited to such an embodiment. Furthermore, the remainder of this disclosure will refer to the condition of a patient's teeth, but it should be understood that other conditions within the user's oral cavity (e.g., stomatitis), as well as medical conditions related to other parts of the body, are within the scope and spirit of the present disclosure.

[0005] 1 is a schematic diagram of a system 100 including a video toothbrush 102 and a mobile computing device 104. The video toothbrush 102 may include a camera 106, an ultraviolet (UV) light 108, and a brush 110 or other working implement. The mobile computing device may include a processor 112 and a non-transitory computer-readable memory or other medium 114 storing instructions that, when executed by the processor 112, cause the mobile computing device 104 to perform one or more steps, operations, methods, processes, algorithms, etc. of the present disclosure. The mobile computing device may further include a display 116 and one or more functional modules 118, 120, which may be embodied in hardware and / or software (e.g., as instructions in the memory 114).

[0006] The video toothbrush 102 is used by a user to brush their teeth with the brush 110 while capturing video with the camera 106. The video toothbrush 102 can be in electronic communication with a mobile computing device 104 and can transmit the captured video feed to the mobile computing device 104. The mobile computing device 104 can then display the video feed substantially in real time on a display 116. As described below, the mobile computing device 104 can also analyze and / or enhance the video stream.

[0007] The functional modules 118, 120 may include an image analysis module 118 that receives a video stream and analyzes the video to determine whether one or more conditions are present. For example, the image analysis module 118 may include a plaque volume sub-module 122 that may determine the presence and location of plaque on the teeth, a gum color sub-module 124 that may determine the color and color change of a user's gums at one or more locations, a tooth movement sub-module 126 that may determine the position and relative movement of teeth, a tonsil space sub-module 128 that may determine the size and size change of a user's upper airway, and / or other sub-modules for other medically relevant or informational determinations.

[0008] The plaque volume sub-module 122 can analyze images in a real-time video feed to determine plaque volume and location. For example, as described with respect to Figures 10, 11, and 12, the plaque volume sub-module 122 can perform processing on and analyze post-UV exposure images.

[0009] The gum color sub-module 124 may obtain multiple sample images of the user's mouth from a given video feed and determine an average gum tissue color associated with that video session (e.g., a brushing session). The gum color sub-module 124 may store the average gum color and compare the average gum tissue color to one or more previously stored gum tissue colors for the user. If the gum color sub-module 124 determines through the comparison that the user's gum tissue color has changed (e.g., deviates from a previous color) by more than a threshold amount, the gum color sub-module 124 may notify the user that their gum color has changed and / or that they may be at risk for periodontal disease.

[0010] The tooth movement sub-module 126 can determine and store the position of each of the user's teeth during a brushing session. The position of a tooth may be determined, for example, relative to other teeth. The tooth movement sub-module 126 can compare the position of each tooth to previously stored positions of the user's teeth. If the tooth movement sub-module 126 determines through the comparison that one or more of the user's teeth have moved beyond a threshold amount, the tooth movement sub-module 126 can cause a notification to be sent to the user that the user's teeth have moved or have moved.

[0011] The tonsil space sub-module 128 may determine the contour of the user's upper airway, including the size and prominence of the user's tonsils, based on the video captured by the video toothbrush 102. The contour of the user's upper airway may be compared to known contour features (e.g., large tonsil size, narrow airway) that indicate the onset of sleep apnea or other breathing disorders. If the tonsil space sub-module 128 determines through the comparison that the contour of the user's upper airway is consistent with a potential breathing disorder, the tonsil space sub-module 128 may cause a notification to be sent to the user that the user's upper airway is consistent with a potential breathing disorder.

[0012] The functional modules 118, 120 may further include an information overlay module 120 that may overlay one or more items of information, graphics, etc., to augment the video feed displayed on the display 116 based on determinations by the image analysis module 118. For example, the information overlay module 120 may add one or more icons to the respective locations of one or more of the user's teeth determined to have plaque present, teeth determined to have moved, discolored gums, etc.

[0013] The mobile computing device 104 may further be in electronic communication with external data storage 130, such as cloud storage. For example, the external data storage 130 may be associated with an application running on the mobile computing device 104, which may include modules 118, 120 and output a video feed on the display 116. The mobile computing device 104 may transmit and retrieve the video feed (in original and / or enhanced format), decisions made by the image analysis module 118, and / or other information to and from the data storage 130. Additionally or alternatively, the mobile computing device 104 may store the video feed, decisions, and / or other information in memory 114.

[0014] The UV light 108 of the video toothbrush 102 can be used to identify plaque on a user's teeth. For example, a user can activate the UV light 108, shine the UV light on one or more portions of the user's mouth, and then deactivate the UV light 108. In some embodiments, the UV light 108 can be automatically deactivated by the video toothbrush 102 after a predetermined time or based on video analysis of the reaction of the user's mouth after sufficient exposure to UV light. Because plaque is easily identifiable in video after UV exposure, the image analysis module 118 can identify plaque exposed to UV light. In some embodiments, the video toothbrush may further include a permanent or retractable filter on the camera 106 that performs a color correction function to identify clean teeth as white and plaque as red after the plaque is illuminated with UV light.

[0015] The UV light 108, in embodiments, can emit light across the UV spectrum. For example, the UV light 108 can be a UV curing light that outputs light in the range of 385-490 nm. In some embodiments, the UV light 108 can emit light at a frequency centered around 405 nm, a frequency particularly useful for irradiating and identifying plaque.

[0016] In addition to or instead of local execution of modules 118, 120 on mobile computing device 104, backend processing source 132 (e.g., a cloud service) can execute one or more aspects of the image analysis module and / or the information overlay module. Thus, mobile computing device 104 can be in electronic communication with backend processing 132, for example, to transmit raw data and / or augmented video between backend processing 132 and mobile computing device 104, and to transmit decisions based on image analysis between backend processing 132 and mobile computing device 104.

[0017] The mobile computing device 104, data storage 130, and / or backend processing 132 may further electronically communicate with one or more care providers (caregiver or treatment provider-utilized devices) 134. For example, the mobile computing device and / or backend processing 132 may, with appropriate consent from the user, send a decision based on image analysis to a care provider (e.g., a dentist or other dental health care professional). Notifications disclosed herein as being sent to the user may instead or additionally be sent to the care provider 134. Furthermore, with appropriate consent from the user, the care provider may access the user's footage or decisions stored in storage 130 and send comments to the user based on the care provider's review of the footage and / or decisions.

[0018] In some embodiments, the video toothbrush 102 can be in electronic communication with one or more of the backend processing 132 and data storage 130 independent of the mobile computing device 104. Thus, the video toothbrush 102 can send video to and / or receive instructions from the backend processing 132 and / or data storage 130 in addition to, or instead of, similar communication with the mobile computing device 104.

[0019] The video toothbrush 102 can record video with a camera 106 included in or attached to the toothbrush 102 and output the video to one or more displays or storage devices (e.g., external data storage 130 or memory 114). In one embodiment, video from the toothbrush may be output and displayed on a mobile computing device 104, which may be a smartphone, tablet, or other device. Additionally or alternatively, video may be output and displayed on a user-wearable device such as a smartwatch, smart glasses, etc. Additionally or alternatively, video may be output and displayed on a hologram projector. Thus, output of video from the toothbrush to a display may be substantially real-time or may be at a time after the video is recorded (e.g., playback or stored video).

[0020] The video toothbrush 102 may, in one embodiment, be used by a consumer as part of a normal hygiene routine. Additionally or alternatively, the video toothbrush 102 may be used by a dental professional as part of a patient's treatment.

[0021] Footage captured by the video toothbrush 102 may, in one embodiment, be displayed by a user of the toothbrush 102 (e.g., on the consumer's mobile computing device 104 or wearable device if the toothbrush is used by a consumer, or on a display associated with a dental professional if the toothbrush is used by a dental professional). Additionally or alternatively, the footage may be displayed by a third party that has received the footage or permission to display the footage from the capture device. For example, such a third party may be a dental professional who has received the footage from a patient.

[0022] Various information may be used to enhance the display of the video captured by the video toothbrush 102. For example, such information may be overlaid on or presented adjacent to the video. Several example modes of enhancement are described in turn below. The various modes are described with reference to a "patient's" mouth. However, it should be understood that this is merely for ease of explanation, and the various modes may be used by either a dental or medical professional or a consumer. These modes may, in embodiments, be implemented via functional modules 118, 120.

[0023] In one or more modes of use, the video may be augmented by overlaying information about the patient's dental condition. For example, known fillings, crowns, implants, etc. may be color-coded, text-labeled, or identified in some additional or alternative manner within the video. Such information, in embodiments, may be based on clinical information provided by a healthcare provider, input from the consumer / patient, or another source external to the video. Additionally or alternatively, such information may be determined by analysis of the video itself. For example, the video may be analyzed to identify and tag the patient's teeth and conditions associated with those teeth (e.g., by the computer processor of the user's mobile electronic device). Such video analysis, as with all video analysis in this disclosure, may be performed by the mobile computing device 104, the processing power of the video toothbrush 102, and / or the back-end processing 134.

[0024] In one or more modes of use, the video captured by the video toothbrush 102 can be used as the basis for creating a model of the patient's teeth or other body part. For example, the video captured by the toothbrush 102 can be used to create a partial or complete three-dimensional model of the patient's teeth. The video can then be analyzed to create the model. In any of the modes of use described below, the video of the patient's teeth can be superimposed on such a model, displayed next to the model, or displayed in place of the actual video feed, based on the camera's field of view and position. Thus, although the following modes of use are described with respect to displaying information with or on the video, such information can additionally or alternatively be displayed on or with the model. In addition to or instead of a model of the patient's teeth, one or more icons or other graphic features may be superimposed on the actual image of the user's teeth, as described below.

[0025] In "live mode," the video may be supplemented to show a wider field of view than is actually captured by camera 106. For example, in embodiments in which the live video typically displays three or four teeth based on the position of camera 106, the field of view of camera 106, etc., the video display may augment the teeth actually captured in the video with representations of the surrounding teeth in the patient's oral cavity (e.g., models that appear to the user to be the teeth themselves). Icons and other image enhancements in accordance with the present disclosure may be overlaid or added to such models, as well as the teeth in the field of view of camera 106. Additionally or alternatively, "live mode" may enhance the display by allowing the user to zoom in on a particular portion of the video frame, for example, by holding the camera lens in one position for a predetermined period of time. Additionally or alternatively, "live mode" may zoom in by default, providing a close-up view of the patient's teeth.

[0026] In "X-ray mode," the video may be augmented by overlaying an x-ray of the user's teeth on the video or displaying the x-ray along with the video. The x-ray may, in embodiments, be a 2D or 3D x-ray. In other embodiments, images from other imaging modalities may be overlaid or displayed along with the video. Icons and other image augmentations according to the present disclosure may be overlaid or added to such x-ray images and the actual video feed from camera 106.

[0027] In "movement mode," a toothbrush (or other instrument) can be used to apply pressure to the teeth or other body parts, and footage can be displayed to highlight tooth movement (e.g., by showing the "original" and "moved" positions of the teeth). Additionally, or alternatively, footage can be displayed to highlight the movement of one or more teeth relative to previously captured footage (e.g., as described above with respect to tooth movement sub-module 126).

[0028] In "Plaque Mode," the video may be overlaid with information about the patient's tooth plaque. For example, plaque may be color-coded on the video to provide the user with instructions on specific areas to brush. "Plaque Mode," in one embodiment, may include the use of a UV light on the toothbrush to indicate plaque. Figures 3 and 9-12 illustrate various aspects of "Plaque Mode," in which a "monster" or other illustration is overlaid on areas of the tooth that have plaque, allowing the user to "attack" the monster by brushing the plaque-covered area. In response to plaque being removed from the tooth, the overlaid monster or other illustration may be removed.

[0029] In "clinical mode," the imagery may be overlaid or displayed with information for performing a clinical procedure. For example, locations to be drilled, filled, incised, lasered, or otherwise treated may be color-coded or otherwise annotated for the clinician to see.

[0030] The footage may also be used to determine the patient's ability to treat a condition at home, or the success of attempts to treat a condition at home. For example, if the patient is unable to adequately cleanse plaque at home or treat another condition identified in the footage, the footage can be sent to a care provider, such as the patient's dentist, to request assistance with the condition (e.g., explanation of the condition, treatment recommendations, scheduling an appointment, etc.).

[0031] The video and tagged information can be used to recommend further treatment. For example, if a patient's teeth change to appear to be creating a negative condition (e.g., early decay, excessive tooth wear, etc.), recommendations can be automatically provided to the patient (e.g., a recommendation to schedule fluoride treatments, a recommendation to brush certain areas more carefully, etc.). Such recommendations can be determined, for example, by a processor in the patient's mobile computing device.

[0032] FIG. 2 is a schematic diagram of the use of the system 100. As shown in FIG. 2, the toothbrush 102 captures video of the user's mouth, including the user's teeth, while the user brushes their teeth. The toothbrush 102 can stream live video to the user's mobile computing device 104, which can display live video of the user's mouth for the user to view while brushing. With reference to FIGS. 1 through 3, the image analysis and information overlay modules 118, 120 can determine the amount of plaque or other information on the user's teeth and overlay one or more icons 302 on the user's teeth based on those determinations. For example, an icon 302 can be overlaid on live video where the amount of plaque on the user's teeth exceeds a threshold. As described with respect to FIGS. 10 and 11, the icon 302 can be animated or otherwise changed based on the user's brushing and interaction with the icon 302. Additionally, the icon can be positioned based on the amount of plaque on the user's teeth, which can be determined in a first example based on images captured by the video toothbrush 102 after irradiating the user's teeth with UV light. This UV light may also be emitted by the imaging toothbrush 102 .

[0033] FIG. 4 illustrates an exemplary user interface portion 400 that may be presented on a user's mobile computing device, such as in a portion of an application that also displays a live video feed from the toothbrush 102 in another portion. The user interface portion 400 may include multiple user-selectable menu items 402, including a calendar menu item, a progress menu item, a bill credit menu item, a dashboard menu item, a live view menu item, and a learning menu item. The user interface portion may allow the user to select among the various modes described above. The user interface portion 400 may report the user's participation in one or more activities, such as daily brushing routines and / or scores for a particular period of time, in a reporting portion 404. Both the score and brushing routine may be determined by the application based on the user's interaction with the icons. For example, a user may add a day to a streak by having a brushing session in which all icons are brushed based on the associated plaque removed. In another example, a user may add a day to a streak by brushing twice in one day with a particular number of icons brushed. Other information described herein, such as notifications based on analysis of the video of the user's mouth, may also be presented to the user in interface portion 400. Additionally or alternatively, such notifications may be sent to the user via email, text message, etc.

[0034] FIGS. 5-8 illustrate various exemplary features of the imaging toothbrush. First, FIGS. 5 and 6 illustrate a dental instrument 500 (which may function as the imaging toothbrush 102) including a body 502 and a dental instrument insert 504 extending from the end of the body 502 for use in a patient's oral cavity. Positioned adjacent to the dental instrument insert 504 may be a camera lens 506. The camera lens 506 may record images or videos and may preferably have a viewing angle that is generally parallel to the dental instrument insert 504 extending from the body 502 of the dental instrument 500 and directed toward the working tip of the dental instrument insert 504. However, the camera lens 506 may also have a relatively wide angle (field of view) to record or show a relatively wide field of view. Thus, the dental instrument 500 may display images and / or videos of the working tip of the dental instrument insert 504, particularly while it is being used in a patient's oral cavity. In a preferred embodiment, the dental instrument 500 can have multiple lights 508, such as, for example, LEDs, ringing the lens 506 and providing sufficient light source for recording and / or viewing images and / or videos. In another embodiment, the lights can be ultraviolet lights or a combination of visible and ultraviolet lights that can assist the user of the dental instrument 500 in observing plaque on a patient's teeth or in treating epoxies, etc., in the patient's mouth.

[0035] 6 shows a cutaway side view of a dental instrument 500 according to the present invention. Specifically, the dental instrument 500 may preferably include a dental instrument insert 504 as described above that is removable from the body 502 so as to be interchangeable with other dental instrument inserts, such as a pick, scaler, flosser, etc. The dental instrument insert 504 may be positioned within an opening in the body 502 and, once positioned therein, may be mechanically connected to a motor and transmission 510 that may be electrically connected to a battery / power source 512 within the body 502. The dental instrument 500 may be powered via the battery / power source 512 and, preferably, recharged via a charging element 514 that may be electrically coupled to a power source for charging the battery / power source 512 as needed.

[0036] The lens 506 and light 508 can be located on the same end of the body as the dental instrument insert 504 so that when used in a patient's mouth, the camera lens 506 and light 508 facilitate recording and / or viewing of the dental instrument insert 504. Coupled to the lens 506 and light 508 is a main PCB board 516 that can control the recording and viewing of images and / or videos through the lens 506 and can also control the light 508. The main PCB board 516 may allow a user to turn the lens 506 on or off for recording and / or viewing or turn the light 508 on or off, as needed.

[0037] In use, a user can insert the dental instrument insert 504 into an opening in the body 502 of the dental instrument 500. The dental instrument 500 may be pre-connected, by wire or preferably wirelessly, to a display device (not shown) for viewing and / or recording images and / or video through the lens 506. The user may have the ability to control the lens 506 and / or light 508 to be on or off via the display device (not shown), via the body 502, or via any other means or mechanism for controlling the same.

[0038] FIG. 7 shows a system 700 including a dental instrument 702 (which may function as the video toothbrush 102), a plurality of dental instrument inserts 704, a base 706, and a display device (e.g., a user's mobile computing device) 104 for displaying the video thereon.

[0039] The base 706 holds the dental instrument 702, the plurality of tool inserts 704, and may also hold the display device 104. Specifically, the base 706 may operate as a charging cradle for holding and / or charging the dental instrument 702 and / or the display device 104. More specifically, the base 706 may include a plurality of openings or cradles for holding the dental instrument 702 and the plurality of dental instrument inserts 704.

[0040] The display device 104 may be mounted on or within a platform 710. The platform 710 may be angled so that the user of the dental instrument 702 and / or the patient can view the display device 104. The platform 710 may be adjustable to tilt or reposition the display device 104 for better viewing thereon, and may have a frame, flange, or other similar holder for holding the display device 104 thereon or therein.

[0041] The display device 104 may preferably be a smartphone such as an iPhone®, an Android® phone, a tablet computer such as an iPad®, or other similar display device capable of wirelessly transmitting streaming video from the dental instrument 702, as described in more detail below. The display device 104 may receive wireless streaming video from the dental instrument 702 via any method apparent to one skilled in the art, preferably via WiFi, a 3G cellular network, Bluetooth®, or other similar data transmission protocol, and via MPEG4, M-JPEG, or other similar video codec.

[0042] The multiple dental instrument insert 704 is any dental instrument that a dental practitioner can use to clean, treat, or otherwise prepare a patient's oral cavity. Common dental instruments include brushes, scalers, mirrors, probes, syringes, drills, burrs, excavators, burnishers, drills, elevators, forceps, curettes, and any other similar instruments that can be usefully used from the dental instrument 702. Additionally, the present invention allows for the simultaneous use of multiple instruments, such as, for example, a dental mirror and a scaler to aid in the use of the scaler.

[0043] 8 shows a cutaway perspective view of a dental instrument 702. The dental instrument 702 generally includes a housing 720 that can accommodate a camera 722 having a lens 724 for capturing video and / or still photographs, one or more processing boards 726, a rechargeable battery 728, a power cable 730, a charging input 732, and a dental instrument insert opening 734 for holding one or more dental instrument inserts 704. One or more light sources (not shown) can further be included to direct illumination into the dental instrument inserts 704 and / or the patient's oral cavity. As will be apparent to one skilled in the art, the housing 720 can be hermetically sealed and made of a material that can withstand a wet environment, such as metal or plastic.

[0044] During use, a user, such as a dentist, can insert the dental instrument insert 704 into the dental instrument opening 734, which can securely hold the dental instrument therein via clamping means, friction-resistant means, or the like. The camera lens 724 is disposed at the end of the housing 720 and is aimed toward the tip of the dental instrument insert 704 to provide a relatively wide cone of view 742 for viewing the patient's oral cavity and the dental instrument being used therein. Specifically, the lens 724 may be adjacent to the shaft of the dental instrument insert 704 and may provide a camera line of sight 740 that is generally parallel to the shaft of the dental instrument insert 704. In this manner, the dental instrument insert 704, and in particular the working tip of the dental instrument insert 704, can be easily viewed by a user of the dental instrument 702 and / or the patient via the camera 722. In a preferred embodiment, the lens 724 can be aimed directly at the working tip of the dental instrument insert 704 to specifically and precisely view the working tip of the dental instrument insert 704. Additionally, one or more light sources, such as white LEDs, can be used to illuminate the dental appliance insert 704 and / or the patient's oral cavity.

[0045] In a preferred embodiment, a dental mirror may be used to enable the camera 722 to record video and / or still photographs of areas, such as the inside of the patient's oral cavity, at 90 degrees or other angles relative to the line of sight 740 of the camera 722. Additionally, illumination sources emanating from the dental instruments 702 may be further reflected off the dental mirror to help illuminate the patient's oral cavity.

[0046] In some embodiments, the video toothbrush 102 can include a UV light for illuminating a user's oral cavity, for example, to reveal plaque on the user's teeth. FIG. 9 shows an exemplary UV light arrangement 900 for including UV light in a video toothbrush 102 according to the present disclosure. The UV light arrangement 900 includes a UV surface-mounted device ("SMD") LED 902 or other LED capable of emitting light in UV frequencies and a light pipe 904 that transmits the output light from the LED 902 to a desired output location and angle on the video toothbrush. For example, the UV light output can be near the camera lens (e.g., as shown for light 508 in FIG. 5). Alternatively, the UV LED can be mounted near its output with a lens to generate the appropriate output angle and pattern, and white light or other visible light LEDs can also be located within the same video toothbrush 102 so that the video toothbrush 102 can selectively output visible light and / or UV.

[0047] 10 is a flow chart illustrating an example method 1000 of operating a video toothbrush system. Method 1000, or one or more aspects of method 1000, may be performed, in embodiments, by one or more components of system 100. For example, one or more of video toothbrush 102, user mobile computing device 104, and backend processing 132 may perform one or more operations of method 1000.

[0048] Method 1000 may include, at operation 1002, receiving video of a tooth captured by a dental tool having a camera and a cleaning tool. The cleaning tool may be within the field of view of the camera. The video may be received by the dental tool itself (e.g., via the camera) and / or by a user mobile computing device, back-end processing, or other processing source.

[0049] Method 1000 may further include outputting the received video to a display in real time at operation 1004. The received video may be output, for example, on a display of a user's mobile computing device, a projector, a pair of VR goggles, or any other suitable display device. In some embodiments, the real-time output may continue while further operations described below are performed.

[0050] Method 1000 may further include analyzing the received video to determine the amount of plaque at the multiple locations at operation 1006. Plaque may be identified by a light-induced fluorescence (LIF) technique based on the red fluorescent properties of porphyrins, which are heterogeneous bacterial metabolites within the plaque, when illuminated with a narrow blue-violet light (e.g., centered at 405 nm wavelength).

[0051] In some embodiments, operation 1006 can include a multi-stage process for identifying plaque in an image. First, object detection can be performed on each image frame to determine whether the input image is an oral cavity image and localize oral cavity regions. Second, instance segmentation can be performed to extract plaque regions.

[0052] For object detection, a ground-truth bounding box technique can be applied to identify the outline of a region of interest that contains only the teeth and gums in each image. A more fine-tuned bounding box detector can then be applied to detect the Region of Interest within the image.

[0053] In instance segmentation, a machine learning model can be applied to precisely extract red-fluorescent plaque regions (e.g., multiple regions) from oral images to identify red fluorescence for individual teeth as well as across or between multiple teeth and determine where the corresponding plaque is located on the tooth. The model can be, for example, or include, a recurrent neural network trained on a set of red fluorescence images. Details of an exemplary model are described in Kim et al., Light-Induced Fluorescence-Based Device and Hybrid Mobile App for Oral Hygiene Management at Home: Development and Usability Study, JMIR Mhealth Uhealth. 2020 Oct 16;8(10):e17881. doi:10.2196 / 17881. Pmid:33064097; pmcid:pmc7600004.

[0054] Operation 1006 may include identifying the width and height of areas of plaque on individual teeth, as well as areas of plaque between teeth and across multiple teeth, and thus may determine the size and approximate shape of each plaque area.

[0055] Method 1000 may further include, at operation 1008, determining that more than a threshold amount of plaque is present at one or more unhygienic locations on the user's teeth. In some embodiments, a specific size of a particular region of plaque may be used as a threshold (e.g., a size below which plaque regions are considered "clean" or not flagged for further cleaning). Such a size threshold may be a square area, width, and / or height. Additionally or alternatively, the threshold amount of plaque may be determined according to the total amount of fluorescence associated with the region of plaque. Such total fluorescence may be, for example, the cumulative intensity of red pixels in the plaque region. More generally, the threshold can be based on the size of the plaque region or the total intensity in an image of the plaque region. Plaque regions exceeding the threshold may be considered "unhygienic."

[0056] Method 1000 may further include overlaying an icon at each unhygienic location at operation 1010. The icon may be a suitable image that draws the user's attention to the plaque area and motivates the user to brush the plaque area. For example, the icon may be a monster (as shown in FIG. 3 ) or other anthropomorphic image, a simple shape (e.g., a square, a circle, etc.), an arrow, or any other icon visible to the user. The icon may be overlaid on the live video frame at locations determined to be unhygienic. The icon may be overlaid such that it moves along the field of view or viewing frame as the user moves the camera and changes the field of view and associated video, remaining visible and overlaid on unhygienic locations on the user's teeth.

[0057] Method 1000 may further include determining that plaque has been removed from the unsanitary location and removing the icon at operation 1012. Operation 1012 may include the same or similar process as operation 1006, determining that the amount of plaque at a location previously determined (at a first time point) to be "unsanitary" at operation 1008 is now (at a second time point) clean, i.e., the amount of plaque no longer exceeds the threshold applied at operation 1008. In response, the icon may be removed from the video view.

[0058] In some embodiments, operation 1012 may include changing the icon when the user brushes on or near the location over which the icon is located. For example, the icon may animate when an associated unhygienic location is being brushed as visual feedback to the user that the correct location is being brushed. In another example, the size, brightness, or other characteristic of the icon may change to indicate a reduction in plaque volume or the presence of a toothbrush in the location of the plaque.

[0059] While the user is brushing, i.e., while the user is moving the camera to remove plaque, operations 1002, 1004, 1006, 1008, 1010, and 1012 are continuously performed for each frame or a predetermined subset of frames (e.g., 1 frame per second, 2 frames per second, etc.) in the live video feed, and the location of plaque is continuously annotated with icons that can be changed or removed while the user is brushing to provide the user with real-time feedback on their brushing progress.

[0060] In some embodiments, method 1000 may further include analyzing the video to determine deviations in gum color, upper airway contours that may indicate a respiratory problem, tooth movement, and / or other physiological conditions that can be determined from the video as described above. If such analysis indicates a potential problem, method 1000 may further include sending a notification to the user, as described with respect to FIG. 1. Additionally, in some embodiments, method 1000 may further include receiving contact information for a care provider (e.g., from the user or care provider), receiving user consent to share diagnostic information with the care provider, and sharing one or more analyses, notifications, images, video, conclusions, etc. of this disclosure with the care provider.

[0061] 11 is a flowchart illustrating an example method 1100 for manipulating an augmented reality output. Method 1100, or one or more aspects of method 1100, may, in an embodiment, be performed by one or more components of system 100. For example, one or more of video toothbrush 102, user mobile computing device 104, and backend processing 132 may perform one or more of the operations of method 1100.

[0062] Method 1100 can include various non-cleaning conditions and scenarios in which the overlaid icon can be animated to draw the user's attention to the icon and encourage the user to brush away plaque associated with the icon. In addition to the example reasons and scenarios provided in method 1100, the icon can be animated in numerous other situations consistent with this disclosure.

[0063] Method 1100 may include determining that a predetermined amount of time has elapsed or occurred since brushing began, at operation 1102. This amount of time may be, for example, 1 minute, 90 seconds, or 2 minutes since brushing began for the entire oral cavity. Additionally or alternatively, this amount of time may be 15, 20, or 30 seconds since the user began brushing a quadrant of the oral cavity.

[0064] Method 1100 may further include, at operation 1104, determining that the brush passed over the unsanitary location without adequately cleaning the location, i.e., without adequately removing plaque. Operation 1104 may include, for example, determining that the amount of plaque (e.g., the amount of red fluorescence) has decreased but not decreased below a threshold for classifying the location as "clean." Additionally or alternatively, operation 1104 may include determining that the brush coincided with a location known to have plaque present, but that the amount of plaque remains above a threshold for classifying the location as "unsanitary."

[0065] Method 1100 may further include, in act 1106, animating the icon in response to either act 1102 or act 1104 (i.e., upon determining that the no-wash condition has been met). Animating the icon may include, for example, moving the icon (e.g., rotating it), making the icon a multi-frame “video” icon so that the icon appears to dance, ripple, wiggle, etc., flashing the icon, changing the size or shape of the icon, etc. In essence, act 1106 may include changing the appearance of the icon in response to act 1102 or 1104. Thus, as small amounts of plaque build up, the icon may change in real time (e.g., the monster may do a death flip, become injured, or lean forward in time with a death scream). If not all of the plaque has been removed, the visual representation may change to a smaller monster or other representation. Early enamel decay or demineralization may be underlying the plaque. This area of ​​possible caries can be marked as something else (igloo, stop sign, etc.) for the dentist to investigate.

[0066] 12 is a flow chart illustrating an example method 1200 for exposing a user's teeth to UV light to assess the presence of plaque. Method 1200, or one or more aspects of method 1200, can be performed, in embodiments, by one or more components of system 100. For example, one or more of video toothbrush 102, user mobile computing device 104, and backend processing 132 can perform one or more operations of method 1200.

[0067] Method 1200 may include, at operation 1202, instructing a user to expose their teeth to UV light. The instruction may be provided via a display, such as a display on the user's mobile computing device. Alternatively, the instruction may be provided via the video toothbrush, by output sounds, recorded verbal output instructions, output light patterns, etc. The UV light may be disposed on the video toothbrush such that the video toothbrush can capture video of the user's mouth being exposed to the UV light.

[0068] Method 1200 may further include, at operation 1202, analyzing output video from the video toothbrush when exposing the user's teeth to UV light. Operation 1202 may include, for example, analyzing exposure of multiple surfaces in the user's mouth to determine whether UV light has been applied to one or more teeth. In some embodiments, operation 1202 may include individually assessing UV exposure for teeth and adding or removing tooth tags when those teeth are sufficiently exposed to UV light. Sufficient exposure of a tooth may be determined, for example, by determining a duration of exposure for the tooth based on the duration of the tooth in a portion of the video frame that correlates with the UV light emission field based on images captured by a camera that include UV light exposure.

[0069] The method 1200 may further include, at operation 1204, determining that the UV exposure is sufficient and, in response, automatically deactivating the UV light. The automatic deactivation may serve as a safety measure to prevent a user from intentionally or unintentionally overexposing themselves to the UV light. The UV exposure may be determined to be sufficient, for example, when each tooth has been sufficiently exposed, when a certain percentage of the teeth have been sufficiently exposed, or when a total exposure time has reached a threshold.

[0070] In some embodiments, methods 1000, 1100, and 1200 can be used in combination. For example, method 1200 can be performed to expose a user's teeth to UV light to enable plaque identification, and then method 1000 can be performed for the user to clean the plaque. During the performance of method 1000, method 1100 can also be performed to add further detail and interest to the augmented reality display.

[0071] In a first aspect of the present disclosure, a method for enhancing video from a dental tool is provided, the method including receiving video of a plurality of teeth, receiving video captured by a dental tool including a camera and cleaning implements, outputting the video in substantially real time to a display visible to a user of the dental tool, analyzing the video to determine an amount of plaque at a plurality of locations on the plurality of teeth, determining at a first time that the amount of plaque exceeds a threshold for an unsanitary location at the plurality of locations, overlaying respective icons within the video at the unsanitary locations such that the respective icons are visible to the user when the unsanitary locations are visible to the user in the substantially real-time output, and determining at a second time that the amount of plaque at the unsanitary locations no longer exceeds the threshold, and in response, removing the icons from the overlaid video.

[0072] In an embodiment of the first aspect, the method further includes determining that a non-cleaning condition for the unsanitary location has been met and, in response, animating an icon associated with the unsanitary location. In a further embodiment, the non-cleaning condition includes the passage of a predetermined time after the first time or a dental implement passing through the unsanitary location without sufficient plaque removal.

[0073] In a first embodiment, analyzing the image to determine the amount of plaque at a plurality of locations on a plurality of teeth includes detecting red fluorescence at each of the plurality of locations.

[0074] In a first embodiment, the icon is a first icon, and removing the first icon from the superimposed image includes replacing the first icon with a second icon.

[0075] A first embodiment involves analyzing the video to determine when a dental tool implement is in an unsanitary position and animating an icon accordingly.

[0076] In a first embodiment, the method provides a dental tool system, further comprising instructing a user to expose the user's teeth to UV light of the dental tool.

[0077] In a second embodiment of the present disclosure, a dental tool system is provided, the dental tool system including a camera, a UV light, and a dental tool having a working tool. The system further includes a non-transitory computer-readable medium storing instructions that, when executed by a processor of the computer system, cause the computer system to perform operations including: instructing a user via a display to irradiate the user's teeth with the UV light; receiving images of the teeth; receiving the images captured by the camera of the dental tool; outputting the images to a display viewable in substantially real time to a user of the dental tool; analyzing the images to determine the amount of plaque at multiple locations on the multiple teeth; first determining that the amount of plaque exceeds a threshold value for an unsanitary location at the multiple locations; superimposing respective icons on the image of the unsanitary locations so that the respective icons are visible to the user when the unsanitary locations are visible to the user in the substantially real-time output; and second determining that the amount of plaque at the unsanitary locations does not exceed a threshold value; and, in an embodiment of the second aspect, the operations further include determining that the amount of plaque at the unsanitary locations does not exceed a threshold value and, in response, removing the icons from the superimposed image.

[0078] In an embodiment of the second aspect, the operations further include determining that a non-cleaning condition for the unsanitary location has been met and animating the icon associated with the unsanitary location accordingly. In a further embodiment of the second aspect, the non-cleaning condition includes the lapse of a predetermined time after the first time or the passage of a working implement of the dental tool at the unsanitary location without sufficient removal of the plaque.

[0079] In an embodiment of the second aspect, analyzing the image to determine the amount of plaque at multiple locations on the multiple teeth includes detecting red fluorescence at each of the multiple locations.

[0080] In an embodiment of the second aspect, the icon is a first icon, and removing the first icon from the superimposed image includes replacing the first icon with a second icon.

[0081] In an embodiment of the second aspect, the operations further include analyzing the video to determine that a working implement of the dental instrument is in an unsanitary location and animating the icon accordingly.

[0082] In a third aspect of the present disclosure, a method is provided for receiving a video of a user's oral cavity, outputting the video captured by a dental instrument including a camera and a cleaning instrument in substantially real time on a display visible to a user of the dental instrument, analyzing the video to determine an amount of plaque at a first plurality of locations on a plurality of teeth in the oral cavity, and determining a color of gingival tissue at a second plurality of locations in the oral cavity, superimposing respective icons in the video based on the determined amount of plaque at unhygienic locations of the first plurality of locations so that the respective icons are visible to the user when the unhygienic locations are visible to the user in the substantially real-time output, determining that the color of the gingival tissue deviates from a previous color of the user's gingival tissue by more than a threshold amount, and responsively outputting an indication of potential periodontal disease to the user on the display.

[0083] In an embodiment of the third aspect, the video is analyzed to determine a position of each of a plurality of teeth relative to adjacent teeth, and the method further includes determining that at least one tooth has moved relative to a previous position of the at least one tooth, and responsively outputting an indication of the moved tooth to a user on a display.

[0084] An embodiment of the third aspect includes analyzing the video to determine a contour of the user's respiratory passages, determining that the contour is indicative of a respiratory problem, and responsively outputting an indication of the potential respiratory problem to the user on a display.

[0085] In an embodiment of the third aspect, the method further comprises outputting instructions to a user to expose the user's teeth and gums to the UV light of the dental instrument.

[0086] In an embodiment of the third aspect, the method further includes sending a notification of the potential periodontal disease to a care provider associated with the user. In a further embodiment of the third aspect, the method further includes receiving contact information for the care provider and receiving consent from the user to share the user's personal medical information with the care provider.

[0087] In an embodiment of the third aspect, analyzing the image to determine the amount of plaque at a plurality of locations on a plurality of teeth comprises detecting red fluorescence at each of the plurality of locations.

[0088] Although this disclosure has described particular embodiments, it will be understood that the claims are not intended to be limited to these embodiments, unless expressly set forth in the claims. To the contrary, this disclosure is intended to cover alternatives, modifications, and equivalents that may be included within the spirit and scope of the disclosure. Furthermore, in the detailed description of the present disclosure, numerous specific details are set forth in order to provide a thorough understanding of the disclosed embodiments. However, it will be apparent to those skilled in the art that systems and methods consistent with the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure various aspects of the present disclosure.

[0089] Some portions of the detailed descriptions of this disclosure are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer or digital system memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art, and a procedure, logic block, process, or the like, is herein generally conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps require physical manipulations of physical quantities. Usually, though not necessarily, these physical manipulations take the form of electrical or magnetic data capable of being stored, transferred, combined, compared, and otherwise manipulated within a computer system or similar electronic computing device. For convenience, and with reference to common usage, such data will be referred to as bits, values, elements, symbols, characters, terms, numbers, or the like in reference to the various embodiments of the invention.

[0090] It should be noted, however, that these terms should be interpreted as referring to physical operations and quantities and are merely convenient labels to be further interpreted in light of terminology commonly used in the art. Unless otherwise indicated, as will be apparent from the discussion herein, throughout the discussion of the present embodiments, discussions using terms such as "determine" or "output" or "transmit" or "record" or "arrange" or "store" or "display" or "receive" or "recognize" or "utilize" or "generate" or "provide" or "access" or "check" or "notify" or "deliver" are understood to refer to the operations and processes of a computer system or similar electronic computing device that manipulates and transforms data. Data is represented as physical (electronic) quantities in the registers and memory of a computer system and is transformed into other data similarly represented as physical quantities in the memory or registers of a computer system, or other information storage, transmission, or display methods described herein or understood by those skilled in the art.

Claims

1. 1. A method of enhancing an image from a dental instrument, comprising: receiving images of a plurality of teeth, the images being captured by dental instruments including a camera and a cleaning instrument; outputting the image to a display viewable by a user of the dental instrument in substantially real time; analyzing the images to determine the amount of plaque at multiple locations on multiple teeth; determining, at a first time, that the amount of plaque exceeds an unsanitary location threshold for the plurality of locations; superimposing the respective icons in the image at the unsanitary location such that the respective icons are visible to the user when the unsanitary location is visible to the user in substantially real-time output; and at a second time, determining that the amount of plaque in the unsanitary location no longer exceeds the threshold, and responsively removing the icon from the superimposed image.

2. The method of claim 1 further comprising: determining that an unclean condition is met at the unsanitary location and animating the icon associated with the unsanitary location accordingly.

3. 3. The method of claim 2, wherein the non-washing conditions include: a predetermined elapsed time after the first time; or The method comprises passing a working tool of the dental instrument over the unsanitary area without adequately removing plaque.

4. 10. The method of claim 1, The method, wherein analyzing the images to determine the amount of plaque at a plurality of locations on the plurality of teeth includes detecting red fluorescence at each of the plurality of locations.

5. 10. The method of claim 1, the icon is a first icon, The method, wherein removing the first icon from the superimposed image includes replacing the first icon with a second icon.

6. 10. The method of claim 1, The method further comprising analyzing the video to determine that a working implement of the dental instrument is in the unsanitary location and animating the icon accordingly.

7. 10. The method of claim 1, further comprising outputting instructions to a user to expose the user's teeth to the UV light of the dental instrument.

8. 1. A dental instrument system comprising: A dental instrument and this dental instrument are Camera and; UV light; and a work implement; A non-transitory computer-readable medium storing instructions that, when executed by a processor of a computer system, cause the computer system to perform an operation, the non-transitory computer-readable medium comprising: instructing the user via the display to irradiate the user's teeth with the UV light; receiving images of a plurality of teeth captured by the camera of the dental instrument; outputting the image on a display viewable in substantially real time to a user of the dental instrument; analyzing the images to determine the amount of plaque at multiple locations on multiple teeth; determining with the dental instrument that the amount of plaque exceeds a threshold value for an unsanitary location among the plurality of locations; superimposing respective icons in the image at the unsanitary locations such that the respective icons are visible to the user when the unsanitary locations are visible to the user in substantially real-time output; and At a second time, the system determines that the amount of plaque in the unsanitary location no longer exceeds the threshold, and accordingly removes the icon from the overlaid footage.

9. 9. The system of claim 8, The operation may further include: determining that a non-clean condition of the unsanitary location has been met, and responsively animating an icon associated with the unsanitary location.

10. 10. The system of claim 9, The non-washed state is a predetermined time has elapsed since the first time; or A system in which dental instrument working tools pass through unsanitary areas without adequately removing plaque.

11. 9. The system of claim 8, The system, wherein analyzing the images to determine an amount of plaque at a plurality of locations on the plurality of teeth includes detecting red fluorescence at each of the plurality of locations.

12. 9. The system of claim 8, The icon is a first icon, and removing the first icon from the superimposed image includes replacing the first icon with a second icon.

13. 9. The system of claim 8, The operation further comprises: analyzing the video to determine that a working implement of the dental instrument is in the unhygienic position and animating the icon accordingly.

14. 1. A method comprising: receiving video of the user's oral cavity, the video captured by dental tools including a camera and cleaning instruments; outputting the image to a display viewable in substantially real time to a user of the dental tool; Analyze the video Determining the amount of plaque at a first plurality of locations on a plurality of teeth in the oral cavity; and determining the color of the gum tissue at a second plurality of locations within the oral cavity; based on the determined amount of plaque in the unsanitary locations of the first plurality of locations, superimposing respective icons within the image such that the respective icons are visible to the user when the unsanitary locations are shown to the user via a substantially real-time output; and determining that the color of the gum tissue deviates from the previous color of the gum tissue of the user by more than a threshold amount, and responsively outputting an indication of potential gum disease to the user on the display; A method comprising:

15. 15. The method of claim 14, Analyzing the images further comprises determining a position of each of the plurality of teeth relative to adjacent teeth; and The method further includes determining that the at least one tooth has moved relative to a previous position of the at least one tooth, and in response outputting a representation of the moved tooth to a user on a display.

16. 15. The method of claim 14, analyzing the image further comprises determining a contour of the user's respiratory passageway; The method further includes determining that the contour is indicative of a respiratory problem, and in response, outputting an indication of a potential respiratory problem to the user on the display.

17. 15. The method of claim 14, further comprising outputting instructions to the user to expose the user's teeth and gums to the UV light of the dental tool.

18. 15. The method of claim 14, further comprising sending a notification of the potential periodontal disease to a care provider associated with the user.

19. 20. The method of claim 18, receiving contact information for the care provider; and A method comprising receiving consent from the user to share personal medical information of the user with the care provider.

20. 15. The method of claim 14, The method, wherein analyzing the images to determine the amount of plaque at a plurality of locations on the plurality of teeth includes detecting red fluorescence at each of the plurality of locations.

Images