Augmented reality display for dental tools

Abstract

A dental tool system includes a dental tool having a camera, a UV lamp, and a work implement. The system also includes related software that receives a video of the user's teeth captured by the dental tool camera, analyzes the video to determine a number of plaques on the plurality of teeth, overlays the icon at the unclean location such that the icon is visible to the user in a substantially real-time display of the user's teeth, and displays the icon at the unclean location. And removing the icon after the user sufficiently wipes the unclean location.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. application No. 63 / 404,451, filed on September 7, 2022, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to augmented reality for videos from medical instruments, such as dental tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a block diagram of an example video dental tool system.

[0005] Figure 2 is in use Figure 1 Schematic diagram of a portion of an example system.

[0006] Figure 3 It is possible to Figure 1 Schematic diagram showing an overlay of systems used together.

[0007] Figure 4 Shows that it can be Figure 1 An example graphical user interface for use with the system.

[0008] Figure 5 Yes, you can Figure 1 A perspective view of the video toothbrush used in the system.

[0009] Figure 6 yes Figure 5 The video shows a perspective view of the toothbrush where a portion of the housing is cut away to show the internal components.

[0010] Figure 7 yes Figure 1 A perspective view of a portion of a system including an example video toothbrush.

[0011] Figure 8 yes Figure 7 Video of a side view of a toothbrush where a portion of the housing is shown as transparent to show the internal components.

[0012] Fig. 9 is a schematic diagram of a light pipe that can be used to transmit ultraviolet light in a video toothbrush.

[0013] Fig.10 is a flow chart illustrating an example method for enhancing a video from a dental tool.

[0014] Fig.11 is a flow chart illustrating an example method for enhancing a video from a dental tool.

[0015] Fig.12 It is shown in Figure 1 A flow chart of an example method for operating an ultraviolet lamp in a system. DETAILED DESCRIPTION

[0016] The inventors have recognized that it may be beneficial to enhance the display of a video from a medical tool to provide information about the patient's teeth to the patient and / or healthcare provider. For example, a video from a dental tool may be enhanced according to the present disclosure. Such a video dental tool is shown and described in U.S. application Ser. No. 14 / 645,145, which is incorporated herein by reference and is disclosed as U.S. Patent Application Publication No. 2015 / 0257636. For ease of description, the remainder of the present disclosure will relate to a video toothbrush, but the present disclosure is not limited to such an embodiment. In addition, the remainder of the present disclosure will relate to the condition of the patient's teeth, but it should be understood that other conditions in the user's mouth (e.g., oral ulcers) and medical conditions associated with other parts of the body are also within the scope and spirit of the present disclosure.

[0017] Figure 1 1 is a schematic diagram of a system 100 including a video toothbrush 102 and a mobile computing device 104. The video toothbrush 102 may include a camera 106, an ultraviolet (UV) lamp 108, and a brush 110 or other working implement. The mobile computing device may include a processor 112 and a non-transitory computer-readable memory or other medium 114, which stores instructions that, when executed by the processor 112, cause the mobile computing device 104 to perform one or more steps, operations, methods, processes, algorithms, etc. of the present disclosure. The mobile computing device may also include a display 116 and one or more functional modules 118, 120, which may be implemented as hardware and / or software (e.g., instructions in the memory 114).

[0018] The video toothbrush 102 may be used by a user to brush their teeth with a brush 110 while capturing video with a camera 106. The video toothbrush 102 may be in electronic communication with a mobile computing device 104 and may send the captured video feed to the mobile computing device 104. The mobile computing device 104 may in turn display the video feed in substantially real time on a display 116. As will be described below, the mobile computing device 104 may also analyze and / or enhance the video stream.

[0019] The functional modules 118, 120 may include an image analysis module 118 that may receive a video stream and analyze the video to determine if one or more conditions exist. For example, the image analysis module 118 may include a plaque quantity submodule 122 that may determine the presence and location of plaque on the teeth, a gum color submodule 124 that may determine the color and color changes of the user's gums at one or more locations, a tooth movement submodule 126 that may determine the location and relative movement of the teeth, a tonsil space submodule 128 that may determine the size and size changes of the user's upper airway, and / or other submodules for other medical related or information determinations.

[0020] The plaque quantity submodule 122 can analyze the images in the real-time video feed to determine the amount of plaque and the location of the plaque. For example, the plaque quantity submodule 122 can process the images after UV exposure and analyze those images, such as reference Fig.10 , Fig.11 and Fig.12 Described.

[0021] The gum color submodule 124 can obtain multiple sample images of the user's mouth from a given video source and determine an average gum tissue color associated with the video segment (e.g., a brushing segment). The gum color submodule 124 can store the average gum color and can compare the average gum tissue color to one or more previously stored gum tissue colors of the user. If the gum color submodule 124 determines through the comparison that the change in the user's gum tissue color (e.g., deviation from a previous color) has exceeded a threshold amount, the gum color submodule 124 can send a notification to the user that the user's gum color has changed and / or that the user may be at risk for gum disease.

[0022] The tooth movement submodule 126 can determine and store the position of each of the user's teeth during the brushing session. For example, the tooth position can be determined relative to other teeth. The tooth movement submodule 126 can compare the position of each tooth with the previously stored positions of the user's teeth. If the tooth movement submodule 126 determines through the comparison that one or more of the user's teeth have moved more than a threshold amount, the tooth movement submodule 126 can send a notification to the user that one or more of the user's teeth have moved.

[0023] The tonsil space submodule 128 can determine the contour of the user's upper airway (including the size and protrusion of the user's tonsils) based on the video captured by the video toothbrush 102. The contour of the user's upper airway can be compared to known contour features that indicate the onset of sleep apnea or other breathing problems (e.g., too large tonsils, too small airway spaces). If the tonsil space submodule 128 determines through the comparison that the user's upper airway contour is consistent with a potential breathing problem, the tonsil space submodule 128 can send a notification to the user that the user's upper airway is consistent with a potential breathing problem.

[0024] The functional modules 118, 120 may also include an information overlay module 120 that may overlay one or more information items, graphics, etc. to enhance the video source displayed on the display 116 based on the determination of the image analysis module 118. For example, the information overlay module 120 may add one or more icons to one or more corresponding locations on the user's teeth where plaque is determined to be present, teeth that are determined to have moved, gums that have become discolored, etc.

[0025] The mobile computing device 104 may also be in electronic communication with an external data storage 130, such as a cloud storage. For example, the external data storage 130 may be associated with an application executing on the mobile computing device 104 that may include the modules 118, 120 and may output a video source on the display 116. The mobile computing device 104 may send the video source (in raw and / or enhanced form), determinations performed by the image analysis module 118, and the like to the storage 130, and may retrieve the video source (in raw and / or enhanced form), determinations performed by the image analysis module 118, and the like from the storage 130. Additionally or alternatively, the mobile computing device 104 may store the video source, determinations, and / or other information in the memory 114.

[0026] The UV light 108 of the video toothbrush 102 can be used to identify dental plaque on the user's teeth. For example, the user can activate the UV light 108, cause the UV light to illuminate one or more portions of the user's mouth, and then deactivate the UV light 108. In some embodiments, the video toothbrush 102 can automatically deactivate the UV light 108 after a predetermined period of time or in response to a determination based on video analysis that the user's mouth is sufficiently exposed to UV light. Because dental plaque is easily identifiable in a video after UV exposure, the image analysis module 118 can identify dental plaque that has been exposed to UV light. In some embodiments, the video toothbrush may also include a fixed or retractable filter on the camera 106 that can perform a color correction function to identify clean teeth as white and dental plaque as red after the plaque is exposed to UV light.

[0027] In an embodiment, the UV lamp 108 can emit light in the entire UV spectrum. For example, the UV lamp 108 can be a UV curing lamp that outputs light in the range of 385nm to 490nm. In some embodiments, the UV lamp 108 can emit light at a frequency centered on a frequency (e.g., 405nm) that is particularly useful for plaque irradiation and identification.

[0028] In addition to or in lieu of local execution of the modules 118, 120 on the mobile computing device 104, a backend processing source 132 (e.g., a cloud service) may perform one or more aspects of the image analysis module and / or the information overlay module. Thus, the mobile computing device 104 may be in electronic communication with the backend processing device 132 to, for example, transfer original and / or enhanced video between the backend processing device 132 and the mobile computing device 104, transfer determinations based on image analysis between the backend processing device 132 and the mobile computing device 104, and the like.

[0029] The mobile computing device 104, data storage 130, and / or backend processing 132 may also be in electronic communication with one or more care providers 134. For example, the mobile computing device and / or backend processing 132 may send a determination based on the image analysis to a care provider (e.g., a dentist or other dental health professional) with appropriate permission from the user. Alternatively or additionally, any notification disclosed herein to be sent to the user may be sent to the care provider 134. Furthermore, the care provider may access the stored video or determination of the user in the storage 130 with appropriate permission from the user, send comments to the user based on the care provider's review of the video and / or determination, etc.

[0030] In some embodiments, the video toothbrush 102 can electronically communicate with one or both of the backend processing device 132 and the data storage device 130 independently of the mobile computing device 104. Thus, in addition to or in lieu of similar communications with the mobile computing device 104, the video toothbrush 102 can send video to the backend processing device 132 and / or the data storage device 130 and / or receive instructions from the backend processing device and / or the data storage device.

[0031] The video toothbrush 102 can record video using a camera 106 included in or mounted on the toothbrush 102, and the video can be output to one or more displays or storage devices (e.g., external storage 130 or memory 114). In an embodiment, the video from the toothbrush can be output to and displayed on a mobile computing device 104, which can be a smart phone, tablet computer, or other device. Additionally or alternatively, the video can be output to and displayed on a user wearable device (e.g., a smart watch, smart glasses, and the like). Additionally or alternatively, the video can be output to and displayed on a holographic projector. Therefore, the output of the video from the toothbrush on the display can be approximately real-time, or can be a considerable time after the video is recorded (e.g., a video that is replayed or stored).

[0032] In an embodiment, the video toothbrush 102 can be used by a consumer as part of their daily hygiene routine. Additionally or alternatively, the video toothbrush 102 can be used by a dental professional as part of treating a patient.

[0033] In an embodiment, the video captured by the video toothbrush 102 can be viewed by a user of the toothbrush 102 (e.g., on a computing device 104 or wearable device of the consumer when the toothbrush is used by a consumer, or on a display associated with a dental professional when the toothbrush is used by a dental professional). Additionally or alternatively, the video can be viewed by a third party that has received the video or permission to view the video from the capturer. For example, such a third party can be a dental professional that received the video from a patient.

[0034] Various information may be used to enhance the display of the video captured by the video toothbrush 102. For example, such information may be superimposed on the video or presented near the video. A number of example modes of enhancement are discussed in turn below. The various modes will be described with reference to the mouth of a "patient". It should be understood that this is for ease of description only and that the various modes may be used by dental or medical professionals or consumers. In an embodiment, these modes may be implemented by functional modules 118, 120.

[0035] In one or more modes of use, the video can be enhanced by superimposing information about the patient's dental condition. For example, known fillings, crowns, implants, etc. can be color-coded in the video, labeled with text, or identified in some additional or alternative manner. In an embodiment, such information can be based on clinical information provided by a medical care provider, input from a consumer / patient, or another source outside the video. Additionally or alternatively, such information can be determined by analyzing the video itself. For example, the video can be analyzed by, for example, a computer processor of a user's mobile electronic device to identify and mark the patient's teeth and the conditions associated with those teeth. Similar to all video analysis in the present disclosure, this video analysis can be performed by the processing power of the mobile computing device 104, the video toothbrush 102, and / or by the back-end processing device 134.

[0036] In one or more use modes, the video captured by the video toothbrush 102 can be used as the basis for creating a model of the patient's teeth or other body parts. For example, the video captured by the toothbrush 102 can be used to create a partial or complete three-dimensional model of the patient's teeth. Therefore, the video can be analyzed to create a model. In any of the use modes described below, based on the field of view and position of the camera, the video of the patient's teeth can be superimposed on such a model, displayed next to the model, or the model can be displayed instead of the actual video source. Therefore, although the use mode will be described below with reference to displaying information with or on the video, such information can be displayed on or with the model in addition to or in lieu of the model of the patient's teeth, one or more icons or other graphical features can be superimposed on the actual image of the user's teeth, as will be described below.

[0037] In "live mode," the video may be supplemented to show a wider field of view than is actually captured by camera 106. For example, in embodiments where the live video typically displays three or four teeth based on the position of camera 106, the field of view of camera 106, etc., the display of the video may augment the teeth actually captured in the video with a representation of the surrounding teeth in the patient's mouth (e.g., a model of the teeth themselves as seen by the user). Icons and other image enhancements in accordance with the present disclosure may be superimposed or added to such models as well as the teeth within the field of view of camera 106. Additionally or alternatively, "live mode" may enhance the display, for example, by enabling a user to zoom in on a particular portion of the video frame by keeping the camera lens in a single position for a given period of time. Additionally or alternatively, "live mode" may zoom in by default to provide an enlarged view of the patient's teeth.

[0038] In "x-ray mode", the video may be enhanced by superimposing an x-ray of the user's teeth on the video or displaying the x-ray with the video. In embodiments, the x-ray may be a 2D or 3D x-ray. In other embodiments, images from other imaging modalities may be superimposed or displayed with the video. Icons and other image enhancements in accordance with the present disclosure may be superimposed or added to such x-ray images as well as the actual video feed from the camera 106.

[0039] In "movement mode", a toothbrush (or other instrument) can be used to apply pressure to teeth or other body parts, and a video can be displayed to record the movement of the teeth (e.g., by showing the "original" and "moved" positions of the teeth). Additionally or alternatively, a video can be displayed to record the movement of one or more teeth relative to a previously captured video (e.g., as described above with reference to the tooth movement submodule 126).

[0040] In "plaque mode," the video may be superimposed with information about the plaque on the patient's teeth. For example, the plaque may be color-coded on the video to provide the user with an indication of specific areas to clean with the brush. In an embodiment, "plaque mode" may include the use of a UV light on the toothbrush to illustrate the plaque. Figure 3 as well as Figures 9 to 12 Various aspects of a "plaque mode" are shown, in which a "monster" or other graphic is superimposed on an area of ​​teeth having plaque, so that a user can "attack" the monster by brushing the plaque-covered area. In response to plaque being cleared from the teeth, the superimposed monster or other graphic can be removed.

[0041] In "clinical mode," the video may be superimposed or displayed with information useful for performing clinical procedures. For example, locations to be drilled, filled, cut, lasered, or otherwise treated may be color-coded or otherwise annotated for the clinician's review.

[0042] The video can also be used to determine the patient's ability to treat a condition at home, or successful attempts to treat a condition at home. For example, in the event that the patient is unable to adequately clean dental plaque at home or treat another condition identified in the video, the video can be transmitted to a care provider (e.g., the patient's dentist) to obtain assistance with the condition (e.g., explanation of the condition, treatment recommendations, scheduling an appointment, etc.).

[0043] The video and marking information may be used to recommend further treatment. For example, where a patient's teeth have changed such that a negative condition (e.g., incipient tooth decay, excessive tooth wear, etc.) appears to be developing, recommendations may be automatically provided to the patient (e.g., recommendations to schedule fluoride treatments, recommendations to brush specific areas more carefully, etc.). For example, such recommendations may be determined by a processor of the patient's mobile computing device.

[0044] Figure 2 is a schematic diagram of the use of system 100. Figure 2 As shown, the toothbrush 102 can capture a video of the user's mouth (including the user's teeth) as the user brushes their teeth. The toothbrush 102 can stream the real-time video to the user's mobile computing device 104, which can display the real-time video of the user's mouth for the user to watch while brushing their teeth. Figures 1 to 3 , the image analysis and information overlay modules 118, 120 can determine the amount of plaque or other information on the user's teeth and overlay one or more icons 302 on the user's teeth based on these determinations. For example, in the event that the user's teeth have an amount of plaque above a threshold, the icon 302 can be overlaid in the real-time video. The icon 302 can be animated or otherwise changed based on the user's brushing interaction with the icon 302, as shown in FIG. Fig.10 and Fig.11 In addition, the icon can be set based on the amount of plaque on the user's teeth. In a first example, the amount of plaque on the user's teeth can be determined based on an image captured by the video toothbrush 102 after the user's teeth are exposed to UV light, which can also be emitted by the video toothbrush 102.

[0045] Figure 4 An example user interface portion 400 is shown that can be presented on a user's mobile computing device, for example, in a portion of an application, which also displays a real-time video feed from a toothbrush 102 in another portion. The user interface portion 400 may include a plurality of user-selectable menu items 402, including a calendar menu item, a progress menu item, an honors menu item, a control panel menu item, a real-time display menu item, and a learning menu item. The user interface portion enables the user to select from the various modes described above. The user interface portion 400 may report the user's participation in one or more activities in a report portion 404, such as a daily brushing streak and / or a score for a specific time period. Both the score and the brushing streak may be determined by the application based on the user's interaction with the icon. For example, a user may increase the streak by one day by performing a brushing segment, in which all icons are brushed off based on the removal of associated plaque. In another example, a user may increase the streak by one day by brushing twice in a day and causing a certain number of icons to be brushed off. Other information discussed herein (e.g., notifications based on analysis of a video of the user's mouth) may also be presented to the user in the interface portion 400. Additionally or alternatively, such notification may be sent to the user via email, text message, or the like.

[0046] Figures 5 to 8Various example features of the video toothbrush are shown. First, Figure 5 and Figure 6 A dental instrument 500 (which can be used as a video toothbrush 102) is shown, which includes a body 502 and a dental instrument insert 504, which extends from the end of the body 502 for use in the patient's oral cavity. A camera lens 506 can be disposed adjacent to the dental instrument insert 504. The camera lens 506 enables recording of images or videos, and can preferably have a line of sight substantially parallel to the dental instrument insert 504 extending from the body 502 of the dental instrument 500, and can aim at the working tip of the dental instrument insert 504. However, the camera lens 506 can also have a relatively wide angle to record or show a relatively large field of view. Therefore, the dental instrument 500 can display an image and / or video of the working tip of the dental instrument insert 504, particularly during use of the working tip of the dental instrument insert in the patient's oral cavity. In a preferred embodiment, the dental instrument 500 can have a plurality of lights 508 (e.g., a plurality of LED lights), which can surround the lens 506 and provide sufficient light source for recording and / or viewing of images and / or videos. In another embodiment, the light may be ultraviolet light or a combination of visible and ultraviolet light, which may help a user of the dental appliance 500 observe dental plaque on a patient's teeth or be used to cure epoxy or the like in a patient's mouth.

[0047] Figure 6 A cut-away side view of a dental instrument 500 according to the present invention is shown. Specifically, the dental instrument 500 may include the aforementioned dental instrument insert 504, which is preferably removable from the body 502 so as to be replaceable with other dental instrument inserts (e.g., a pick tool, a dental flosser, dental floss, etc.). The dental instrument insert 504 may be disposed within a hole in the body 502, and when the dental instrument insert is disposed within the hole in the body, the dental instrument insert may be mechanically connected to a motor and transmission 510, which may be electrically connected to a battery / power supply 512 within the body 502. The dental instrument 500 may be powered via the battery / power supply 512, and may preferably be recharged via a charging element 514, which may be electrically coupled to a power source for charging the battery / power supply 512 as needed.

[0048] The lens 506 and light 508 may be disposed on the same end of the body as the dental instrument insert 504, such that the camera lens 506 and light 508 may facilitate recording and / or viewing of the dental instrument insert 504 when used within the patient's mouth. A main PCB 516 may be coupled to the lens 506 and light 508, which may control the recording and viewing of images and / or videos through the lens 506, and may further control the light 508. The main PCB 516 enables a user to turn the lens 506 on or off for recording and / or viewing of images and / or videos, or enables a user to turn the light 508 on or off as desired.

[0049] Thus, when in use, a user may insert the dental instrument insert 504 into the hole of the body 502 of the dental instrument 500. The dental instrument 500 may be pre-wired or preferably wirelessly coupled to a display device (not shown) to view and / or record images and / or video via the lens 506. The user may have the ability to control the opening or closing of the lens 506 and / or light 508 via the display device (not shown), via the body 502, or via any other device or mechanism.

[0050] Figure 7 A system 700 is shown that includes a dental tool 702 (which may be used as a video toothbrush 102), a plurality of dental tool instrument tool inserts 704, a base 706, and a display device (eg, a user mobile computing device) 104 for displaying a video thereon.

[0051] The base 706 can hold the dental tool 702, the plurality of tool inserts 704, and can also hold the display device 104. Specifically, the base 706 can be used as a charging stand for holding the dental tool device 702 and / or the display device 104, and / or charging the dental tool device and / or the display device. More specifically, the base 706 can include a plurality of holes or stands for holding the dental tool 702 and the plurality of dental tool inserts 704.

[0052] The display device 104 can rest on or within the platform 710. The platform 710 can be angled so that a user and / or patient of the dental tool apparatus 702 can see the display device 104. The platform 710 can be adjustable to tilt or otherwise reposition the display device 104 for better viewing on the display device, and the platform can have a frame, flange, or other similar retainer for holding the display device 104 thereon or therein.

[0053] The display device 104 may preferably be a smart phone (eg Phones, tablets (eg ) or other similar display device that enables wireless transmission of streaming video from the dental tool 702, as described in more detail below. Preferably, the display device 104 can be connected to the dental tool 702 via any means obvious to those skilled in the art (e.g., via WiFi, 3G cellular telephone network, or other similar data transmission protocols) and receiving wireless streaming video from the dental tool 702 via any video codec (e.g., MPEG4, M-JPEG, or other similar video codec).

[0054] The plurality of dental tool inserts 704 may be any dental tool instrument that may be used by a dental practitioner to clean, repair, or otherwise care for a patient's oral cavity. Common dental tool instruments may include brushes, scalers, mirrors, probes, syringes, drills, drill bits, picks, polishers, lifters, tweezers, curettes, and any other similar instrument that may be usefully employed from the dental tool instrument 702. In addition, it should be noted that the present invention enables the use of multiple instruments, such as a dental mirror and scaler, at the same time to facilitate the use of the scaler.

[0055] Figure 8 A cutaway perspective view of a dental tool 702 is shown. The dental tool 702 can generally include a housing 720, which can contain a camera 722 having a lens 724 for taking video and / or still photos, one or more processing boards 726, a rechargeable battery 728, a power cord 730, a charging input device 732, and a dental instrument insertion hole 734 for holding one or more dental instrument inserts 704. One or more light sources (not shown) can also be provided to direct illumination to the dental instrument insert 704 and / or the patient's oral cavity. The housing 720 can be sealed and made of a material that is resistant to humid environments (e.g., metal or plastic, as is apparent to one skilled in the art).

[0056] In use, a user (e.g., a dental practitioner) can insert the dental instrument insert 704 into the dental instrument hole 734, which can, for example, securely hold the dental instrument therein via a clamping device, a friction resistance device, etc. The camera lens 724 is arranged on the end of the housing 720 and oriented toward the tip of the dental instrument insert 704 to provide a relatively wide viewing cone 742 for viewing the patient's oral cavity and the dental tools used in the patient's oral cavity. Specifically, the lens 724 can be arranged adjacent to the axis of the dental instrument insert 704 and can provide the camera with a line of sight 740 that is substantially parallel to the axis of the dental instrument insert 704. Therefore, the dental instrument insert 704 (particularly the working tip of the dental instrument insert 704) can be easily seen by the user and / or patient of the dental tool 702 via the camera 722. In a preferred embodiment, the lens 724 can be directly aligned with the working tip of the dental instrument insert 704 to specifically and accurately observe the working tip of the dental instrument insert 704. One or more light sources (eg, white LEDs, etc.) may also be used to illuminate the dental instrument insert 704 and / or the patient's oral cavity.

[0057] In a preferred embodiment, a dental mirror can be used to enable the camera 722 to record video and / or still photographs of an area within the patient's mouth that is, for example, 90 degrees or any other angle to the line of sight 740 of the camera 722. In addition, the illumination source emitted from the dental tool 702 can also be reflected from the dental mirror to help illuminate the patient's mouth.

[0058] In some embodiments, for example, the video toothbrush 102 may include a UV light for illuminating the user's mouth to show plaque on the user's teeth. Fig. 9 An example UV light fixture 900 is shown for including a UV light in a video toothbrush 102 in accordance with the present disclosure. The UV light fixture 900 includes a UV surface mount device (SMD) LED 902 or other LED capable of emitting light at UV frequencies, and a light pipe 904 that transmits output light from the LED 902 to a desired output position and angle on the video toothbrush. For example, the UV light output may be close to a camera lens (e.g., Figure 5 Alternatively, the UV LED may be mounted with its output adjacent to the lens to form an appropriate output angle and pattern, and white light or other visible light LEDs may also be arranged in the same video toothbrush 102 so that the video toothbrush 102 can selectively output visible light and / or UV.

[0059] Fig.101 is a flow chart illustrating an example method 1000 of operating a video toothbrush system. In embodiments, the method 1000 or one or more aspects of the method 1000 may be performed by one or more components of the system 100. For example, one or more of the operations of the method 1000 may be performed by one or more of the video toothbrush 102, the user mobile computing device 104, and the backend processing device 132.

[0060] Method 1000 may include receiving a video of teeth captured by a dental tool having a camera and a cleaning appliance in operation 1002. The cleaning appliance may be located within the field of view of the camera. The video may be received by the dental tool itself (e.g., via the camera of the dental tool) and / or may be received by a user mobile computing device, a backend processing device, or other processing source.

[0061] Method 1000 may also include outputting the received video to a display in real time in operation 1004. The received video may be output, for example, on a display of a user's mobile computing device to a projector, a pair of VR glasses, etc., or to any other suitable display device. In some embodiments, the real-time output may continue while performing other operations described below.

[0062] The method 1000 may also include analyzing the received video to determine the amount of dental plaque at multiple locations in operation 1006. When illuminated with a narrow blue-violet light (e.g., centered at a wavelength of 405 nm), dental plaque can be identified by light-induced fluorescence (LIF) technology, which is based on the red fluorescent properties of porphyrins (metabolites of foreign bacteria within dental plaque).

[0063] In some embodiments, operation 1006 may include a multi-stage process for identifying dental plaque in an image. First, object detection may be performed on each image frame to determine whether the input image is an oral cavity image and locate the oral cavity region. Second, instance segmentation may be performed to extract the dental plaque region.

[0064] In object detection, a ground-truth bounding box technique may be applied to outline a region of interest that includes only teeth and gums that can be identified for each image, and then a more fine-tuned bounding box detector may be applied to detect the RoI (predicted bounding box) within the image.

[0065] In instance segmentation, in order to finely extract plaque areas (e.g., multiple areas) that emit red fluorescence from an oral image, a machine learning model can be applied to identify red fluorescence on a single tooth and red fluorescence across multiple teeth or between teeth to determine the location of the corresponding plaque on the tooth. The model can be or can include, for example, a recurrent neural network trained on a set of red fluorescence images. Details of the example model are provided in the following literature: Kim et al., "Light-Induced Fluorescence-Based Device and Hybrid Mobile App for OralHygiene Management at Home: Development and Usability Study", JMIR MhealthUhealth, October 16, 2020; 8(10): e17881. doi: 10.2196 / 17881. PMID: 33064097; PMCID: PMC7600004.

[0066] Operation 1006 may include identifying the width and height of areas of plaque on individual teeth and between teeth, as well as areas of plaque that span multiple teeth. Thus, operation 1006 may include determining the size and approximate shape of each plaque area.

[0067] Method 1000 may also include determining in operation 1008 that there is more than a threshold amount of dental plaque at one or more unclean locations on the user's teeth. In some embodiments, a certain size of a particular area of ​​dental plaque can be used as a threshold (e.g., a plaque area below this size is considered "clean" or not marked for further cleaning). Such a size threshold can be a square area, width, and / or height. Additionally or alternatively, a threshold amount of dental plaque can be determined based on the total fluorescence associated with the dental plaque area. Such a total fluorescence can be, for example, the cumulative intensity of red pixels in the dental plaque area. More generally, the threshold can be based on the size of the dental plaque area or the total intensity in an image of the dental plaque area. Plaque areas that exceed a threshold can be considered "unclean."

[0068] Method 1000 may also include superimposing an icon at each dirty location in operation 1010. The icon may be an appropriate image to draw the user's attention to the dental plaque area and motivate the user to brush the dental plaque area. For example, the icon may be a monster (such as Figure 3The icons may be superimposed on the live video frame at locations determined to be unclean. The icons may be superimposed so that when the user moves the camera and changes the field of view and associated video, the icons may move with the field of view or viewing frame so that the icons remain displayed and superimposed on the unclean locations on the user's teeth.

[0069] Method 1000 may also include determining that plaque has been removed from the dirty location and removing the icon in operation 1012. Operation 1012 may include the same or similar process as operation 1006, and determining that the amount of plaque at the location previously determined as "dirty" in operation 1008 (at the first time) is now (at the second time) clean, i.e., the amount of plaque no longer exceeds the threshold applied in operation 1008. In response, the icon may be removed from the video view.

[0070] In some embodiments, operation 1012 may include changing an icon when the user brushes at or near a location where the icon is superimposed. For example, when an associated dirty location is brushed, the icon may be animated as visual feedback to the user that the correct location is being brushed. In another example, the size, brightness, or other characteristics of the icon may be changed to indicate a decrease in the amount of plaque or the presence of a toothbrush at a plaque location.

[0071] As the user brushes (during which time the user both moves the camera and removes plaque), operations 1002, 1004, 1006, 1008, 1010, and 1012 may be performed continuously for each frame in the live video feed or a predetermined subset of frames (e.g., one frame per second, two frames per second, etc.) such that the plaque locations are continuously marked with icons, and those icons change or are removed as the user brushes to provide the user with real-time feedback on their brushing progress.

[0072] In some embodiments, method 1000 may additionally include analyzing the video to determine gum color deviations, potential upper airway contours indicative of breathing problems, tooth movement, and / or other physiological conditions that can be determined from the video, as discussed above. Where such analysis indicates a potential problem, method 1000 may also include sending a notification to the user, such as regarding Figure 1 Discussed. In addition, in some embodiments, method 1000 may also include receiving contact information of a care provider (e.g., from a user or from a care provider), receiving permission from the user to share diagnostic information with the care provider, and sharing one or more analyses, notifications, images, videos, conclusions, etc. of the present disclosure with the care provider.

[0073] Fig.111 is a flow chart illustrating an example method 1100 for operating an augmented reality output. In embodiments, the method 1100 or one or more aspects of the method 1100 may be performed by one or more components of the system 100. For example, one or more of the operations of the method 1100 may be performed by one or more of the video toothbrush 102, the user mobile computing device 104, and the backend processing device 132.

[0074] Method 1100 may include various unclean situations and scenarios in which the superimposed icons may be animated to draw the user's attention to the icons and cause the user to brush the dental plaque associated with the icons. In accordance with the present disclosure, in addition to the example reasons and scenarios given in method 1100, icons may be animated in many other situations.

[0075] Method 1100 may include determining that a predetermined time period has passed since brushing began, i.e., a predetermined time has passed, in operation 1102. The time period may be, for example, one minute, ninety seconds, or two minutes since brushing began for the entire mouth. Additionally or alternatively, the time period may be 15 seconds, 20 seconds, or 30 seconds since the user began brushing one quarter of their mouth.

[0076] Method 1100 may also include determining, in operation 1104, that the brush has passed over a dirty location without adequately cleaning the dirty location, i.e., without adequately removing dental plaque. Operation 1104 may include, for example, determining that the amount of dental plaque (e.g., the amount of red fluorescence) has decreased, but has not decreased below a threshold for classifying the location as "clean." Additionally or alternatively, operation 1104 may include determining that the brush has just landed at a location where dental plaque is known to exist (previously annotated with an icon), but the amount of dental plaque remains above a threshold to classify the location as "dirty."

[0077] Method 1100 may also include animating the icon in response to operation 1102 or operation 1104 (i.e., in response to determining that the unclean condition has been met) in operation 1106. Animating the icon may include, for example, moving the icon (e.g., rotating), turning the icon into a multi-frame "video" icon so that the icon appears to be dancing, waving, swinging, etc., pulsating the icon, changing the icon size or shape, etc. In short, operation 1106 may include changing the appearance of the icon in response to operation 1102 or 1104. Therefore, when the dental plaque is removed to a smaller amount, the icon can change in real time (e.g., the monster performs a death flip, is injured, listless and emits a death scream, etc.). If all the dental plaque is not removed, the visual representation can be changed to a smaller monster or other representation. Incipient decay or decalcification of the enamel may occur under the dental plaque. The possible decay area can be marked as something else (igloo, stop sign, etc.) for inspection by a dentist.

[0078] Fig.12 1 is a flow chart illustrating an example method 1200 of exposing a user's teeth to UV light to assess the presence of dental plaque. In embodiments, the method 1200 or one or more aspects of the method 1200 may be performed by one or more components of the system 100. For example, one or more of the operations of the method 1200 may be performed by one or more of the video toothbrush 102, the user mobile computing device 104, and the backend processing device 132.

[0079] Method 1200 may include instructing a user to expose their teeth to UV light in operation 1202. The instruction may be made via a display (e.g., a display of a user's mobile computing device). Alternatively, the instruction may be made by the video toothbrush, for example, by outputting a sound, recording a verbal output instruction, outputting a light pattern, etc. A UV light may be provided on the video toothbrush so that the video toothbrush can capture a video of the user's mouth while the user's mouth is exposed to UV light.

[0080] Method 1200 may also include, in operation 1202, analyzing the output video from the video toothbrush when the user exposes their teeth to UV light. Operation 1202 may include, for example, analyzing the exposure of multiple surfaces within the user's mouth to determine whether UV light has been applied to one or more teeth. In some embodiments, operation 1202 may include evaluating the UV exposure of teeth individually and adding or removing labels on teeth when those teeth are sufficiently exposed to UV light. Sufficient exposure of teeth can be determined by, for example, determining the exposure duration of the teeth based on an image captured by a camera that includes UV light exposure, based on the duration of the teeth in a portion of a video frame associated with an emission field of UV light, and the like.

[0081] The method 1200 may also include determining that the UV exposure is sufficient in operation 1204, and automatically deactivating the UV light in response. Automatic deactivation may be used as a safety measure to prevent the user from intentionally or accidentally overexposing to UV light. For example, the UV exposure may be determined to be sufficient when each tooth has been sufficiently exposed, when a certain percentage of teeth have been sufficiently exposed, or when the total exposure time has reached a threshold.

[0082] In some embodiments, methods 1000, 1100, and 1200 may be used in combination. For example, method 1200 may be performed to expose a user's teeth to UV light to enable plaque identification, and then method 1000 may be performed to allow the user to clean the plaque. During the execution of method 1000, method 1100 may also be performed to add further detail and focus to the augmented reality display.

[0083] In a first aspect of the present disclosure, a method for enhancing a video from a dental tool is provided. The method includes: receiving a video of a plurality of teeth, the video being captured by a dental tool including a camera and a cleaning device; causing the video to be outputted substantially in real time to a display visible to a user of the dental tool; analyzing the video to determine the amount of dental plaque at a plurality of locations on the plurality of teeth; determining at a first time that the amount of dental plaque exceeds a threshold for an unclean location among the plurality of locations; superimposing a corresponding icon at the unclean location in the video, so that when the unclean location is visible to the user in the substantially real-time output, the corresponding icon is visible to the user; and determining at a second time that the amount of dental plaque at the unclean location no longer exceeds the threshold, and in response removing the icon from the superimposed video.

[0084] In an embodiment of the first aspect, the method further comprises determining that a dirty condition of the dirty position has been met, and in response, animating an icon associated with the dirty position. In another embodiment, the dirty condition comprises a predetermined time having passed after the first time, or a working tool of the dental tool has passed over the dirty position without adequately removing dental plaque.

[0085] In an embodiment of the first aspect, analyzing the video to determine an amount of dental plaque at a plurality of locations on a plurality of teeth comprises detecting red fluorescence at each of the plurality of locations.

[0086] In an embodiment of the first aspect, the icon is a first icon, and removing the first icon from the superimposed video includes replacing the first icon with the second icon.

[0087] In an embodiment of the first aspect, the video is analyzed to determine that a working instrument of the dental tool is in a dirty position, and in response the icon is animated.

[0088] In an embodiment of the first aspect, the method further comprises outputting instructions for a user to expose the user's teeth to a UV light of the dental tool.

[0089] In a second aspect of the present disclosure, a dental tool system is provided, which includes a dental tool having a camera, a UV lamp, and a working tool. The system also includes a non-transitory computer-readable medium storing the following instructions, which, when executed by a processor of a computing system, causes the computing system to perform operations including the following: instructing a user via a display to irradiate the user's teeth with a UV lamp; receiving a video of a plurality of teeth, the video being captured by a dental tool camera; causing the video to be outputted to a display visible to a user of the dental tool in substantially real time; analyzing the video to determine the amount of dental plaque at a plurality of locations on a plurality of teeth; determining at a first time that the amount of dental plaque exceeds a threshold for an unclean location among the plurality of locations; superimposing a corresponding icon at the unclean location in the video, so that when the unclean location is visible to the user in the substantially real-time output, the corresponding icon is visible to the user; and determining at a second time that the amount of dental plaque at the unclean location no longer exceeds the threshold, and removing the icon from the superimposed video in response.

[0090] In an embodiment of the second aspect, the operation further comprises determining that a dirty condition of the dirty position has been met, and in response, animating an icon associated with the dirty position. In another embodiment of the second aspect, the dirty condition comprises a predetermined time having passed after the first time, or a working implement of the dental tool has passed over the dirty position without adequately removing dental plaque.

[0091] In an embodiment of the second aspect, analyzing the video to determine an amount of dental plaque at a plurality of locations on a plurality of teeth comprises detecting red fluorescence at each of the plurality of locations.

[0092] In an embodiment of the second aspect, the icon is a first icon, and removing the first icon from the superimposed video includes replacing the first icon with the second icon.

[0093] In an embodiment of the second aspect, the operations further include analyzing the video to determine that a working instrument of the dental tool is in a dirty position, and in response animating the icon.

[0094] In a third aspect of the present disclosure, a method is provided, the method comprising: receiving a video of a user's mouth, the video being captured by a dental tool comprising a camera and a cleaning implement; causing the video to be output in substantially real time to a display visible to a user of the dental tool; analyzing the video to determine an amount of dental plaque at a plurality of first locations on a plurality of teeth in the mouth, and determining a color of gum tissue at a plurality of second locations in the mouth; superimposing corresponding icons in the video based on the determined amount of dental plaque at unclean locations among the plurality of first locations, so that when the unclean locations are visible to the user in the substantially real-time output, the corresponding icons are visible to the user; and determining that a deviation of the color of the gum tissue from a previous color of the user's gum tissue exceeds a threshold amount, and in response outputting an indication of potential gum disease to the user on the display.

[0095] In an embodiment of the third aspect, analyzing the video is also used to determine a respective position of each of a plurality of teeth relative to adjacent teeth, and the method further comprises determining that at least one of the teeth has moved relative to a previous position of the at least one of the teeth, and in response outputting an indication of the moved tooth to a user on a display.

[0096] In an embodiment of the third aspect, analyzing the video is further used to determine a contour of the user's airway, and the method further comprises determining that the contour is indicative of a breathing problem, and in response outputting an indication of the potential breathing problem to the user on a display.

[0097] In an embodiment of the third aspect, the method further comprises outputting instructions to the user to expose the user's teeth and gums to UV light of the dental tool.

[0098] In an embodiment of the third aspect, the method further comprises sending a notification of potential gum disease to a care provider associated with the user.In another embodiment of the third aspect, the method further comprises receiving contact information of the care provider, and receiving permission from the user to share the user's personal medical information with the care provider.

[0099] In an embodiment of the third aspect, analyzing the video to determine an amount of dental plaque at a plurality of locations on a plurality of teeth comprises detecting red fluorescence at each of the plurality of locations.

[0100] Although the present disclosure has described some embodiments, it should be understood that unless explicitly stated in the claims, the claims are not intended to be limited to these embodiments. On the contrary, the present disclosure is intended to cover alternatives, modifications and equivalents that may be included in the spirit and scope of the present disclosure. In addition, in the detailed description of the present disclosure, many specific details are set forth to provide a thorough understanding of the disclosed embodiments. However, it is obvious to those skilled in the art that the system and method according to the present disclosure can be implemented without these specific details. In other examples, well-known methods, processes, components and loops are not described in detail to avoid unnecessary ambiguity in various aspects of the present disclosure.

[0101] Some parts of the detailed description of the present disclosure have been presented according to the process, logic block, processing and other symbolic representations of the operation of the data bit in the computer or digital system memory. These descriptions and representations are the most effective means used by the technicians in the field of data processing to convey the essence of their work to other technicians in the field. Process, logic block, processing, etc. are generally considered to be the self-consistent sequence of steps or instructions that lead to the desired result. Steps are those required physical operations of physical quantities. Usually, although not necessarily, these physical operations are in the form of electrical or magnetic data that can be stored, transmitted, combined, compared and otherwise operated in a computer system or similar electronic computing device. For convenience and with reference to common usage, with reference to various embodiments of the present invention, such data are referred to as bits, values, elements, symbols, characters, terms, numbers or the like.

[0102] However, it should be remembered that these terms should be interpreted as referring to physical operations and physical quantities, and are merely convenient labels that should be further interpreted according to terms commonly used in the art. Unless otherwise specifically stated, as is apparent from the discussion herein, it should be understood that throughout the discussion of this embodiment, discussions utilizing terms such as "determining" or "outputting" or "transmitting" or "recording" or "locating" or "storing" or "displaying" or "receiving" or "identifying" or "utilizing" or "generating" or "providing" or "accessing" or "checking" or "notifying" or "delivering" refer to the actions and processes of a computer system or similar electronic computing device that operates and transforms data. Data is represented as physical (electronic) quantities within the registers and memories of a computer system, and is converted into other data similarly represented as physical quantities within the computer system memories or registers (or other such information storage, transmission or display devices as described herein or understood by those skilled in the art).

Claims

1. A method for enhancing a video from a dental tool, the method comprising: receiving a video of a plurality of teeth, the video being captured by a dental tool including a camera and a cleaning implement; causing the video to be output in substantially real time to a display visible to a user of the dental tool; analyzing the video to determine an amount of dental plaque at a plurality of locations on the plurality of teeth; determining at a first time that the amount of dental plaque exceeds a threshold value for an unclean location among the plurality of locations; superimposing a corresponding icon at the dirty location in the video so that the corresponding icon is visible to the user when the dirty location is visible to the user in the substantially real-time output; as well as At a second time it is determined that the amount of dental plaque at the dirty location no longer exceeds the threshold, and in response the icon is removed from the superimposed video.

2. The method according to claim 1, further comprising: A determination is made that a dirty condition for a dirty location has been met, and in response, an icon associated with the dirty location is animated.

3. The method according to claim 2, wherein: The unclean conditions include: a predetermined time elapses after said first time; or The working tool of the dental tool passes over the dirty location without adequately removing dental plaque.

4. The method according to claim 1, wherein: Analyzing the video to determine an amount of dental plaque at a plurality of locations on the plurality of teeth includes detecting red fluorescence at each of the plurality of locations.

5. The method according to claim 1, wherein: The icon is a first icon; and Removing the first icon from the superimposed video includes replacing the first icon with a second icon.

6. The method according to claim 1, further comprising: The video is analyzed to determine that a working implement of the dental tool is located at the dirty location, and the icon is animated in response.

7. The method of claim 1, further comprising outputting instructions for a user to expose the user's teeth to UV light of the dental tool.

8. A dental tool system comprising: A dental tool, comprising: camera; UV lamps; and work equipment; and A non-transitory computer-readable medium storing instructions that, when executed by a processor of a computing system, cause the computing system to perform operations including: instructing the user via a display to irradiate the user's teeth with the UV lamp; receiving a video of a plurality of teeth, the video being captured by a camera of the dental tool; causing the video to be output in substantially real time to a display visible to a user of the dental tool; analyzing the video to determine an amount of dental plaque at a plurality of locations on the plurality of teeth; determining at a first time that the amount of dental plaque exceeds a threshold value for an unclean location among the plurality of locations; superimposing a corresponding icon at the dirty location in the video so that the corresponding icon is visible to the user when the dirty location is visible to the user in the substantially real-time output; and At a second time it is determined that the amount of dental plaque at the dirty location no longer exceeds the threshold, and in response the icon is removed from the superimposed video.

9. The system according to claim 8, wherein: The operations also include: A determination is made that a dirty condition for a dirty location has been met, and in response, an icon associated with the dirty location is animated.

10. The system according to claim 9, wherein: The unclean conditions include: a predetermined time elapses after said first time; or The working tool of the dental tool passes over the dirty location without adequately removing dental plaque.

11. The system according to claim 8, wherein: Analyzing the video to determine an amount of dental plaque at a plurality of locations on the plurality of teeth includes detecting red fluorescence at each of the plurality of locations.

12. The system according to claim 8, wherein: The icon is a first icon; and Removing the first icon from the superimposed video includes replacing the first icon with a second icon.

13. The system according to claim 8, wherein: The operations also include: The video is analyzed to determine that a working implement of the dental tool is located at the dirty location, and the icon is animated in response.

14. A method comprising: receiving a video of a user's mouth, the video captured by a dental tool including a camera and a cleaning implement; causing the video to be output in substantially real time to a display visible to a user of the dental tool; The video was analyzed to: determining an amount of dental plaque at a plurality of first locations on a plurality of teeth in the mouth; as well as determining the color of gum tissue at a plurality of second locations in the mouth; superimposing a corresponding icon in the video based on the determined amount of dental plaque at a dirty location among the plurality of first locations, such that the corresponding icon is visible to the user when the dirty location is visible to the user in the substantially real-time output; as well as It is determined that the color of the gum tissue deviates from a previous color of the user's gum tissue by more than a threshold amount, and in response an indication of potential gum disease is output to the user on the display.

15. The method according to claim 14, wherein: Analyzing the video is also used to determine the respective position of each of the plurality of teeth relative to adjacent teeth; and The method also includes determining that at least one of the teeth has moved relative to a previous position of the at least one of the teeth, and in response, outputting an indication of the moved tooth to the user on the display.

16. The method according to claim 14, wherein: Analyzing the video is also used to determine the contour of the user's airway; and The method also includes determining that the contour is indicative of a breathing problem, and in response, outputting an indication of a potential breathing problem to the user on the display.

17. The method of claim 14, further comprising outputting instructions for a user to expose the user's teeth and gums to the UV light of the dental tool.

18. The method according to claim 14, further comprising: A notification of potential gum disease is sent to a care provider associated with the user.

19. The method according to claim 18, further comprising: receiving contact information of the care provider; as well as Permission is received from the user to share the user's personal medical information with the care provider.

20. The method according to claim 14, wherein: Analyzing the video to determine an amount of dental plaque at a plurality of locations on the plurality of teeth includes detecting red fluorescence at each of the plurality of locations.

Images