Motion tracking of a dental care appliance

Abstract

A method of tracking a user's dental care activity includes receiving video images of the user's face during a brushing session and identifying predetermined features of the user's face in each of a plurality of frames of the video images. The features include at least two invariant landmarks associated with the user's face and one or more landmarks selected from at least a mouth feature location and an eye feature location. A predetermined marker feature of a dental care implement in use, such as a toothbrush, is identified in each of the plurality of frames of the video images. A measure of inter-landmark distance is determined from the at least two invariant landmarks associated with the user's nose. An implement length normalized by the inter-landmark distance is determined. One or more implement-to-face feature distances each normalized by the inter-landmark distance are determined from the one or more landmarks selected from at least a mouth feature location and an eye feature location. An implement-to-nose angle and one or more implement-to-face feature angles are determined. Each frame is classified as corresponding to one of a plurality of possible regions of teeth being brushed using the determined angles, normalized implement length, and normalized implement-to-face feature distances.

Description

Technical Field

[0001] This disclosure relates to tracking the movement of oral hygiene devices or appliances, such as electric or manual toothbrushes, during oral hygiene routines, which are generally referred to herein as dental care or brushing routines. Background Technology

[0002] The effectiveness of a person's brushing routine can vary significantly depending on many factors, including the duration of brushing in each part of the mouth, the total brushing duration, the extent to which each surface of a single tooth and all areas of the mouth are brushed, and the angle and direction of the back-and-forth motion of the toothbrush. Numerous systems have been developed to track the movement of the toothbrush in the user's mouth to provide feedback on brushing technique and help users achieve an optimal brushing routine.

[0003] Some of these toothbrush tracking systems have the drawback of requiring motion sensors, such as accelerometers, to be built into the toothbrush. Adding such motion sensors to other low-cost and relatively disposable items such as toothbrushes can be expensive, and may also require associated signal transmission hardware and software to transfer data from sensors on or within the toothbrush to appropriate processing and display devices.

[0004] Another problem arises because the posture of a person's head while brushing their teeth affects the contact position of the oral cavity relative to the established toothbrush position. Some toothbrush tracking systems also attempt to track the movement of the user's head in order to better determine the oral cavity areas that may be brushed, but this can also be a complex task involving some form of 3D imaging system to track the position and orientation of the user's head.

[0005] US 2020359777 AA (Dentlytec GPL Ltd.) discloses a dental device tracking method comprising: acquiring at least a first image using an imager of the dental device, the first image including an image of at least one user body part outside the user's oral cavity; identifying the at least one user body part in the first image; and using the at least first image to determine the position of the dental device relative to the at least one user body part.

[0006] CN 110495962 A (HI P Shanghai Household Appliances Co., Ltd., 2019) discloses a smart toothbrush used in a method for monitoring toothbrush position. The method involves acquiring an image including a face and a toothbrush, and using this image to detect the position of the face and establish a face coordinate system. The image including the face and toothbrush is used to detect the position of the toothbrush. The position of the toothbrush in the face coordinate system is analyzed, and a first classification region is determined; toothbrush posture data is obtained, and a second classification region is determined. Based on the image including the face and the image including the face and toothbrush, the first classification region of the toothbrush is obtained; the second classification region of the toothbrush is obtained through a multi-axis sensor and a classifier to determine the position of the toothbrush. This allows for determination of brushing effectiveness, and the effective brushing time in each second classification region is statistically analyzed to guide the user in thoroughly cleaning their oral cavity.

[0007] KR 2015 0113647 A (Rpboprint Ltd.) relates to a toothbrush with a built-in camera and a dental medical examination system using the toothbrush. The toothbrush with the built-in camera takes pictures of the teeth before and after brushing, displays the tooth condition on a display unit, and allows for real-time visual confirmation. The captured dental images are sent to telemedicine support devices in hospitals, etc., enabling remote treatment.

[0008] In a research article published as Marcon M., Sarti A., Tubaro S. (2016) Smart Toothbrushes: Inertial Measurement Sensors Fusion with Visual Tracking. In: Hua G., Jégou H. (editors) Computer Vision – ECCV 2016 Workshops. ECCV 2016. Lecture Notes in Computer Science, Vol. 9914, Springer, Cham., the authors compared two previously known smart toothbrushes to identify their advantages and disadvantages and to examine their accuracy and reliability, with the aim of clarifying how the next generation of smart toothbrushes can be fully utilized for oral care in children, adults, and people with dental conditions.

[0009] It is desirable to be able to track the movement of a toothbrush or other dental care appliance within a user's oral cavity without requiring electronic sensors built into or applied to the toothbrush itself. It is also desirable to be able to track the movement of the toothbrush relative to the user's mouth using relatively conventional video imaging systems, such as those found on ubiquitous smartphones or other widely available consumer devices like computer tablets. It is desirable if the video imaging system used does not need to be a three-dimensional imaging system, such as those using stereoscopic imaging. It is also desirable to provide a toothbrush or other dental care appliance tracking system that can provide real-time feedback to the user based on the areas of the oral cavity that have been brushed or treated during a brushing or dental care period. This invention can achieve one or more of the above objectives. Summary of the Invention

[0010] According to one aspect, the present invention provides a method for tracking a user's dental care activities, comprising:

[0011] - Receive video images of the user's face during dental care sessions;

[0012] - Identify predetermined features of a user’s face in each of multiple frames of a video image, the features including at least two invariant landmarks associated with the user’s face and one or more landmarks selected from at least oral cavity feature locations and eye feature locations;

[0013] - Identify predetermined marker features of the dental care appliance in use in each of the plurality of frames of the video image;

[0014] - Determine a measure of the distance between the landmarks from the at least two invariant landmarks associated with the user's face;

[0015] - Determine the length of the dental care appliance normalized by the distance between the landmarks;

[0016] - Determine one or more device-to-facial feature distances, each normalized to the distance between the said landmarks, from one or more landmarks selected from at least oral cavity feature locations and eye feature locations;

[0017] - Determine the angle of the instrument to the nose and the angle of one or more instruments to facial features;

[0018] - Using the determined angle, the normalized appliance length, and the normalized appliance-to-facial feature distance, each frame is classified into one of a plurality of possible tooth regions treated with the dental care appliance.

[0019] Dental care activities may include brushing teeth. Dental care appliances may include a toothbrush. The at least two invariant landmarks associated with the user's face may include landmarks on the user's nose. The distance between the landmarks may be the length of the user's nose.

[0020] The distance from the one or more devices to the facial features, which is each normalized to the nose length, may include any one or more of the following:

[0021] (i) Distance from the mouth to the instrument normalized to the length of the nose;

[0022] (ii) Distance from the nose to the eye, normalized by the length of the nose;

[0023] (iii) The distance from the nose bridge to the instrument normalized to the nose length;

[0024] (iv) Distance from the nose-length normalized instrument to the left corner of the mouth;

[0025] (v) Distance from the nose-length normalized instrument to the right corner of the mouth;

[0026] (vi) The distance from the nose to the left eye, normalized by the length of the nose;

[0027] (vii) Distance from the nose to the right eye, normalized by the instrument.

[0028] (viii) Distance from the nose-length normalized instrument to the left corner of the eye;

[0029] (ix) Distance from the right corner of the eye to the instrument normalized to the length of the nose.

[0030] The angle of the one or more devices to the facial features may include any one or more of the following:

[0031] (i) Angle from the instrument to the oral cavity;

[0032] (ii) The angle from the instrument to the eye;

[0033] (iii) The angle between the vector from the instrument mark to the bridge of the nose and the vector from the bridge of the nose to the tip of the nose;

[0034] (iv) The angle between the vector from the instrument mark to the left corner of the mouth and the vector from the left corner of the mouth to the right corner of the mouth;

[0035] (v) The angle between the vector from the appliance mark to the right corner of the mouth and the vector from the left corner of the mouth to the right corner of the mouth;

[0036] (vi) The angle between the vector from the instrument mark to the center of the left eye and the vector from the center of the left eye to the center of the right eye;

[0037] (vii) The angle between the vector from the instrument mark to the center of the right eye and the vector from the center of the left eye to the center of the right eye.

[0038] The at least two landmarks associated with the user's nose may include the bridge of the nose and the tip of the nose. The features of the device may include a generally spherical mark attached to or forming part of the device. The spherical mark may have multiple colored segments or quadrants arranged around a longitudinal axis. The segments or quadrants of the mark may be separated from each other by contrasting color bands. The generally spherical mark may be positioned at the end of the device, its longitudinal axis aligned with the longitudinal axis of the device. Identifying predetermined features of the device in use in each of the plurality of frames of the video image may include: determining the position of the generally spherical mark in the frame; cropping the frame to capture the mark; resizing the cropped frame to a predetermined pixel size; using a trained orientation estimator to determine the pitch, roll, and yaw angles of the mark; and using the pitch, roll, and yaw angles to determine the angular relationship between the device and the user's head. Identifying predetermined features of an appliance in use in each of the plurality of frames of the video image may include: identifying bounding box coordinates for each of a plurality of candidate appliance marker detections, each bounding box coordinate having a corresponding detection likelihood score; determining the spatial position of the appliance relative to the user's head based on the detection likelihood score having a value greater than a predetermined threshold and / or the coordinates of the bounding box with the highest score. The method may further include ignoring frames in which the bounding box coordinates are spatially separated from at least one of the predetermined features of the user's face by a value greater than a threshold separation. Candidate appliance marker detection may be determined by a trained convolutional neural network. Determining the appliance length may include determining the distance between the generally spherical marker and one or more landmarks associated with the user's mouth, the distance being normalized by the distance between the landmarks.

[0039] Classifying each frame into one of a plurality of possible tooth regions being processed may further include using the determined angle, the normalized appliance length, and the normalized appliance-to-facial feature distance, and one or more of the following as input to the trained classifier:

[0040] (i) Head pitch angle, roll angle, and yaw angle;

[0041] (ii) Oral landmark coordinates;

[0042] (iii) Eye landmark coordinates;

[0043] (iv) Coordinates of the nose landmark;

[0044] (v) The pitch angle, roll angle, and yaw angle of the instrument derived from the instrument's marking features;

[0045] (vi) Coordinates of the instrument's location;

[0046] (vii) Confidence score for instrument tag detection;

[0047] (viii) Confidence score for the estimated angle of the instrument marking;

[0048] (ix) The sine and cosine values ​​of the angle of the instrument;

[0049] The output of the trained classifier provides the tooth region based on the input.

[0050] Classifying each frame into one of several possible tooth regions being processed may include using any of the following as input to the trained classifier:

[0051] (i) The instrument marks pitch, roll and yaw angles;

[0052] (ii) The instrument marks the sine and cosine values ​​of pitch, roll, and yaw angles;

[0053] (iii) Confidence scores for instrument marking pitch, roll, and yaw angles;

[0054] (iv) Confidence score of instrument marking detection;

[0055] (v) Head pitch angle, roll angle, and yaw angle;

[0056] (vi) The length of the instrument, normalized to the length of the nose, is estimated as the distance between the instrument marker and the coordinates of the oral cavity center;

[0057] (vii) The angle between two vectors and their sine and cosine: one vector from the instrument mark to the bridge of the nose, and the other vector from the bridge of the nose to the tip of the nose (nasal line).

[0058] (viii) The length of the vector between the instrument mark and the bridge of the nose, normalized by the length of the nose;

[0059] (ix) Angle between two vectors: one vector from the appliance mark to the left corner of the mouth, and the other vector from the left corner of the mouth to the right corner of the mouth (mouth line).

[0060] (x) The length of the vector between the instrument mark and the left corner of the mouth, normalized by the length of the nose;

[0061] (xi) Angle between two vectors: one vector from the instrument mark to the right corner of the mouth, and the other vector from the left corner of the mouth to the right corner of the mouth (mouth line);

[0062] (xii) The length of the vector between the instrument mark and the right corner of the mouth, normalized by the length of the nose;

[0063] (xiii) Angle between two vectors: one vector from the instrument mark to the center of the left eye, and the other vector from the center of the left eye to the center of the right eye (eyeliner);

[0064] (xiv) The length of the vector between the instrument mark and the center of the left eye, normalized by the length of the nose;

[0065] (xv) Angle between two vectors: one vector from the device mark to the center of the right eye, and the other vector from the center of the left eye to the center of the right eye (eyeliner).

[0066] (xvi) The length of the vector between the instrument mark and the center of the right eye, normalized by the length of the nose;

[0067] The output of the trained classifier provides the tooth region based on the input.

[0068] The tooth region may include any of the following: left lateral, left upper crown inner, left lower crown inner, center lateral, center upper inner, center lower inner, right lateral, right upper crown inner, right lower crown inner.

[0069] According to another aspect, the present invention provides a dental care appliance activity tracking device, comprising:

[0070] - A processor configured to perform the steps defined above.

[0071] The dental care appliance activity tracking device may further include a camera for generating multiple frames of the video images. The dental care appliance activity tracking device may also include an output device configured to provide indications of the categorized tooth areas being treated during dental care activities.

[0072] The dental care appliance activity tracking device may be included in a smartphone.

[0073] According to another aspect, the present invention provides a computer program that can be distributed via electronic data transmission, comprising computer program code components that, when the program is loaded onto a computer, are adapted to cause the computer to perform any of the methods defined above; or a computer program product comprising a computer-readable medium having computer program code components thereon, wherein, when the program is loaded onto a computer, the computer program code components are adapted to cause the computer to perform any of the methods defined above.

[0074] According to another aspect, the present invention provides a dental care appliance including a generally spherical mark attached to or forming part of the appliance, the generally spherical mark having a plurality of colored segments or quadrants disposed around a longitudinal axis defined by the dental care appliance, the generally spherical mark including a flat end to form a plane at the end of the appliance.

[0075] Each of the colored segments extends from one pole of the generally spherical marker to the opposite pole of the marker, with the axis between the poles aligned with the longitudinal axis of the dental care appliance. The segments or quadrants may be separated from each other by contrasting colored bands. The diameter of the generally spherical marker may be 25 mm to 35 mm, and the width of the band may be 2 mm to 5 mm.

[0076] Dental care appliances may include toothbrushes.

[0077] The flat end of the generally spherical mark may define a plane with a diameter of 86% to 98% of the total diameter of the sphere. Attached Figure Description

[0078] Embodiments of the invention will now be described by way of example and with reference to the accompanying drawings, wherein:

[0079] - Figure 1 A schematic functional block diagram of the components of a toothbrush tracking system is shown;

[0080] - Figure 2 It shows the result of Figure 1 A flowchart of the toothbrush tracking process implemented by the system;

[0081] - Figure 3 A perspective view of a toothbrush marker structure suitable for tracking the position and orientation of a toothbrush is shown;

[0082] - Figure 4 It shows the mounting on the toothbrush handle Figure 3 A perspective view of the toothbrush mark. Detailed Implementation

[0083] The examples described below typically involve brushing teeth, but the principles described can generally be extended to any other form of dental care activity using dental care appliances. Such dental care activities may include, for example, the application of teeth whitening agents or the application of dental or oral medications or materials, such as enamel slurries, using any suitable form of dental care appliance, where it is necessary to track the dental care appliance as it travels over the tooth surface.

[0084] The term "toothbrush" as used here is intended to include both manual and electric toothbrushes.

[0085] refer to Figure 1 A toothbrush motion tracking system 1 for tracking a user's brushing activities may include a camera 2. The term "camera" is intended to encompass any image capture device suitable for obtaining a series of images of a user using a toothbrush during a brushing period. In one arrangement, the camera may be a camera conventionally found in smartphones or other computing devices.

[0086] Camera 2 communicates with data processing module 3. Data processing module 3 may be provided, for example, within a smartphone or other computing device, which may be suitably programmed or otherwise configured to implement the processing modules described below. Data processing module 3 may include face tracking module 4, configured to receive a series of video frames and determine from them various features or landmarks on the user's face and the orientation of the user's face. Data processing module 3 may also include toothbrush mark position detection module 5, configured to receive a series of video frames and determine the position of the toothbrush in each frame. Data processing module 3 may also include toothbrush mark orientation estimation module 6, configured to receive a series of video frames and determine / estimate the orientation of the toothbrush in each frame. The term "series of frames" is intended to encompass a generally temporally ordered sequence of frames, which may or may not constitute each frame captured by the camera, and is intended to encompass periodically sampled frames and / or a series of aggregated or averaged frames.

[0087] The corresponding outputs 7, 8, and 9 of the face tracking module 4, toothbrush mark position detection module 5, and toothbrush mark orientation detection module 6 can be provided as input to the brushed oral cavity region classifier 10, which is configured to determine the oral cavity region being brushed. In one example, the classifier 10 is configured to classify each video frame of the user's brushing action as corresponding to brushing one of the following oral cavity regions / tooth surfaces: left lateral, left upper crown inner, left lower crown inner, center lateral, center upper inner, center lower inner, right lateral, right upper crown inner, and right lower crown inner. This list of oral cavity regions / tooth surfaces is the currently preferred classification, but if needed and depending on the resolution of the classifier training data, the classifier can be configured to classify the brushing action into fewer or more oral cavity regions / tooth surfaces.

[0088] A suitable storage device 11 can be provided for the program and brushing data. Storage device 11 may include, for example, the internal memory of a smartphone or other computing device, and / or may include remote memory. A suitable display 12 can provide the user with, for example, visual feedback on the real-time progress of the brushing time and / or reports on the effectiveness of the current and historical brushing time periods. Another output device 1, such as a speaker, can provide audio feedback to the user. Audio feedback may include real-time verbal instructions on the ongoing actions during the brushing time, such as instructions on when to move to another oral cavity area or guidance on brushing movements. An input device 14 can be provided for the user to input data or commands. The display 12, output device 13, and input device 14 can be provided, for example, via an integrated touchscreen and audio output of a smartphone.

[0089] Now refer to Figure 2 Describe the functions of modules 4–6 and 10 above.

[0090] 1. Face tracking module

[0091] The face tracking module 4 can receive each consecutive frame or selected frame as input from the camera 2 (box 20). In one arrangement, the face tracking module 4 takes a 360 × 640 pixel RGB color image and attempts to detect faces within it (box 21). If a face is detected (box 22), the face tracking module 4 estimates the XY coordinates of multiple facial landmarks within it (box 23). The resolution and type of the image can be changed and selected according to the requirements of the imaging processing.

[0092] In one example, up to 66 facial landmarks can be detected, including the edges or other features of the mouth, nose, eyes, cheeks, ears, and chin. Preferably, the landmarks include at least two landmarks associated with the user's nose, and preferably include at least one or more landmarks selected from mouth feature locations such as the corners of the mouth and the center of the mouth, and eye feature locations such as the corners of the eyes and the center of the eyes. The face tracking module 4 also preferably uses the facial landmarks to estimate some or all of the head pitch angle, roll angle, and yaw angle (box 27). The face tracking module 4 can use conventional face tracking techniques, such as those described in E.S. et al. (2016). "Cascaded Regression with Sparsified FeatureCovariance Matrix for Facial Landmark Detection", Pattern Recognition Letters.

[0093] If face tracking module 4 fails to detect a face (box 22), module 4 can be configured to loop back (path 25) to obtain the next input frame and / or deliver an appropriate error message. If no face landmarks are detected, or an insufficient number of face landmarks are detected (box 24), the face tracking module can loop back (path 26) to obtain the next frame for processing and / or deliver an error message. If face detection has been implemented in a previous frame, a search window for estimating landmarks has been defined, and landmarks can be tracked in subsequent frames, for example, the location of the landmarks can be accurately predicted (box 43), then the face detection process can be omitted (boxes 21, 22).

[0094] 2. Toothbrush Marker Position Detection Module

[0095] The toothbrush used has toothbrush marking features that can be recognized by the toothbrush marking position detection module 5. These toothbrush marking features can be, for example, clearly defined shapes and / or color patterns on a portion of the toothbrush, which are typically kept in the field of vision during brushing. The toothbrush marking features can form an integral part of the toothbrush, or they can be applied to the toothbrush, for example, during manufacturing or by the user after purchase.

[0096] A particularly advantageous approach is to provide a structure at the end of the toothbrush handle, i.e., at the opposite ends of the bristles. This structure can be formed as an integral part of the toothbrush handle, or it can be applied after manufacturing as an accessory or "software dongle." One particularly successful form of structure is a generally spherical marker 60 with multiple colored quadrants 61a, 61b, 61c, 61d arranged around a longitudinal axis (corresponding to the longitudinal axis of the toothbrush). Figure 3 ). In such Figure 3 In some of the arrangements shown, each of quadrants 61a, 61b, 61c, and 61d is separated from the adjacent quadrant by strongly contrasting color bands 62a, 62b, 62c, and 62d. The generally spherical marker may have a flat end 63 away from the toothbrush handle receiving end 64, the flat end 63 defining a plane such that the toothbrush can stand upright on the flat end 63.

[0097] This combination of features offers numerous advantages, as symmetrical features have been found to provide easier spatial location tracking, while asymmetrical features have been found to provide better location tracking. Different colors enhance the performance of the structure and are preferably chosen to have high color saturation values ​​for easy segmentation under poor and / or uneven lighting conditions. The color selection can be optimized for the specific model of the camera in use. Figure 4As shown, marker 60 can be considered to have a first pole 71 attached to the end of the toothbrush handle 70 and a second pole 72 in the center of the flat end 63. Quadrants 61 may each provide a uniform color or color pattern extending uninterruptedly from the first pole 71 to the second pole 72, the color or color pattern being strongly separated from at least the adjacent quadrants, and preferably strongly separated from all other quadrants. In this arrangement, there may be no equatorial color change boundary between the poles. Also as... Figure 4 As shown, the axis of the mark extending between the first and second poles 71, 72 is preferably substantially aligned with the axis of the toothbrush / toothbrush handle 70.

[0098] In one arrangement, the toothbrush mark location detection module 5 receives facial position coordinates from the face tracking module 4 and crops (e.g., 360 × 360 pixels) a segment from the input image such that the face is positioned in the middle of the segment (box 28). The resulting image is then used by a convolutional neural network in the toothbrush mark detection module 5 (box 29), which returns a list of bounding box coordinates for candidate toothbrush mark detections, each accompanied by a detection score, for example, ranging from 0 to 1.

[0099] The detection score indicates the confidence level of a specific bounding box surrounding a toothbrush marker. In one arrangement, if the detection confidence level is higher than a predefined threshold, the system can provide the bounding box with the highest returned confidence level corresponding to the correct location of the marker within the image (box 30). If the highest returned detection confidence level is lower than the predefined threshold, the system can determine that the toothbrush marker is not visible. In this case, the system can skip the current frame and loop back to the next frame (path 31) and / or pass an appropriate error message. In general, the toothbrush marker location detection module exemplifies components for identifying predetermined marker features of a toothbrush in use in each of multiple frames of a video image, based on which the toothbrush position and orientation can be established.

[0100] If a toothbrush marker is detected (box 30), the toothbrush marker detection module 5 checks the distance between the oral cavity landmark and the toothbrush marker coordinates (box 32). If these are found to be too far apart, the system may skip the current frame and loop back to the next frame (path 33) and / or return an appropriate error message. The toothbrush-to-oral distance tested in box 32 can be a distance normalized to the length of the nose, as discussed further below.

[0101] To detect when someone hasn't brushed their teeth, the system can also track the toothbrush marker coordinates over time and estimate the marker movement value (box 34). If this value is below a predefined threshold (box 35), the toothbrush marker detection module 5 can skip the current frame, loop back to the next frame (path 36), and / or return an appropriate error message.

[0102] The toothbrush marker detection module 5 is preferably trained on a dataset comprising labeled real-life toothbrush marker images under various orientations and lighting conditions obtained from brushing videos collected for training purposes. Each image in the training dataset can be annotated semi-automatically using toothbrush marker coordinates. The toothbrush marker detector can be based on an existing pre-trained object detection convolutional neural network, which can be retrained to detect toothbrush markers. This can be achieved by adjusting the object detection network using images from the toothbrush marker dataset, a technique known as transfer learning.

[0103] 3. Toothbrush Mark Orientation Estimator

[0104] The toothbrush marker coordinates, or toothbrush marker bounding box coordinates (box 37), are passed to the toothbrush orientation detection module 6, which can crop and resize (box 38) the toothbrush marker image to a pixel count that can be optimized for the operation of the neural network in the toothbrush marker orientation detection module 6. In one example, the image is cropped / resized to 64 × 64 pixels. The resulting toothbrush marker image is then passed to the toothbrush marker orientation estimator convolutional neural network (box 39), which returns a set of pitch, roll, and yaw angles for the toothbrush marker image. Similar to the toothbrush marker position detection CNN, the toothbrush marker orientation estimation CNN can also output a confidence level ranging from 0 to 1 for each estimated angle.

[0105] A toothbrush marker orientation estimation CNN can be trained on any suitable dataset of labeled images under a wide range of possible orientations and background variations. Each image in the dataset can be accompanied by a corresponding marker pitch angle, roll angle, and yaw angle.

[0106] 4. Oral cavity area classifier

[0107] The oral cavity region classifier 10 accumulates the data generated by the three modules (face tracking module 4, toothbrush mark position detection module 5, and toothbrush mark orientation detection module 6) to extract a feature set specifically designed for the oral cavity region classification task, thereby generating a prediction of the oral cavity region (box 40).

[0108] The feature data used as input to the classifier preferably includes:

[0109] (i) Face tracker data, including facial landmark coordinates and one or more of head pitch, roll and yaw angles;

[0110] (ii) Toothbrush marker detector data, which includes one or more of the toothbrush marker coordinates and toothbrush marker detection confidence scores;

[0111] (iii) Toothbrush mark orientation estimator data, which includes toothbrush mark pitch angle, roll angle and yaw angle and toothbrush mark angle confidence score.

[0112] Many features used in oral cavity region classification can be derived individually from face tracker data or from a combination of face tracker and toothbrush marker detector data. These features are designed to improve oral cavity region classification accuracy and reduce the impact of unwanted variability in the input image, such as variability irrelevant to oral cavity region classification that could otherwise confuse the classifier and lead to incorrect predictions. Examples of such variability include facial size, location, and orientation.

[0113] To achieve this, oral cavity region classifiers can use face tracker data in several ways:

[0114] (i) Head pitch, roll, and yaw angles enable the classifier to learn to distinguish between oral cavity regions under various head rotations in three-dimensional space relative to the camera's viewpoint;

[0115] (ii) Use oral landmark coordinates to estimate the length of the toothbrush projection (relative to the camera viewpoint) as the length of the vector between the marker (toothbrush tip) and the center of the oral cavity;

[0116] (iii) Use oral landmark coordinates to estimate the position of the toothbrush relative to the left and right corners of the mouth;

[0117] (iv) Use eye landmark coordinates to estimate the toothbrush position relative to the center of the left and right eyes;

[0118] (v) Use nose landmark coordinates to estimate the position of the toothbrush relative to the nose. These coordinates can also be used to calculate the projected nose length as the Euclidean distance between the bridge and tip of the nose.

[0119] The projected nose length is used to normalize all oral region classification features derived from the distance.

[0120] The nose length normalization of distance-derived features makes the oral cavity region classifier 10 less sensitive to changes in the distance between the brusher and the camera, which affects the projected facial dimensions. It preferably works by measuring all distances within a portion of the person's nose length rather than absolute pixel values, thereby reducing the variability of corresponding features caused by the person's distance from the camera.

[0121] While the projected nose length is variable due to individual anatomy and age, it has been found to be the most stable measure of how far a person is from the camera and is least affected by facial expressions. It is found to be relatively unaffected or constant when the face is rotated relative to the camera between left, center, and right orientations. This contrasts with overall face height, which can also be used for this purpose but is easily altered by variable chin position, depending on the width of the mouth opening during brushing. Interocular distance can also be used, but this can be more susceptible to uncorrectable variations as the face turns from side to side and can also lead to tracking failure when the eyes are closed. Therefore, while any pair of facial landmarks that remain constant in their relative positions can be used to generate a normalization factor for normalizing oral region classification features derived from distance, the projected nose length has been found to achieve better results. Thus, in general, any at least two invariant landmarks associated with a user's face can be used to determine the inter-landmark distance for normalizing classification features derived from distance, with nose length being the preferred option.

[0122] Example feature sets that have been found to achieve optimal accuracy in classifying brushed oral cavity regions include at least some or all of the following:

[0123] (i) The toothbrush marks the pitch angle, roll angle, and yaw angle;

[0124] (ii) The sine and cosine values ​​of the pitch, roll, and yaw angles marked on the toothbrush;

[0125] (iii) Confidence scores for estimating pitch, roll, and yaw angles of the toothbrush mark;

[0126] (iv) Confidence score of toothbrush mark detection;

[0127] (v) Head pitch angle, roll angle, and yaw angle;

[0128] (vi) The toothbrush length, normalized to the nose length and estimated as the distance between the toothbrush mark and the center coordinates of the oral cavity;

[0129] (vii) The angle between two vectors and their sine and cosine: one vector runs from the toothbrush mark to the bridge of the nose, and the other vector runs from the bridge of the nose to the tip of the nose (nasal line).

[0130] (viii) The length of the vector between the toothbrush mark and the bridge of the nose, normalized by the length of the nose;

[0131] (ix) Angle between two vectors: one vector runs from the toothbrush mark to the left corner of the mouth, and the other vector runs from the left corner of the mouth to the right corner of the mouth (mouth line).

[0132] (x) The length of the vector between the toothbrush mark and the left corner of the mouth, normalized by the length of the nose;

[0133] (xi) Angle between two vectors: one vector runs from the toothbrush mark to the right corner of the mouth, and the other vector runs from the left corner of the mouth to the right corner of the mouth (mouth line);

[0134] (xii) The length of the vector between the toothbrush mark and the right corner of the mouth, normalized by the length of the nose;

[0135] (xiii) Angle between two vectors: one vector runs from the toothbrush mark to the center of the left eye, and the other vector runs from the center of the left eye to the center of the right eye (eyeliner);

[0136] (xiv) The length of the vector between the toothbrush mark and the center of the left eye, normalized by the length of the nose;

[0137] (xv) Angle between two vectors: one vector runs from the toothbrush mark to the center of the right eye, and the other vector runs from the center of the left eye to the center of the right eye (eyeliner);

[0138] (xvi) The length of the vector between the toothbrush mark and the center of the right eye, normalized by the length of the nose.

[0139] Once extracted, some, or preferably all, of the features listed above are passed to the brushed oral region support vector machine (SVM) (box 41) in the brushed oral region classifier 10 as classifier input. Classifier 10 outputs the index of the most likely oral region currently being brushed based on the current image frame or frame sequence.

[0140] Facial landmark coordinates, such as the positions of the eyes, nose, and mouth, as well as the toothbrush coordinates, are preferably not directly fed into the classifier 10, but are used to calculate various relative distances and angles of the toothbrush relative to the face, as well as other features as described above.

[0141] The toothbrush length is the projected length, meaning it varies as a function of distance from the camera and angle relative to the camera. Head angle helps the classifier account for variable angles, and nose length normalization of the toothbrush length helps accommodate the variability of the projected toothbrush length caused by distance from the camera. Together, these two methods help the classifier better determine the extent to which the toothbrush is hidden in the mouth, independent of the camera angle / distance, which is directly related to which oral region is being brushed. It has been found that classifiers trained on a specific toothbrush or other appliance length work well on other appliances of similar length with corresponding labels attached. It is also expected that classifiers trained on manual toothbrushes will operate accurately on electric toothbrushes of similar length with corresponding labels attached.

[0142] The oral cavity region classifier can be trained on a dataset of labeled videos capturing people brushing their teeth. Each frame in the dataset is labeled by the action depicted by the frame. These can include "idle" (no brushing), "label not visible," "other," and nine brushing actions, each corresponding to a specific oral cavity region or tooth surface region. In a preferred example, these regions correspond to: left lateral, left upper crown inner, left lower crown inner, central lateral, central upper inner, central lower inner, right lateral, right upper crown inner, and right lower crown inner.

[0143] The training dataset can include two video sets. The first video set can be recorded from a single viewpoint, with the camera mounted in front of the person at eye level, capturing unrestricted brushing. The second video set captures restricted brushing, instructing participants which oral regions to brush, when to brush, and for how long. These videos can be recorded from multiple different viewpoints. In one example, four different viewpoints are used. Increasing the number and range of viewing positions can improve classification accuracy.

[0144] The toothbrush tracking system illustrated above enables purely vision-based tracking of the toothbrush and facial features to predict oral cavity regions. It eliminates the need to place sensors on the toothbrush (although such sensor data would enhance the technology described here). It also eliminates the need to place sensors on the person brushing their teeth (although such sensor data would enhance the technology described here). This technology can be robustly implemented with sufficient performance using currently available mobile phone technology. It can also be performed using conventional 2D camera video images.

[0145] By tracking not only the location and orientation of the toothbrush, but also the position and orientation of the oral cavity relative to the normalized characteristics of the face, allowing the toothbrush position to be directly correlated with the position of the mouth and head, the aforementioned system provides superior performance in predicting / detecting the brushed oral cavity area.

[0146] Throughout this specification, the term "module" is intended to cover a functional system that may include computer code executing on a general-purpose or custom processor, or a hardware machine implementation of a function, such as on an application-specific integrated circuit.

[0147] Although the functions of, for example, the face tracking module 4, the toothbrush mark position detection module 5, the toothbrush mark orientation estimator / detector module 6, and the brushed oral cavity region classifier 10 have been described as different modules, their functions can be combined into single or multi-threaded processing within a suitable processor, or can be divided differently among different processors and / or processing threads. Functionality can be provided on a single processing device or on a distributed computing platform, such as implementing some processing on a remote server.

[0148] At least some of the functions of the data processing system can be implemented through smartphone applications or other processing performed on mobile telecommunications devices. Some or all of the described functions can be provided on the smartphone. Some functions can be provided by a remote server using the remote communication facilities of the smartphone, such as cellular phone networks and / or wireless internet connections.

[0149] It has been found that the above-mentioned techniques are particularly effective in reducing the influence of variable or unknown person-to-camera distance and variable or unknown person-to-camera angle, which are difficult to assess using only 2D imaging equipment. The feature set designed and selected for the input of the brushed oral region classifier 10 preferably includes head pitch angle, roll angle, and yaw angle to indicate the orientation of the person's head relative to the camera, and nose length normalization (or other normalized distance between two invariant facial landmarks) to indicate the variable distance between the person and the camera.

[0150] If the influence of variable person-to-camera distance is minimized by nose length normalization and the angle of person-to-camera is explained by head angle, then the use of facial points improves oral region classification by calculating the relative position of toothbrush markers with respect to these points.

[0151] Can be combined Figure 3 and Figure 4 The design of the described toothbrush mark features is modified. For example, although the mark 60 shown is divided into four quadrants, each extending from one pole of the generally spherical mark to the other, a different number of segments 61 can be used, as long as they enable the orientation detection module 6 to detect orientation with sufficient resolution and accuracy, such as three, five, or six segments arranged around the longitudinal axis. The strip 62 separating the segments 61 can extend the entire circumference of the toothbrush mark, for example, from one pole to the other, or it can extend only a portion of the circumference. The strip 62 can have any suitable width to optimize the recognition of the mark features and the orientation detection module's detection of orientation. In a preferred example, the diameter of the mark 60 is 25 to 35 mm (approximately 28 mm in a particular example), and the width of the strip 62 can be 2 mm to 5 mm (3 mm in a particular example).

[0152] The contrast color for each segment can be selected to optimally contrast with the skin tone of the user using the toothbrush. In the example shown, red, blue, yellow, and green are used. The color and color area size can also be optimized for the imaging device used (e.g., a smartphone imaging device). Color optimization can take into account the characteristics and limitations of the imaging sensor and processing software. For example, as illustrated above, marker size optimization can also consider a specific working distance range from the imaging device to the toothbrush marker 60, to ensure that a minimum number of pixels can be captured for each color patch, especially those with boundary stripes, while ensuring that the marker size is not so large as to degrade the performance of the face tracking module 4 due to excessive occlusion of facial features during tracking.

[0153] The dimensions of the flat end 63 can be configured to provide the necessary stability for the toothbrush or other dental care appliance when it stands on the flat end. In the examples above, the flat end may be created by removing 20% ​​to 40% of the longitudinal dimension of the sphere. In the specific example above with a sphere of 28 mm in diameter, the planar section defined by the flat end 63 is approximately 7 to 8 mm along the longitudinal axis, that is, the longitudinal dimension of the sphere (between poles) is shortened by approximately 7 to 8 mm. In other examples, the flat end 63 may define a plane with a diameter in the range of 24 to 27.5 mm or a diameter of 86% to 98% of the total diameter of the sphere, and in the specific example above, the flat end 63 may define a plane with a diameter of 26 mm or a diameter of 93% of the total diameter of the sphere.

[0154] As used herein, the term "generally spherical" is intended to cover a mark having a spherical main surface (or, for example, a mark defining an oblate spheroid), wherein a portion of the spherical main surface within the aforementioned range is removed / absent in order to define a small plane thereon.

[0155] Use such as Figure 3 and Figure 4 The results of marker 60 shown, i.e., those with flat ends 63 and contrasting colors for adjacent segments / quadrants 61 and separation bands 62 between segments / quadrants 61, have been compared with spherical markers with more limited color variation, and significant improvements in orientation estimation results and classification accuracy are shown in Table 1 below:

[0156] Table 1

[0157]

[0158] The Z-oriented data shows the number of samples that achieve the angular measurement error threshold for the left-hand column about the Z-axis (corresponding to the axis extending between the first and second poles of the marker, and therefore the long axis of the toothbrush), while the X-oriented data shows the number of samples that achieve the angular measurement error threshold for the left-hand column about the X-axis (corresponding to one of the axes extending orthogonally to the Z / longitudinal axis of the marker / toothbrush). It should be understood that the Y-oriented accuracy data generally correspond to the X-axis accuracy data. These levels of accuracy have been found to be sufficient for classifying the oral cavity regions and tooth surfaces as illustrated above.

[0159] Other embodiments are intentionally within the scope of the appended claims.

Claims

1. A computer-implemented method for tracking a user's dental care activities, comprising: - Receive video images of the user's face during dental care sessions; - Identify predetermined features of the user's face in each of a plurality of frames of the video image, the predetermined features including at least two invariant landmarks associated with the user's face and one or more landmarks selected from at least oral cavity feature locations and eye feature locations, wherein the at least two invariant landmarks are invariant in relative position to each other, wherein the at least two invariant landmarks associated with the user's face include a landmark on the user's nose; - Identify predetermined marking features of a dental care appliance used by a user in each of the plurality of frames of the video image, wherein the predetermined marking features of the dental care appliance include a generally spherical mark attached to or forming part of the dental care appliance, the generally spherical mark having a plurality of colored segments or quadrants arranged around a longitudinal axis; - Determine a measure of the distance between the landmarks based on the at least two invariant landmarks associated with the user's face; - Determine the length of a dental care appliance, wherein the length of the dental care appliance is evaluated as the distance between the generally spherical marker and one or more landmarks associated with the user's oral cavity, the length of the dental care appliance being normalized by the distance between the landmarks; - Determine one or more dental care appliances to facial feature distances, each normalized to the distance between the respective landmarks, based on one or more landmarks selected from at least oral feature locations and eye feature locations; - Determine the angle of the dental care appliance to the nose and the angle of one or more dental care appliances to facial features; - Using the determined angle from the dental care appliance to the nose and one or more angles from the dental care appliance to the facial features, the normalized length of the dental care appliance and the normalized distance from the dental care appliance to the facial features, each frame is classified as corresponding to one of a plurality of possible tooth regions processed with the dental care appliance.

2. The method of claim 1, wherein the dental care activity includes brushing teeth, and the dental care appliance includes a toothbrush.

3. The method according to claim 1, wherein the distance between the landmarks is the length of the user's nose.

4. The method of claim 3, wherein the distance from each of the one or more dental care appliances to the facial feature, each free from the length normalized version of the user's nose, includes one or more of the following: (i) The distance from the dental care appliance to the oral cavity, normalized by the length of the user's nose; (ii) The distance from the dental care appliance to the eye, normalized to the length of the user's nose; (iii) The distance from the dental care appliance to the bridge of the nose, normalized to the length of the user's nose; (iv) The distance from the dental care appliance to the left corner of the mouth, normalized to the length of the user's nose; (v) The distance from the dental care appliance to the right corner of the mouth, normalized to the length of the user's nose; (vi) The distance from the dental care appliance to the left eye, normalized by the length of the user's nose; (vii) The distance from the dental care appliance to the right eye, normalized by the length of the user's nose; (viii) The distance from the dental care appliance to the left corner of the eye, normalized to the length of the user's nose; (ix) The distance from the dental care appliance to the right corner of the eye, normalized by the length of the user's nose.

5. The method according to claim 3 or claim 4, wherein the angle of the one or more dental care appliances to facial features includes one or more of the following: (i) Angle of dental care appliance to the oral cavity; (ii) The angle from the dental care appliance to the eye; (iv) The angle between the vector from the generally spherical marker to the left corner of the mouth and the vector from the left corner of the mouth to the right corner of the mouth; (v) The angle between the vector from the generally spherical marker to the right corner of the mouth and the vector from the left corner of the mouth to the right corner of the mouth; (vi) The angle between the vector from the generally spherical marker to the center of the left eye and the vector from the center of the left eye to the center of the right eye; (vii) The angle between the vector from the generally spherical marker to the center of the right eye and the vector from the center of the left eye to the center of the right eye.

6. The method of claim 1, wherein the at least two landmarks associated with the user's nose include the bridge of the nose and the tip of the nose.

7. The method according to claim 1, wherein each segment or quadrant is separated by a contrasting color band.

8. The method of claim 1, wherein the generally spherical marker is positioned at the end of the dental care appliance, its longitudinal axis being aligned with the longitudinal axis of the dental care appliance.

9. The method of claim 1 or claim 7, wherein identifying a predetermined marker feature of a dental care appliance in use in each of the plurality of frames of the video image comprises: - Determine the position of the generally spherical marker in the frame; - Crop the frame to capture the generally spherical marker; - Resize the cropped frame to a predetermined pixel size; - Use a trained orientation estimator to determine the pitch, roll, and yaw angles of the generally spherical marker; - The pitch angle, roll angle, and yaw angle are used to determine the angular relationship between the dental care appliance and the user's head.

10. The method of claim 1, wherein identifying a predetermined marker feature of a dental care appliance in use in each of the plurality of frames of the video image comprises: - For each identification bounding box coordinate in the detection of multiple candidate spherical marks, each bounding box coordinate has a corresponding detection likelihood score; - The spatial position of the dental care appliance relative to the user's head is determined based on the detection likelihood score that is greater than a predetermined threshold and / or the coordinates of the bounding box with the highest score.

11. The method according to any one of the preceding claims, wherein the generally spherical mark includes a flat end to form a plane at the end of the dental care appliance.

12. The method of claim 11, wherein the flat end of the generally spherical mark defines a plane with a diameter of 86% to 98% of the total diameter of the sphere.

13. A tracking device for the movement of dental care appliances, comprising: A processor configured to perform the method according to any one of claims 1 to 12.

14. A computer program product comprising a computer-readable medium having a computer program thereon, the computer program including instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 12.

Images