Oral health care

Abstract

The proposed concepts are therefore directed to providing protocols, solutions, concepts, designs, methods and systems relating to oral health care routines assisting a user. It has been recognized that captured videos of a user and / or personal care device during execution of an oral care routine may be analyzed to obtain motion data, which may then be utilized to determine at least one parameter value of the personal care routine. That is, the user's performance of an oral health care routine may be insighted based on movement of the user's body and / or personal care device, such as a toothbrush. Such video may be obtained using existing or conventional devices that have been owned by the user, including cameras.

Description

Technical Field

[0001] The present invention relates to the field of oral health care routines, and in particular to the field of assisting a user in performing an oral health care routine. Background Art

[0002] Oral care devices such as electric toothbrushes or dental braces are used frequently (e.g., daily), and oral health care routines involving these devices often include several complex steps. As a result, users performing oral health care routines often cannot accurately judge their own performance of the routine. A particularly problematic example is the routine of properly brushing and flossing teeth.

[0003] Guided brushing is a feature that provides better oral care for toothbrush users. Users often tend to deviate, brushing one side of their mouth more than the other, or focusing on the front teeth and ignoring the back teeth. Therefore, better oral care can be achieved by providing users with important brushing information, such as the total time spent brushing each area of ​​the mouth, or whether basic oral hygiene procedures (such as flossing) are performed. This information is particularly useful for parents to track their children's teeth cleaning routines.

[0004] Currently, guided brushing is provided using sensors embedded in the toothbrush, such as gyroscopes. However, these sensors are expensive and are therefore only available in high-end, expensive toothbrushes. Most conventional oral care devices do not include any sensors suitable for monitoring an oral health care routine. In addition, few solutions exist to track flossing performance.

[0005] US2020 / 201272A1 describes a system and method for operating a personal beauty / household appliance, comprising providing a personal beauty / household appliance having at least one physical sensor.

[0006] “Method and System for Measuring Effectiveness of Tooth Brushing ED–Darl Kuhn” describes a method and system for measuring a user’s effectiveness at brushing their teeth by dynamically assessing the degree of plaque removal and the practice of proper brushing technique.

[0007] WO 2021 / 197801 A1 describes a method of tracking a user's dental care activities, comprising receiving a video image of the user's face during, for example, a brushing process.

[0008] US2018 / 132602A1 describes an oral care system that may include a toothbrush having physical properties and a programmable processor configured to receive physical property data.

[0009] US2020 / 179089A1 describes an oral hygiene monitoring system for tracking the movement and orientation of an oral hygiene device. A control system can process data output from a motion sensor to determine the position and orientation of the oral hygiene device. Summary of the invention

[0010] The present invention is defined by the claims.

[0011] According to an example of one aspect of the invention, a method for assisting a user's oral health care routine is provided.

[0012] The method includes obtaining video data from a captured video of a user performing an oral health care routine using a personal care device; processing the video data to obtain motion data describing motion of a portion of the user during performance of the oral health care routine; and analyzing the motion data to determine at least one parameter value of the oral health care routine, wherein the portion of the user includes a hand of the user.

[0013] Therefore, the proposed concepts are aimed at providing schemes, solutions, concepts, designs, methods and systems related to assisting users in their oral health care routine.

[0014] Specifically, embodiments are directed to determining at least one parameter value of an oral health care routine based on the movement of a portion of a personal care device and / or a user during the execution of the oral health care routine. Information about the movement of a portion of a personal care device and / or a user can be obtained from a video of the user and / or personal care device captured during the execution of the oral health care routine. That is, in the example of a user brushing their teeth, a video of the user brushing their teeth can be captured and then analyzed. By analyzing the movement of the user and their toothbrush, it can be understood how effectively they are brushing their teeth, i.e., spending the appropriate amount of time on each tooth. Guidance can then be provided to the user to inform them of the quality of their execution. In another example, where the oral health care routine is the user flossing, the movement of one or both of the user's hands can be analyzed to determine whether the user is flossing correctly or comprehensively, and guidance can be provided to the user to inform them how to floss more effectively in the future.

[0015] In other words, it is proposed that video of a user and / or a personal care device captured during performance of an oral health care routine can be analyzed to obtain motion data, which can then be utilized to determine at least one parameter value of the personal health care routine. That is, insight into the user's performance of an oral health care routine can be gained based on the movement of the user's body and / or a personal care device (such as a toothbrush or flossing device). Such video can be obtained using existing or conventional equipment that the user already owns, including a camera.

[0016] By providing a computer vision-based method for analyzing an oral health care routine, feedback similar to guided brushing can be provided to the user, regardless of the type of toothbrush used. However, embodiments are not limited to electric or manual toothbrushes. Personal care devices may also include braces, dental floss devices, or any other personal oral care device. Thus, one or more of the proposed concepts may be employed in a range of different personal care devices. Thus, embodiments may have a wide range of applications in the field of personal care devices and are particularly relevant to dentist recommendations. For example, by reducing unnecessary tissue damage, improved cleaning of the user's teeth, gums, tongue, etc. is achieved. Thus, embodiments may be used in dental treatments to provide support to dental health care professionals when providing treatments to subjects.

[0017] By integrating into the user's normal brushing habits, embodiments may support improved dental care.Thus, an improved oral health care routine may be provided by the proposed concept.

[0018] For example, by automatically analyzing portions of a user during the execution of an oral health care routine, one or more learnings or statistics may be determined that may be useful to the user. An example parameter value may, for example, be related to user deviations. For example, if the oral health care routine is the user brushing their teeth, the parameter value may be the ratio of time spent brushing the left side of the mouth to the right side of the mouth, or alternatively, the percentage of the mouth that is cleaned. Such learning may enable feedback to be provided to the user, thereby reinforcing good habits and bringing attention to bad behavior.

[0019] Using only video data to understand a user's performance of an oral health care routine may permit assistance to the user in their daily routine without the need for specialized sensors in the personal care device, which would be costly and would increase the complexity of the personal care device required. Thus, assistance may be provided to a user performing an oral health care routine regardless of whether the personal care device in question has sensors suitable for monitoring an oral health care routine.

[0020] Ultimately, the proposed concept(s) may support the user in improving the performance of an oral health care routine.

[0021] In some embodiments, at least one parameter value may include at least one of: user deviation; a measure of the completion of the oral health care routine; a measure of the completion of a subroutine of the oral health care routine; and duration. An example of a user deviation may be the ratio of the time spent brushing the upper teeth to the time spent brushing the lower teeth. An example of a measure of the completion of the oral health care routine may be the number of teeth remaining to be properly cleaned. An example of a measure of the completion of a subroutine of the oral health care routine may be whether the user has flossed. An example of duration may be the total time spent brushing each area of ​​the mouth. These parameter values ​​may enable feedback to be provided to the user so that they are aware of deficiencies in their performance of the oral health care routine, thereby enabling conscious improvements.

[0022] In some embodiments, the obtained motion data also describes the motion of the personal care device. This can allow tracking of the motion of, for example, a toothbrush or flossing device, which can allow further understanding of the user's performance of an oral health care routine.

[0023] In some embodiments, processing video data to obtain motion data describing the movement of a personal care device may include providing the video data as input to a first convolutional neural network (CNN), the first CNN being trained to predict motion data indicating a series of positions of the personal care device for a personal care device associated with the video data.

[0024] Compared to simple bounding boxes, using CNN instead of common object detection algorithms can achieve pixel-level segmentation of personal care devices. This can facilitate more accurate localization and movement tracking of personal care devices.

[0025] In some embodiments, a series of positions of a personal care device may describe a region of the personal care device. For example, the position and motion of a toothbrush top (e.g., a brush head) may have particular use in analyzing motion data to determine parameter values ​​for an oral health care routine.

[0026] In some embodiments, the first CNN can be trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with a personal care device and the corresponding known outputs include motion data indicating a series of positions of the personal care device. In this manner, the first CNN can be trained to output motion data indicating a series of positions of the personal care device when provided with video data associated with the personal care device during an oral health care routine.

[0027] In some embodiments, the first CNN is a pre-trained model that has been further trained on videos of subjects using the personal care device that have been manually annotated. This makes the CNN particularly good at recognizing personal care devices.

[0028] In some embodiments, processing the video data to obtain motion data describing motion of a portion of the user includes:

[0029] providing the video data as input to a second neural network, the second neural network being trained to predict, for a portion of a user associated with the video data, motion data indicative of a sequence of positions of a palm of a hand of the user;

[0030] The video data is provided as input to a third neural network that is trained to predict, for a portion of the user associated with the video data, motion data indicative of a series of positions of a landmark on a hand of the user.

[0031] The split model separately identifies the location of the palm of the user's hand and the location of the landmarks on the user's hand, allowing for fast and accurate hand tracking without the use of any specialized hardware (such as a depth-sensing camera). The third neural network can quickly identify the location of the landmarks on the hand by searching only within the bounding box identified by the second neural network.

[0032] In some embodiments, the second neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with a portion of the user and the corresponding known outputs include motion data indicative of a series of positions of a palm of the user's hand. This causes the second neural network to become adept at recognizing the user's palm.

[0033] In addition, a third neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with the portion of the user and the corresponding known outputs include motion data indicative of a series of positions of a landmark on the user's hand, so that the third neural network becomes adept at recognizing the landmark on the user's hand.

[0034] In some embodiments, wherein the second neural network comprises a single shot multi-box detector architecture and the third neural network comprises a feature pyramid network. The single shot multi-box detector architecture allows multiple objects present in an image to be detected in a single forward pass of the network. This allows multiple objects, such as two palms, to be detected quickly. The feature pyramid network is particularly good at recognizing small objects, such as knuckles or finger joints, and is therefore well suited for detecting landmarks on a user's hand.

[0035] In some embodiments, processing the video data to obtain motion data describing motion of a portion of the user includes:

[0036] providing the video data as input to a fourth neural network, the fourth neural network being trained to predict, for a portion of a user associated with the video data, motion data indicative of a sequence of positions of a face of the user;

[0037] The video data is provided as input to a fifth neural network, which is trained to predict, for a portion of a user associated with the video data, motion data indicative of a series of positions of landmarks on the user's face.

[0038] The split model separately identifies the location of the user's face and the location of the landmarks on the user's face, thereby allowing fast and accurate face tracking without the use of any specialized hardware (such as a depth-sensing camera). By first identifying the user's face, the landmarks on the face can then be more quickly identified by the fifth neural network by searching only within the bounding box detected by the fourth neural network.

[0039] In some embodiments, the fourth neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with portions of a user and the corresponding known outputs include motion data indicating a series of positions of the user's face. This causes the fourth neural network to become adept at recognizing the user's face.

[0040] In addition, a fifth neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with portions of the user and the corresponding known outputs include motion data indicating a series of positions of the landmarks on the user's face. This causes the fifth neural network to become adept at recognizing the landmarks on the user's face.

[0041] In some embodiments, the fourth neural network includes a single-shot multi-box detector architecture and the fifth neural network includes a feature pyramid network. The single-shot multi-box detector architecture allows multiple objects present in an image to be detected in a single forward pass of the network. This allows single or multiple objects, such as one or more faces, to be quickly detected. The feature pyramid network is particularly good at identifying small objects, such as nostrils and eyes, and is therefore well suited for detecting landmarks on a user's face.

[0042] In some embodiments, analyzing the motion data to determine at least one parameter value of the oral health care routine includes:

[0043] The motion data is provided as input to a machine learning algorithm that is trained to predict at least one parameter value of an oral health care routine for an oral health care routine associated with the motion data. From the personal care device position, hand and facial landmark positions, and pattern matching, the machine learning algorithm can infer useful information. For example, in the case of a user brushing their teeth, the machine learning algorithm can predict the angle of the personal care device relative to the user's mouth.

[0044] In some embodiments, the machine learning algorithm includes a supervised classifier model. This is particularly useful for classifying detected behaviors, such as brushing a specific area of ​​a user's mouth.

[0045] In some embodiments, a machine learning algorithm is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include motion data associated with an oral health care routine and the corresponding known outputs include at least one parameter value of the oral health care routine. This enables the machine learning algorithm to become particularly good at determining the parameter value of the oral health care routine.

[0046] In some embodiments, analyzing the motion data to determine at least one parameter value of the oral health care routine includes:

[0047] The motion data is provided as input to a rule-based algorithm designed to predict at least one parameter value of the oral health care routine for the oral health care routine associated with the motion data. This allows the parameter value to be inferred from the motion data without the use of a machine learning algorithm, which can reduce computational requirements.

[0048] In some embodiments, analyzing the motion data to determine at least one parameter value of the oral health care routine includes:

[0049] Predicting the contact location between the personal care device and the user's surface. For example, in the case of a user brushing their teeth, from the contact location between the personal care device and the user's surface, the area where brushing is taking place can be inferred. For example, it can be inferred that the user is currently brushing their left molars.

[0050] In some embodiments, analyzing the motion data to determine at least one parameter value of the oral health care routine includes:

[0051] Predict the distance between each of the user's palms and the user's face,

[0052] And optionally, wherein the predicted distance between each palm of the user and the user's face is compared to a predetermined threshold.

[0053] This allows detection of a user flossing since they would need to have both hands close to their mouth when flossing.

[0054] In some embodiments, a computer program is provided comprising code means for implementing any of the methods described above when the program is run on a processing system.

[0055] According to another aspect of the present invention, there is provided a system for assisting a user's oral health care routine, the system comprising:

[0056] an input interface configured to obtain video data from a captured video of a user performing a personal health care routine using a personal care device;

[0057] A processor device configured to:

[0058] processing the video data to obtain motion data describing the motion of a portion of a user during performance of an oral health care routine;

[0059] The motion data is analyzed to determine at least one parameter value of an oral health care routine, wherein the portion of the user includes a hand of the user.

[0060] Thus, a concept for assisting a user's oral health care routine may be presented, and this may be done based on visually observed user movements during the performance of the oral health care routine. Determining parameter values ​​for the oral health care routine may help inform the user of the quality of their oral health care routine performance and allow them to adjust their performance accordingly.

[0061] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] For a better understanding of the invention, and to more clearly show how the same may be carried into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0063] Figure 1 is a simplified flow chart of a method for assisting a user's oral health care routine according to a proposed embodiment;

[0064] Figure 2 is a more in-depth flow chart of a method for assisting a user's oral health care routine according to a proposed embodiment;

[0065] Figure 3a is a simplified diagram of tracking personal care devices according to a proposed embodiment;

[0066] Figure 3b is a simplified diagram of tracking a personal care device at alternative angles according to a proposed embodiment;

[0067] Figure 4is a simplified block diagram of a system for assisting a user's oral health care routine according to a proposed embodiment; and

[0068] Figure 5 An example of a computer is shown in which one or more portions of the embodiments may be employed. DETAILED DESCRIPTION

[0069] The present invention will be described with reference to the accompanying drawings.

[0070] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, system, and method, are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the invention will be better understood from the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0071] Variations to the disclosed embodiments may be understood and implemented by those skilled in the art in practicing the claimed invention by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0072] It should be understood that the drawings are merely schematic and not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0073] Implementations according to the present disclosure involve various techniques, methods, schemes and / or solutions related to assisting a user's oral health care routine. Based on the proposed concepts, a variety of possible solutions can be implemented individually or in combination. That is, although these possible solutions may be described separately below, two or more of these possible solutions can be implemented in one or more combinations.

[0074] Embodiments of the present invention are directed to assisting a user in an oral health care routine by analyzing the motion (i.e., movement) of the user and a personal care device during the performance of the oral health care routine. This may be accomplished by obtaining video data from a captured video of the user performing the oral health care routine using the personal care device. The video data may then be processed to obtain motion data describing at least one of: the motion of the personal care device and a portion of the user during the performance of the oral health care routine. The motion data may then be analyzed to determine at least one parameter value of the oral health care routine.

[0075] Thus, the proposed concept is directed to determining at least one parameter value of an oral health care routine based on the movement of a portion of a personal care device and / or a user during the performance of the oral health care routine. Information about the movement of a portion of a personal care device and / or a user can be obtained from a video of the user and / or the personal care device captured during the performance of the oral health care routine. That is, in the example of a user brushing their teeth, a video of the user brushing their teeth can be captured and then analyzed. By analyzing the movement of the user and / or their toothbrush, it can be understood how effectively they are brushing their teeth, i.e., spending the appropriate amount of time on each tooth. Guidance can then be provided to the user to inform them of the quality of their performance.

[0076] See now Figure 1 , wherein a flow chart of a method 100 for assisting a user's oral health care routine according to a proposed embodiment is depicted.

[0077] The method begins at step 110 by obtaining video data from a captured video of a user performing an oral health care routine using a personal care device. Such video may be obtained using an existing or conventional device that the user already owns, including a camera. For example, the conventional device may be one of the following: a smartphone, a tablet, a laptop, or any other suitable device. By providing a computer vision-based method for analyzing an oral health care routine, feedback similar to guided brushing may be provided to the user regardless of the type of toothbrush or personal care device used. Using only video data to understand the user's performance of the oral health care routine may allow the user to be assisted in their daily routine without the need for dedicated sensors in the personal care device, which are costly and increase the complexity of the required personal care device. Thus, assistance may be provided to a user performing an oral health care routine regardless of whether the personal care device in question has sensors suitable for monitoring the oral health care routine.

[0078] In step 120, the video data is processed to obtain motion data describing at least one of: motion of the personal care device and parts of the user during performance of the oral health care routine. Details of the processing of the video data will be explained in more detail later.

[0079] In step 130, the motion data is analyzed to determine at least one parameter value of the oral health care routine. An example parameter value may, for example, be related to user deviations. For example, if the oral health care routine is the user brushing their teeth, the parameter value may be the ratio of time spent brushing the left side of the mouth to the right side of the mouth, or alternatively, the percentage of the mouth that is cleaned. Such knowledge may enable feedback to be provided to the user, thereby reinforcing good habits and bringing attention to bad behaviors.

[0080] In some embodiments, at least one parameter value may include at least one of: user deviation; a measure of the completion of the oral health care routine; a measure of the completion of a subroutine of the oral health care routine; and duration. An example of a user deviation may be the ratio of the time spent brushing the upper teeth to the time spent brushing the lower teeth. An example of a measure of the completion of the oral health care routine may be the number of teeth remaining to be properly cleaned. An example of a measure of the completion of a subroutine of the oral health care routine may be whether the user has flossed. An example of duration may be the total time spent brushing each area of ​​the mouth. These parameter values ​​may enable feedback to be provided to the user so that they are aware of deficiencies in their performance of the oral health care routine, thereby enabling conscious improvements.

[0081] See now Figure 2 , which depicts a more in-depth flow chart of a method 200 for assisting a user in performing an oral health care routine according to an exemplary embodiment. The method begins at step 110 by obtaining video data from a captured video of a user performing an oral health care routine using a personal care device, as described above.

[0082] In step 210, the video data is processed to predict motion data indicative of a sequence of positions of the personal care device. The video data may be conveniently processed by providing the video data as input to a first convolutional neural network (CNN), the first CNN being trained to predict motion data of the personal care device, the motion data indicative of a sequence of positions of the personal care device. In this manner, the motion of the personal care device captured during the performance of the oral health care routine is converted / translated into motion data.

[0083] Compared to simple bounding boxes, using CNN instead of common object detection algorithms can achieve pixel-level segmentation of personal care devices. This can facilitate more accurate localization and movement tracking of personal care devices.

[0084] The structure of artificial neural networks (or neural networks for short) is inspired by the human brain. Neural networks consist of layers, each containing multiple neurons. Each neuron contains a mathematical operation. Specifically, each neuron can include a different weighted combination of a single type of transformation (e.g., the same type of transformation, sigmoid, etc., but with different weights). In the process of processing input data, each neuron performs a mathematical operation on the input data to produce a numerical output, and the output of each layer in the neural network is fed to the next layer in turn. The last layer provides the output.

[0085] There are several types of neural networks, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). This exemplary embodiment of the present invention employs a CNN-based learning algorithm because CNNs have proven to be particularly successful in analyzing videos and are able to classify frames within a video with much lower error rates than other types of neural networks.

[0086] CNNs typically contain several layers, including convolutional layers, pooling layers, and fully connected layers. Convolutional layers consist of a set of learnable filters and extract features from the input. Pooling layers are a form of nonlinear downsampling that reduces the size of the data by combining the outputs of multiple neurons in one layer into a single neuron in the next layer. Fully connected layers connect each neuron in one layer to all neurons in the next layer.

[0087] Methods for training machine learning algorithms are well known. Typically, such methods include obtaining a training data set, including training input data entries and corresponding training output data entries. An initialized machine learning algorithm is applied to each input data entry to generate a predicted output data entry. The error between the predicted output data entry and the corresponding training output data entry is used to modify the machine learning algorithm. The process can be repeated until the error converges and the predicted output data entry is sufficiently similar to the training output data entry (e.g., ±1%). This is generally referred to as a supervised learning technique.

[0088] For example, the weights of the mathematical operations of each neuron can be modified until the error converges. Known methods for modifying neural networks include gradient descent, back propagation algorithms, and the like.

[0089] The training input data entries of the first CNN used in method 200 correspond to example video data associated with a personal care device. The training output data entries correspond to motion data indicating a series of positions of the personal care device. In addition, several pre-processing methods can be employed to improve the training samples. In other words, the first CNN can be trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with the personal care device, and the corresponding known outputs include motion data indicating a series of positions of the personal care device. In this way, when provided with video data associated with the personal care device during an oral health care routine, the first CNN is trained to output motion data indicating a series of positions of the personal care device.

[0090] In some embodiments, the first CNN is a pre-trained model that is further trained on videos of subjects using personal care devices that have been manually annotated. This makes the CNN particularly good at identifying personal care devices. In some embodiments, the pre-trained model is trained on the COCO dataset based on the RESNET34, RESNET 50 and / or RESNET101 backbone architecture.

[0091] In some embodiments, the first CNN is more specifically a Mask-RCNN deep neural model trained based on custom toothbrush annotations.

[0092] In some embodiments, a series of positions of a personal care device describes a region of the personal care device. For example, the position and motion of a top edge of a toothbrush (eg, a brush head) may have particular use during analysis of motion data to determine parameter values ​​for an oral health care routine.

[0093] In step 220, the video data is processed to predict motion data indicative of a series of positions of a portion of a user during performance of an oral health care routine. In step 220, the portion of the user is a palm of a user's hand. Processing is facilitated by providing the video data as an input to a second neural network, which is trained to predict motion data indicative of a series of positions of a palm of a user's hand for the portion of the user associated with the video data. In an embodiment, the second neural network includes a single-shot multi-box detector architecture. The single-shot multi-box detector architecture allows multiple objects present in an image to be detected in a single forward pass of the network. This allows multiple objects, such as two palms, to be quickly detected.

[0094] The second neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with a portion of the user and the corresponding known outputs include motion data indicative of a series of positions of a palm of the user's hand. This causes the second neural network to become adept at recognizing the user's palm and drawing a bounding box around it.

[0095] In step 230, the video data is processed to predict motion data indicative of a series of positions of a part of a user during performance of an oral health care routine. In step 230, the part of the user is the position of a landmark on the user's hand. Processing is facilitated by providing the video data as an input to a third neural network, which is trained to predict motion data indicative of a series of positions of landmarks on the user's hand for the part of the user associated with the video data. In one embodiment, the third neural network comprises a feature pyramid network. Feature pyramid networks are particularly good at recognizing small objects, such as knuckles or finger joints, and are therefore well suited for detecting landmarks on the user's hand. In one embodiment, at least 21 landmarks on the hand are detected.

[0096] The third neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with the portion of the user and the corresponding known outputs include motion data indicative of a series of positions of a landmark on the user's hand. This causes the third neural network to become adept at recognizing the landmark on the user's hand. In one embodiment, the coordinates of the recognized landmark on the user's hand are saved to a memory for further analysis.

[0097] The split model separately identifies the location of the palm of the user's hand and the location of the landmarks on the user's hand, thereby allowing fast and accurate hand tracking, such as tracking of the hand skeleton, without the use of any specialized hardware (such as a depth-sensing camera). The third neural network can quickly identify the location of the landmarks on the hand by searching only within the bounding box identified by the second neural network. An encoder-decoder architecture can be used to identify the landmarks.

[0098] In step 240, the video data is processed to predict motion data indicative of a series of positions of a portion of a user during performance of the oral health care routine. In step 240, the portion of the user is a face of the user. Processing is facilitated by providing the video data as an input to a fourth neural network, which is trained to predict motion data indicative of a series of positions of a face of the user for the portion of the user associated with the video data. In one embodiment, the fourth neural network includes a single-shot multi-box detector architecture. The single-shot multi-box detector architecture allows multiple objects present in an image to be detected in a single forward pass of the network. This allows single or multiple objects, such as one or more faces, to be quickly detected.

[0099] The fourth neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with portions of a user and the corresponding known outputs include motion data indicative of a series of positions of a face of the user. This causes the fourth neural network to become adept at recognizing the face of the user.

[0100] In step 250, the video data is processed to predict motion data indicative of a series of positions of a portion of a user during performance of an oral health care routine. In step 250, the portion of the user is the location of landmarks on the user's face. Processing is facilitated by providing the video data as input to a fifth neural network, which is trained to predict motion data indicative of a series of positions of landmarks on the user's face for the portion of the user associated with the video data. In one embodiment, the fifth neural network comprises a feature pyramid network. Feature pyramid networks are particularly good at recognizing small objects, such as nostrils and eyes, and are therefore well suited for detecting landmarks on the user's face. In one embodiment, at least 230 landmarks on the face are detected.

[0101] The fifth neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with portions of a user and the corresponding known outputs include motion data indicating a series of positions of landmarks on the user's face. This causes the fifth neural network to become adept at recognizing landmarks on the user's face. In one embodiment, the coordinates of the recognized landmarks on the user's face are saved to a memory for further analysis.

[0102] The split model separately identifies the location of the user's face and the location of the landmarks on the user's face, allowing for fast and accurate face tracking without the use of any specialized hardware (such as a depth sensing camera). By first identifying the user's face, the landmarks on the face can then be more quickly identified by the fifth neural network by searching only within the bounding box detected by the fourth neural network. An encoder-decoder architecture can be used to identify the landmarks.

[0103] In step 260, the motion data determined by steps 210, 220, 230, 240, and 250 are analyzed to determine at least one parameter value of the oral health care routine. This can be done in one of two ways:

[0104] i.) The motion data may be provided as input to a machine learning algorithm that is trained to predict at least one parameter value of an oral health care routine for an oral health care routine associated with the motion data. From the personal care device position, hand and facial landmark positions, and pattern matching, the machine learning algorithm may infer useful information. For example, in the case of a user brushing their teeth, the machine learning algorithm may predict the angle of the personal care device relative to the user's mouth.

[0105] In one embodiment, the machine learning algorithm includes a supervised classifier model. This is particularly useful for classifying detected behaviors such as brushing a specific area of ​​a user's mouth. The brushing area can be inferred based on personal care devices, hand and facial landmark coordinates, and pattern matching.

[0106] The machine learning algorithm is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include motion data associated with an oral health care routine and the corresponding known outputs include at least one parameter value of the oral health care routine. This enables the machine learning algorithm to become particularly good at determining the parameter value of the oral health care routine.

[0107] ii.) The motion data may be provided as input to a rule-based algorithm designed to predict at least one parameter value of the oral health care routine for the oral health care routine associated with the motion data. This allows the parameter value to be inferred from the motion data without the use of a machine learning algorithm, which may alleviate computational requirements. For example, if the toothbrush head is closer to the right edge of the oral cavity than the left edge, and the angle between the toothbrush and a line parallel to the plane of the face is 95 degrees, then it may be determined that brushing occurs on the right side of the oral cavity.

[0108] In one embodiment, analyzing the motion data by any method to determine at least one parameter value of the oral health care routine includes predicting a contact location between the personal care device and a surface of the user. For example, in the case of a user brushing their teeth, from the contact location between the personal care device and the surface of the user, the area where the brushing is occurring can be inferred. For example, it can be inferred that the user is currently brushing their left molars.

[0109] In one embodiment, analyzing the motion data to determine at least one parameter value of the oral health care routine includes predicting the distance between each palm of the user and the user's face. The predicted distance between each palm of the user and the user's face can then be compared to a predetermined threshold. This allows for detection of a user flossing because they need to bring both hands close to their mouth when flossing, but is not limited to this purpose.

[0110] See now Figure 3a and Figure 3b , which depicts simplified diagrams of tracking a personal care device according to a proposed embodiment. The diagrams depict a user performing an oral health care routine, such as brushing teeth. In both diagrams, a personal care device 320 is held up to the user's face 310. The top edge 340 and bottom edge 350 of the personal care device are used to find the angle 360 ​​of the personal care device relative to a line 330 that is parallel to the plane of the face and passes through the top edge.

[0111] By processing the video data to obtain motion data indicating a series of positions of the personal care device 320, the first CNN outputs a mask of the personal care device. In this example, the personal care device is a toothbrush. From the output mask, the top edge 340 (i.e., the minimum Y coordinate at the top of the video frame, Y=0) and the bottom edge 350 (i.e., the maximum Y coordinate) of the toothbrush can be found.

[0112] In one embodiment, the first CNN is a Mask-RCNN, but in this embodiment, the mask of the personal care device 320 generated by the Mask-RCNN may consist of many sub-masks. To overcome this problem and determine the top edge 340 and the bottom edge 350 of the personal care device, the following algorithm is implemented:

[0113] First, extract the top edge 340 by finding the minimum Y coordinate of the mask (the top of the video frame where Y=0) and finding its corresponding X coordinate. Second, extract the bottom edge 350 by finding the maximum Y coordinate of the mask and finding its corresponding X coordinate. Third, if there is more than one mask of the personal care device 320, iteratively find the top and bottom edges of each sub-mask and repeat the first two steps to reach the top and bottom edges of the personal care device.

[0114] After finding the top edge 340 and bottom edge 350 of the personal care device 320, the angle 360 ​​of the personal care device relative to the plane 330 of the face 310 can be found. The angle between the toothbrush and the face is used to determine the position of brushing. For example, if the user is brushing the sides of the mouth, the toothbrush is almost perpendicular to the plane of the face, and if the user is brushing the front of the mouth, the toothbrush is almost parallel to the face. In order to determine the angle, the top edge and the bottom edge of the toothbrush are connected with a straight line. Then, this line intersects with the reference line 330 parallel to the plane 310 of the face at the top edge, which enables the angle between the two lines to be determined using the dot product. This angle is equal to the angle between the toothbrush and the face.

[0115] Considering the two lines as two vectors, the angle between the two vectors can be calculated using the following formula (i):

[0116]

[0117] where a→.b→ is the dot product between two vectors, and |a→||b→| is the magnitude of the vectors.

[0118] In one embodiment, the output coordinates of the first to fifth neural networks are saved in a memory. The saved coordinates are then analyzed and a rule-based or machine learning-based method is used to predict the brushing position.

[0119] The memory may include any one or a combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic tape, compact disk read-only memory (CD-ROM), magnetic disk, floppy disk, magnetic cassette, cassette, etc.). In addition, the memory may incorporate electrical, magnetic, optical, and / or other types of storage media. Note that the memory may have a distributed architecture in which various components are located remote from each other but can be accessed by the processor.

[0120] In some embodiments, a computer program is provided comprising code means for implementing any of the methods described above when the program is run on a processing system.

[0121] See now Figure 4 , depicting a simplified block diagram of a system 400 for assisting a user's oral health care routine according to an embodiment. The system configured for assisting a user's oral health care routine includes an input interface 410 and one or more processors 430 .

[0122] The system 400 is configured to analyze the motion of a user and the motion of a personal care device during performance of an oral health care routine. A captured video 415 of a user performing an oral health care routine is obtained. The system outputs an output 450, which includes at least one parameter value of the oral health care routine. The output 450 is generated based on the analysis of the motion data and is used to provide feedback to the user regarding the performance of their oral health care routine.

[0123] In more detail, the input interface 410 receives video data 415 from a captured video of a user during performance of an oral health care routine. The input interface 410 provides the video data 415 to one or more neural networks 420 for predicting the location of the user's part(s) within the video data, and for predicting the location of the personal health care device within the video data. This facilitates motion pattern analysis of the oral health care routine.

[0124] Based on the identified positions, the processor(s) 430 then processes the video data using a motion detection algorithm to obtain motion data describing the motion of at least one of the personal care device and the part(s) of the user during the performance of the oral health care routine. To this end, a multi-person detection network can be used to avoid external interference by focusing only on the user performing the oral health care routine.

[0125] The resulting motion data is then processed by processor(s) 430 to generate an output 450 that includes at least one parameter value of the oral health care routine.

[0126] Figure 5 An example of a computer 500 is illustrated in which one or more portions of the embodiments may be employed. The various operations discussed above may utilize the capabilities of the computer 500. In this regard, it should be understood that the system functional blocks may be executed on a single computer, and may also be distributed across multiple computers and locations (e.g., connected via the Internet).

[0127] Computer 500 includes but is not limited to PC, workstation, notebook computer, PDA, handheld device, server, memory and the like. Generally, in terms of hardware architecture, computer 500 may include one or more processors 510, memory 520 and one or more I / O devices 530, which are communicatively coupled via a local interface (not shown). The local interface may be, for example but not limited to, one or more buses or other wired or wireless connections, as known in the art. The local interface may have other elements for implementing communication, such as controllers, buffers (cache), drivers, repeaters and receivers. In addition, the local interface may include address, control and / or data connections to implement appropriate communication between the above components.

[0128] Processor 510 is a hardware device for executing software that may be stored in memory 520. Processor 510 may be nearly any custom or commercially available processor, central processing unit (CPU), digital signal processor (DSP), or auxiliary processor among several processors associated with computer 500, and processor 510 may be a semiconductor-based microprocessor (in the form of a microchip) or a microprocessor.

[0129] The memory 520 may include any one or a combination of volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic tape, compact disk read-only memory (CD-ROM), magnetic disk, floppy disk, magnetic cassette, cassette, etc.). In addition, the memory 520 may incorporate electrical, magnetic, optical, and / or other types of storage media. Note that the memory 520 may have a distributed architecture in which various components are located away from each other but can be accessed by the processor 510.

[0130] The software in the memory 520 may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. The software in the memory 520 includes a suitable operating system (O / S) 550, a compiler 560, a source code 570, and one or more applications 580 according to an exemplary embodiment. As illustrated, the application 580 includes numerous functional components for implementing the features and operations of the exemplary embodiment. The application 580 of the computer 500 may represent various applications, computing units, logic, functional units, processes, operations, virtual entities, and / or modules according to exemplary embodiments, but the application 580 is not meant to be limiting.

[0131] The operating system 550 controls the execution of other computer programs and provides scheduling, input and output control, file and data management, memory management, and communication control and related services. The inventors contemplate that the application 580 used to implement the exemplary embodiments can be adapted to all commercially available operating systems.

[0132] Application 580 may be a source program, an executable program (object code), a script, or any other entity containing a set of instructions to be executed. When it is a source program, the program is typically translated via a compiler (such as compiler 560), an assembler, an interpreter, etc., which may or may not be included in memory 520 to operate correctly in conjunction with O / S 550. In addition, application 580 may be written in an object-oriented programming language having data and method classes, or a procedural programming language having routines, subroutines, and / or functions, such as, but not limited to, C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP scripts, JavaScript, FORTRAN, COBOL, Perl, Java, ADA, .NET, and the like.

[0133] I / O devices 530 may include input devices such as, for example, but not limited to, a mouse, keyboard, scanner, microphone, camera, etc. In addition, I / O devices 530 may also include output devices such as, but not limited to, a printer, a display, etc. Finally, I / O devices 530 may also include devices for transmitting input and output, such as, but not limited to, a NIC or a modulator / demodulator (for accessing remote devices, other files, devices, systems, or networks), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc. I / O devices 530 also include components for communicating over various networks such as the Internet or an intranet.

[0134] If the computer 500 is a PC, workstation, intelligent device, etc., the software in the memory 520 may also include a basic input and output system (BIOS) (omitted for simplicity). The BIOS is a set of basic software routines used to initialize and test the hardware at startup, start the O / S 550, and support the transfer of data between hardware devices. The BIOS is stored in some type of read-only memory, such as ROM, PROM, EPROM, EEPROM, etc., so that the BIOS can be executed when the computer 800 is started.

[0135] When the computer 500 is in operation, the processor 510 is configured to execute software stored in the memory 520, transfer data to and from the memory 520, and generally control the operation of the computer 500 according to the software. The application 580 and the O / S 550 are read in whole or in part by the processor 510, possibly buffered within the processor 510, and then executed.

[0136] When the application 580 is implemented in software form, it should be noted that the application 580 can be stored on substantially any computer-readable medium for use by or in connection with any computer-related system or method. In the context of this document, a computer-readable medium may be an electronic, magnetic, optical, or other physical device or component that can contain or store a computer program for use by or in connection with a computer-related system or method.

[0137] Application 580 may be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, a system containing a processor, or other system that can obtain instructions from an instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a "computer-readable medium" may be any component that can store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0138] Figure 1-Figure 2 Methods and Figure 4The system may be implemented in hardware or software, or may be implemented in a mixture of the two (e.g., as firmware running on a hardware device). In the context of an embodiment that is partially or entirely implemented in software, the functional steps illustrated in the flowchart may be performed by a suitably programmed physical computing device, such as one or more central processing units (CPUs) or graphics processing units (GPUs). Each process (and its individual component steps illustrated in the flowchart) may be performed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program, the computer program including a computer program code configured to cause one or more physical computing devices to perform the encoding or decoding method as described above when the program is run on one or more physical computing devices.

[0139] Storage media may include volatile and non-volatile computer memory, such as RAM, PROM, EPROM and EEPROM, optical disks (such as CD, DVD, BD), magnetic storage media (such as hard disks and tapes). The various storage media may be fixed within the computing device, or may be removable so that one or more programs stored thereon can be loaded into the processor.

[0140] Insofar as the embodiments are implemented partially or entirely in hardware, Figure 4 The blocks shown in the block diagrams may be separate physical components, or logical subdivisions of a single physical component, or may all be implemented in one physical component in an integrated manner. The functions of a block shown in the figure may be divided between multiple components in one implementation, or the functions of multiple blocks shown in the figure may be combined in a single component in one implementation. Hardware components suitable for use in embodiments of the present invention include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). One or more blocks may be implemented as a combination of dedicated hardware that performs certain functions and one or more programmed microprocessors and related circuits that perform other functions.

[0141] Those skilled in the art can understand and implement variations to the disclosed embodiments when practicing the claimed invention by studying the drawings, the disclosure and the attached claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "one" or "an" does not exclude a plurality. A single processor or other unit can implement the functions of several items stated in the claims. The fact that certain measures are stated in mutually different dependent claims does not in itself indicate that the combination of these measures cannot be fully utilized. If a computer program is discussed above, it can be stored / distributed on a suitable medium, such as an optical storage medium or solid-state medium provided with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems. If the term "suitable for" is used in a claim or specification, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". Any reference symbols in the claims should not be interpreted as limiting the scope.

[0142] The flowchart and block diagram in the figure illustrate the possible implementation architecture, functionality and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing (multiple) specified logical functions. In some alternative implementations, the functions indicated in the block may not occur in the order indicated in the figure. For example, the two blocks shown in succession can actually be executed substantially at the same time, or sometimes in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagram and / or flowchart and the combination of blocks in the block diagram and / or flowchart can be implemented by a dedicated hardware system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions. According to various embodiments of the present invention, the possible implementation architecture, functionality and operation of the system, method and computer program product. In this regard, each block in the flowchart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for implementing (multiple) specified logical functions. In some alternative implementations, the functions indicated in the block may not occur in the order indicated in the figure. For example, two blocks shown in succession may actually be executed substantially simultaneously, or may sometimes be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow charts, and combinations of blocks in the block diagrams and / or flow charts, may be implemented by a dedicated hardware system that performs the specified functions or actions or executes a combination of dedicated hardware and computer instructions.

Claims

1. A method for assisting a user's oral health care routine, the method comprising: obtaining (110) video data from a captured video of the user performing the oral health care routine using a personal care device; processing (120) the video data to obtain motion data describing motion of a portion of the user during performance of the oral health care routine; as well as analyzing (130) the motion data to determine at least one parameter value of the oral health care routine; Wherein the part of the user comprises a hand of the user.

2. The method of claim 1, wherein the at least one parameter value comprises at least one of: user deviation; a measure of completion of the oral health care routine; a measure of completion of a subroutine of the oral health care routine; and duration.

3. The method according to claim 1 or 2, wherein the obtained motion data also describes the motion of the personal care device.

4. The method of claim 3, wherein processing the video data to obtain motion data describing the motion of the personal care device comprises: providing the video data as input to a first convolutional neural network (CNN), the first CNN being trained to predict, for the personal care device associated with the video data, motion data indicative of a series of positions of the personal care device, And optionally, wherein the series of positions of the personal care device describes a zone of the personal care device.

5. The method of claim 4, wherein the first CNN is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with a personal care device and the corresponding known outputs include motion data indicating a series of positions of the personal care device.

6. The method of claim 4, wherein the first CNN is a pre-trained model that is further trained on videos of subjects using the personal care device that have been manually annotated.

7. A method according to any one of the preceding claims, wherein processing the video data to obtain motion data describing the motion of a part of the user comprises: providing the video data as input to a second neural network trained to predict, for the portion of the user associated with the video data, motion data indicative of a sequence of positions of a palm of a hand of the user; The video data is provided as input to a third neural network that is trained to predict, for the portion of the user associated with the video data, motion data indicative of a series of positions of a landmark on a hand of the user.

8. The method of claim 7, wherein the second neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with the portion of the user and the corresponding known outputs include motion data indicative of a sequence of positions of a palm of the user's hand; and wherein the third neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with portions of the user and the corresponding known outputs include motion data indicative of a series of positions of landmarks on the user's hand.

9. The method of claim 7 or 8, wherein the second neural network comprises a single-shot multi-box detector architecture and the third neural network comprises a feature pyramid network.

10. A method according to any one of the preceding claims, wherein processing the video data to obtain motion data describing the motion of a part of the user comprises: providing the video data as input to a fourth neural network, the fourth neural network being trained to predict, for the portion of the user associated with the video data, motion data indicative of a sequence of positions of the user's face; The video data is provided as input to a fifth neural network that is trained to predict, for the portion of the user associated with the video data, motion data indicative of a series of positions of landmarks on the user's face.

11. The method of claim 10, wherein the fourth neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with the portion of the user and the corresponding known outputs include motion data indicative of a sequence of positions of the user's face; and wherein the fifth neural network is trained using a training algorithm configured to receive an array of training inputs and corresponding known outputs, wherein the training inputs include video data associated with portions of the user and the corresponding known outputs include motion data indicative of a series of positions of landmarks on the user's face.

12. The method of claim 10 or 11, wherein the fourth neural network comprises a single-shot multi-box detector architecture and the fifth neural network comprises a feature pyramid network.

13. The method of any one of the preceding claims, wherein analyzing the motion data to determine at least one parameter value of the oral health care routine comprises: providing the motion data as input to a machine learning algorithm trained to predict at least one parameter value of the oral health care routine for the oral health care routine associated with the motion data, And optionally, wherein the machine learning algorithm comprises a supervised classifier model.

14. A computer program comprising code means for implementing the method of any preceding claim when said program is run on a processing system.

15. A system for assisting a user's oral health care routine, the system comprising: an input interface (410) configured to obtain video data from a captured video of the user performing a personal health care routine using a personal care device; A processor device (430), the processor device being configured to: processing the video data to obtain motion data describing motion of one or more parts of the user during performance of the oral health care routine; analyzing the motion data to determine at least one parameter value of the oral health care routine; Wherein the part of the user comprises a hand of the user.

Images