Region detection

Abstract

Apparatus for determining a region in a user's mouth that is being brushed by a toothbrush is disclosed. The apparatus comprises an accelerometer (32) arranged to sense acceleration of the toothbrush to produce accelerometer data, and a magnetometer (34) arranged to sense a magnetic field to produce magnetometer data. Means (48) are provided for combining the accelerometer data and the magnetometer data to produce orientation data indicating an orientation of the toothbrush. A region detection model (60) is arranged to determine a region of the mouth being brushed from the orientation data. The disclosed apparatus may allow the region of the mouth being brushed to be determined with a greater degree of granularity.

Description

[0001] REGION DETECTION

[0002] The present invention relates to a method and apparatus for determining a region in a user’s mouth that is being brushed with a toothbrush.

[0003] A toothbrush is an oral hygiene instrument used to clean the teeth and gums. It consists of a head of tightly clustered bristles mounted on a handle. In the case of a manual toothbrush, the brushing motion is produced by the user. In the case of an electric toothbrush, the toothbrush causes vibrations of the bristles in the brush head in order to clean the teeth and gums.

[0004] In order for a user to maintain good oral health it is important that all areas of the mouth are brushed correctly and for a reasonable amount of time. To help a user brush their teeth, devices have been developed which monitor how the user brushes their teeth and relay this information back to the user. The monitoring device may be integral with the toothbrush, or attachable to the toothbrush. Data from the monitoring device may be sent to an application (app) or computer program, which is run on an external device such as a mobile phone, tablet or other portable device. The app or program then provides feedback to the user about how they have brushed their teeth.

[0005] WO 2017 / 029469 A1 , the subject matter of which is incorporated herein by reference, discloses a manual toothbrush system which has an accelerometer fitted to a toothbrush holder. Acceleration data from the accelerometer are processed to determine whether the toothbrush is in one of six orientations. The determination of the orientation may be used as inputs to control a game on a mobile device.

[0006] WO 2019 / 034854 A1 , the subject matter of which is incorporated herein by reference, discloses a device for providing an indication of brushing activity of a toothbrush. The device comprises an accelerometer configured to produce acceleration data from motion of the toothbrush. A clustering process is performed to determine an area which is being brushed. Feedback can be provided to a user based on the determination of brushing area. US 2016 / 0143718 A1 , the subject matter of which is incorporated herein by reference, discloses a tooth brushing monitoring system that includes a toothbrush with sensors and a base station. The sensors record data regarding the quality, quantity and location of brushing and the system can analyse the data to provide feedback on the quality of brushing. Brushing data from each usage may be compared to reference data to evaluate its quality and determine the position of the toothbrush.

[0007] It has been found that the existing devices are able to determine a general area of the mouth which is being brushed. However, in order to improve the feedback provided to the user, it would be desirable to be able to detect the part of the teeth being brushed with a higher degree of granularity.

[0008] According to one aspect of the present invention there is provided apparatus for determining a region in a user’s mouth that is being brushed by a toothbrush, the apparatus comprising: an accelerometer arranged to sense acceleration of the toothbrush to produce accelerometer data; a magnetometer arranged to sense a magnetic field to produce magnetometer data; means for combining the accelerometer data and the magnetometer data to produce orientation data indicating an orientation of the toothbrush; and a region detection model arranged to determine a region of the mouth being brushed from the orientation data.

[0009] The present invention may provide the advantage that, by combining the accelerometer data and the magnetometer data to produce orientation data, and using a region detection model to determine a region of the mouth being brushed from the orientation data, it may be possible to determine an area of the mouth being brushed with a greater degree of granularity. This may allow better feedback to be provided to the user about their oral care.

[0010] The means for combining the accelerometer data and the magnetometer data may be embodied as an orientation (or attitude) determination unit. The means for combining the accelerometer data and the magnetometer data may comprise a sensor fusion algorithm. This may allow the orientation data to be produced with less uncertainty than would be the case if either of the sensors were used individually.

[0011] Raw magnetometer data, and to a certain extent raw accelerometer data, may vary between locations and user orientations. It has been found that, in order to provide a transferable region detection model that can be used in different locations and user orientations, it is desirable to obtain a relative measure of the brush orientation with respect to the user’s orientation. This can be achieved by obtaining an absolute measure of toothbrush orientation in three-dimensional space (i.e., relative to the earth’s frame of reference) which can then be rotated using a known user heading. Thus, the sensor fusion algorithm may be arranged to produce an estimate of the orientation of the toothbrush in three-dimensional space.

[0012] The sensor fusion algorithm may use a least squares optimization to combine the magnetometer data and the gravitational components. This may allow the accelerometer data and the magnetometer data to be combined in a computationally efficient manner. For example, in one embodiment the sensor fusion algorithm may use a Super-Fast Attitude Determination (SAAM) algorithm, or similar. However, other algorithms, such as Bayesian networks, Kalman filters, particle filters such Sequential Monte Carlo (SMC), or any other appropriate sensor fusion algorithm may be used instead or as well.

[0013] The sensor fusion algorithm may be arranged to produce an orientation quaternion uniquely defining the orientation of the toothbrush in three- dimensional space (relative to the earth’s frame of reference). This may facilitate subsequent processing of the orientation data.

[0014] In order to obtain an accurate estimate of the region of the mouth being brushed, it is desirable to know the orientation of the toothbrush relative to the user. Thus, the apparatus may further comprise storage means for storing a reference orientation indicating the user’s orientation in three-dimensional space, and rotating means for rotating the orientation data using the reference orientation to obtain an estimate of the orientation of the toothbrush relative to the user (heading). This may allow the input to the region detection model to be relative to the user’s orientation in space (the user’s frame of reference). This may facilitate accurate determination of the region of the mouth being brushed.

[0015] The apparatus may be arranged to obtain the reference orientation using a mobile processing device, such as a mobile phone or tablet. For example, the reference orientation may be obtained while the user is viewing a screen of the mobile processing device, so that the orientation of the user relative to the device is known. The reference orientation may be obtained, for example, during an onboarding process, or when the user starts using the toothbrush in a new location, or at any other time. The mobile processing device may contain additional and / or more accurate sensors (such as gyroscopes, accelerometers, magnetometers and / or satellite positioning sensors) and / or more powerfully processing than is available on a toothbrushing device, and thus may be able to provide a relatively accurate measure of the user’s orientation. Alternatively, it may be possible for the toothbrushing device itself to produce the reference orientation, for example, during an onboarding process.

[0016] The rotating means may be arranged to produce an orientation quaternion defining the orientation of the toothbrush in three-dimensional space relative to the user. This may facilitate the process of rotating the orientation data.

[0017] While quaternion representation is useful for data transformations such as rotations, Euler angles (roll, pitch and yaw) may be more useful for displaying data to the user and when continuous output is beneficial, such as when providing inputs to machine learning algorithms. Thus, the apparatus may further comprise means for converting the estimate of the orientation of the toothbrush to Euler angles.

[0018] When used to represent the orientation of the toothbrush, the Euler angles may contain discontinuities at -180° and 180° (corresponding to -IT and IT) which may cause similar angles to output distant values. It has been found that in practice this may have an impact on the performance of the region detection model. In embodiments of the invention, this problem is addressed by calculating a trigonometric function of an angle of the orientation data, and using the result as an input to the region detection model.

[0019] Thus, the apparatus may further comprise means for calculating a trigonometric function of an angle of the orientation data. For example, where the estimate of the orientation of the toothbrush comprises Euler angles, the apparatus may comprise means for calculating at least one trigonometric function of the Euler angles. The trigonometric function may be at least one of sine, cosine and tangent. The output of the trigonometric function (for example, the sine and / or cosine of the Euler angles of the orientation data) may be used as an input to the region detection model. Thus, the region detection model may be arranged to determine the region of the mouth being brushed based at least in part on the calculated trigonometric function. This may help to provide a continuous input to the region detection model, which may help to provide improved results.

[0020] The region detection model may be a mathematical model relating orientation data to the region of the mouth. For example, the model may comprise an algorithm for inferring the region of the mouth being brushed from the orientation of the toothbrush.

[0021] The region detection model may be a pre-trained machine learning model. The machine learning model may be generated by a supervised machine learning algorithm and may use computational methods to build the model directly from data without relying on a predetermined equation. This can allow the model to be readily adapted to different situations.

[0022] In principle it would be possible to use a neural network such as a recursive neural network (RNN) as a region detection model. These may produce good results, but may be computationally expensive and thus difficult to implement on a toothbrush device or a mobile device.

[0023] In one embodiment, the machine learning model is a classification model. Classification is a predictive modelling technique which involves assigning class labels to training data. It has been found that a classification model may produce accurate results using relatively low processing resources. However, other types of machine learning model, such as artificial neural networks (ANNs), decision trees, logistic regression (LR), support-vector machines (SVMs), Bayesian networks, Gaussian processes, or any other appropriate machine learning model, could be used instead or as well.

[0024] The machine learning model may be pre-trained using supervised learning. Supervised learning builds a mathematical model of a set of data that contains both the inputs and the desired outputs. In one embodiment, the machine learning model is trained in advance using data collected during a training process with the user. For example, the user may be invited to perform a training process as part of an onboarding process when they first use the toothbrush and / or at subsequent times, for example if their brushing habits change. During a training process, the user may be guided to brush different regions of the mouth, for example using an application on a mobile device. As the user brushes each region, accelerometer data and magnetometer data may be collected and annotated with the region of the mouth being brushed. The collected data together with the annotations may then be used to train the machine learning model.

[0025] Thus, the apparatus may be arranged to perform a training process in which accelerometer data and magnetometer data are collected and annotated with the region of the mouth being brushed. The machine learning model may then be trained using the collected data and the annotations. This may allow a robust model to be produced which is less sensitive to individual brushing habits.

[0026] During the training process, a reference orientation indicating the user’s orientation in three-dimensional space may be obtained, for example, from a mobile processing device such as a mobile phone, and stored in storage means. This may allow the apparatus to estimate the orientation of the toothbrush relative to the user during subsequent brushing sessions.

[0027] In principle, the machine learning model may be generated by the apparatus itself. However, due to the resources required, it may be desirable for the machine learning model to be generated by a remote device such as a backend server. Thus, the apparatus may further comprise means for transmitting data collected during a training process to a remote device and / or means for receiving the region detection model from a remote device.

[0028] While an individualised region detection model is likely to perform better, if desired, the apparatus may be provided with a generic region detection model instead or as well. For example, the apparatus may be provided with a generic region detection model when it is shipped. This may allow the user to use the apparatus without the need for a training session. If the user’s brushing habits do not deviate significantly from those assumed by the generic model, then this may allow reasonable results to be produced. However, the user may be invited to perform a training session in order to produce a more individualised model, for example, during an onboarding process, at a subsequent stage, or if inaccurate results are produced.

[0029] The accelerometer may be configured to produce acceleration data in three orthogonal directions. Likewise, the magnetometer may be configured to produce magnetometer data in three orthogonal directions. This may facilitate an estimation of toothbrush orientation in three-dimensional space.

[0030] The magnetometer will generally be arranged to sense a magnetic field as experienced by the toothbrush. This can allow the orientation of the toothbrush relative to the earth’s magnetic field to be estimated. However, magnetometers are susceptible to interference from local magnetic fields. Furthermore, although magnetometers may be calibrated in the factory, the desired calibration may change during assembly and / or over time. Thus, the apparatus may further comprise means for calibrating the magnetometer data. This may help to produce more accurate results.

[0031] Similarly, the desired calibration for an accelerometer may change during assembly or over time. Thus, the apparatus may further comprise means for calibrating the accelerometer data.

[0032] If desired, the (calibrated) accelerometer data and / or the (calibrated) magnetometer data may be normalised. Thus, the apparatus may further comprise means for normalising the magnetometer data and / or means for normalising the accelerometer data. Normalisation may be achieved by dividing the vectors by their magnitude. This may allow data to be processed in a form which is most suitable for subsequent stages such as sensor fusion algorithms and machine learning models, and thus may help to achieve the best results.

[0033] In order to allow orientation data to be obtained, the (calibrated and / or normalised) accelerometer data may be filtered to extract gravity components of the acceleration data. Thus, the apparatus may further comprise a filter arranged to filter the accelerometer data to produce accelerometer gravity data. The filter may be a low pass filter, and may remove some or all of the acceleration due to movement of the toothbrush from the accelerometer data, leaving a signal substantially consisting of the gravitational components. For example, in one embodiment, the filter may be a low pass filter arranged to pass frequencies below 6Hz, 5Hz, 4Hz, 3Hz, 2Hz or 1 Hz (for example, with a 3dB point at or below any of these values), although other values could be used instead. The accelerometer gravity data may give an indication of the orientation of the toothbrush relative to the Earth’s gravitational field. The accelerometer gravity data may be provided as an input to the combining means.

[0034] Similarly, the (calibrated and / or normalised) magnetometer data may be filtered to extract magnetic heading. Thus, the apparatus may further comprise a filter for filtering the magnetometer data to produce magnetic heading data. The filter may be a low pass filter, and may remove some or all of the components due to movement of the toothbrush from the magnetometer data, leaving a signal substantially consisting of the magnetic heading. For example, in one embodiment, the filter may be a low pass filter arranged to pass frequencies below 6Hz, 5Hz, 4Hz, 3Hz, 2Hz or 1 Hz (for example, with a 3dB point at or below any of these values), although other values could be used instead. The magnetic heading data may give an indication of the orientation of the toothbrush relative to the Earth’s magnetic field. The magnetic heading data may be provided as an input to the combining means.

[0035] Due to the low pass filtering, the accelerometer gravity data and magnetic heading data may provide long term averages which may provide relatively stable representations of the brush orientation in space. The accelerometer gravity data and the magnetic heading data may each comprise data in three orthogonal directions (for example, in the form of an accelerometer gravity vector and a magnetic heading vector).

[0036] If desired, two or more filters may be used to produce the accelerometer gravity data. For example, the accelerometer data could be filtered with a first low pass filter with a first (higher) cutoff frequency, and a second low pass filter with a second (lower) cutoff frequency to produce the accelerometer gravity data. Similarly, two or more filters may be used to produce the magnetic heading data. For example, the magnetometer data could be filtered with a first low pass filter with a first (higher) cutoff frequency, and a second low pass filter with a second (lower) cutoff frequency to produce the magnetic heading data. This may help to provide stable results and improve performance, although it may not be necessary. Furthermore, other filter configurations could be used instead.

[0037] In principle, it should be sufficient to use the orientation data as an input to the region detection model. However, it has been found that, in some circumstances, improved results may be obtained if one or more additional inputs to the region detection model are used.

[0038] For example, it has been found that, in some cases, improved results may be obtained if accelerometer gravity data (for example, the accelerometer gravity data provided as an input to the combining means) are also used as an input to the region detection model. Thus, in one embodiment of the invention, the region detection model is arranged to receive accelerometer gravity data and to determine the region of the mouth being brushed based further on the accelerometer gravity data.

[0039] It has been found that, in some cases, a ratio between a pair (and optionally each pair) of the three orthogonal components of the accelerometer gravity data (or equivalent) may be a useful feature for the region detection model. This ratio may be represented, for example, by the corresponding angle. Thus, the apparatus may further comprise means for calculating a value representing a ratio between a pair of orthogonal components of the accelerometer gravity data, for input to the region detection model. For example, the apparatus may comprise angle calculating means for calculating an angle corresponding to a ratio of a pair of orthogonal components (and optionally each pair of orthogonal components) of the accelerometer gravity data to produce accelerometer angle data. In this case, the region detection model may be arranged to determine the region of the mouth being brushed based further on the accelerometer angle data. It has been found that this may help to improve the performance of the region detection model.

[0040] The angle corresponding to the ratio of the pair of orthogonal components of the accelerometer gravity data may be calculated, for example, using an inverse trigonometric function, such as the arctangent function, which may be implemented, for example, using the Atan2 function.

[0041] In some cases, the accelerometer gravity data which are provided as an input to the combining means may also be provided as an input to the angle calculating means. This may avoid the need to provide a separate filter. However, in some embodiments, a separate (low pass) filter is provided to filter the accelerometer data to produce accelerometer gravity data for input to the angle calculating means. Thus, the apparatus may further comprise means for filtering the accelerometer data to produce accelerometer gravity data for input to the angle calculating means. This may allow separate tuning of the filters, which may help to improve results.

[0042] The apparatus may further comprise means for calculating a trigonometric function of the angle corresponding to the ratio of the pair of orthogonal components of the accelerometer gravity data. The trigonometric function may be at least one of sine, cosine and tangent. The output of the trigonometric function may be used as an input to the region detection model. This may help to provide a continuous input to the region detection model, which may help to provide improved results.

[0043] While ideally each brushing region is represented by a distinct brush orientation, in practice some brushing regions may be represented by similar or overlapping orientations. In such cases, statistical features may provide additional insights into the brushing motion and improve the accuracy of the region detection model. These statistical features can include features of any of the previously mentioned values, such as the accelerometer data and / or the orientation data, at any of the various stages of processing. Thus, the apparatus may further comprise means for calculating a statistical feature of at least one of the accelerometer data and the orientation data, and the region detection model may be arranged to determine the region of the mouth being brushed based at least in part on the calculated statistical feature. This may help to further improve the prediction accuracy.

[0044] The statistical feature may be any suitable measure of the properties of a dataset, which may comprise any appropriate number of samples. For example, the statistical feature may be a measure of the amount of variation of the values in the dataset about a mean or average value. Thus, the statistical feature may be standard deviation, or any other measure of variability such as average absolute deviation (AAD). However, any other appropriate statistical feature could be used such as a measure of central tendency (for example, mean, median or mode), measure of variability (for example, range, variance, standard deviation, skewness or kurtosis), or other statistical feature, for example, percentile, frequency distribution, probability distribution or confidence interval.

[0045] The statistical feature may be calculated from a dataset acquired over any suitable (predetermined) time period. For example, the statistical feature may be calculated over a sliding time window, which may comprise, for example, 10, 20, 30, 40, 50 or any other appropriate number of samples. Alternatively, the statistical feature may be calculated over all or part of a brushing session.

[0046] In one embodiment, the statistical feature is the standard deviation of the accelerometer data (for example, the calibrated or normalised acceleration data) and / or the standard deviation of a trigonometric function of an angle of the orientation data (for example, the standard deviation of the sine and / or cosine of the Euler angles of the orientation data). However, the statistical feature may be a feature of any of the other values described above (for example, the accelerometer gravity data or the Euler angles of the orientation data), at any of the various stages of processing. In any of the arrangements described above, the apparatus may be provided as part of a toothbrush device and optionally a separate mobile processing device such as a mobile phone or tablet. Thus, according to another aspect of the invention there is provided a toothbrushing system comprising a toothbrush device and optionally a mobile processing device, the toothbrushing system comprising apparatus in any of the forms described above. The toothbrush device may comprise a processor arranged to carry out any of the processing functions described herein.

[0047] Corresponding methods may also be provided. Thus, according to another aspect of the invention there is provided a method of determining a region in a user’s mouth that is being brushed by a toothbrush, the method comprising: sensing acceleration of the toothbrush using an accelerometer to produce accelerometer data; sensing a magnetic field using a magnetometer to produce magnetometer data; combining the accelerometer data and the magnetometer data to produce orientation data indicating an orientation of the toothbrush; and determining a region of the mouth being brushed from the orientation data using a region detection model.

[0048] The accelerometer data and the magnetometer data may be combined using a sensor fusion algorithm.

[0049] The method may further comprise calculating a statistical feature of at least one of the accelerometer data and the orientation data, and determining a region of the mouth being brushed based further on the statistical feature.

[0050] The method may further comprise collecting accelerometer data and magnetometer data during a training process, and annotating the collected accelerometer data and magnetometer data with the region of the mouth being brushed. During the training process, the user may be guided to brush different parts of the mouth. The user’s orientation may also be obtained during the training process. The method may further comprise training the region detection model using the collected accelerometer data, the collected magnetometer data and the annotations.

[0051] Features of one aspect may be used in combination with any other aspect. Any of the apparatus features may be provided as method features and vice versa.

[0052] Preferred features of the present invention will now be described, purely by way of example, with reference to the accompanying drawings, in which:

[0053] Figure 1 shows parts of a toothbrushing system;

[0054] Figure 2 shows parts of a toothbrush device;

[0055] Figure 3 illustrates four different orientations of a toothbrush;

[0056] Figure 4 shows accelerometer data;

[0057] Figure 5 illustrates how a user’s mouth may be divided into a number of different regions;

[0058] Figure 6 shows some physical components of a toothbrushing system in an embodiment of the invention;

[0059] Figure 7 shows parts of a region detection system in an embodiment of the invention;

[0060] Figure 8 shows an parts of a region detection process in an embodiment of the invention;

[0061] Figure 9 illustrates various stages in the processing of data from an accelerometer;

[0062] Figure 10 illustrates various stages in the processing of data from a magnetometer;

[0063] Figure 11 illustrates various stages in the processing of orientation data;

[0064] Figure 12 illustrates various stages in the processing of data to produce a region estimation;

[0065] Figure 13 shows parts of a toothbrush region detection system in another embodiment; and

[0066] Figure 14 shows an example of how feedback may be provided to the user. Overview

[0067] Figure 1 shows parts of a toothbrush system in an embodiment of the invention. Referring to Figure 1 , the system comprises a toothbrush device 10, an external processing device 30, and a backend processor and database 40. The toothbrush device 10 is in the form of an electric toothbrush or an attachment to a manual or electric toothbrush. The external device 30 is typically in the form of a mobile phone, tablet, laptop or similar portable computing device. The backend processor and database 40 are typically implemented using a server and storage device located at a remote location in the cloud.

[0068] In the arrangement of Figure 1 , the toothbrush device 10 has the ability to acquire toothbrushing data, potentially perform some initial data processing, and then transfer the data to the external device 30. The external device may perform further data processing and / or pass the data on to the backend processor 40. Aggregated data can be processed to extract useful information which can be presented to the user on the external device 30.

[0069] Systems such as that shown in Figure 1 can typically operate in either online or offline mode. In online mode, the toothbrush device 10 is connected to an external device 30 (such as a mobile phone), for example, using Bluetooth Low Energy. In this case, brushing times and data captured during brushing can be transferred directly to the external device. The external device can then run the algorithms for detecting the mouth regions which are being brushed.

[0070] On the other hand, offline mode allows use of the toothbrush without it being connected to an external device, which may be more convenient for the user. In this case, brushing times and data captured during toothbrushing may be stored on the toothbrush device 10. The data stored can then be transferred via BLE to a mobile device and to the cloud for further processing and long-term storage.

[0071] In the system of Figure 1 , the toothbrush device 10 may also have the ability to process the acquired brushing data, to store the processed data, and / or to present information to the user. Furthermore, algorithms for detecting the mouth regions which are being brushed may be transferred from the external device to the toothbrush device itself. This can allow the toothbrush device to know which parts of the mouth are being brushed. As a consequence, rather than storing raw sensor data, it may be possible to store the teeth segment being brushed at every time interval, together with their timestamps. This abbreviated data (compared to raw sensor data) can allow more brushing sessions to be stored and allow the brushing data to be transferred more quickly to the external device.

[0072] Figure 2 shows parts of a toothbrush device in an embodiment of the invention. Referring to Figure 2, the toothbrush device 10 in this embodiment is an electric toothbrush with a removable toothbrush head 12. The toothbrush comprises a housing which accommodates a brushing motor 14, a rechargeable battery 16, and a circuit board 18. The circuit board 18 comprises a plurality of sensors 20, a microcontroller 22 and a communications module 24, together with other components used for operating the toothbrush device. A button 26 is provided to turn the device on and off and to adjust the settings. In addition, a ring of LEDs 28 is provided.

[0073] In operation, the battery 16 supplies power to the brushing motor 14 to cause vibration of the toothbrush head 12. The sensors 20 sense motion of the toothbrush and provide sensor data to the microcontroller 22. The microcontroller executes an algorithm to estimate an area of the mouth being brushed from the sensor data. The area of the mouth being brushed may be stored in memory and / or indicated to the user using the LEDs 28. Alternatively or in addition, brushing data may be transferred to an external device using the communications module 24, and the external device may execute an algorithm to estimate an area of the mouth being brushed. The external device may also be used to provide feedback to the user regarding how they are brushing their teeth.

[0074] As an alternative, the toothbrushing device could be in the form of an attachment to either a manual toothbrush or an electric toothbrush. For example, the toothbrushing device could be in the form of a toothbrush holder such as that disclosed in WO 2017 / 029469. In this case, the circuit board 18 may be provided in the attachment, rather than in the toothbrush itself.

[0075] Existing toothbrush monitoring devices typically use an accelerometer to sense motion of the toothbrush. The acceleration data may be processed to determine the area of the mouth which is being brushed. However, the existing techniques are typically only able to distinguish between a limited number of areas. For example, in some cases, it is only possible to distinguish between four areas of the mouth, corresponding to four different orientations of the brushing head: up, down, left and right. These four orientations are illustrated in Figure 3.

[0076] Figure 4 illustrates acceleration data sensed by an accelerometer while brushing in a number of different regions of the mouth. The accelerometer is a three-axis accelerometer which produces an output in three orthogonal directions: x, y and z. From Figure 4, it can be seen that different regions may have very similar data. As a consequence, it is difficult to determine the region of the mouth being brushed with a high degree of granularity.

[0077] In order to improve the feedback provided to the user about their oral care, it would be desirable to detect the part of the teeth being brushed with a higher granularity. For example, Figure 5 illustrates how the mouth may be divided into a number of different regions. In this example, the mouth is divided into top teeth and bottom teeth. Each of the top teeth and bottom teeth are divided into left, right and centre parts. The centre parts are divided into inner surface and outer surface. The left and right parts are divided into inner surface, outer surface and chewing surface. This provides sixteen separately identifiable regions of the mouth. Of course, it will be appreciated that different regions and a different degree of granularity could be used instead.

[0078] In embodiments of the invention, techniques are provided which are able to determine which of the regions illustrated in Figure 5 (or a similar set of regions) is being brushed. To help achieve this higher granularity, it is beneficial to be able to orient the toothbrush with respect to the user’s orientation, in particular with respect to their jaws. One way of dealing with this is to place both in an “outside world” (Earth) reference frame and then to perform orientation transformations from absolute (Earth) to relative (with respect to jaws) orientation. The “outside world” reference may be obtained using an orientation sensor such as a magnetometer sensor. A magnetometer may be chosen for its low cost and low current consumption, which are important considerations for oral hygiene products, although other orientation sensors such as a gyroscope could be used instead or as well.

[0079] Figure 6 shows some physical components of a toothbrushing system in an embodiment of the invention. Referring to Figure 6, the system comprises accelerometer 32, magnetometer 34, microcontroller 22, communications module 24, and mobile device 30. The accelerometer 32, magnetometer 34, microcontroller 22 and communications module 24 may be provided as part of a toothbrush device 10 such as that shown in Figure 2, or as part of an attachment for connection to a toothbrush. The accelerometer 32 is a three-axis accelerometer which produces acceleration data in three orthogonal directions x, y and z. The magnetometer 34 is a three-axis magnetometer which produces magnetic field data in three orthogonal directions x, y and z. The microcontroller 22 includes a processor, memory and input / output peripherals. The communications module 24 is used to communicate with external devices such as the mobile device 30. The communications module 24 may use any suitable wireless or wired communications protocol, such as Bluetooth, WiFi, ZigBee, LoRa, or an optical transmission protocol. In some embodiments, the communications module 24 may be integrated with the microcontroller 22. The mobile device 30 may be, for example, a mobile phone, tablet, laptop, or any other suitable processing device.

[0080] In operation, the accelerometer 32 produces acceleration data caused by movement of the toothbrush. The acceleration data are fed to the microcontroller 22. At the same time, the magnetometer 34 senses the local magnetic field to produce magnetometer data. The magnetometer data are also fed to the microcontroller 22. The microcontroller 22 is programmed to execute algorithms which determine the orientation of the toothbrush in three-dimensional space from the acceleration data and the magnetometer data, as will be explained below. If desired, some or all of the processing may be performed using a processor 31 in the mobile device 30. The microcontroller 22 and / or mobile device 30 are also programmed to determine a region of the mouth being brushed from the orientation data, as will be explained below. The mobile device 30 may also be in communication with a backend device 40, for example via WiFi or a cellular network. In this case, some of the processing may be performed by the backend device. The backend device 40 may comprise a processor 41 and a database 42 along with other components such as a communications module.

[0081] In embodiments of the invention, a machine learning approach is used to infer the region of the mouth being brushed from the accelerometer and magnetometer data. The process consists of the following steps:

[0082] 1. Data collection

[0083] 2. Data (pre)processing

[0084] 3. Model generation (training)

[0085] 4. Region inference (prediction).

[0086] Data collection

[0087] In order to build a data model, it is necessary first to collect the data. This may be done initially as part of an onboarding process when the user first uses the toothbrush. During the onboarding process, an application is run on the mobile device, and the user is guided to brush different regions of the mouth. As part of this process, the user is instructed to hold the brush in a specific way, with visual reminder cues during the brushing session. Raw sensor data are collected from the toothbrush while also being annotated with the corresponding brushing region. In addition to raw accelerometer and magnetometer data from the toothbrush, a reference (user) orientation is captured at the start of the brushing session through the mobile device’s sensors and stored along with the session data. The collected data are forwarded to the backend device which stores the data for later processing.

[0088] During subsequent brushing sessions, the user may again be guided to brush different regions of the mouth using the application on the mobile device. The collected data may be annotated with the brushing region and forwarded to the backend device. This can allow the model to be updated over time.

[0089] Data processing

[0090] In order to obtain input features useful for region detection, the raw sensor data needs to be processed first. In embodiments of the invention, some or all of the following steps are carried out: • data clean up (removal of outliers, non-brushing data, region transitions)

[0091] • sensor calibration (accelerometer and magnetometer)

[0092] • sensor normalisation (removing offset and / or scaling factor)

[0093] • low pass filtering (gravity and smoothing)

[0094] • sensor fusion (obtaining orientation from accelerometer and magnetometer data)

[0095] • setting reference orientation necessary to obtain relative orientation

[0096] • calculating roll, pitch and yaw

[0097] • calculating gravity angles

[0098] • calculating sine and cosine of all angles (to remove discontinuities of angles at 0 and 360 deg)

[0099] • calculating statistical features, such as standard deviation of some of the above signals.

[0100] These steps are discussed in more detail below.

[0101] Model generation

[0102] In principle, a recurrent neural network (RNN) could be used for region prediction. However, RNNs tend to be computationally expensive, both for training and inference. Therefore, in embodiments of the invention, a machine learning classification approach is used.

[0103] The pre-processed and annotated samples are fed into a machine learning classifier. The machine learning classifier may be located at the backend device where sufficient processing and storage is available. However, it would also be possible for it to be located on the mobile device and / or the toothbrush device.

[0104] The machine learning classifier “learns” the labels (regions) from the data samples. This process can include training multiple models, tuning of model parameters and model validation - ensuring it performs well enough for all regions, and selecting the best performing one. This feedback loop can be also used to fine tune the feature generation and selection (preprocessing stage) to achieve the best performance. The generated model is agnostic to the user location and orientation. In embodiments of the invention, logistic regression and SVM classifiers are used for their good trade-off between performance and processing and memory requirements as well as easy and efficient microcontroller implementation for offline inference.

[0105] Region inference

[0106] The trained classification model can be used for region inference (prediction). In one embodiment, the trained classification model is downloaded to the mobile device. Real time samples from the accelerometer and magnetometer are transformed in a compatible manner in the preprocessing stage to generate features which are fed into the classification model. The output of the model is the predicted brushing region which can be used for user feedback.

[0107] In order to achieve accurate region inference, it is desirable for the toothbrush magnetometer to be calibrated and the reference orientation set.

[0108] Implicit magnetometer calibration and reference orientation setting

[0109] In an ideal scenario, the user would explicitly calibrate the magnetometer and set the reference orientation before each brushing session, in order to achieve the highest magnetometer and orientation accuracy and therefore the best model performance. However, under some real-life scenarios, magnetometer calibration data may be missing or inaccurate. In this case, it may still be possible to obtain (or infer) the magnetometer calibration implicitly, given magnetometer data from the whole or a substantial part of a brushing session which contains a wide range of device motion. By knowing the characteristics of the incomplete magnetometer data from a brushing session it is possible to utilise the symmetry of the device and brushing motion to extrapolate available data and obtain “good enough” magnetometer calibration. Reference orientation may be inferred in a similar fashion. Brushing sessions with the magnetometer calibrated (and the reference orientation set) in such way can then be used for model training. Region inference is also possible. However, region inference may only be available after part or all of the session has finished, whereas live inference may be available during brushing in case of manual calibration I reference orientation setting before a session is started. Region detection system

[0110] Figure 7 shows parts of a toothbrush region detection system in an embodiment of the invention. Referring to Figure 7, the region detection system comprises accelerometer 32, magnetometer 34, accelerometer calibration unit 36, accelerometer normalisation unit 37, low pass filter 38, smoothing filter 39, magnetometer calibration unit 44, magnetometer normalisation unit 45, low pass filter 46, smoothing filter 47, attitude determination unit 48, rotation unit 50, reference orientation unit 52, quaternion-to-Euler transformation unit 54, sine calculation unit 56, cosine calculation unit 58, region detection model 60, low pass filter 62, arctangent calculation unit 64, sine calculation unit 66, and cosine calculation unit 68. The accelerometer 32 and magnetometer 34 are provided as part of the toothbrush or toothbrush attachment. The other parts shown in Figure 7 may be implemented as one or more software routines executing on the microcontroller 22 or processor 31 shown in Figure 6 or if appropriate on another processor and / or as hardware or firmware.

[0111] In operation, as the user brushes their teeth, the accelerometer 32 produces acceleration data caused by movement of the toothbrush. The output of the accelerometer is a set of samples which can be expressed as: d = a + g + e where d is the raw (measured) acceleration data, a is the acceleration due to movement of the accelerometer, g is the gravity component, and e is an error term. In the case of a three-axis accelerometer, d, a, g and e are each three- dimensional vectors. The output of the accelerometer 32 is fed to the accelerometer calibration and normalisation unit 36.

[0112] Three-axis accelerometers are usually calibrated by the manufacturer in the factory. However, the factory calibration may change during assembly and over time. Therefore, for best performance, calibration of the accelerometer may be desirable. The accelerometer calibration unit 36 calibrates the output of the accelerometer to correct for errors. This is achieved using a calibration model where offset and scale factors are calculated and applied to the raw accelerometer data. The offset and scale factors may be calculated, for example, from test data sampled while moving the sensor multiple times around the x, y and z axes, or from brushing data, or in any other way. Suitable accelerometer calibration techniques are known in the art, and therefore are not described further. The calibrated acceleration data are then normalised (divided by the magnitude) in the accelerometer normalisation unit 37, to produce normalised accelerometer data A in three orthogonal directions (x, y and z).

[0113] The normalised acceleration data A are fed to the low pass filter 38. The low pass filter 38 filters the normalised acceleration data to remove components due to brushing motion. In this embodiment, an exponential moving average filter is used for its memory efficiency. This is a first-order infinite impulse response filter that applies weighting factors which decrease exponentially.

[0114] In one embodiment the low pass filter 38 is implemented using the following equations: where xtare input samples, ytare output samples, and a is the filter coefficient. In this embodiment, a = 0.3552, which is equivalent to a filter cut off frequency (3dB point) of about 5.78 Hz (at a sampling frequency of 20 Hz). The x, y and z components of the normalised accelerometer data are filtered separately. The output of the low pass filter 38 is an accelerometer signal with frequencies due to non-brushing motion removed. Thus, this signal contains a gravity component, and potentially some brushing motion components. The output of the low pass filter 38 is fed to smoothing filter 39.

[0115] The smoothing filter 39 is a low pass filter which is used to smooth out long term averages of the accelerometer signal. In this embodiment, the smoothing filter 39 is implemented as an exponential moving average filter with a filter coefficient a = 0.9177, giving a filter cut off frequency of around 0.29 Hz (at a sampling frequency of 20 Hz). The smoothing filter 39 removes substantially all of the acceleration components due to movement of the toothbrush, to produce filtered gravitational components GF of the acceleration data (filtered accelerometer gravity data) in three orthogonal directions. The filtered accelerometer gravity data GF give an indication of how the toothbrush is oriented with respect to the Earth’s gravitational field. The filtered accelerometer gravity data GF are fed to the attitude determination unit 48.

[0116] At the same time, the magnetometer 34 measures the strength of the local magnetic field at the toothbrush. The output of the magnetometer 34 is a set of samples of the magnetic field in three orthogonal directions x, y and z. The output of the magnetometer 34 is fed to the magnetometer calibration unit 44.

[0117] Magnetometers are usually calibrated by the manufacturer in the factory. However, the factory calibration may change during assembly and over time. Furthermore, magnetometers are susceptible to interference from local magnetic fields. Therefore, for best performance, calibration of the magnetometer is desirable.

[0118] The magnetometer calibration and normalisation unit 44 calibrates the output of the magnetometer 34 to correct for distortion and other errors. This is achieved using a calibration model where offset and scale factors are calculated and applied to the raw magnetometer data. The offset and scale factors may be calculated, for example, from test data sampled while moving the toothbrush multiple times around the x, y and z axes, or from brushing data, or in any other way. Suitable magnetometer calibration techniques are known in the art, and therefore are not described further. The calibrated magnetometer data are then normalised in the magnetometer normalisation unit 45, to produce normalised magnetometer data M in three orthogonal directions.

[0119] The normalised magnetometer data M are fed to the low pass filter 36. The low pass filter 46 filters the normalised magnetometer data to remove components due to non-brushing motion. In this embodiment, an exponential moving average filter is used with a filter coefficient a = 0.3552, giving a filter cut off frequency of approximately 5.78 Hz (at a sampling frequency of 20 Hz). The x, y and z components of the magnetometer data are filtered separately. The output of the low pass filter 38 is a magnetometer signal with frequencies due to non-brushing motion removed. Thus, this signal contains a magnetic heading component, and brushing motion components. The output of the low pass filter 46 is fed to smoothing filter 47.

[0120] The smoothing filter 47 is a low pass filter which is used to smooth out long term averages of the magnetometer signal. In this embodiment, the smoothing filter 47 is implemented as an exponential moving average filter with a filter coefficient a = 0.9177, giving a filter cut off frequency of around 0.29 Hz (at a sampling frequency of 20 Hz). The smoothing filter 47 removes substantially all of the motion components from the magnetometer signal. The smoothing filter 47 outputs filtered magnetic heading components HF of the magnetometer signal (filtered magnetic heading data) in three orthogonal directions. The filtered magnetic heading data HF give an indication of how the toothbrush is oriented with respect to the Earth’s magnetic field. The filtered magnetic heading data HF are fed to the attitude determination unit 48, along with the filtered accelerometer gravity data GF.

[0121] Raw magnetometer data varies between locations and user orientations, depending on the local magnetic field. For a transferable model it is therefore desirable to obtain a relative measure of the brush orientation with respect to the user’s orientation. This can be achieved by obtaining an absolute orientation of the brush in three dimensions and rotating it using a known user heading. The absolute measure of brush orientation can be obtained utilising sensor fusion algorithms which combine accelerometer and magnetometer data into an orientation quaternion, uniquely defining the brush orientation in space.

[0122] In the arrangement of Figure 7, the attitude determination unit 48 receives the filtered magnetic heading data HF along with the filtered accelerometer gravity data GF, and implements a sensor fusion algorithm to obtain an estimation of the toothbrush orientation OA in three-dimensional space. The sensor fusion algorithm uses least squares optimization to combine the filtered magnetic heading data and the filtered accelerometer gravity data. A suitable algorithm is the Super-Fast Attitude Determination (SAAM) algorithm described in the article Wu et al“ Super Fast Attitude Determination Algorithm for Consumer-Level Accelerometer and Magnetometer”, IEEE Transactions on Consumer Electronics, vol. 64, no. 3, August 2018, the subject matter of which is incorporated herein by reference. However, other types of least squares optimization and other types of sensor fusion algorithm could be used instead, including but not limited to Kalman filters, Bayesian networks, Dempster-Shafer algorithms, convolutional neural networks, Gaussian processes, etc.

[0123] By combining acceleration data and magnetometer data using a sensor fusion algorithm, a more accurate estimate of the absolute orientation OA can be obtained than would otherwise be the case. The output of the attitude determination unit 48 is an orientation quaternion, uniquely defining the toothbrush orientation in three-dimensional space.

[0124] In order to determine the region of the mouth being brushed with a high degree of granularity, it is beneficial to determine the orientation of the toothbrush with respect to the user, and in particular with respect to their jaws. In the arrangement of Figure 6, this is achieved using the rotation unit 50. The rotation unit 50 receives the absolute orientation OA from the attitude determination unit 48 and a reference orientation OREF from the reference orientation unit 52. The reference orientation O EF is a quaternion defining the direction the user is facing while brushing, i.e. their heading. The reference orientation OREF may be captured in advance during a training session using the mobile device’s sensors, or may be determined at the start of a toothbrushing session or at some other time. The rotation unit 50 rotates the reference frame of the absolute orientation quaternion OA using the reference orientation quaternion OREF to obtain the orientation of the toothbrush relative to the user’s orientation in space. The output of the rotation unit 50 is a relative orientation quaternion OR defining the orientation of the toothbrush relative to the user.

[0125] While quaternion representation is useful for data transformations such as rotations, Euler angles (roll, pitch and yaw) are more useful for displaying data to the user and when continuous output is beneficial, as is the case with inputs to machine learning classifiers. The quaternion-to-Euler transformation unit 54 receives the relative orientation quaternion OR and converts it to Euler angles. The conversion may be achieved using the following formula: where cp, 0 and i are roll, pitch and yaw. The output of the quaternion-to-Euler transformation unit 54 is the relative orientation OR specified as Euler angles (roll, pitch and yaw). This output is fed to the sine calculation unit 56 and the cosine calculation unit 58.

[0126] The Euler angles output from the quaternion-to-Euler transformation unit 54 may contain discontinuities at -1 and 1 (corresponding to -IT and IT), which may cause similar angles to output distant values. These discontinuities may be avoided by wrapping the angle features in sine and cosine. Both sine and cosine are used to ensure continuity. This can allow a continuous input to be provided to the region detection model 60, giving improved results.

[0127] In the arrangement of Figure 7, the sine calculation unit 56 calculates the sine of the roll, pitch and yaw angles from the quaternion-to-Euler transformation unit 54. Thus, the sine calculation unit 56 outputs the values sin(cp), sin(0) and sin(ip). Likewise, the cosine calculation unit 58 calculates the cosine of the roll, pitch and yaw angle from the quaternion-to-Euler transformation unit 54. The cosine calculation unit 58 outputs the values cos(cp), cos(0) and cos(ip). The sine and cosine values of the relative orientation are fed to the region detection model 60.

[0128] The region detection model 60 is a pre-trained classification model which relates toothbrush orientations to regions of the mouth. Real time samples from the accelerometer and magnetometer are transformed in the preprocessing stages described above to generate features which are fed into the classification model 60. The output of the region detection model 60 is the predicted brushing region R which can be used for user feedback.

[0129] In principle it should be sufficient to have relative orientation as the only input to the region detection model. However, it has been found that including additional features as inputs to the model may help to improve accuracy. In the arrangement of Figure 7, the filtered accelerometer gravity data GF from the smoothing filter 39 are also used as inputs to the region detection model 60. The filtered accelerometer gravity data GF give an indication of how the toothbrush is oriented with respect to the Earth’s gravitational field, and thus may be a useful classifier input in addition to the relative orientation.

[0130] In addition, it has been found that the ratio between each pair of the three orthogonal components of the filtered accelerometer gravity data may be a valuable classifier input. The ratio can be represented by the corresponding angle. The angle can be obtained using the arctangent function.

[0131] Still referring to Figure 7, the normalised accelerometer data A from the accelerometer normalisation unit 37 are also fed to the low pass filter 62. In this embodiment, the low pass filter 62 is implemented as an exponential moving average filter with a filter coefficient a = 0.9177, giving a filter cut off frequency of around 0.29 Hz (at a sampling frequency of 20 Hz). The low pass filter 62 removes the acceleration components due to movement of the toothbrush from the normalised accelerometer data. Thus, the low pass filter 62 outputs gravitational components of the acceleration data in three orthogonal directions (filtered accelerometer gravity data). The filtered accelerometer gravity data from the low pass filter are fed to the arctangent calculation unit 64.

[0132] The arctangent calculation unit 64 calculates the angle corresponding to the ratio of each pair of the three orthogonal components of the filtered accelerometer gravity data from the filter 62 (the “gravity angles”). For example, assuming the filtered accelerometer gravity data are in the form of vectors with values (x, y, z), then the arctangent calculation unit 64 may calculate the values:

[0133] However, it would be possible for some or all of the numerators and denominators to be the other way around. In this example the arctangent calculation unit 62 calculates the values 0xy, 0XZand 0yz(the gravity angles) using the Atan2 function. The output of the arctangent calculation unit 64 is fed to the sine calculation unit 66 and the cosine calculation unit 68.

[0134] The sine calculation unit 66 calculates the sines of the gravity angles from the arctangent calculation unit 64. Thus, the sine calculation unit 66 outputs the values sin(0xy), sin(0yz) and sin(0zx). The cosine calculation unit 68 calculates the cosines of the gravity angles from the arctangent calculation unit 64. The cosine calculation unit 68 outputs the values cos(0xy), cos(0yz) and cos(0zx). Wrapping the angle features in sine and cosine can help to avoid discontinuities which may occur in the output of the Atan2 function.

[0135] The sines and cosines of the gravity angles output from the sine calculation unit 66 and the cosine calculation unit 68 are fed to the region detection model 60, together with the sine and cosine values of the relative orientation and the filtered gravitational components GF. Including the gravity angles as inputs to the region detection model in this way may help to improve the accuracy of predictions.

[0136] In principle, the filtered magnetic heading components HF could also be used as inputs to the region detection model. However, as the local magnetic field will tend to vary, this may be less valuable as a classifier input.

[0137] It will be appreciated that the embodiment described above and shown in Figure 7 is given by way of example only, and various modifications are possible. For example, the values of the filter coefficients are given purely as examples, and may be changed as appropriate. Other types of filter could be used instead of moving average filters. In some cases, it may be possible to combine various filters such as the low pass filter 38 and the smoothing filter 39. If desired, the output of the smoothing filter 39 could be fed to the arctangent calculation unit 64, in which case the low pass filter 62 may be dispensed with. In some cases, one or both of the sine and cosine calculation units may be dispensed with, and / or the angles fed directly into the model. In some cases, the arctangent calculation unit 64 could be dispensed with, and ratios between pairs of the three orthogonal components of the accelerometer gravity data could be fed directly into the model. The arctangent calculation unit 62 may calculate different arctangent values from those shown. Various other components may be included as well as or instead of those shown. Furthermore, various other features could be used as inputs to the region detection model as well as or instead of those shown.

[0138] In the arrangement described above, the region detection model 60 is a pretrained classification model which relates the input variables to the region of the mouth being brushed. Classification is a supervised machine learning technique in which the model tries to predict the correct label of a given input data. The model is fully trained using training data and then evaluated on test data. It can then be used to perform predictions on new unseen data.

[0139] Each individual has their own brushing habits and it is challenging to achieve good system performance across a number of users with a single region detection model. While it is possible to use a generalised model, it has been found that personalised models perform better.

[0140] In embodiments of the invention, the region detection model 60 uses logistic regression and support vector machine (SVM) classifiers to estimate the relationship between the input variables and the region of the mouth being brushed. These have been found to achieve a good trade-off between performance and processing and memory requirements, and allow easy and efficient microcontroller implementation for offline inference. However, other machine learning approaches, such as artificial neural networks (ANNs), regression analysis, Bayesian networks, Gaussian processes, etc. may be used instead or as well.

[0141] In order to generate the region detection model 60, training data is first collected using an application running on the mobile device during an onboarding process. During this process, the user is guided to brush different regions of the mouth. Sensor data (accelerometer data and magnetometer data) are collected from the toothbrush and annotated with the corresponding bushing region. The annotated data (including calibration data and reference orientation) are transferred for example via WiFi or a cellular network to the backend device 40. At the backend device, the annotated samples are fed into a machine learning classifier which “learns” the labels (regions) from the data samples. This process can include training multiple models, tuning of model parameters and model validation. This can ensure it performs well enough for all regions, and allow the best performing one to be selected. The generated model is agnostic to the user location and orientation. The generated model is then downloaded to the mobile device (or potentially the toothbrush device) where it can be used for region inference.

[0142] If desired, a generalised model could first be provided on the mobile device, to allow the user to use the region detection functionality without the need for an onboarding process. The user could then be presented with the option of performing the onboarding process, which may allow more accurate results to be achieved.

[0143] Furthermore, the generated model may be updated over time as the user uses the toothbrush. For example, if the user carries out a coaching process, in which they are guided to brush different regions of the mouth using an application on the mobile device, then the brushing data collected in this way could also be annotated and uploaded to the backend device. The backend device may then use the newly annotated samples to update the generated model. The updated model may be downloaded to the mobile device and / or toothbrush device for region inference.

[0144] In order to be able to build a robust model, insensitive to brushing in different user orientations, it is desirable to know the direction the user is facing while brushing, i.e. their heading. The heading can be obtained through the mobile device’s sensors during the onboarding session. The thus obtained heading is stored as the reference orientation OREF in the reference orientation unit 52.

[0145] In some cases, it may be assumed that the user will subsequently brush their teeth while facing in the same direction, for example, facing a bathroom mirror. However, if the user’s orientation changes, then the reference orientation may be updated. For example, the user may be prompted to update their orientation using the mobile device, or to select between one of a plurality of pre-stored reference orientations. Due to constrained memory and processing capabilities of a microcontroller, for offline inference (using the toothbrushing device) the model may be converted to a more efficient format. Two classifiers can be implemented fairly easily while maintaining an ability for user customisation:

[0146] • LogisticRegression

[0147] • LinearSVC

[0148] However, other classifiers may be used instead or as well. Such classifiers are generally known in the art, and therefore not described in more detail here.

[0149] Region detection process

[0150] Figure 8 shows an overview of a region detection process in an embodiment of the invention. Referring to Figure 8, in step 100 training data is collected from the sensors in the toothbrush device. The training data may be collected as part of an onboarding process, in which the user is guided to brush different parts of their teeth. As part of this step, a reference orientation and data for calibrating the magnetometer and accelerometer may also be collected.

[0151] In step 102 the raw accelerometer data and magnetometer data are pre- processed in the manner described above, and annotated with the brushing region. The pre-processed and annotated data are uploaded to the backend device.

[0152] In step 104, the backend device uses the pre-processed and annotated data to generate the region detection model. The region detection model is then downloaded to the mobile device (or toothbrush device) for use in region inference.

[0153] In step 106, brushing data is collected as part of a brushing session. If desired, calibration data and / or orientation data may also be collected as part of this step. In step 108, the brushing data is pre-processed in the manner described above.

[0154] In step 110, the region detection model on the mobile device (or toothbrush device) is used to detect the region being brushed. The pre-processed brushing data may also be uploaded to the backend device. The backend device may use the brushing data to update the region detection model.

[0155] Figure 9 is a diagram illustrating various stages in the pre-processing of data from the accelerometer in one embodiment. Referring to Figure 9, the acceleration of the toothbrush is sensed by the accelerometer to produce raw accelerometer data 200 in three orthogonal directions x, y and z. The raw accelerometer data 200 are then calibrated to produce calibrated accelerometer data 202. The calibrated accelerometer data 202 are then normalised to produce normalised accelerometer data 204. The normalised accelerometer data are low pass filtered to remove components due to non-brushing motion and produce accelerometer gravity data 206. The accelerometer gravity data 206 are low pass filtered to produce filtered accelerometer gravity data 208. The filtered accelerometer gravity data 208 are output to the sensor fusion algorithm and the region detection model.

[0156] The normalised acceleration data 204 are also low pass filtered to produce filtered accelerometer data 210 in three orthogonal directions. The angles corresponding to the ratios of pairs of orthogonal components of the filtered accelerometer data 210 are calculated to produce accelerometer angle data (gravity angles) 212. The sines of the accelerometer (gravity) angles are calculated to produce sines of the accelerometer angles 214. The cosines of the accelerometer (gravity) angles are calculated to produce cosines of the accelerometer angles 216. The sines and cosines 214, 216 of the accelerometer angles are output to the region detection model.

[0157] Figure 10 is a diagram illustrating various stages in the pre-processing of data from the magnetometer in one embodiment. Referring to Figure 10, the magnetic field at the toothbrush is sensed by the magnetometer to produce raw magnetometer data 220 in three orthogonal directions x, y and z. The raw magnetometer data 220 are then calibrated to produce calibrated magnetometer data 222. The calibrated magnetometer data 222 are then normalised to produce normalised magnetometer data 224. The normalised magnetometer data 224 are low pass filtered to produce magnetometer heading data 226. The magnetometer heading data 226 are filtered to produce filtered magnetometer heading data 228. The filtered magnetometer heading data 228 are output to the sensor fusion algorithm.

[0158] Figure 11 is a diagram illustrating various stages in the processing of orientation data in one embodiment. Referring to Figure 11 , the filtered accelerometer gravity data 208 and the filtered magnetometer heading data 228 are input to a sensor fusion algorithm, which in this example is a SAAM estimator. The sensor fusion algorithm combines the filtered accelerometer gravity data 208 and the filtered magnetometer heading data 228 to produce an estimation of the toothbrush orientation in three-dimensional space. The output of the sensor fusion algorithm is an absolute orientation quaternion 230, uniquely defining the toothbrush orientation in three-dimensional space.

[0159] The absolute orientation quaternion 230 is then rotated in three-dimensional space using the reference orientation quaternion to produce a relative orientation quaternion 232 defining the orientation of the toothbrush relative to the user. The relative orientation quaternion 232 is then converted to Euler angles to produce relative orientation data 234 specified as Euler angles (roll, pitch and yaw). The sines of the relative orientation roll, pitch and yaw angles 234 are calculated to produce sines of the relative orientation angles 236. The cosines of the relative orientation roll, pitch and yaw angles 234 are calculated to produce cosines of the relative orientation angles 238. The sines 236 and cosines 238 of the relative orientation angles are output to the region detection model.

[0160] Figure 12 is a diagram illustrating stages in the processing of data to produce a region estimation in one embodiment. Referring to Figure 12, the filtered accelerometer gravity data 208, the sines of the accelerometer angles 214, the cosines of the accelerometer angles 216, the sines of the relative orientation angles 236 and the cosines of the relative orientation angles 238 are all input to the brushing region detection model. The brushing region detection model is a pre-trained classification model, which has been trained in advance using training data produced for example during an onboarding process with the user. The brushing region detection model may be trained, for example, using logistic regression and support vector machine (SVM) classifiers. Alternatively, the brushing region detection model may be a generic model which has been produced in advance for use by a user without the need for an onboarding process. The output of the brushing region detection model is a brushing region prediction 240. The brushing region prediction 240 indicates which brushing region of the mouth is currently being brushed. For example, the brushing region prediction 240 may indicate which of the sixteen regions shown in Figure 5 is being brushed, although other regions and other degrees of granularity could be used instead.

[0161] It will be appreciated that the various steps in Figures 8 to 12 are shown by way of example only, and the steps may be carried out in a different order, some of the steps may be omitted, and / or additional steps may be carried out.

[0162] Statistical features

[0163] While ideally each brushing region is represented by a distinct brush orientation, in practice some brushing regions may be represented by similar or overlapping orientations. In such cases, statistical features may provide additional insights into the brushing motion and improve the accuracy of the region detection model. These statistical features can be features of any of the previously mentioned values, for example, statistical features of the acceleration data and / or orientation data at any of the various stages of processing. A statistical feature may be, for example, a measure of the amount of variation of a value about its mean during a certain period of time, such as during a sliding time window or during a brushing session. As an example, a statistical feature may be a standard deviation, or any other appropriate measure of variability such as average absolute deviation (AAD). The statistical features can be fed into the region detection model as additional features to further improve the prediction accuracy.

[0164] Figure 13 shows parts of a toothbrush region detection system in another embodiment of the invention. Referring to Figure 13, the region detection system comprises accelerometer 32, magnetometer 34, accelerometer calibration unit 36, accelerometer normalisation unit 37, low pass filter 38, smoothing filter 39, magnetometer calibration unit 44, magnetometer normalisation unit 45, low pass filter 46, smoothing filter 47, attitude determination unit 48, rotation unit 50, reference orientation unit 52, quaternion-to-Euler transformation unit 54, sine calculation unit 56, cosine calculation unit 58, region detection model 60, low pass filter 62, arctangent calculation unit 64, sine calculation unit 66, and cosine calculation unit 68, all of which may be substantially in the form described above with reference to Figure 7. In addition, the toothbrush region detection system of Figure 13 includes an acceleration standard deviation calculation unit 70 and an orientation standard deviation calculation unit 72. The acceleration standard deviation calculation unit 70 and the orientation standard deviation calculation unit 72 are arranged to calculate standard deviation values which are used as additional inputs to the region detection model 60.

[0165] In operation, the toothbrush region detection system of Figure 13 operates essentially in the manner of that shown in Figure 7. However, in the system of Figure 13, the acceleration standard deviation calculation unit 70 receives the calibrated accelerometer data from the accelerometer calibration unit 36. The acceleration standard deviation calculation unit 70 calculates the standard deviation of the calibrated accelerometer data over a sliding time window. The sliding window may be, for example, 1 .0, 1 .5 or 2.0 seconds, or 20, 30 or 40 samples at a sampling rate of 20 Hz, although other values could be used instead. The standard deviation of the calibrated accelerometer data is fed to the region detection model 60, in addition to the other values described above with reference to Figure 7.

[0166] The orientation standard deviation calculation unit 72 receives the sine values of the relative orientation (sin(cp), sin(0), sin(ip)) from the sine calculation unit 56, and the cosine values of the relative orientation (cos(cp), cos(0) and cos(ip)) from the cosine calculation unit 58. The orientation standard deviation calculation unit 72 calculates the standard deviations of the sine and the cosine values. The standard deviations are calculated over a sliding time window, such as 1 .0, 1 .5 or 2.0 seconds, or 20, 30 or 40 samples at a sampling rate of 20 Hz, although other values could be used instead. The standard deviations of the sine and the cosine values are fed to the region detection model 60, in addition to the other values described above.

[0167] In general, for a dataset comprising a set of A / samples xi, ... XN, the standard deviation a may be calculated using the equation: where

[0168] The statistical features described above may capture subtle differences in brushing patterns that are characteristic of particular regions, even when the orientation data alone may be ambiguous. Thus, using the statistical features as additional inputs into the region detection model may help to further improve the prediction accuracy. It will be appreciated that the statistical features may also be used as part of the training process to generate and / or update the region detection model.

[0169] While Figure 13 shows the standard deviations of the calibrated accelerometer data and the standard deviations of the sine and cosine values of the relative orientation being used as additional inputs to the region detection model, any of the other values in the arrangement of Figure 13 could be used instead or as well. For example, the standard deviation of the normalised accelerometer data A, the standard deviation of the filtered accelerometer gravity data GF, or the standard deviation of the relative orientation OR could be used instead or as well. Furthermore, any other appropriate statistical feature, such any appropriate measure of variability or central tendency, or other statistical feature, such as percentile, frequency distribution, probability distribution or confidence interval, could be used instead or as well.

[0170] In any of the arrangements described above, the region of the mouth being brushed may be presented to the user during brushing, for example, using lights on the toothbrush and / or the screen of the mobile device. Alternatively or in addition, a brushing report may be produced at the end of a brushing session. The brushing report may contain information such as the time the user spent brushing each part of the mouth. If brushing pressure is available, then the pressure applied to each region of the mouth may also be recorded. The brushing pressure may be determined, for example, using the techniques disclosed in WO 2021 / 044129, the subject matter of which is incorporated herein by reference. The brushing report may be sent from the toothbrushing device 10 to the mobile device 30 to allow feedback to be provided to the user.

[0171] Figure 14 shows an example of how feedback may be provided to the user on the screen of the mobile device. Referring to Figure 14, the screen of the mobile device 30 is used to show a representation of the teeth in the user’s mouth. In one embodiment, each region of the mouth is displayed in a particular colour, with the colour representing the time the user has spent brushing that region. Of course, other ways of presenting the data, such as shading or numerical values, could be used instead or as well. If desired, pressure information for each region could also be displayed in a similar way. Preferred features of the invention have been described above with reference to various embodiments. However, it will be appreciated that the invention is not limited to these embodiments, and variations in detail may be made within the scope of the appended claims.

Claims

CLAIMS1 . Apparatus for determining a region in a user’s mouth that is being brushed by a toothbrush, the apparatus comprising: an accelerometer arranged to sense acceleration of the toothbrush to produce accelerometer data; a magnetometer arranged to sense a magnetic field to produce magnetometer data; means for combining the accelerometer data and the magnetometer data to produce orientation data indicating an orientation of the toothbrush; and a region detection model arranged to determine a region of the mouth being brushed from the orientation data.

2. Apparatus according to claim 1 , wherein the means for combining the accelerometer data and the magnetometer data comprises a sensor fusion algorithm.

3. Apparatus according to claim 2, wherein the sensor fusion algorithm is arranged to produce an estimate of the orientation of the toothbrush in three- dimensional space.

4. Apparatus according to claim 2 or 3, wherein the sensor fusion algorithm uses a least square optimization to combine the magnetometer data and the gravitational components.

5. Apparatus according to any of claims 2 to 4, wherein the sensor fusion algorithm is arranged to produce an orientation quaternion uniquely defining the orientation of the toothbrush in three-dimensional space.

6. Apparatus according to any of the preceding claims, further comprising: means for storing a reference orientation indicating a user’s orientation in three-dimensional space; and means for rotating the orientation data using the reference orientation to obtain an estimate of the orientation of the toothbrush relative to the user.

7. Apparatus according to claim 6, wherein the apparatus is arranged to obtain the reference orientation using a mobile processing device.

8. Apparatus according to claim 6 or 7, further comprising means for converting the estimate of the orientation of the toothbrush to Euler angles.

9. Apparatus according to any of the preceding claims, further comprising means for calculating a trigonometric function of an angle of the orientation data, wherein the region detection model is arranged to determine the region of the mouth being brushed based at least in part on the calculated trigonometric function.

10. Apparatus according to any of the preceding claims, wherein the region detection model is a mathematical model relating orientation data to the region of the mouth.11 . Apparatus according to any of the preceding claims, wherein the region detection model is a pre-trained machine learning model, such as a classification model.

12. Apparatus according to claim 11 , wherein the machine learning model is trained in advance with data collected during a training process with the user.

13. Apparatus according to 11 or 12, wherein the apparatus is arranged to perform a training process in which accelerometer data and magnetometer data are collected and annotated with the region of the mouth being brushed.

14. Apparatus according to any of the preceding claims, further comprising means for transmitting data collected during the training process to a remote device and / or means for receiving the region detection model from a remote device.

15. Apparatus according to any of the preceding claims, wherein the accelerometer is configured to produce acceleration data in three orthogonaldirections and / or the magnetometer is configured to produce magnetometer data in three orthogonal directions.

16. Apparatus according to any of the preceding claims, further comprising means for calibrating the magnetometer data and / or means for calibrating the accelerometer data.

17. Apparatus according to any of the preceding claims, further comprising means for normalising the magnetometer data and / or means for normalising the accelerometer data.

18. Apparatus according to any of the preceding claims, further comprising a filter arranged to filter the accelerometer data to produce accelerometer gravity data, wherein the accelerometer gravity data are provided as an input to the combining means.

19. Apparatus according to any of the preceding claims, further comprising a filter for filtering the magnetometer data to produce magnetic heading data, wherein the magnetic heading data are provided as an input to the combining means.

20. Apparatus according to any of the preceding claims, wherein the region detection model is arranged to receive accelerometer gravity data and to determine the region of the mouth being brushed based further on the accelerometer gravity data.21 . Apparatus according to any of the preceding claims, further comprising angle calculating means for calculating an angle corresponding to a ratio of a pair of orthogonal components of accelerometer gravity data to produce accelerometer angle data, wherein the region detection model is arranged to determine the region of the mouth being brushed based further on the accelerometer angle data.

22. Apparatus according to claim 21 , further comprising means for filtering the accelerometer data to produce accelerometer gravity data for input to the angle calculating means.

23. Apparatus according to claim 21 or 22, further comprising means for calculating a trigonometric function of the angle corresponding to the ratio of the pair of orthogonal components of the accelerometer gravity data, wherein an output of the trigonometric function is used as an input to the region detection model.

24. Apparatus according to any of the preceding claims, further comprising means for calculating a statistical feature of at least one of the accelerometer data and the orientation data, wherein the region detection model is arranged to determine the region of the mouth being brushed based at least in part on the calculated statistical feature.

25. A toothbrushing system comprising a toothbrush device and optionally a mobile processing device, the toothbrushing system comprising apparatus according to any of the preceding claims.

26. A method of determining a region in a user’s mouth that is being brushed by a toothbrush, the method comprising: sensing acceleration of the toothbrush using an accelerometer to produce accelerometer data; sensing a magnetic field using a magnetometer to produce magnetometer data; combining the accelerometer data and the magnetometer data using a sensor fusion algorithm to produce orientation data indicating an orientation of the toothbrush; and determining a region of the mouth being brushed from the orientation data using a region detection model.

Images