Wearable device and method for detection of mouth movements

Abstract

Wearable device and method for detection of mouth movements This wearable device (1) worn by a user for detection of mouth movements of said user, wherein said wearable device (1) comprises: at least one sensor (2) disposed in an auricular and / or temporoparietal region of a head of said user, said at least one sensor (2) being configured to monitor mouth movements of said user, so as to obtain signal data; a processing unit, configured to analyze said signal data and to extract at least one parameter representative of said mouth movements

Description

[0001] Wearable device and method for detection of mouth movements

[0002] FIELD OF THE INVENTION

[0003] This invention relates to the field of wearable devices and methods to detect mouth movements. In particular, it refers to wearable devices able to detect any activity involving the temporal muscle and / or the auricular muscles and / or the temporoparietal muscle, especially chewing, speaking, drinking or yawning patterns.

[0004] BACKGROUND OF THE INVENTION

[0005] There has been a surge these last few years in the use of smart devices with embedded sensors to help users monitor their health and general wellbeing. More and more of these devices include the use of machine learning (ML) algorithms to analyze the data collected by the devices, detect patterns, and generate personalized recommendations.

[0006] Many diet management mobile applications and services, like “myfitnesspal”, have been developed to track a user’s eating habits and offer health advice. However, most of them require the user to manually input on a daily basis a wide range of information, including the nature and amount of the food taken. This task must be meticulously undertaken by the user and may become over time quite bothersome. These problems often lead to a premature stop of the use of diet management services. There is therefore a need for a device that would automatize the recording of a user’s food and beverage intake.

[0007] US2019038186 discloses a device to detect chewing and swallowing, comprising a sensor, preferably a piezo sensor, positioned on the bridge of the frame of glasses and able to sense vibrations. The device includes a chewing analyzer and a drinking analyzer that aim to detect the type of aliment or drink consumed using pattern analysis of the vibrations recorded. This solution may however be difficult to implement since patterns from different aliments may be similar and no method to properly differentiate them is suggested.

[0008] US2019171282 describes a head mounted display with an embedded acceleration sensor able to determine whether a user is chewing or speaking. A fatigue degree of the user may also be estimated, notably from an image of the food and cutlery used, acquired thanks to a camera embedded in the head mounted display. Nonetheless, acceleration sensors have a high-power consumption, which limits their use in a head mounted display that would be worn continuously by a user. The device of US2019171282 also offers very little information about the eating habits of the user and may not record for example his calorie intake.

[0009] Document US 2021 / 369187 Al discloses a non-contact chewing sensor and portion estimator.

[0010] Document US 2021 / 345959 Al discloses systems and methods for monitoring food intake.

[0011] Document US 2022 / 415476 Al discloses a wearable device and system for nutritional intake monitoring and management.

[0012] Document US 2023 / 335253 Al discloses devices, systems and methods, including augmented reality eyewear, for estimating food consumption and providing nutritional coaching.

[0013] There is thus a need for a device with low power consumption that would be able to detect, monitor and analyze mouth movements of a user.

[0014] SUMMARY OF THE INVENTION

[0015] To this end, the invention proposes a wearable device worn by a user for detection of mouth movements of said user, according to claim 1.

[0016] By using at least one sensor situated in an auricular and / or temporoparietal region of a head of a user and thus able to sense repetitive movements of auricular and / or temporal and / or temporoparietal muscles, the invention makes it possible to detect and monitor the mouth movements of the user. The integration of a processing unit in the device ensures the wearable device is able to analyze the signal data obtained from the at least one sensor.

[0017] The wearable device of the invention is also non-invasive.

[0018] User

[0019] A “user” refers to any human being to whom the wearable device may be fitted. The mouth movements of the user may be associated with different specific activities, especially chewing / eating, swallowing / drinking, speaking, yawning, and grinding one’s teeth. The invention helps the user monitor his diet and hydration level, by detecting chewing and swallowing sequences, his tiredness, by detecting yawning patterns, and his stress and anxiety, by detecting sequences in which the user grinds his teeth.

[0020] Wearable device

[0021] The wearable device may be a head-worn device, notably an eyewear. Preferably, the wearable device is an eye wear, for instance a pair of glasses or an eyeglasses clip.

[0022] The wearable device may comprise a non-transitory memory unit, for instance a random-access memory (RAM), configured to store the signal data and the extracted at least one parameter.

[0023] The wearable device may further comprise a parameter transfer unit, configured to transfer said extracted at least one parameter to a remote device or to its associated cloud backend.

[0024] The parameter transfer unit may use one technology of the Bluetooth, Wi-Fi or Radio technologies. In another non-limiting embodiment, the parameter transfer unit may use cellular communications.

[0025] The remote device may be external. It may be another smart wearable device, like a smart watch. It may be one of a mobile phone, a tablet computer, a microcomputer, a personal computer or a computer display.

[0026] In a non-limiting embodiment, the remote device may comprise a user-interface. The user-interface may display the extracted at least one parameter. It may use at least one graph and / or at least one image to display information. The user of the interface may be a different person from the user wearing the device. The user may be a decisionmaker of various nature. For instance, the user of the interface may be a clinician.

[0027] The remote device may be configured to compare the at least one parameter to reference parameters and generate recommendations. The remote device may display these recommendations on the user-interface.

[0028] The wearable device may comprise at least one camera and / or video camera.

[0029] Sensor The at least one sensor may monitor mouth movements of the user by detecting movements of the auricular muscles and / or the temporal muscle and / or the temporoparietal muscle, situated in the auricular and temporoparietal regions of the head of the user. The at least one sensor may detect movements of other muscles involved in the mastication process, for example the masseter or the lateral pterygoid or the medial pterygoid. The device of the invention aims to detect thanks to the at least one sensor, any repetitive movements of these muscles.

[0030] The at least one sensor may be disposed between an outer canthus of an eye of the user and a top of the ear which is the closest one to said eye.

[0031] The at least one sensor may be disposed at a distance from at least one temple of the user, which is below 3 cm, better below 2 cm, even better below 1 cm. Such a distance allows the sensor to be able to detect the movements of the auricular and / or temporal and / or temporoparietal muscles with good accuracy.

[0032] In an embodiment of the invention where the device is a pair of glasses, the at least one sensor may be embedded in one of its arms, preferably on a portion of the arm extending between the outer canthus of an eye of the user and a top of the ear which is the closest one to said eye, in particular on a portion of the arm in contact with one temple of the user.

[0033] The at least one sensor may be an optical sensor.

[0034] The at least one sensor may be a capacitive sensor. A “capacitive sensor” refers to a non-contact device able to sense the presence or measure the position or change of position of a target. The at least one capacitive sensor may be a capacitive proximity sensor. The capacitive proximity sensor can detect the presence of nearby targets without any physical contact by emitting an electromagnetic field and monitoring for changes in the field.

[0035] Capacitive sensors may be formed by two electrodes. The capacitance of capacitive sensors may vary depending on the distance between the two electrodes.

[0036] A temple area of the user may act as the first electrode of the capacitive sensor. The second electrode may be embedded in a part of the wearable device located near the temple area of the user acting as the first electrode. When the wearable device is an eyewear, especially a pair of glasses, the second electrode may be embedded in one of its arms.

[0037] The second electrode may be formed by a copper plate, preferably embedded in a printed circuit board integrated into the capacitive sensor. The copper plate may have a surface bigger than 100 mm2. The bigger the surface of the copper plate is, the more sensitive the sensor is.

[0038] When the auricular and / or temporal and / or temporoparietal muscles of a user are used, the distance between the user’s temple acting as the first electrode and the second electrode embedded in the wearable device changes. The capacitive sensor may detect movements of the auricular and / or temporal and / or temporoparietal muscles by measuring the variation of its capacitance implied by any variation of the distance between the user’ s temple acting as the first electrode and the second electrode embedded in the wearable device.

[0039] Capacitive sensors are particularly advantageous because they are low power, consuming far less energy than accelerometers or optical sensors for example. They are moreover highly precise, being able to measure distances of only a few millimeters. They also require less processing time from the processing unit, than optical sensors or accelerometers.

[0040] The wearable device may comprise several sensors of the same type. Alternatively, the wearable device may comprise several sensors of different types.

[0041] In a particular embodiment of the invention, the wearable device comprises only one sensor, which is a capacitive sensor.

[0042] The at least one sensor may be further configured to detect a worn state of said wearable device.

[0043] Many smart wearable devices already include an optical sensor or a capacitive sensor to detect a worn state of the device. Configuring an optical sensor or a capacitive sensor included in a wearable device for this purpose to also detect mouth movements would bring new features to the device without increasing its ecological footprint, while limiting power consumption.

[0044] Signal data obtained from sensors & parameter representative of mouth movements

[0045] The signal data may be one dimensional or multi-dimensional. The signal data may be continuous or discrete. The signal data may be time-dependent, meaning that it is collected by the at least one sensor in a successive manner. In an embodiment of the invention where the at least one sensor is a capacitive sensor, the amplitude of the signal data may be proportional to the intensity of the demand on the auricular and / or temporal and / or temporoparietal muscles. The greater the demand on the muscles is, the closer the electrodes of the at least one capacitive sensor are, and the higher the amplitude of the signal data is. In a chewing sequence, the demand on the muscles may depend on the density of the food chewed.

[0046] The signal data may be pre-processed before being analyzed by the processing unit, for instance it may be pre-processed for noise removal. The pre-processing step may include filtering the signal data.

[0047] The signal data may be analyzed live by the processing unit. The signal data and / or the at least one parameter extracted may be stored in a non-transitory memory unit of the wearable device. By “at least one parameter representative of said mouth movements”, it is meant that an assessment of an activity associated with said mouth movements can be inferred from the at least one parameter. The at least one parameter may be compared for instance with reference values for the activity associated with said mouth movements. As defined previously, activities include chewing, swallowing, speaking, yawning, and grinding one’s teeth.

[0048] The at least one parameter may be included in a group comprising frequency and duration of the mouth movements.

[0049] When the activity associated with said mouth movements corresponds to a chewing sequence, the at least one parameter may be included in a group comprising an average number of chewing per minute, a total number of che wings, a total chewing duration during food intake, a chewing intensity, and a timestamp of the chewing sequence.

[0050] A low average number of chewing per minute expresses a slow chewing, which is beneficial to avoid weight taking and to facilitate digestion as showed in the article “ Improvement in chewing activity reduces energy intake in one meal and modulates plasma gut hormone concentrations in obese and lean young Chinese men” by Li et al. The total number of chewings and the total chewing duration during food intake also have an interest as eating too quickly will impact weight loss / taking. Health guidelines recommend, based on studies like “ Effects of Longer Seated Lunch Time on Food Consumption and Waste in Elementary and Middle School-age Children, A Randomized Clinical Trial” by Burg et al, that the main food intake during a day lasts at least for 20 minutes. Finally, the moment of a main food intake has an influence on the user’s ability to lose weight as late meals tend to make this process harder as demonstrated in “Timing of Breakfast, Lunch, and Dinner. Effects on Obesity and Metabolic Risk" by Lopez-Minguez et al.

[0051] The chewing intensity can be defined as the amplitude of the signal data in a chewing sequence. As explained previously, the amplitude of the signal data is proportional to the demand on the muscles.

[0052] Processing unit

[0053] The processing unit may comprise at least one processor. In addition, it may comprise one or several general or special purpose microprocessors, microcontrollers or both, or any other kind of CPU, GPU, FPGA, quantum processor and any combination of those devices.

[0054] The processing unit may be configured to detect with at least one algorithm one or several sequences among the signal data associated with a specific activity of the user, in particular it may be configured to detect chewing and / or swallowing sequences.

[0055] The invention is not limited to a specific type for the at least one algorithm. The at least one algorithm may be a pattern matching algorithm. It may be based on at least one ML algorithm. The invention is not limited to a specific type or architecture for the at least one ML algorithm.

[0056] The processing unit may be configured to send an alert to the remote device when detecting a beginning of a chewing sequence. The alert may trigger an action of the remote device, for instance display an advisory message.

[0057] The processing unit may be configured to trigger the activation of the at least one camera and / or video camera in the wearable device when detecting a beginning of a chewing sequence.

[0058] The at least one video camera may be configured to start recording what the user is watching when activated.

[0059] The at least one camera may be configured to take at least one picture when activated. The at least one camera may be configured to take at least one picture when the head of the user is lowered when activated, in order to better image a table and / or a plate in front of the user. The wearable device may comprise a sensor to detect when the head of a user is lowered. This sensor may be an accelerometer or an inertial measurement unit comprising a gyroscope.

[0060] The processing unit may be configured to run at least one ML algorithm, notably a neural network (NN), to analyze one or several recordings by the at least one video camera and / or one or several pictures by the at least one camera, in order to detect aliments and / or beverages, identify respective types and assess a quantity of these aliments and / or beverages. The processing unit may combine results of the at least one ML algorithm with at least one extracted parameter representative of the chewing sequence, for instance chewing time or chewing intensity, to obtain more accurate results. The chewing time and chewing intensity vary depending on the type of food, for instance a meat will take longer to chew, and the chewing intensity will be higher than when eating a fish.

[0061] The invention is not limited to a specific type or architecture for the at least one ML algorithm. However, some types may be particularly well adapted for processing images. Preferably, the at least one ML algorithm is based at least in part on a convolutional neural network (CNN). The at least one ML algorithm may be one of the algorithms described in the article “Applications of Deep Learning in Food : review”, by Zhou et al, Compr Rev Food Sci Food Saf. 2019 Nov;18(6):1793-1811.

[0062] By detecting, identifying, and assessing the quantity of aliments and / or beverages, the processing unit may be able to assess a global calorie intake of a chewing and / or swallowing sequence. The processing unit may be able to estimate a quantity of one of carbohydrate, lipid, protein, salt, saturated fat, and unsaturated fat in the aliments and / or beverages. The processing unit may detect the presence of allergens, for instance gluten. Besides, the processing unit may estimate the CO2 impact of the aliments and / or beverages.

[0063] The processing unit may be configured to transfer part, or all the information described above to the remote device. The remote device may be configured to display part, or all of this information.

[0064] The remote device may be configured to generate recommendations based on information sent by the parameter transfer unit. The remote device may display these recommendations via a user interface.

[0065] Method Another aspect of the invention relates to a method for detection of mouth movements of an individual by means of a wearable device worn by said individual, according to claim 10.

[0066] The advantages of the method are similar to those of the device and are thus not repeated here.

[0067] The method may comprise, before the step of extracting the at least one parameter, detecting with at least one algorithm one or several sequences among the signal data associated with a specific activity of the user, in particular detecting chewing and / or swallowing sequences.

[0068] The method may comprise starting recording with at least one video camera and / or taking at least one picture with at least one camera embedded in the wearable device, after a detection of a beginning of a chewing sequence.

[0069] The method may comprise running at least one ML algorithm, notably a neural network (NN), to analyze one or several recordings by the at least one video camera and / or one or several pictures by the at least one camera, in order to detect aliments and / or beverages, identify respective types and assess a quantity of these aliments and / or beverages.

[0070] The method may further comprise transferring the extracted at least one parameter to a remote device, by means of a parameter transfer unit included in the wearable device.

[0071] The method may further comprise detecting a worn state of said wearable device, by means of said at least one sensor.

[0072] Non-transitory memory unit

[0073] Another aspect of the invention relates to a non-transitory memory unit, storing one or more sequences of instructions that are accessible to a processor and that, when executed by said processor, cause said processor to implement a method as previously defined.

[0074] Computer program product

[0075] Another aspect of the invention relates to a computer program product, comprising one or more sequences of instructions that are accessible to a processor and that, when executed by said processor, cause said processor to implement a method as previously defined.

[0076] BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Other features and advantages of the invention will become clear upon reading the following detailed description and by examining the attached drawing, in which:

[0078] Figure 1 is a view in perspective of the wearable device according to the invention,

[0079] Figure 2 is an example of signal data analyzed by the processing unit of the wearable device of Figure 1, and

[0080] Figure 3 is a schematic representation of the wearable device and its connection to the remote device.

[0081] DETAILED DESCRIPTION

[0082] Using Figures 1 to 3, the functioning of a wearable device according to the invention is explained.

[0083] The Figure 1 represents in perspective a head-worn device 1 according to the invention. The head-worn device 1 is a pair of glasses.

[0084] The head-worn device 1 comprises a capacitive sensor 2. The capacitive sensor 2 has two electrodes. The first electrode is a temple area of the user wearing the head- worn device 1. The second electrode is a thin copper plate embedded in a printed circuit board in the capacitive sensor 2. The capacitive sensor 2 is embedded in one of the arm of the pair of glasses, in a portion of the arm intended to be in contact with the temple of the user acting as the first electrode, so that the capacitive sensor 2 may be able to detect the movements of the auricular and / or temporal and / or temporoparietal muscles with good accuracy. The capacitive sensor 2 is in addition configured to detect when a user is wearing the device 1.

[0085] When monitoring mouth movements of a user, the capacitive sensor 2 outputs signal data 8, as illustrated in Figure 2. The signal data 8 is stored in a non-transitory memory unit 11, for example a RAM, of the wearable device 1.

[0086] The head-worn device 1 comprises a processing unit 3, for instance made up of one processor, as depicted in Figure 3. The processing unit 3 analyzes the signal data 8 live using a pattern matching algorithm and identifies successively a chewing sequence 9 and a yawning sequence 10. From each of these sequences 9 and 10, the processing unit 3 extracts at least one parameter representative of the sequence, stored in the non-transitory memory unit 11.

[0087] The head- worn device 1 comprises a parameter transfer unit 4, that transfers the parameters to a remote device 6. The remote device 6 displays the parameters on a userinterface 7.

[0088] When the beginning of the chewing sequence 9 is identified by the processing unit 3, the processing unit 3 triggers the activation of a camera 5 embedded in the wearable device 1, as shown in Figure 3. The camera 5 takes a picture of a plate of the user when his head is lowered. The picture is stored in the non-transitory memory unit 11. The processing unit 3 detects, identifies, and quantifies aliments and beverages in the picture using a machine learning algorithm based on a convolutional neural network, that has been previously trained and validated on other food datasets. The processing unit 3 transfers the information extracted from the picture to the remote device 6 that displays it on the userinterface 7. The remote device 6 may generate advisory messages for the user, based on the information sent by the parameter transfer unit 4.

[0089] In other applications of the invention, the sensor 2 may be an accelerometer.

[0090] In a first application where the sensor 2 is an accelerometer, the accelerometer may be used to detect from facial movements that the wearer of the head- worn device 1 is speaking.

[0091] To do so, classification methods may be used to detect peaks in the spectral density of the signal output by the accelerometer, after filtering in the frequency range [0.5 Hz; 3 Hz]. In that range, the walking pattern and the heart activity can be clearly identified. If there is another signal, this is the wearer’s voice.

[0092] This makes it possible to detect that the facial muscles are moving, without trying to understand the words spoken. This method is thus very advantageous for preserving privacy, for example. In addition, such detection of speaking makes it possible to enable or disable audio enhancement features only when necessary, for preserving battery life and offering the wearer a better experience, as it is better to switch off audio treatment when the wearer is speaking. In a second application where the sensor 2 is an accelerometer, the accelerometer may detect that a predetermined trigger word or phrase has been said by the wearer, such as those required for starting a request to a search engine. At least one record of the trigger word or phrase is first stored. Then, when the wearer is equipped with the head- worn device 1, a time window of the signal output by the accelerometer is analyzed continuously by the processing unit 3, by filtering the frequency range [0.5 Hz; 3 Hz] and by using metrics such as DTW (Dynamic Time Warping, which is an algorithm for pattern recognition described in the URL htps: / / eii.wikipedia.org / wiki / Dyiianiic time warping), in order to determine whether the trigger word or phrase has been said in the recorded window. Other algorithms for pattern recognition than DTW can of course be applied, even if DTW is more efficient compared to other Euclidian metrics. A significant advantage of this method is that the trigger word or phrase detection is carried out only by the wearer himself and not by someone around who may say the trigger word or phrase as well. Additionally, as only facial movement of the wearer is being analyzed, privacy of conversations is increased, as audio recordings are not constantly being analyzed or transmitted for analysis, while trying to detect trigger words or phrases. Another significant advantage of this method is that it is insensitive to the sound level, i.e. the level of noise, of the environment. It is possible to recognize the pattern even in a very noisy environment such as an airport, a concert, or a construction area.

[0093] The invention is not limited to the examples that have just been described.

[0094] The wearable device may take other forms, like an eyeglasses clip. It may comprise other types of sensors, like an optical sensor.

[0095] The signal data output by the sensor or sensors may be different. The components of the wearable device, like the processing unit, may take various forms.

Claims

CLAIMS1. A wearable device (1) worn by a user for detection of mouth movements of said user, wherein said wearable device (1) comprises: at least one sensor (2) disposed in an auricular and / or temporoparietal region of a head of said user, said at least one sensor (2) being configured to monitor mouth movements of said user, so as to obtain signal data (8); a processing unit (3), configured to analyze said signal data (8) and to extract at least one parameter representative of said mouth movements and to infer from said at least one parameter an assessment of an activity associated with said mouth movements, by comparing said at least one parameter with reference values for the activity associated with said mouth movements.

2. A wearable device (1) according to claim 1, further comprising a parameter transfer unit (4), configured to transfer said extracted at least one parameter to a remote device (6).

3. A wearable device (1) according to claim 1 or 2, wherein said at least one sensor (2) is a capacitive sensor.

4. A wearable device (1) according to any one of the preceding claims, wherein said at least one sensor (2) is an optical sensor.

5. A wearable device (1) according to any one of the preceding claims, wherein said at least one sensor (2) is disposed at a distance from at least one temple of said user which is below 3 cm, better below 2cm, even better below 1 cm.

6. A wearable device (1) according to any one of the preceding claims, wherein said at least one sensor (2) is disposed between an outer canthus of an eye of said user and a top of that ear which is the closest one to said eye.

7. A wearable device (1) according to any one of the preceding claims, wherein said at least one sensor (2) is further configured to detect a worn state of said wearable device (1).

8. A wearable device (1) according to any one of the preceding claims, wherein said wearable device (1) is a head- worn device, notably an eyewear.

9. A wearable device (1) according to any one of the preceding claims, wherein said at least one parameter is included in a group comprising frequency and duration of said mouth movements.

10. A method for detection of mouth movements of an individual by means of a wearable device (1) worn by said individual, wherein said method comprises: monitoring mouth movements of said individual, by means of at least one sensor (2) embedded in the wearable device and disposed in an auricular and / or temporoparietal region of a head of said individual, so as to obtain signal data; analyzing said signal data (8) and extracting at least one parameter representative of said mouth movements, by means of a processing unit included in the wearable device and inferring from said at least one parameter an assessment of an activity associated with said mouth movements, by comparing said at least one parameter with reference values for the activity associated with said mouth movements.

11. A method according to claim 10, further comprising transferring said extracted at least one parameter to a remote device (6), by means of a parameter transfer unit (4).

12. A method according to claim 10 or 11, further comprising detecting a worn state of said wearable device, by means of said at least one sensor.

13. A non-transitory memory unit, storing one or more sequences of instructions that are accessible to a processor and that, when executed by said processor, cause said processor to implement a method according to any one of claims 10 to 12.

14. A computer program product, comprising one or more sequences of instructions that are accessible to a processor and that, when executed by said processor, cause said processor to implement a method according to any one of claims 10 to 12.

Images