Image processing system and method for processing image from head of personal care device

Abstract

An image processing system processes an image received from an imaging device mounted at a head of a personal care device. The personal care device has a personal care element that is movable relative to the imaging device while implementing its personal care function. An image of a user over time is received from an imaging device. The image is segmented and three-dimensional data is generated based on the temporal evolution of the segmented shape. The 3D volume enables determination of current motion characteristics of the personal care element and / or future motion characteristics of the personal care element.

Description

Technical Field

[0001] The present invention relates to integrating imaging into the head of a personal care device such as an electric toothbrush, shaver or brush. Background Art

[0002] Cameras are becoming more and more common in the personal care field, especially as electric toothbrushes with integrated cameras continue to appear on the market. For example, camera images are processed to provide feedback to the user about his / her brushing routine, or to provide diagnostic information. For example, caries and plaque can be determined. Imaging can also be used to determine the position of the oral care head in the mouth (more accurately than with motion sensors alone) so that the user's oral care routine can be analyzed more accurately. This information can be used to guide the consumer in a way that he / she reaches all positions during the consumer's oral care routine. The user's oral health history can be collected.

[0003] Thus, imaging-assisted oral care devices may be able to enable image-based diagnosis, superior position sensing for the head of the oral care device, oral care treatment planning (e.g., orthodontics, braces), or treatment monitoring (e.g., periodontitis). These new capabilities may be seamlessly integrated into the user's normal brushing hygiene regimen, replacing the need for a separate device, such as a smartphone or dental endoscope.

[0004] Other oral care devices, such as mouthpieces and oral irrigators, are also being developed with imaging capabilities.

[0005] Similarly, a shaver or epilator can use imaging to detect shaving or epilator performance or detect skin conditions. For example, imaging can be used to optimize the action and quality of shaving or epilator (local beard orientation, individual hair geometry, etc.), characterize skin type to suggest and optimize treatment, and enable skin condition prediction, such as acne detection, acne avoidance during shaving, acne occurrence prediction, etc.

[0006] Hairbrushes may also be equipped with imaging technology, for example, to enable a determination of hair condition.An integrated camera in a hairbrush could be an enabler in the fields of cosmetics and personal wellness.

[0007] Furthermore, in professional nursing services, it is documented in the scientific literature that a large portion of nurses’ time (20% to 40% in total) is occupied by direct patient care tasks. Any technology that can achieve a reduction in the time required for such tasks can greatly increase nurse productivity. Personal hygiene related tasks account for a significant portion of this type of care (e.g., brushing teeth, shaving, combing hair, etc.). The use of imaging sensors in these brushing actions is an enabler for the optimization (i.e., time reduction) of such tasks. Imaging sensors and / or imaging cameras can help in both the automation of tasks (automatic brushing using visual information) and any pre-diagnostic analysis of the patient’s physiology.

[0008] With this type of device comes a new generation of problems related to the information that these sensors are able to detect. In particular, brushing devices equipped with cameras have a disadvantage that the presence of the bristles causes interference and occlusions within the camera image. This is because the camera is located outside the head of the personal care device to avoid excessive interference from the brush head geometry and composition (i.e., the bristles). The same problems apply to the case of shavers if the blades are within the field of view of the imaging camera because the camera is integrated into the shaving head. These problems also imply that there is algorithmic complexity that arises from the image processing required to handle complex images.

[0009] Acquiring images and video streams without such interferences and occlusions will greatly improve both the quality of the user experience (usability, pleasure, and aesthetic appeal of the images) and the quality and quantity of information obtained from the oral images themselves, e.g., for diagnostic purposes.

[0010] A miniature camera can be easily placed almost anywhere in a device, such as the head of a personal care device. For example, in an electric toothbrush, the key design features are the type of oscillation (i.e., vibration in the plane of the toothbrush head) and the geometry of the bristles. There is a lot of variability in the geometry of the bristles, either because there are many different head types specialized for specific purposes (gum types, etc.) or simply because new more optimized versions are available. The type of oscillation is usually in a well-defined plane, such as the vertical plane (when brushing the sides of the teeth).

[0011] The oscillating pattern causes the bristles to contact the teeth, drag and bend during the oscillation, depending on the force and type of movement applied by the user. The problem is that this mechanical behavior will cause some of the bristles to enter the field of view of the camera mounted in the head of the oral care device. This blocking or occlusion of the image is a problem to be solved.

[0012] Bristle location, geometry, density, etc. vary with head type. The camera and optics could be designed to have minimal obstruction to specific bristle characteristics, but this would result in expensive and laborious adaptation of the optics or camera design or mounting to the specific bristle design.

[0013] If the skin condition is to be imaged, the use of a brush may also result in the brushed hairs obstructing or obscuring the underlying skin.

[0014] The present invention aims to solve this problem in a more general way.

[0015] US2019200746A1 describes a method for promoting compliance with an oral hygiene regimen, comprising displaying a representation of at least a portion of a set of teeth of a user on a display device. This document does not mention a solution to address blocking or occlusion of the image. Summary of the invention

[0016] The invention is defined by the claims.

[0017] According to an example of one aspect of the present invention, there is provided an image processing system for processing an image received from an imaging device, the imaging device being mounted at a head of a personal care device, the personal care device having a personal care element that can move relative to the imaging device to implement a personal care function, the image processing system comprising:

[0018] A processor configured to:

[0019] receiving a sequence of images of a user over time from an imaging device at a head of the personal care device;

[0020] segmenting personal care components from an image sequence;

[0021] constructing a three-dimensional 3D volume representing the temporal evolution of the segmented 2D shape; and

[0022] Process 3D volumes to:

[0023] Determining user-induced motion characteristics of a personal care element and / or personal care device; and / or

[0024] Predicting future motion characteristics of a personal care element and / or personal care device.

[0025] The 3D volume essentially provides the temporal evolution of the presence / absence of the personal care element in the field of view (FOV) of the imaging device, as captured in the 2D image. The changes in presence / absence are caused by user movement. For example, a user presses a toothbrush against their teeth causing the bristles to bend into the FOV of the imaging device. In a second example, when a user uses a razor, the rotation plane of the razor will adapt to the surface of the skin.

[0026] It has been recognised that the time evolution may be processed to determine the user-induced motion characteristics and to determine the future motion characteristics of the personal-care element.

[0027] The imaging device may be, for example, a camera or a laser scanner. Other suitable imaging devices capable of capturing an image of the user may be used.

[0028] Personal care elements are elements in a personal care device that are intended to perform a personal care function (e.g., bristles in a toothbrush for cleaning teeth, razors in a shaver for shaving hair, bristles in a hairbrush for combing hair, etc.).

[0029] The processor may be configured to process the 3D volume by determining topological characteristics of the 3D volume and inputting the topological characteristics into a regression algorithm, wherein the regression algorithm is configured to output a user-induced motion characteristic and / or a future motion characteristic.

[0030] The use of topological properties enables a reduction in the data that needs to be processed (eg as input to a regression algorithm) while maintaining an accurate description of the 3D volume. This reduces the time required to process the 3D volume.

[0031] Topological properties may include one or more of connectivity, distance matrix, topological signature, local shape, and point signature.

[0032] The processor may be further configured to receive motion-related sensor data from a sensor arrangement on the personal care device, the sensor data indicating motion of the personal care device and / or the personal care element, and the processor may be further configured to use the sensor data in processing the 3D volume.

[0033] The sensor data may include data of the personal care device, such as gyroscope data, acceleration / velocity data, pressure data, drive current or drive voltage for a motor, etc. This data may improve the accuracy of the output of the regression algorithm when processing a 3D volume.

[0034] The processor may be further configured to generate a clean image from the sequence of images using the segmented personal care elements from the sequence of images, wherein the clean image is an image of the user with the personal care elements removed.

[0035] The processor may be configured to use the future motion characteristic and / or the user-induced motion characteristic in segmenting the personal-care element from the sequence of images.

[0036] The processor may be further configured to use the user-initiated motion characteristics and / or future motion characteristics to adjust one or more parameters of one or more of: an imaging device, a light source of a personal care device, a personal care device, a personal care element, and a personal care function.

[0037] The personal care device may be an electric toothbrush, and the processor may adjust the cleaning frequency of the electric toothbrush based on the future motion characteristics. For example, the future motion characteristics may indicate that the electric toothbrush is about to move to a specific area or orientation, and the cleaning frequency may be adjusted accordingly.

[0038] The personal care device may be a manual toothbrush, and the processor may adjust the bristle length and / or stiffness according to the anatomical structure to be cleaned next. For example, future motion characteristics may be used to predict which anatomical structure will be cleaned next.

[0039] Acquisition parameters of the imaging device may also be adjusted based on the anatomy to be cleaned (and therefore imaged) next.

[0040] The personal care device may be an electric razor, and the processor may adjust the razor motor current according to the anatomical structure to be cleaned next. For example, if the skin is about to move to a more sensitive area of ​​the user (e.g., redness is detected), the movement of the razor element may become gentler. In contrast, if there is more / thicker hair in the next predicted area, the shaving power of the motor current may be increased.

[0041] The personal care device may be a manual razor, and the processor may adjust the stiffness of the razor element depending on the anatomy to be imaged next.

[0042] The personal care device may be an electric hairbrush, and the processor may adjust the amplitude modulation of the bristles on the electric hairbrush based on the anatomical structure to be imaged next. In one example, the processor may be further configured to predict the arrival of one or more hair knots based on the future motion characteristics and a clean image of the user's anatomy containing one or more hair knots. Therefore, the processor may be further configured to adjust the amplitude modulation of the bristles based on the predicted arrival of the one or more hair knots.

[0043] The personal care device may be a manual hairbrush, and the processor may be configured to adjust the length and / or stiffness of the bristles based on the future motion characteristics.

[0044] The motion characteristics can be measured using sensors on the personal care device or sensors on the user (such as sensors on a smart watch, etc.).

[0045] The present invention also provides a personal care system, the personal care system comprising:

[0046] Personal care devices, having personal care components;

[0047] An imaging device mounted at the head of the personal care device;

[0048] The aforementioned image processing system; and

[0049] A light source is adapted to illuminate the field of view of the imaging device.

[0050] The personal care device may be, for example, a toothbrush, a hairbrush, a shaver or a skin cleansing brush.

[0051] The light source may be a light emitting diode (LED).

[0052] The present invention also provides an image processing method for processing an image received from an imaging device, wherein the imaging device is mounted at the head of a personal care device, the personal care device having a personal care element, and the personal care element can move relative to the imaging device to achieve a personal care function, the method comprising:

[0053] receiving a sequence of images of a user over time from an imaging device at a head of the personal care device;

[0054] segmenting personal care components from an image sequence;

[0055] constructing a three-dimensional 3D volume representing the time evolution of the segmented shape; and

[0056] Process 3D volumes to:

[0057] determining user-induced motion characteristics of a personal care element; and / or

[0058] Predict the future motion characteristics of personal care components.

[0059] Processing the 3D volume may include determining a topological characteristic of the 3D volume and inputting the topological characteristic into a regression algorithm, wherein the regression algorithm is configured to output a user-induced motion characteristic and / or a future motion characteristic.

[0060] Topological properties may include one or more of connectivity, distance matrix, topological signature, local shape, and point signature.

[0061] The method may further include receiving motion-related sensor data from a sensor arrangement on the personal-care device, the sensor data indicating motion of the personal-care device and / or the personal-care element; and processing the 3D volume using the sensor data.

[0062] The method may further include using the user-induced motion characteristic and / or the future motion characteristic to adjust one or more parameters of one or more of: an imaging device, a light source, a personal care device, a personal care element, and a personal care function.

[0063] The present invention also provides a computer program, which includes computer program codes. When the program is run on a computer, the computer program codes are suitable for implementing the above-mentioned image processing method.

[0064] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] For a better understanding of the invention, and in order to more clearly show how the invention may be put into practice, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0066] Figure 1 A camera is shown placed on a toothbrush having bristles;

[0067] Figure 2 A camera is shown placed on a hairbrush having bristles;

[0068] Figure 3 shows a camera placed on a shaver having shaving elements;

[0069] Figure 4 A method for predicting the motion of a personal care element is shown;

[0070] Figure 5 A snapshot of the 3D volume from a first perspective is shown;

[0071] Figure 6 A snapshot of the 3D volume from a second viewing angle is shown;

[0072] Figure 7 A snapshot of the 3D volume from a third perspective is shown;

[0073] Figure 8 showing a snapshot of the 3D volume from a fourth perspective; and

[0074] Fig. 9 A standard deviation projection of the 3D volume from a fourth viewing angle is shown. DETAILED DESCRIPTION

[0075] The present invention will be described with reference to the accompanying drawings.

[0076] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the apparatus, system, and method, are intended for illustrative purposes only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the apparatus, system, and method of the invention will be better understood through the following description, the appended claims, and the accompanying drawings. It should be understood that the drawings are merely schematic and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the drawings to indicate the same or similar components.

[0077] The present invention provides an image processing system that processes images received from an imaging device mounted at the head of a personal care device. The personal care device has a personal care element that can move relative to the imaging device when performing its personal care function. An image of a user over time is received from the imaging device. The image is segmented, and three-dimensional data is generated based on the time evolution of the segmented shape. The 3D volume enables the determination of current motion characteristics of the personal care element (including user-induced motion characteristics) and / or future motion characteristics of the personal care element.

[0078] Personal care devices enable consumers to perform their personal health care routine in a simple way. Therefore, it is important to guide the consumer in a way that the device reaches all places or use the device in an optimal way. One way to achieve this goal is to perform image and video acquisition during the personal care routine. This image data can be used, for example, to guide the consumer to places that have not been adequately brushed / shaved, or to check for possible lesions by creating a (video) history of the user's anatomy / physiology.

[0079] Image processing algorithms exist for identifying, locking onto and predicting moving objects. It is recommended to use these algorithms for enabling the removal of moving elements / rotating elements (e.g. bristles, blades, etc.) from pictures and videos acquired with an imaging device (e.g. a camera) installed in the brush head / shaving head, including when the user disturbs the natural / free movement of the bristles / blades with her / his movements (including applying pressure, moving, etc.). Furthermore, the recommended algorithms may instead be used to enable the removal of images of the user's hair from the field of view.

[0080] This can greatly improve the quality of the user experience (usability, enjoyment and aesthetic appeal of the image) as well as the quality and quantity of information derived from the device image itself. This can also improve any end use of the image for diagnostic purposes.

[0081] A miniature camera with current technology can be easily placed almost anywhere in a personal care device such as a toothbrush, shaver, hairbrush, etc. There are both mechanical and optical properties to consider.

[0082] The problem to be solved originates from the positioning of the camera in the head of the device.

[0083] Figure 1 , Figure 2 and Figure 3 Cameras placed on different personal care devices are shown. Figure 1 A camera 104 is shown positioned on a toothbrush 100 having bristles 102 . Figure 1 Also schematically shown is a processor 110 for performing the image processing described below.

[0084] Figure 2 A camera 204 is shown positioned on a hairbrush 200 having bristles 202 . Figure 3 A camera 304 is shown positioned on a shaver 300 having shaving elements 302 .

[0085] For the purposes of the following description, it may be assumed that the optical environment for image acquisition is optimal in terms of lighting, lens, etc., or at least provides an adequate image for the purposes of image processing. This may be achieved in part by, for example, having lighting devices 106, 206, and 306 on the personal care device. The lens arrangement on the camera 104, 204, and 304 may also be optimized for a particular use.

[0086] More importantly, there are three main mechanical aspects to consider for personal care devices:

[0087] - The intended functional movement of the personal-care elements 102, 202 and 302. This may include the type of oscillation, the plane of oscillation, the rate of rotation, etc. This is particularly important for powered personal-care devices (e.g. powered toothbrushes, shavers, etc.).

[0088] - The geometry of the personal care elements 102 , 202 and 302 .

[0089] - Typically random, user-initiated modification of the movement of the personal-care elements 102 , 202 , and 302 .

[0090] The camera and the personal care element are typically both rigidly connected to the personal care device.Due to this interlock, overall movement of the personal care device initiated by the user during use of the personal care device does not significantly interfere with the functional movement of the personal care element.

[0091] However, there is movement variability caused by user actions that directly and / or indirectly affect the functional movement of the personal care element. For example, the oscillation frequency and phase of the rotating personal care element may be affected. The plane of movement (e.g., the rotation plane of the shaving element 302) may also be affected.

[0092] In the case of bristles, pressure applied by the user may also change the linear movement of the bristles, such as causing the bristles 102 to bend.

[0093] Therefore, there is a need to address the changing dynamics of personal care elements during use of a personal care device.

[0094] In toothbrushes, such as Figure 1 The toothbrush 100 shown may have a high degree of variability in the bristles 102, as the bristles 102 may potentially have many different geometries tailored for specific purposes (eg, gum type, etc.), or simply as a new, more optimized version.

[0095] However, with respect to the functional movement of the bristles 102 (in the context of a powered toothbrush), the plane of rotation of the bristles is typically well defined in a vertical oscillating position (relative to the plane of the user's teeth). This is because powered toothbrushes are typically constrained to the plane in which they can rotate.

[0096] In the case of a shaver, Figure 3 The shaver 300 shown, the actual situation is almost the opposite. In this case, the rotation frequency is usually well defined, but the rotation plane is not well defined. The change of the rotation plane of the shaver element 302 can change how many shaver elements block the field of view of the camera 304.

[0097] This is because the shaver 300 conforms to the user's anatomy based on user input. This causes the shaver element 302 to tilt in various directions to conform to the user's anatomy. The tilt of the shaver element 302 can change how much and / or where and when the shaver element 302 blocks the field of view of the camera 304.

[0098] For the sake of brevity, the following description will be given with respect to a powered toothbrush. However, it should be understood that the same considerations are valid, with minor adjustments, for the general case of personal care appliances including shavers, hairbrushes, etc.

[0099] In the case of a powered toothbrush, the oscillation of the bristles means that the bristles will contact the teeth and may drag and bend during the oscillation. The amount of drag and bend on the bristles will depend at least in part on the force and type of movement applied by the user. This mechanical behavior will likely cause some of the bristles to enter the camera's field of view (FOV). In addition, due to the (pseudo) randomness of the user's interaction with the toothbrush, the amount, shape, and geometry of the bristles in the camera's FOV will also likely change over time.

[0100] This type of variable occlusion of teeth in the camera's image is the problem addressed herein. The same reasoning applies to a hairbrush with a built-in camera (i.e., the bristles of the hairbrush will variably occlude the user's anatomy over time) and a shaver with a built-in camera (i.e., changes in the plane of rotation will variably occlude the user's anatomy over time). In fact, the same reasoning applies to any personal care device with a personal care element that variably occludes the FOV of a camera integrated into the personal care device over time while the personal care device is in use.

[0101] Regarding the bristles of a toothbrush, it is worth noting that the location of the bristles as well as the geometry, density, etc. of the bristles may change depending on the type of brush head on the toothbrush. These variations may mean that the unobstructed FOV of the head-mounted camera is reduced. Of course, this problem can be solved by using appropriate camera optics (e.g. read focal length, field of view, etc.), but this requires expensive and laborious adjustments of the optics to these mechanical properties. Furthermore, the random nature of user interaction makes this mechanical approach inefficient as it can only be optimized once before the toothbrush is delivered.

[0102] Instead, it is recommended to use software algorithms that can automatically adapt to the variability of bristle oscillations and remove the bristle interference from the acquired images and videos.

[0103] The proposed algorithm allows for clearer imaging of the underlying user anatomy.

[0104] The use of algorithms may also be an enabler for more accurate image analysis and classification for, for example, detecting caries or plaque, or general problems in dentures or the general anatomy being cared for (skin, etc.).

[0105] Figure 1 , Figure 2 and Figure 3 The lighting devices 106, 206, and 306 shown in the drawings, respectively, may be light emitting diodes (LEDs). LEDs may be white light sources / visible light sources with known physical characteristics (including lighting frequency, duty cycle, etc.).

[0106] The processor can be used to perform a hybrid classification regression task of the position of the bristles in the image from the camera and their predicted oscillation / rotation frequency. This will be described in more detail below.

[0107] Figure 4 A method for predicting the motion of a personal care element is shown.In step 402, a sequence of images of a user's anatomy (ie, teeth, skin, hair, etc.) is obtained during use of a personal care device (ie, brushing, shaving, etc.).

[0108] Suitable locations for the camera should be considered. Since the focus is on eliminating the bristles from the camera's FOV when vibrating, potential locations include placing the camera in or around the brush head, in the same plane as the actual brush head, or in a different plane than the brush head.

[0109] In these solutions, the functional movement of the bristles can be of any type (eg fixed, linear, rotational, etc.).

[0110] A light source may be used to illuminate the area of ​​which the photograph is being taken. Ideally this is an LED. The advantage of LEDs is that they can be easily triggered, facilitating the taking of subsequent images with the same or similar bristle orientation.

[0111] Additionally, the following information may be useful during the following processing steps:

[0112] - Free movement / functional movement of the bristles (e.g. free vibration, rotation frequency or linear speed, etc.);

[0113] - The geometry of the bristles;

[0114] - Camera acquisition characteristics (e.g., pixel resolution, exposure time, frames per second, acquisition time, global or rolling shutter, etc.); and

[0115] - Camera optical properties (reading focal length, field of view, etc.).

[0116] In most cases, it may be assumed that the camera acquisition and / or optical characteristics are optimal for the conditions. However, the characteristics may be used to filter the images. For example, if the vibration frequency of the bristles is consistent with the frame rate of the camera, the bristles may appear to be stationary in the image sequence. Therefore, these images can be filtered out. In some cases, the frame rate (or other characteristics) can also be adjusted based on the image sequence, or even the field of view of the camera can be adapted to the content of the image.

[0117] If available, data from various onboard sensors available in the toothbrush (eg, inertial monitoring unit, pressure, motor current, etc.) may also be used in the following processing steps.

[0118] The central assumption is that user-induced motion induces physical modifications that are reflected in the shape of the envelope of bristles, which is related to the frequency of motion they now exhibit.

[0119] The algorithm begins accumulating a sequence of images from the camera and, potentially, performs simple image processing adjustments and / or analysis on them (eg, to remove blurry images, etc.).

[0120] In step 404, the bristles are segmented from the image sequence. This can be performed by an online video semantic segmentation task using a pre-trained unsupervised deep learning network (i.e., unsupervised domain adaptation, reinforcement learning, etc.) that uses both video frames and sensor data as input. An envelope representing the shape of the bristles from the camera's perspective is obtained. In other words, an envelope showing the location of the bristles in the image is obtained.

[0121] During the segmentation step, the geometry of the bristles (and / or any physical properties of the bristles) may be used in order to improve the accuracy of the segmentation. For example, the geometry of the bristles and / or the color of the bristles may be used as additional constraints for the segmentation step.

[0122] The envelope of the global shape of the extracted bristles is then represented in a 3D dataset. This dataset is 3D because it includes 2D image data captured by a camera over time and therefore includes three dimensions. This three-dimensional dataset will be referred to as a "3D volume", but it should be understood that the data volume has two spatial dimensions and one temporal dimension. In other words, the 3D volume is a stack of the 2D envelopes output at the segmentation step.

[0123] A 3D volume is generated in step 406. This volume therefore represents the time evolution of the 2D shape of the bristles in the captured images. It has been found that the topological characteristics (connectivity etc.) of such volumes are directly related to the motion properties of the bristles and vice versa.

[0124] For a new image frame, the newly extracted bristle shape envelope is merged into the semantic image / frame segmentation result and then added to the global 3D volume (two spatial dimensions and one temporal dimension) obtained from the previous segmentation of the video.

[0125] At the start of the operation, the global 3D volume may be empty. Thus, for each new image in the image sequence (or a new set of frames), the 3D volume may be updated with the shape envelope of the new image.

[0126] In a general sense, the 3D volume is then processed in step 408 to extract the current user-induced motion (e.g., movement type, speed, frequency, etc.) and / or future motion (i.e., a prediction of motion at a future time) of the bristles in step 410. In practice, an example of how to process a 3D volume includes determining topological properties of the 3D volume and processing the topological properties to output current motion and / or future motion. This is because, although the 3D volume includes all the necessary information for determining the current user-induced motion and future motion, the 3D volume consists of a large amount of data. Processing the entire 3D volume would require a large amount of processing power. Therefore, it is recommended to calculate the topological properties of the 3D volume and use them as input data to determine the current user-induced motion and / or predicted motion.

[0127] The topological properties of the 3D volume (ie, connectivity in 3D, distance matrix, topological signature, local shape, point signature, etc.) can be determined and updated with new data and predicted position differences from previous video frames.

[0128] As explained above, the extracted topological features can be used as input to a regression algorithm (e.g., a motion flow algorithm, a trained machine learning / deep learning algorithm, etc.) to determine the current user-induced bristle motion and the predicted bristle motion. The motion of the bristles can include stationary position, any linear movement, any rotational movement, any oscillating movement, frequency, speed, etc.

[0129] Data from the toothbrush's sensors (e.g., IMU, pressure, etc.) can also be input into the regression algorithm along with the topological properties.

[0130] In practice, the output of the entire algorithm (i.e., including both the segmentation algorithm and the regression algorithm) is as follows:

[0131] - Detect and eliminate bristles or blades from video frames / images.

[0132] -Determine physical quantities related to the actual motion of the bristles or blades (including user-induced motion).

[0133] -Predict the future position of bristles or blades based on estimated future motion.

[0134] If sensor data from a toothbrush is input into a regression algorithm, the determination of the user-induced motion of the bristles can be improved. For example, knowing the pressure of the toothbrush against the teeth can improve the determination of the degree of bristle bending.

[0135] Similarly, knowledge of the free movement of the bristles (eg, frequency of rotation in the absence of user-induced motion) can further help the regression algorithm distinguish user-induced motion from motion caused by the free movement of the bristles.

[0136] Figure 5 A snapshot of a 3D volume from a first perspective is shown. The darker areas represent the shape of the bristles that obstruct the FOV of the camera on the toothbrush. The x-axis and y-axis are spatial axes, while the t-axis is the time axis.

[0137] Figure 6 A snapshot of the 3D volume from a second viewing angle is shown. As can be seen from this view, the shape of the bristles remains somewhat constant, but does change. The random changes in the 3D volume Figure 7 More obvious in.

[0138] Figure 7 A snapshot of the 3D volume from a third perspective is shown. In this view, random changes and sudden changes can be seen in the 3D volume, while the overall general shape remains somewhat similar over time.

[0139] Figure 8 A snapshot of the 3D volume from a fourth perspective is shown. This view is similar to Figure 5 , but from the other side of the 3D volume.

[0140] Fig. 9 A standard deviation projection of the 3D volume from a fourth viewing angle is shown. Darker areas generally represent smaller standard deviations, thus indicating areas that are more likely to remain covered by bristles over time.

[0141] Figures 5 to 9 It is shown how changes in the position of the bristles (ie caused by the movement of the bristles) can manifest themselves in the 3D volume. These changes in the 3D volume affect its topological properties.

[0142] Essentially, the regression algorithm discussed previously is designed to estimate the relationship between a 3D volume (or preferably, its topological properties) and time. Other variables may also be considered in this relationship, such as the free movement of the bristles and data from sensors in the toothbrush.

[0143] One of the unknown variables considered by the regression algorithm may be the user-induced motion of the bristles. User-induced motion can be distinguished from the overall motion of the bristles (represented by changes in the 3D volume) because the overall motion is (usually) a combination of the user-induced motion and the free movement of the bristles. User-induced motion is the motion of the bristles (or other personal cleansing element) that is induced by the user's motion, rather than by the free movement of the bristles (i.e., how the bristles would be expected to move in the absence of any user-induced motion).

[0144] Once the relationship between the 3D volume and time has been estimated, the relationship can be extended into the future to predict what the shape of the 3D volume will be (or at least what the topological properties will be) in the future. This prediction provides a forecast of the future movement of the bristles. Generally speaking, the predicted future movement will indicate the movement expected if the user does not induce any further movement.

[0145] In general, regression algorithms usually have two different goals: prediction and estimation of the relationship between variables. In this case, these two goals can be used to predict the movement of the bristles and determine the impact of user-induced movement on the relationship between 3D volume and time.

[0146] The form of the regression algorithm (e.g., using a pre-existing motion flow regression algorithm or training a deep learning algorithm for this purpose) may depend on the specific implementation. For example, the intended processor to be used may factor in the complexity of the algorithm, since the more complex the algorithm, the more processing power required.

[0147] The user-induced motion and / or predicted bristle motion can be used in a variety of ways. One of the uses of this information is to generate a clean image of an anatomical structure (e.g., a user's mouth or a user's skin). The predicted future motion can be used as an input to a segmentation algorithm to improve the accuracy and / or speed of segmentation. In fact, the predicted motion can be used in the aforementioned segmentation algorithm. This can then enable the output of a clean image of the anatomical structure (i.e., no moving parts), which can then be used further.

[0148] Thus, the predicted future motion may enable improved segmentation of the personal care element from the image (eg, for the next frame).

[0149] User-induced motion can also be used to improve segmentation of personal care elements from an image. For example, user-induced motion can be used to indicate how similar a clean image is expected to be to a previous clean image. For example, if the user-induced motion indicates that the user has not moved significantly between frames, then it can be expected that the next segmented image will be similar to the previous segmented image.

[0150] The clean image can be used to generate a map of the anatomical structure and / or detect the presence of any potential lesions. This information can be used to adjust the free movement of personal care elements. For example, the frequency of bristles in a toothbrush can be adjusted based on the map of the anatomical structure obtained and / or the presence of known (general) lesions. In areas where it is known that the user has sensitive anatomical structures (e.g., bleeding or redness during use), the free movement can be adjusted to reduce the impact of the personal care elements on the area. The map of the anatomical structure can also be used to modify the free movement based on the area being treated (e.g., the frequency of bristles on the gums is lower than the frequency of bristles on the teeth). As will be understood, understanding the predicted future motion enables the free movement of the personal care device to be adjusted in advance.

[0151] Of course, understanding the user-induced motion also provides further information for when the free movement should be adjusted. For example, when a user applies relatively high pressure on the bristles of a toothbrush against their teeth (e.g., indicated by a large bend in the bristles), it can be expected that the user is still cleaning that area, and therefore will not move to the next area until the pressure is reduced to some extent.

[0152] The map of the anatomical structure and / or the presence of any lesions can also be used to adjust the physical properties of personal care elements, or even recommend personal care devices with more suitable physical properties. For example, the length and / or stiffness of the bristles in a toothbrush can be adjusted based on the map of the anatomical structure and / or the presence of any lesions. Similarly, the stiffness of the blade in a manual razor can be adjusted, or different blade stiffnesses can be recommended. As above, knowing the predicted future motion and / or the current user-induced motion can enable pre-adjustment of physical properties.

[0153] The predicted movement of the bristles can be used to anticipate the anatomical structure that will be imaged next, and adjust the physical properties in advance accordingly. Similarly, user-induced movement can indicate whether the user is focused on using the personal care device on a particular area, or whether they will likely move to the next area. For example, if there is no or little user-induced movement, this may indicate that the user has recently moved to this area and will begin using the personal care device in this area, or that the user has finished with this area and will likely move on to the next area.

[0154] The map of the anatomical structure and / or the presence of lesions can also be used to adjust the camera acquisition characteristics. For example, for areas that are known to be darker or include lesions, the illumination of the camera LED can be increased (and / or the exposure time of the camera can be increased). The frame rate (i.e., the acquisition frequency) can also be adjusted based on the anatomical structure being imaged or to be imaged next. This can also use the predicted future movement of the bristles to anticipate which area of ​​the anatomical structure will be imaged next, and thereby change the camera acquisition characteristics in advance.

[0155] In the case of a shaver, the predicted future motion can be used to anticipate the area to be shaved next. If the anticipated area is known to have sensitive skin, the motor current driving the shaver can be adjusted accordingly (i.e., the motor speed is reduced to provide a gentler shave). Similarly, if the anticipated area is known to have more or thicker hair, the motor speed can be increased.

[0156] In the case of a shaver, user-induced motion may be used to determine whether a property of the shaver needs to be adjusted. For example, if the plane of the shaver element is tilted by a relatively large amount, this may indicate that the user is attempting to shave a relatively difficult area and is therefore inducing more motion than usual on the shaver element. Thus, a large amount of user-induced motion on the shaver element may indicate that the motor current driving the shaver element may need to be increased.

[0157] In some cases (e.g., hairbrushes and shavers), the presence of hair can be further considered. For example, the hair bundle shapes present in the image can be correlated with the optimal bristle movement (linear and rotational) required to untangle the hair bundle with the optimal amount of time, energy, and primarily lower bristle-hair bundle interlock time, which may result in perceived pain.

[0158] The predicted motion can be used to anticipate the hairbrush reaching a knot / tress. The user may be alerted to this, and / or the stiffness / length of the hairbrush's bristles may be reduced accordingly.

[0159] User-induced motion can be used to determine the presence of a hair knot (e.g., not visible in the image). For example, if a subset of bristles on a hairbrush bends more than the rest (i.e., the user-induced force is higher for a subset of bristles), this may indicate the presence of a hair knot.

[0160] Notably, hair can also be segmented from the image to obtain a clean image of anatomical structures covered by hair and bristles. A method for identifying hair (and thereby enabling its segmentation) can include using hyperspectral imaging to distinguish hair from skin (or other anatomical features). For example, an image can be separated into a red channel, a green channel, and a blue channel and compared to identify hair. Color-based segmentation can also be used, based on hair color and skin color.

[0161] Those skilled in the art will be able to easily develop a processor for performing any of the methods described herein. Therefore, each step of the flowchart may represent a different action performed by the processor and may be performed by a corresponding module of the processor.

[0162] As discussed above, the system utilizes a processor to perform data processing. The processor can be implemented in a variety of ways using software and / or hardware to perform the various functions required. The processor typically employs one or more microprocessors that can be programmed using software (e.g., microcode) to perform the required functions. The processor can be implemented as a combination of dedicated hardware that performs some functions and one or more programmed microprocessors and associated circuitry that performs other functions.

[0163] Examples of circuit devices that may be used in various embodiments of the present disclosure include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0164] In various implementations, the processor may be associated with one or more storage media, such as volatile and non-volatile computer memory, such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that, when executed on one or more processors and / or controllers, perform the desired functions. The various storage media may be fixed within the processor or controller, or may be transportable so that one or more programs stored thereon may be loaded into the processor.

[0165] Variations to the disclosed embodiments may be understood and effected by those skilled in the art in practicing the claimed invention, through a study of the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0166] The functions implemented by a processor may be implemented by a single processor or by multiple separate processing units, which may be collectively considered to constitute a “processor.” In some cases, such processing units may be remote from each other and communicate with each other in a wired or wireless manner.

[0167] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0168] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or solid-state medium provided with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0169] If the term "suitable for" is used in the claims or the description, it should be noted that the term "suitable for" is intended to be equivalent to the term "configured to". If the term "arranged" is used in the claims or the description, it should be noted that the term "arranged" is intended to be equivalent to the term "system" and vice versa.

[0170] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. An image processing system for processing an image received from an imaging device (104, 204, 304), wherein the imaging device is mounted at the head of a personal care device (100, 200, 300), wherein the personal care device has a personal care element (102, 202, 302), wherein the personal care element can move relative to the imaging device to implement a personal care function, the image processing system comprising: A processor (110), the processor being configured to: receiving a sequence of images of a user over time from the imaging device at the head of the personal care device; segmenting the personal care element from the sequence of images; constructing a three-dimensional 3D volume representing a time evolution of the segmented 2D shape; as well as The 3D volume is processed to: determining a user-induced motion characteristic of the personal-care element and / or the personal-care device; and / or Future movement characteristics of the personal-care element and / or the personal-care device are predicted.

2. The system of claim 1 , wherein the processor is configured to process the 3D volume by: determining a topological property of the 3D volume; and The topological characteristic is input into a regression algorithm, wherein the regression algorithm is configured to output the user-induced motion characteristic and / or the future motion characteristic.

3. The system of claim 2, wherein the topological characteristics include one or more of the following: Connectivity; Distance matrix; Topological signature; local shape; and Click Sign.

4. A system according to any one of claims 1 to 3, wherein the processor is further configured to receive motion-related sensor data from a sensor device on the personal care device, the sensor data indicating movement of the personal care device and / or the personal care element, and the processor is further configured to use the sensor data to process the 3D volume.

5. The system of any one of claims 1 to 4, wherein the processor is further configured to generate a clean image from the image sequence using the personal care elements segmented from the image sequence, wherein the clean image is an image of the user with the personal care elements removed.

6. The system of any one of claims 1 to 5, wherein the processor is configured to use the future motion characteristic and / or the user-induced motion characteristic in segmenting the personal-care element from the sequence of images.

7. The system of any one of claims 1 to 6, wherein the processor is further configured to use the user-induced motion characteristic and / or the future motion characteristic to adjust one or more parameters of one or more of: The imaging device; A light source (106, 206, 306) of the personal care device; said personal care device; said personal care element; and The personal care function.

8. A personal care system comprising: A personal care device (100, 200, 300) having a personal care element (102, 202, 302); An imaging device (104, 204, 304) mounted at the head of the personal care device; An imaging processing system according to any one of claims 1 to 7; and A light source (106, 206, 306) is adapted to illuminate a field of view of the imaging device.

9. The personal care system of claim 8, wherein the light source is a light emitting diode (LED).

10. A computer-implemented image processing method for processing an image received from an imaging device, the imaging device being mounted at a head portion of a personal care device, the personal care device having a personal care element that is movable relative to a user of the imaging device to perform a personal care function, the method comprising: receiving (402) a sequence of images of a user over time from the imaging device at the head of the personal care device; segmenting (404) the personal care element from the sequence of images; constructing (406) a three-dimensional 3D volume based on a time evolution of the segmented shape; and The 3D volume is processed (408) to: determining a user-induced motion characteristic of the personal-care element; and / or Future motion characteristics of the personal care element are predicted.

11. The method of claim 10, wherein processing the 3D volume comprises: determining a topological property of the 3D volume; as well as The topological characteristic is input into a regression algorithm, wherein the regression algorithm is configured to output the user-induced motion characteristic and / or the future motion characteristic.

12. The method of claim 11, wherein the topological characteristics include one or more of the following: Connectivity; Distance matrix; Topological signature; local shape; and Click Sign.

13. The method according to any one of claims 10 to 12, further comprising: receiving motion-related sensor data from a sensor arrangement on the personal-care device, the sensor data indicating motion of the personal-care device and / or the personal-care element; as well as The 3D volume is processed using the sensor data.

14. The method according to any one of claims 10 to 13, further comprising: Using the user-initiated motion characteristic and / or the future motion characteristic to adjust one or more parameters of one or more of: The imaging device; a light source for the personal care device; said personal care device; said personal care element; and The personal care function.

Images