Image Processing System and Method for Processing Images from the Head of a Personal Care Device

Abstract

An image processing system processes an image received from an imaging device attached to the head of a personal care device. The personal care device has a personal care element that can move relative to the imaging device while realizing a personal care function. An image of the user over time is received from the imaging device. The image is segmented, and 3D data is generated based on the temporal change in the shape of the segmentation. The 3D volume enables the determination of the current motion characteristics of the personal care element and / or the future motion characteristics of the personal care element.

Description

Technical Field

[0001] The present invention relates to the integration of imaging into the head of a personal care device such as an electric toothbrush, a shaver, or a brush.

Background Art

[0002] Cameras have generally become ubiquitous in the personal care field, and in particular, electric toothbrushes equipped with integrated cameras have been successively introduced to the market. Camera images are processed, for example, to provide feedback to the user regarding the brushing routine or to provide diagnostic information. For example, dental caries and dental plaque can be determined. Imaging can also be used to determine the position of the oral care head in the oral cavity (more accurately than simply using a motion sensor), so that the user's oral care routine can be analyzed more accurately. This information can be used to guide the consumer in a manner that allows the consumer to reach all areas during the oral care routine. A history of the user's oral health can be collected.

[0003] Accordingly, image-assisted oral care devices can enable image-based diagnosis, excellent position sensing of the head of the oral care device, oral care treatment planning (e.g., orthodontics, aligners), or treatment monitoring (e.g., periodontitis). These new functions can be seamlessly integrated into the user's normal brushing hygiene routine instead of requiring a separate device such as a smartphone or a dental endoscope.

[0004] Other oral care devices such as cleaning mouthpieces and oral irrigators have also been developed with imaging capabilities.

[0005] Similarly, shavers or hair removal devices may use imaging to detect shaving or hair removal performance, or to detect the condition of the skin. For example, imaging can be used to optimize shaving (such as the local direction of the beard, the shape of a single hair, etc.) or hair removal action and quality, to characterize skin type and propose and optimize treatments, and to enable skin condition prediction such as acne detection, acne avoidance during shaving, and acne appearance prediction.

[0006] For example, a hairbrush can also be equipped with imaging technology to enable a determination regarding the state of the hair. A camera integrated into the hairbrush can be enabling in the cosmetics and personal health fields.

[0007] Furthermore, in specialized care services, it is recorded in scientific literature that a disproportionate amount (20% to 40% in total) of a nurse's time is occupied by direct care work for patients. Any technology that can shorten the time required for such work can significantly improve the productivity of nurses. A significant portion of such care is occupied by personal hygiene-related work (toothbrushing, shaving, hair brushing, etc.). The use of imaging sensors in these brushing operations is enabling for the optimization (i.e., time reduction) of such work. Imaging sensors and / or cameras can be useful for the automation of the work (automatic brushing using visual information) and any pre-diagnostic analysis of the patient's physiology. SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0008] Such devices give rise to a new generation of problems related to the information that the sensors can detect. In particular, brushing devices with cameras have the drawback that the presence of the bristles causes interference and occlusion in the camera image. This is because the camera is placed outside the head of the personal care device in order to avoid excessive interference due to the shape and configuration of the brush head (such as bristles). In the case of shavers, the same problem occurs when the blade enters the field of view of the imaging camera, because the camera is integrated into the shaving head. These problems mean that there is algorithmic complexity resulting from the image processing necessary to handle complex images.

[0009] Obtaining images and video streams without such interference and occlusion can significantly improve both the quality of the user experience (the usability, enjoyment and aesthetic appeal of the images) and the quality and quantity of information that can be obtained from the oral cavity images themselves, for example for diagnostic purposes.

[0010] Miniaturized cameras can be easily installed almost anywhere on the device, such as on the head of a personal care device. As an example, in an electric toothbrush, the type of vibration (i.e., in-plane vibration of the toothbrush head) and the shape of the bristles are important design features. There is a large variation in the shape of the bristles, because there are many different head types for specific purposes (such as gum types) or for new versions that are simply more optimized. The type of vibration is usually in a clearly defined plane, such as a vertical plane (when polishing the sides of the teeth).

[0011] The vibration mode causes the bristles in contact with the teeth to be dragged and bent during vibration, based on the force and type of movement applied by the user. The problem lies in the fact that this mechanical behavior causes some of the bristles to enter the field of view of the camera attached to the head of the oral care device. This interference or occlusion of the image is a problem to be addressed.

[0012] The position, shape, density, etc. of the bristles vary depending on the head type. It may be possible to design the camera and optical system to have minimal occlusion for specific hair characteristics, but this incurs costs and effort to adapt the design or mounting of the optical system or camera to the specific hair design.

[0013] The use of the brush results in the brushed hair causing an obstruction or occlusion to the underlying skin when the skin condition is imaged.

[0014] The object of the present invention is to address this problem in a more general manner.

[0015] US2019200746A1 describes a method for promoting compliance with an oral hygiene regimen, which includes displaying on a display device a representation of at least a portion of a user's set of teeth. This document does not mention a solution for dealing with image obstruction or occlusion.

Means for Solving the Problem

[0016] The present invention is defined by the claims.

[0017] According to an example according to one aspect of the present invention, there is provided an image processing system for processing an image received from an imaging device, which is attached to the head of a personal care device having a personal care element that can move relative to the imaging device to realize a personal care function, and this is has a processor, and this processor receives a sequence of images of the user over time from the imaging device on the head of the personal care device, segments the personal care element from the sequence of images, constructs a three-dimensional 3D volume representing the temporal change of the segmented 2D shape, processes the 3D volume to determine the personal care element and / or the user-induced movement characteristics of the personal care device, and / or Predict the future motion characteristics of the above personal care element and / or the above personal care device.

[0018] The 3D volume essentially provides the temporal variation of the presence or absence of the personal care element within the field of view (FOV) of the imaging device, captured in a 2D image. The change in presence or absence is caused by the user's movement. For example, the user pressing the toothbrush against the teeth causes the bristles to bend into the FOV of the imaging device. In a second example, when the user uses a shaver, the rotational plane of the shaver adapts to the surface of the skin.

[0019] It is understood that the temporal variation can be processed to determine user-induced motion characteristics and to determine future motion characteristics of the personal care element.

[0020] The imaging device can be, for example, a camera or a laser scanner. Other suitable imaging devices that can capture an image of the user can be used.

[0021] The personal care element is an element of a personal care device intended to perform a personal care function (e.g., the bristles of a toothbrush for cleaning teeth, the blade of a shaver for shaving hair, the bristles of a hairbrush for detangling hair, etc.).

[0022] The processor can be configured to process the 3D volume by determining the topological characteristics of the 3D volume and inputting the topological characteristics into a regression algorithm, where the regression algorithm is configured to output the user-induced motion characteristics and / or the future motion characteristics.

[0023] The use of topological characteristics allows for the reduction of the data that needs to be processed (e.g., input into the regression algorithm) while maintaining an accurate depiction of the 3D volume. This shortens the time taken to process the 3D volume.

[0024] Topological characteristics include one or more of connectivity, distance matrix, topological signature, local shape, and point signature.

[0025] The processor may be further configured to receive motion-related sensor data from a sensor arrangement on a personal care device, the sensor data indicating movement of the personal care device and / or the personal care element, and the processor is configured to use the sensor data when processing a 3D volume.

[0026] The sensor data can include data of the personal care device such as gyro data, acceleration / velocity data, pressure data, drive current or drive voltage to a motor. This data can improve the accuracy of the output of the regression algorithm when processing a 3D volume.

[0027] The processor may be further configured to generate a clean image from a sequence of images using segmented personal care elements from the sequence of images, the clean image being an image of the user with the personal care elements removed.

[0028] The processor may be configured to use future motion characteristics and / or user-induced motion characteristics in segmenting personal care elements from a sequence of images.

[0029] The processor may be further configured to adapt one or more parameters in one or more of an imaging device, a light source of the personal care device, the personal care device, the personal care element, and the personal care function using user-induced motion characteristics and / or future motion characteristics.

[0030] The personal care device may be an electric toothbrush, and the processor can adapt the cleaning frequency of the electric toothbrush based on future motion characteristics. For example, the future motion characteristics can indicate that the electric toothbrush moves in a specific area or direction, and accordingly, the cleaning frequency can be adapted.

[0031] The personal care device may be a manual toothbrush, and the processor can adapt the hair length and / or stiffness as a function of the biological structure to be cleaned next. For example, future motion characteristics can be used to predict which biological structure will be cleaned next.

[0032] The imaging parameters of the imaging device can also be adapted as a function of the biological structure to be cleaned next (and thus imaged next).

[0033] The personal care device may be an electric shaver, and the processor can adapt the shaver motor current as a function of the biological structure to be cleaned next. For example, when the skin is about to move to a more sensitive area of the user (e.g., detected redness), the movement of the shaver element can be made smoother. In contrast, when there is a lot / thick hair in the area predicted next, the shaving force by the motor current may increase.

[0034] The personal care device may be a manual shaver, and the processor can adapt the hardness of the shaver element as a function of the biological structure to be imaged next.

[0035] The personal care device may be an electric hairbrush, and the processor can adapt the hair modulation in the electric hairbrush as a function of the biological structure to be imaged next. In one example, the processor may be further configured to predict arrival at one or more knots based on future motion characteristics and a clean image of the user's biological structure including one or more hair knots. Accordingly, the processor may be further configured to adapt the hair modulation based on the predicted arrival at one or more knots.

[0036] The personal care device may be a manual hairbrush, and the processor may be configured to adapt the hair length and / or stiffness based on future motion characteristics.

[0037] The motion characteristics can be measured using sensors on a personal care device or on the user (e.g., sensors on a smartwatch, etc.).

[0038] The present invention also provides a personal care system, which a personal care device having a personal care element, an imaging device attached to the head of the personal care device, the aforementioned image processing system, and a light source configured to illuminate the field of view of the imaging device.

[0039] The personal care device can be, for example, a toothbrush, a hairbrush, a shaver, or a skin cleansing brush.

[0040] The light source can be a light emitting diode LED.

[0041] The present invention also provides an image processing method for processing an image received from an imaging device attached to the head of a personal care device having a personal care element that can move relative to a user of the imaging device to realize a personal care function, the method comprising: receiving a sequence of images of the user over time from an imaging device on the head of the personal care device; segmenting the personal care element from the sequence of images; constructing a three-dimensional (3D) volume based on the temporal change in the shape of the segmentation; processing the 3D volume to determine the user-induced motion characteristics of the personal care element and / or predict the future motion characteristics of the personal care element.

[0042] The step of processing the 3D volume can include a step of determining topological characteristics of the 3D volume and a step of inputting the topological characteristics into a regression algorithm, and the regression algorithm is configured to output the user-induced motion characteristics and / or the future motion characteristics.

[0043] The topological characteristics include one or more of connectivity, distance matrix, topological signature, local shape, and point signature.

[0044] The method further includes a step of receiving motion-related sensor data from sensor arrangements on a personal care device, where the sensor data indicates movement of the personal care device and / or personal care element, and a step of processing the 3D volume using the sensor data.

[0045] The method may further include a step of adapting one or more parameters in one or more of an imaging device, a light source, a personal care device, a personal care element, and a personal care function using the user-induced motion characteristics and / or the future motion characteristics.

[0046] The present invention also provides a computer program including computer program code configured to implement the above-described image processing method when executed on a computer.

[0047] These and other aspects of the present invention will become apparent from the embodiments described below and will be described with reference to the embodiments.

Brief Description of the Drawings

[0048]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Best Mode for Carrying Out the Invention

[0049] For a better understanding of the present invention and to more clearly show the method of its realization, the accompanying drawings, which are merely illustrative, are referred to.

[0050] The present invention will be described with reference to the drawings.

[0051] It should be understood that the detailed description and specific examples, while showing exemplary embodiments of the apparatus, system and method, are for illustrative purposes only and are not intended to limit the scope of the present invention. These and other features, aspects and advantages of the apparatus, system and method of the present invention will be more preferably understood from the following description, the appended claims and the accompanying drawings. It should be understood that the figures are merely schematic and are not drawn to scale. It should also be understood that the same reference numbers are used throughout the figures to indicate the same or similar parts.

[0052] The present invention provides an image processing system that processes an image received from an imaging device attached to a head of a personal care device. The personal care device has a personal care element that can move relative to the imaging device while realizing a personal care function. An image of the user over time is received from the imaging device. The image is segmented, and 3D data is generated based on a temporal change in the shape of the segmentation. The 3D volume enables determination of the current motion characteristics of the personal care element, including the motion induced by the user, and / or the future motion characteristics of the personal care element.

[0053] Personal care devices enable consumers to easily perform personal health management routines. Therefore, it is important to guide the consumer in such a way that the consumer reaches all spots or uses the device in an optimal manner. One way to achieve this goal is to perform image and video acquisition during the personal care routine. This image data can be used, for example, to guide the consumer to spots where brushing / shaving is insufficient or to check for possible pathologies by creating a (video) history of the user's body structure / physiology.

[0054] There are image processing algorithms that can identify, lock, and predict moving objects. These algorithms are proposed to be used to remove moving / rotating elements (such as hairs, blades, etc.) from photos and videos acquired by an imaging device (such as a camera) attached to a brushing / shaving head, including when the user's movement (including applying pressure, moving, etc.) interferes with the natural / free movement of the hair / blade. Further, the proposed algorithms can alternatively be used to remove an image of the user's hair from the field of view.

[0055] This can significantly improve both the quality of the user experience (the usability, enjoyment, and aesthetic appeal of the images) and the quality and quantity of information obtained from the device images themselves. This can also improve any eventual use of the images for diagnostic purposes.

[0056] Cameras miniaturized with current technology can be easily placed almost anywhere on most personal care devices (such as toothbrushes, shavers, hairbrushes, etc.). Consideration needs to be given to mechanical and optical properties.

[0057] The problem to be addressed arises from placing the camera on the head of the device.

[0058] Figures 1, 2, and 3 show cameras arranged on various personal care devices. Figure 1 shows a camera 104 arranged on a toothbrush 100 with bristles 102. Figure 1 also schematically shows a processor 110 for performing the image processing described below.

[0059] Figure 2 shows a camera 204 arranged on a hairbrush 200 with bristles 202. Figure 3 shows a camera 304 arranged on a shaver 300 with a shaving element 302.

[0060] For the following explanations, it can be assumed that the optical environment for image acquisition is optimal in terms of lighting, lenses, etc., or at least provides an image suitable for the purpose of image processing. This can be achieved in part, for example, by providing lighting devices 106, 206, and 306 on the personal care devices. The lens arrangements in cameras 104, 204, and 304 can also be optimized for specific applications.

[0061] More importantly, there are three main mechanical aspects to consider with respect to personal care devices. The intended functional movement of the personal care elements 102, 202, and 302. This includes the type of vibration, the plane of vibration, the rotational speed, etc. This is particularly important for powered personal care devices (powered toothbrushes, shavers, etc.). The shape of the personal care elements 102, 202, and 302. Generally random and user-induced modification of the movement of the personal care elements 102, 202, and 302.

[0062] Both the camera and the personal care element are typically rigidly connected to the personal care device. Due to this linkage, the overall movement of the personal care device induced by the user during the use of the personal care device does not significantly interfere with the functional movement of the personal care element.

[0063] However, there is movement variability resulting from the user's actions that directly and / or indirectly affect the functional movement of the personal care element. For example, the vibration frequency and phase of a rotating personal care element may be affected. The plane of movement (e.g., the plane of rotation of the shaver element 302) may also be affected.

[0064] In the case of bristles, the pressure applied by the user can also change the linear movement of the bristles, for example, causing the bristles 102 to bend.

[0065] Therefore, it is necessary to address the variations in the dynamics of the personal care element during the use of the personal care device.

[0066] In a toothbrush such as the toothbrush 100 shown in FIG. 1, the bristles 102 can potentially have a high variability by having many different shapes, either specialized for a particular purpose (e.g., type of gum, etc.) or simply a newer and more optimized version.

[0067] However, with respect to the functional movement of the bristles 102 (in the context of a powered toothbrush), the plane of rotation of the bristles is generally clearly defined at the location of the vibration perpendicular (to the user's tooth surface). This is because a powered toothbrush is typically limited in the plane in which it can rotate.

[0068] In the case of a shaver such as the shaver 300 shown in FIG. 3, almost the opposite is true. In this case, the rotational frequency is generally clearly defined, but the plane of rotation is not. Changes in the plane of rotation of the shaver element 302 can change how many shaver elements block the field of view of the camera 304.

[0069] This is because the shaver 300 adapts to the user's body structure based on the user's input. This results in the shaver element 302 tilting in various directions to conform to the user's body structure. The tilt of the shaver element 302 can change the number, and / or location, and time that the shaver element 302 blocks the field of view of the camera 304.

[0070] For the sake of brevity, the following description is given for a powered toothbrush. However, it should be understood that the same considerations are valid with minor adjustments for general personal care devices including shavers, hairbrushes, etc.

[0071] In the case of a powered toothbrush, the vibration of the bristles means that the bristles may contact the teeth and be dragged and bent during vibration. The amount of drag and bending in the bristles depends at least in part on the force and type of movement applied by the user. This mechanical behavior is likely to bring some of the bristles into the field of view (FOV) of the camera. Furthermore, due to the (pseudo) randomness of the user's interaction with the toothbrush, the amount, shape, and geometry of the bristles within the camera's FOV are also likely to change over time.

[0072] The variable occlusion of teeth in the images from the camera is the problem addressed in this document. The same reasoning applies to a hairbrush with an embedded camera (i.e., the hairs from the hairbrush variably occlude the user's body structure over time) and a shaver with an embedded camera (i.e., the changes in the rotational plane variably occlude the user's body structure over time). In fact, the same reasoning applies to any personal care device that includes a personal care element that variably occludes the FOV of a camera integrated into the personal care device as the personal care device is used.

[0073] Regarding the bristles of a toothbrush, it should be noted that the position and shape, density, etc. of the bristles can vary with the type of toothbrush head. These changes mean a reduction in the unoccluded FOV of the head-mounted camera. Of course, this problem can be solved by using an appropriate camera optical system (e.g., focal length, field of view, etc.), but this requires cost and effort to adapt the optical system to these mechanical characteristics. Furthermore, the random nature of user interaction makes this mechanical approach inefficient because it can only be optimized once before the toothbrush is supplied.

[0074] Instead, it is proposed to use a software algorithm that can automatically adapt to the changing nature of the changes in the bristle vibration and cancel the bristle interference from the acquired images and videos.

[0075] The proposed algorithm enables clearer imaging of the underlying user's body structure.

[0076] The use of the algorithm can be an enabler for more accurate image analysis, for example, for detecting dental caries or plaque, common problems in dentures, or general body structures (such as skin) during care.

[0077] The lighting devices 106, 206, and 306 shown in FIGS. 1, 2, and 3, respectively, may be light emitting diodes (LEDs). The LEDs can be white / visible light sources having known physical characteristics (including lighting frequency, duty cycle, etc.).

[0078] A processor can be used to perform a mixed classification regression task of the positions of the hairs in the image from the camera and their predicted vibration / rotation frequencies. This is described in more detail below.

[0079] FIG. 4 shows a method for predicting the movement of a personal care element. In step 402, during the use of the personal care device (i.e., brushing, shaving, etc.), a sequence of images of the user's biological structure (i.e., teeth, skin, hair, etc.) is acquired.

[0080] The appropriate position of the camera should be considered. Since emphasis is placed on removing the hairs from the FOV of the camera during vibration, potential positions include placing the camera in or around the brush head, in the same plane as the actual brush head or a different plane from the brush head.

[0081] In these solutions, the functional movement of the hairs can be of any type (e.g., fixed, linear, rotational, etc.).

[0082] A light source can be used to illuminate the area to be photographed. Ideally, this is an LED. The advantage of LEDs is that they can be easily triggered and it is made easier to take subsequent images with the same or similar hair orientations.

[0083] Furthermore, the following information is useful in subsequent processing steps. The free / functional movement of the hairs (e.g., free vibration, rotation frequency, or linear velocity, etc.); The shape of the hairs; The camera shooting characteristics (e.g., pixel resolution, exposure time, number of frames acquired per second, shooting time, global shutter or rolling shutter, etc.); and The optical characteristics of the camera (focal length, field of view, etc.).

[0084] In most cases, it is highly likely that the shooting and / or optical characteristics of the camera are optimal for the conditions. However, it is possible to filter images using those characteristics. For example, when the frequency of the bristles matches the frame rate of the camera, the bristles will appear stationary in the sequence of images. Therefore, these images can be filtered. In some cases, it is also possible to adapt the frame rate (or other characteristics) based on the sequence of images, or even make the field of view of the camera adaptable to the content of the image.

[0085] If available, data from various on-board sensors available in the toothbrush (such as inertial monitoring units, pressure, motor / current, etc.) can also be used in subsequent processing steps.

[0086] The central hypothesis is that user-induced motion results in physical changes that are reflected in the shape of the envelope of the bristles, which correlates with the frequency of the motion currently exhibited by the bristles.

[0087] The algorithm begins with the accumulation of a sequence of images from the camera and, potentially, performs simple image processing adjustments and / or analysis on it (such as removing blurred images, etc.).

[0088] In step 404, the bristles are segmented from the sequence of images. This can be performed by an online video semantic segmentation task using a pre-trained teacherless deep learning network (i.e., teacherless domain adaptation, reinforcement learning, etc.) that uses both the video frame and sensor data as inputs. An envelope representing the shape of the bristles is obtained from the field of view of the camera. In other words, an envelope indicating the position of the bristles in the image is obtained.

[0089] To improve the accuracy of segmentation, the hair shape (and / or any physical property of the hair) can be used during the segmentation step. For example, the hair shape and / or hair color can be used as additional constraints for the segmentation step.

[0090] The envelope of the overall shape of the extracted hair is represented in a 3D dataset. This dataset is 3D in that it has 2D image data captured by a camera over time and has three dimensions. This three-dimensional dataset is called a "3D volume", but it should be understood that the data volume has two spatial dimensions and one temporal dimension. In other words, the 3D volume is a stack of 2D envelopes output by the segmentation step.

[0091] In step 406, a 3D volume is generated. This volume represents the temporal change in the 2D shape of the hair in the captured images. It has been found that the topological features (such as connectivity) of such a volume are directly correlated with the motion characteristics of the hair, and vice versa.

[0092] For a new image frame, the envelope of the newly extracted hair shape is merged into the result of the semantic image / frame segmentation and added to the global 3D volume (two spatial dimensions and one temporal dimension) obtained from the previous segmentation of the video.

[0093] At the start of the operation, the global 3D volume may be empty. Thus, for each new image in the image sequence (or new group of frames), the 3D volume can be updated with the shape envelope of the new image.

[0094] In a general sense, the 3D volume is then processed in step 408 to extract the current, user-induced movement of the hair (e.g., type of movement, speed, frequency, etc.) and / or in step 410 to extract future movement (i.e., prediction of movement at a future time). In practice, examples of ways to process the 3D volume include determining the topological characteristics of the 3D volume and processing the topological characteristics to output the current movement and / or future movement. This is because although the 3D volume contains all the information necessary to determine the current user-induced movement and future movement, the 3D volume is composed of a large amount of data. Processing the entire 3D volume requires a large amount of processing power. Thus, it is proposed to calculate the topological characteristics of the 3D volume and use them as input data to determine the current user-induced movement and / or the predicted movement.

[0095] The topological characteristics of the 3D volume (connectivity, distance matrix, topological signature, local shape, point signature in 3D, etc.) can be determined and updated using new data and the predicted position differences from past video frames.

[0096] The topological characteristics extracted as described above can be used as input to a regression algorithm (e.g., motion flow algorithm, trained machine learning / deep learning algorithm, etc.) to determine both the current, user-induced movement of the hair and the predicted movement of the hair. The movement of the hair includes a stationary position, any linear movement, any rotational movement, any vibrational movement, frequency, speed, etc.

[0097] Data from the toothbrush sensor (e.g., IMU, pressure, etc.) can also be input into the regression algorithm with topological characteristics.

[0098] In practice, the output of the entire algorithm (including both the segmentation and regression algorithms) is as follows. Detection and removal of hair or blades from video frames / images. Determination of physical quantities related to the actual movement of hair or blades, including user-induced movement. Prediction of the future position of hair or blades based on the estimated future movement.

[0099] When sensor data from a toothbrush is input into a regression algorithm, the determination of user-induced hair movement can be improved. For example, knowledge of the pressure of the toothbrush on the teeth can improve the determination of how much the hair is bent.

[0100] Similarly, knowledge of the free movement of the hair (e.g., the rotational frequency without user-induced movement) can further assist the regression algorithm to distinguish between user-induced movement and movement caused by the free movement of the hair.

[0101] Figure 5 shows a snapshot of a 3D volume from the first view. The dark areas represent the shape of the hair that shields the FOV of the camera on the toothbrush. The x-axis and y-axis are spatial axes, and the t-axis is the time axis.

[0102] Figure 6 shows a snapshot of a 3D volume from the second view. As can be seen from this figure, the shape of the hair is somewhat constant but changing. In Figure 7, the random changes in the 3D volume are more clearly visible.

[0103] Figure 7 shows a snapshot of a 3D volume from the third view. In this view, random and sudden changes are seen in the 3D volume, but the overall general shape remains somewhat similar over time.

[0104] Figure 8 shows a snapshot of a 3D volume from the fourth view. This view is similar to Figure 5 but from the opposite side of the 3D volume.

[0105] Figure 9 shows the standard deviation projection of the 3D volume from the fourth view. Dark regions generally represent areas where the standard deviation is small and thus areas that are likely to remain covered with hair over time.

[0106] Figures 5 through 9 show how changes at the hair positions (i.e., those caused by hair movement) appear in the 3D volume. These changes in the 3D volume affect its topological properties.

[0107] In summary, the regression algorithm described above is designed to estimate the relationship between the 3D volume (or preferably its topological properties) and time. Other variables such as the free movement of the hair and data from sensors in the toothbrush can also be considered in this relationship.

[0108] One of the unknown variables considered in the regression algorithm can be the hair movement induced by the user. In that the overall movement is (generally) a combination of user-induced movement and free hair movement, user-induced movement can be distinguished from the overall movement of the hair (represented by changes in the 3D volume). User-induced movement is the movement of the hair (or other personal grooming elements) induced by the user's movement and not the movement induced by free hair movement (i.e., the movement of the hair expected when there is no movement induced by the user).

[0109] Once the relationship between the 3D volume and time is estimated, this relationship can be extended into the future to predict what the shape of the 3D volume will be in the future (or at least what its topological properties will be). This prediction provides a prediction of future hair movement. Generally, the predicted future movement indicates the movement expected when the user does not induce any further movement.

[0110] Regression algorithms generally typically have two different purposes: predicting and estimating the relationships between variables. In this case, both purposes can be utilized to predict hair movement and determine the influence of user-induced movement in the relationship between the 3D volume and time.

[0111] The form of the regression algorithm (e.g., using an existing motion flow regression algorithm or training a deep learning algorithm for this purpose) may depend on the specific implementation. For example, the intended processor to be used can affect the complexity of the algorithm. This is because the more complex the algorithm, the higher the required processing power.

[0112] User-induced movement and / or predicted hair movement can be used in various ways. One use of this information is the generation of clean images of biological structures (e.g., the user's mouth or the user's skin). The predicted future movement can be used as input to a segmentation algorithm to improve the accuracy and / or speed of segmentation. In fact, the predicted movement can be used in the aforementioned segmentation algorithm. This enables the output of a clean image of the biological structure (i.e., an image without moving parts), which can be further utilized.

[0113] Therefore, the predicted future movement can enable improved segmentation of personal care elements from an image (e.g., for the next frame).

[0114] User-induced movement can also be used to improve the segmentation of personal care elements from an image. For example, user-induced movement can be used to indicate how similar the clean image is expected to be to past clean images. For example, if user-induced movement indicates that the user has not moved significantly between frames, the next segmented image is expected to be similar to the past segmented image.

[0115] The clean image can be used to generate a map of the biological structure and / or detect the presence of any potential lesions. This information can be used to modulate the free movement of the personal care element. For example, the frequency of the bristles in a toothbrush can be adapted based on the obtained map of the biological structure and / or the known presence of (general) pathology. In areas where the user is known to have sensitive biological structures (e.g., bleeding or redness during use), the free movement can be adapted to reduce the effect of the personal care element in that area. A map of the biological structure can also be used to change the free movement based on the area being treated (e.g., the frequency of the bristles on the gums is lower than that of the bristles on the teeth). As understood, knowledge of the predicted future movement enables the free movement of the personal care device to be pre-adapted.

[0116] Of course, knowledge of the user-induced movement also provides additional information about when the free movement should be adapted. For example, when the user applies a relatively high pressure to the bristles of the toothbrush against the teeth (e.g., as indicated by the bristles bending significantly), it is expected that the user is still cleaning that area and thus will not move to the next area until the pressure is somewhat reduced.

[0117] The map of the biological structure and / or the presence of any pathology can also be used to adapt the physical characteristics of the personal care element or to recommend a personal care device with more appropriate physical characteristics. For example, the length and / or hardness of the bristles in a toothbrush can be adapted from the map of the biological structure and / or the presence or absence of any pathology. Similarly, the hardness of the blade in a manual shaver can be adapted or different blade hardnesses can be recommended. As described above, knowledge of the predicted future movement and / or the current user-induced movement can enable proactive adaptation of the physical characteristics.

[0118] The predicted hair movement can be used to predict the next imaged biological structure and thus anticipate and adapt physical characteristics accordingly. Similarly, user-induced movement can indicate whether the user is concentrating the use of the personal care device in a specific area or is likely to move to the next area. For example, if there is no or little user-induced movement, this can indicate that the user has recently moved into this area and started using the personal care device in this area, or that the user has finished using it in this area and is likely to move to the next area.

[0119] Maps of biological structures and / or the presence of pathologies can also be used to adapt the camera imaging characteristics. For example, for areas known to be dark or areas containing pathologies, the illuminance of the camera's LEDs can be increased (and / or the camera's exposure time can be lengthened). The frame rate (i.e., the imaging frequency) can also be adapted as a function of the biological structure being imaged or the next biological structure to be imaged. This can also use predicted future hair movement to predict which areas of the biological structure will be imaged next, thereby pre-adjusting the camera's imaging characteristics.

[0120] In the case of a shaver, the predicted future movement is used to predict the next area to be shaved. If the expected area is known to have sensitive skin, the motor current driving the shaver can be adapted accordingly (i.e., the motor speed can be decreased to provide a gentler shave). Similarly, if the expected area is known to have more or coarser hair, the motor speed can be increased.

[0121] In the case of a shaver, user-induced movement can be used to determine whether the characteristics of the shaver need to be adapted. For example, if the plane of the shaver element is inclined relatively greatly, this may indicate that the user is trying to shave a relatively difficult area and thus a greater movement than normal is induced in the shaver element. Thus, a large user-induced movement in the shaver element may indicate that the motor current driving the shaver element needs to be increased.

[0122] In some cases (such as a hairbrush and a shaver), the presence of hair can be further considered. For example, the shape of the hair bundle present in the image can be associated with the optimal hair movement (linear and rotational movement) required to untangle the hair bundle in a shorter time of hair-hair bundle interaction time that may result in an optimal amount of time, energy, and especially perceived pain.

[0123] The predicted movement can be used to predict the arrival of the hairbrush at the hair knot / bundle. The user can be warned about this and / or the hair hardness / length of the hairbrush can be reduced accordingly.

[0124] User-induced movement can be used to determine the presence of knots (for example, not visible in the image). For example, if a subset of the hairs in the hairbrush bends more than the rest (i.e., when the force induced by the user on the subset of hairs is large), this may indicate the presence of a knot.

[0125] Note that it is also possible to segment hair from an image in order to obtain a clean image of the hair and the biological structure covered by the hair. A method of identifying hair (thereby enabling segmentation) can have the use of hyperspectral imaging to distinguish hair from skin (or other biological structure features). For example, an image can be separated into red, green, and blue channels and compared to identify hair. Color-based segmentation can also be used based on the color of the hair and the color of the skin.

[0126] One skilled in the art would be able to easily develop a processor for performing any of the methods described herein. Accordingly, each step of the flowchart may represent different actions to be performed by a processor and can be performed by individual modules of the processor.

[0127] As described above, this system utilizes a processor for data processing. The processor can be implemented in various ways using software and / or hardware to perform the various functions required. The processor typically uses one or more microprocessors and can be programmed using software (e.g., microcode) to perform the necessary functions. The processor can be implemented as a combination of dedicated hardware for performing some functions and one or more programmed microprocessors and associated circuitry for performing other functions.

[0128] Examples of circuits that can be employed in various embodiments of the present application include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs).

[0129] In various implementations, a processor can be associated with one or more storage media such as volatile and non-volatile computer memories such as RAM, PROM, EPROM, and EEPROM. The storage media may be encoded with one or more programs that perform the required functions when executed by one or more processors and / or controllers. The various storage media may be fixed within the processor or controller, or may be transportable such that one or more programs stored therein can be loaded into the processor.

[0130] Variations to the disclosed embodiments can be understood and effected by persons skilled in the art in light of the drawings, disclosure, and appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0131] The functions realized by the processor can be realized by a single processor or by a plurality of individual processing units that are considered to together constitute a "processor". Such processing units may in some cases be physically separated from each other and can communicate with each other by wire or wirelessly.

[0132] The mere fact that certain means are recited in mutually different dependent claims does not indicate that a combination of these means cannot be used advantageously.

[0133] A computer program can be stored / distributed on a suitable medium such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0134] It should be noted that in the claims or the specification, when the term "adapted to" is used, the term "adapted to" is intended to be equivalent to the term "configured to". It should also be noted that in the claims or the specification, when the term "arrangement" is used, the term "arrangement" is intended to be equivalent to the term "system", and vice versa.

[0135] Any reference signs appearing in the claims shall not be construed as limiting the scope of the invention.

Claims

1. An image processing system for processing images received from an imaging device, wherein the imaging device is attached to the head of a personal care device having a personal care element that can move relative to the imaging device to realize a personal care function, and the system has a processor, and the processor receives a sequence of images of the user over time from the imaging device on the head of the personal care device, segments the personal care element from the sequence of images, constructs a three-dimensional (3D) volume representing the temporal change of the segmented 2D shape, processes the 3D volume to determine the personal care element and / or the user-induced motion characteristics of the personal care device, and / or predict the future motion characteristics of the personal care element and / or the personal care device.

2. The processor determines the topological characteristics of the 3D volume and processes the 3D volume by inputting the topological characteristics into a regression algorithm, which outputs the user-induced motion characteristics and / or the future motion characteristics. The system according to claim 1.

3. The topological characteristics include connectivity, distance matrix, topological signature, local shape, and point signature, one or more of which are included in the system according to claim 2.

4. The processor further receives sensor data related to motion from a sensor arrangement on the personal care device, the sensor data indicating the motion of the personal care device and / or the personal care element, and the processor further processes the 3D volume using the sensor data. The system according to any one of claims 1 to 3.

5. The processor uses the segmented personal care element from the sequence of images to generate a clean image from the sequence of images, the clean image being an image of the user with the personal care element removed. The system according to any one of claims 1 to 4.

6. The processor uses the future motion characteristics and / or the user-induced motion characteristics in the segmentation of the personal care element from the sequence of images. The system according to any one of claims 1 to 5.

7. The processor further uses the user-induced motion characteristics and / or the future motion characteristics to the imaging device; the light source of the personal care device; the personal care device, the elements of the personal care; and adapt one or more parameters in one or more of the personal care functions, the system according to any one of claims 1 to 6. **Claim 8** A personal care system, comprising a personal care device having a personal care element, an imaging device attached to the head of the personal care device, the image processing system according to any one of claims 1 to 7, and a light source for illuminating the field of view of the imaging device, a personal care system. **Claim 9** The personal care system according to claim 8, wherein the light source is a light emitting diode LED. **Claim 10** In a computer-implemented image processing method for processing an image received from an imaging device, the imaging device is attached to the head of a personal care device having a personal care element that can move relative to a user of the imaging device to implement a personal care function, and the method includes receiving a sequence of images of the user over time from the imaging device on the head of the personal care device; segmenting the personal care element from the sequence of images; constructing a three-dimensional (3D) volume based on the temporal change of the segmented shape; processing the 3D volume to determine the user-induced motion characteristics of the personal care element and / or predict the future motion characteristics of the personal care element. **Claim 11** The step of processing the 3D volume includes determining the topological characteristics of the 3D volume, and inputting the topological characteristics into a regression algorithm, the regression algorithm outputting the user-induced motion characteristics and / or the future motion characteristics, the method according to claim 10. **Claim 12** The topological characteristics include connectivity, distance matrix, topological signature, local shape, and one or more of point signatures, the method according to claim 11. **Claim 13** Receiving motion-related sensor data from a sensor arrangement on the personal care device, wherein the sensor data indicates movement of the personal care device and / or the personal care element; The method according to any one of claims 10 to 12, further comprising processing the 3D volume using the sensor data. **Claim 14** Using the user-induced motion characteristics and / or the future motion characteristics to The imaging device; The light source of the personal care device; The personal care device; The personal care element; and The method according to any one of claims 10 to 13, further comprising adapting one or more parameters in one or more of the personal care functions.

Images