Personal Care Device with Integrated Camera

Abstract

A personal care device has an illumination system that projects a known illumination pattern onto a surface to be imaged, and an image sensor that captures an image of the surface to be imaged, wherein the illumination pattern is detected in the captured image and the orientation of the surface to be imaged is derived from the shape of the illumination pattern.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION The present invention relates to personal care devices, and in particular to devices incorporating imaging capabilities, for example to provide feedback on the effectiveness of personal care functions or to allow other diagnostic assessments to be made. [Background technology]

[0002] There is a trend to incorporate cameras or other image sensors into personal care devices. For example, cameras are known to be integrated into devices such as electric toothbrushes, shaving devices, intense pulsed light (IPL) hair removal devices, and other skin care devices such as facial cleansing brushes.

[0003] Such imaging-assisted personal care devices can be enablers for remote image-based diagnosis, superior position detection, treatment planning (eg, orthodontics, aligners in oral care), or treatment monitoring (eg, periodontal disease).

[0004] The imaging functionality can be seamlessly integrated into a user's normal personal care routine, instead of requiring the use of a separate device such as a smartphone or handheld intraoral scanner. Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable for the imaging functionality to be able to determine the orientation of the surface being imaged relative to the personal care device, since personal care devices often have relative orientations in which they function best and others in which they function less well, and therefore knowledge of the orientation of the surface being imaged is important for intervening in the functioning of such personal care devices.

[0006] Furthermore, it would be desirable for the imaging function to enable 3D modeling of the surface being imaged, so that the surface contours can be analyzed and / or more accurate position detection can be enabled. One approach to 3D imaging is to provide the camera with an image sensor and a depth sensor as well as an inertial monitoring unit, so that the position and orientation of the camera relative to the object being imaged is obtained. A 3D image can be constructed from multiple 2D images (+depth) from different known viewpoints.

[0007] Another known approach to reconstructing the 3D shape of human tissue is to project 2D calibration grids or structured light patterns onto the tissue surface and process the 2D images of these patterns to obtain 3D information. However, this requires large and / or multiple projector systems or additional laser light sources to generate the required interference patterns. Such approaches are therefore not suitable for compact integration into handheld personal care devices.

[0008] Furthermore, the movement of the handheld device and the movement of the object being imaged (such as a person) means that 3D reconstruction using 2D images with superimposed distorted light patterns, or with device orientation measurements using an inertial measurement unit, is not accurate and suffers from drift.

[0009] Therefore, there is a need for a simple solution that allows for the determination of the orientation of the surface being imaged relative to the personal care device, and further, for a simple solution that measures the distance to the surface (e.g., allowing for the determination of the magnification factor), and optionally even allows for 3D modeling of the surface imaged by the imaging system, which can be integrated into the personal care device.

[0010] US2020 / 0359777 discloses a dental device with an imager that captures images outside a user's oral cavity from which a body part can be identified. The position of the dental device can be determined relative to the body part. An anatomical model can also be generated using additional intraoral images. [Means for solving the problem]

[0011] The invention is defined by the claims.

[0012] According to an example embodiment of the present invention, there is provided a personal care device comprising:

[0013] a head that provides a personal care function to a tissue surface;

[0014] an in-head illumination system that projects a known illumination pattern onto the surface being imaged;

[0015] an in-head image sensor that captures an image of the surface being imaged;

[0016] a processor, the processor comprising:

[0017] Detecting the illumination pattern in the captured image;

[0018] The illumination pattern is configured to derive the orientation of the imaged surface from the shape of the illumination pattern.

[0019] The device utilizes an on-board lighting system to provide a pattern of lighting that is imaged by an on-board image sensor. The positional relationship between the lighting system and the image sensor is known, so that the sensed shape of the lighting pattern can be used to identify (at least) the orientation of the surface being imaged. This information can be used to determine the manner in which the personal care device is being used, which can be used to provide feedback or guidance to the user.

[0020] The personal care device is, for example, a toothbrush or a shaver.

[0021] In a first set of examples, the illumination system includes a light source and a collimating lens. The collimating lens projects a constant pattern (i.e., a constant size and shape) for all depths. Thus, at any depth, deformation of the pattern indicates the orientation of the surface being imaged. In some examples, the position of the pattern within the field of view of the image sensor can be used to obtain depth information.

[0022] In a second set of examples, the illumination system comprises a light source and a lens system for creating a depth-focused light beam, and the processor is further configured to derive a distance to the surface being imaged from the size of the illumination pattern.

[0023] In this way, surface orientation information and depth information can be obtained. The distance of the pattern from the focal plane affects the size of the pattern (because it has different degrees of defocus). The depth information is used to create a more accurate 3D model of the imaged surface.

[0024] The known illumination pattern may be a single unitary shape. The single shape may be a closed shape (e.g., a circle or a square), and it may be solid or may be just the outline of the shape. Therefore, only the single unitary shape needs to be analyzed; the rest of the image is not covered by the illumination pattern. For example, the illumination pattern typically covers less than half of the image, or less than a quarter of the image.

[0025] The processor may further be configured to derive the distance between the head and the surface being imaged from the position, size and / or shape of the illumination pattern, for example using the approaches outlined above.

[0026] If there is a gap between the illumination system and the image sensor, the illumination pattern will appear at different positions in the image depending on the depth of the surface being imaged.

[0027] In some cases, the illumination pattern may become larger (less focused) as depth increases, and thus the size of the pattern can be used to determine depth. Similarly, the shape of the pattern may change with depth.

[0028] Therefore, depth can be determined solely using the lighting system and the image sensor.

[0029] The processor may further be configured to determine an up-to-scale point cloud of the imaged surface by applying structure-from-motion to the captured images and scaling the up-to-scale point cloud using the distance between the head and the imaged surface.

[0030] Generating point clouds from moving images is a known concept. However, it only generates up-to-scale point clouds (not absolute measurements). In this case, a single depth measurement associated with a given direction corresponding to an image pixel can be used to provide the scaling of the point cloud, allowing the shape of the imaged surface to be discovered.

[0031] The lens system can have multiple lenses to create a set of depth-focused light beams, so that different light beams are used to distinguish different depths, improving depth resolution.

[0032] The set of different depth-focusing light beams may each have a depth-focusing light beam with a different focal depth. However, there may be depth-focusing light beams with multiple different focal depths, and there may be multiple focused light beams for at least one focal depth. These multiple beams with the same focal depth can be used to determine the more macroscopic topography of the surface tissue.

[0033] The illumination system may include a light source and a lens system to create a pattern that is in focus at all depths of interest but has different pattern sizes at different depths. Thus, different pattern sizes (rather than different amounts of defocus) can be used to encode depth information. The lens system may, for example, include an axicon lens to create an annular illumination pattern.

[0034] In all examples, the processor may be further configured to obtain an orientation of the imaged surface when the motion of the personal care device is relatively small, or when the motion of the personal care device is relatively small relative to the surface being imaged. The motion of the personal care device may be, for example, an intrinsic motion of the device caused by an actuator within the device, rather than a user's motion of the device along the tissue surface. Imaging when there is little intrinsic motion reduces noise introduced by the device motion.

[0035] In one example, the head has an actuator that is moved at an operating frequency and the lighting system has a strobe light source that is strobed at the operating frequency of the actuator, thus the motion is caused by the movement of the personal care device itself rather than motion imparted by a user.

[0036] In all examples, the illumination system may further comprise a steering system for steering the illumination pattern over time, which allows for scanning of the surface and provides a high resolution image.

[0037] The personal care device may have a motion sensor for sensing movement of the personal care device and a proximity sensor for sensing the surface being imaged, which allows the distance and orientation to the surface being imaged to be tracked.

[0038] The processor can be further configured to predict the movement of the personal care device from image analysis, which can be achieved without the need for other sensor-based localization techniques.

[0039] The present invention also provides a method of imaging a surface using a personal care device having a head, an in-head illumination system for projecting a known illumination pattern onto the surface to be imaged, and an in-head image sensor, the method comprising: receiving an image captured by the image sensor; detecting an illumination pattern in the captured image; The method includes the step of obtaining the orientation of the surface to be imaged from the shape of the illumination pattern.

[0040] The method may further include obtaining a distance to the surface being imaged from a size of the illumination pattern, wherein the illumination pattern is: a depth-focused light beam, or It has patterns that are in focus at all depths of interest, but with different pattern sizes at different depths.

[0041] The present invention also provides a computer program comprising computer program code means adapted to implement the method defined above when executed on a computer.

[0042] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. [Brief explanation of the drawings]

[0043] [Figure 1] 1 is a schematic diagram of a personal care device; [Figure 2] 1A and 1B show the imaging system of the personal care device in a top view (top image) and a side view (bottom image). [Figure 3] FIG. 1 is a diagram illustrating a first example of an imaging system. [Figure 4] FIG. 10 is a diagram illustrating a second example of an imaging system. [Figure 5] FIG. 10 is a diagram illustrating a third example of an imaging system. [Figure 6] FIG. 10 is a diagram illustrating a fourth example of an imaging system. [Figure 7] FIG. 10 shows plots of toothbrush head angle and angular velocity and associated motor drive signals. [Figure 8] FIG. 10 illustrates timing plots of angles and angular velocities when illumination patterns are generated and image capture occurs. [Figure 9] FIG. 1 illustrates a method for imaging a surface using a personal care device. DETAILED DESCRIPTION OF THE INVENTION

[0044] For a better understanding of the present invention, and in order to show more clearly how it may be carried into effect, reference will now be made to the accompanying drawings, which are given by way of example only, in which:

[0045] The present invention will now be described with reference to the figures.

[0046] It should be understood that the detailed description and specific examples, while indicating exemplary embodiments of the devices, systems, and methods, are for purposes of illustration only and are not intended to limit the scope of the invention. These and other features, aspects, and advantages of the devices, systems, and methods of the present invention will be better understood from the following description, appended claims, and accompanying drawings. It should be understood that the figures are schematic only and are not drawn to scale. It should also be understood that the same reference numerals are used throughout the figures to indicate the same or similar parts.

[0047] The present invention provides a personal care device having an illumination system for projecting a known illumination pattern onto a surface to be imaged, and an image sensor for capturing an image of the surface to be imaged, wherein the illumination pattern is detected in the captured image and the shape of the illumination pattern derives (at least) the orientation of the surface to be imaged.

[0048] 1 schematically illustrates a personal care device 100 having a head 102 and a handle 104. The head 102 provides a personal care function to a tissue surface. The personal care function may be teeth cleaning (the head is a toothbrush head or an oral irrigator head), shaving or hair trimming (the head is a shaver or hair trimmer head), hair removal (the head is an epilator head or an intense pulsed laser treatment head), hair brushing (the head is a brush head), or indeed any other personal care function that interacts with a user's tissue surface.

[0049] The personal care device may be entirely conventional with respect to its personal care function, and as such, a detailed description of the typical structure and operation of a personal care device, such as an electric toothbrush or electric shaver, will not be provided.

[0050] The head has an imaging system 110 for capturing 2D images, which are fed to a processor 112. The imaging system projects a known illumination pattern onto the surface to be imaged and captures the images, and the processor detects the illumination pattern in the captured images and derives the orientation of the imaged surface from the shape of the illumination pattern. The processor 112 can also run a 3D reconstruction algorithm to generate a 3D map of the body surface using the captured images and characteristics of the illumination pattern in the captured images.

[0051] In particular, the size and / or shape of the illumination pattern varies as a function of the distance between the imaging system and the surface onto which the illumination pattern is projected (i.e., the depth of the surface relative to the imaging system), which provides an absolute measure of depth.

[0052] When a circle is projected onto an inclined surface, distance can be derived from the degree of asymmetry of the ellipse-based shape that is formed. This asymmetry is therefore a property of the shape of the illumination pattern. If the surface is close, a very large asymmetry occurs, and if the surface is far, a small asymmetry occurs. Similarly, when a square is projected onto an inclined surface, the ratio of the lengths of the square's opposing sides provides an indicator of how close the faces are.

[0053] If sufficient texture features are present in the images, the set of captured images can be input to, for example, a structure-from-motion (SfM) algorithm that outputs an up-to-scale point cloud of the imaged structure. 3D reconstruction can then be achieved by using the up-to-scale point cloud and scaling it using absolute depth measurements. For example, a mesh can be generated based on the scaled point cloud.

[0054] 2 shows imaging system 110 from a top view (top image) and a side view (bottom image). Imaging system 110 has an illumination system including a light source 120, appropriate optics 121 (e.g., lens, aperture, pinhole, etc.) for projecting a known illumination pattern onto the surface being imaged, and an in-head image sensor 122, which is spatially offset from the illumination system. The image sensor captures an image of the surface being imaged. FIG. 2 also shows an optional motion sensor 124, such as an inertial monitoring unit.

[0055] In one example, lighting systems 120, 121 project a circular dot onto a surface, and a processor analyzes an image of the projection from an offset image obtained via image sensor 122. The resulting center position of the approximately elliptical shape of the dot in the image can provide a measure of depth, since the relative orientations of lighting systems 120, 121 and the image sensor are known, and therefore depth can be obtained via triangulation.

[0056] A second method for determining depth is to measure the size of the dots. In some cases, the illumination system 120, 121 can project shapes that decrease in size with increasing depth. Thus, by measuring the size of the projected dots in the image, depth can be determined.

[0057] In general, depth can be determined when the illumination systems 120, 121 project shapes whose shape and / or position change predictably with depth.

[0058] In this example, the surface orientation relative to the optical axis of the image sensor 122 can be calculated from the orientation of the ellipse and / or the ratio of the minor and major axes of the ellipse. Calculating the surface orientation can be ambiguous because, for a given elliptical shape, two solutions are equally likely for the surface orientation. By combining measurements of the moving image sensor 122 over time, this ambiguity can likely be resolved through filtering.

[0059] Additionally, distortion of the ellipse may indicate that the surface is tilted relative to the illumination system 120, 121. An algorithm can be trained to determine the orientation of a surface based on the shape of the ellipse, including any distortion that may result from tilt. For example, an inertial measurement unit (IMU) can be used to provide a ground truth orientation to train the algorithm, so that after training, the IMU is no longer needed. The same concept can be applied to train an algorithm that determines depth based on the shape / size / position of the ellipse, using a depth sensor for ground truth measurements.

[0060] The illumination system of the imaging system may, for example, comprise an LED arrangement and a lens system for illuminating the tissue surface and generating the projection pattern. Alternatively, the imaging system may comprise a superluminescent diode or a laser diode, and instead of a lens, a simple focusing system such as an aperture or pinhole may be used to generate the illumination pattern.

[0061] The illumination system may, for example, be oblique to the image sensor, which may comprise a camera or a CMOS sensor used to acquire images from the tissue surface.

[0062] The patterns include light dots, crosses, diamonds, slits, stars, parallelograms, circular rings, etc., which are used as a means of reconstructing a 3D image from multiple 2D images.

[0063] The pattern can have a single unitary shape (solid or hollow), in which case a simple, singular shape can be used to obtain surface orientation and depth.

[0064] One or more of the position, size, and shape of the illumination pattern can be used to enable computationally efficient 3D image reconstruction of, for example, the outer skin or the oral cavity. The illumination system and image sensor are offset (in spatial position and, optionally, in orientation angle) relative to each other. As a result, triangulation can be used to calculate the depth of the imaging point. A larger distance between the illumination system and the image sensor position allows for more accurate depth measurements.

[0065] If 3D reconstruction is required, the captured images (over time) can be used to generate an up-to-scale point cloud using an SfM algorithm as described above. Depth measurements can be used to scale the point cloud.

[0066] For example, if the shaver's illumination source projects a guidance dot onto the skin, the intensity and shape of the dot will vary based on the orientation and distance of the shaver relative to the skin and the relative movement of the shaver in space. If a user shaves in the recommended manner of small, repetitive circular motions, the shaving may produce a 3D trajectory such as a spiral pattern.

[0067] Such device movement results in dynamic in-plane or out-of-plane changes in the shape and / or intensity of the illumination pattern (e.g., light dots changing from round to oval, from bright to dull, or showing asymmetric distortion when an inclined surface is imaged). The captured information allows the 3D topographical profile of the imaged surface (and the orientation of the device relative to the skin surface) to be determined. All of this information can be used to perform 3D reconstruction. The 3D reconstruction result can remain sparse, i.e., with the 3D positions of a sparse set of points, or can be converted into a dense 3D model, for example, via converting the point cloud into a mesh representation.

[0068] When the optional motion sensor is used, the combination of the device's movement and orientation with image analysis allows for more accurate 3D reconstruction. The inertial monitoring unit (IMU) provides a reference coordinate system (i.e., a coordinate system relative to the gravity vector). Especially when camera movements are relatively small, the accuracy of SfM algorithms can decrease. Therefore, the IMU provides more accurate information about the camera's position (when capturing images), thereby increasing the accuracy of the point cloud.

[0069] It is worth noting that SfM algorithms tend to assume that the imaged structures are stationary. Therefore, it is preferable to use only a subset of captured images taken over a relatively small time window (e.g., 20 frames over 2 seconds). Of course, the exact time window depends on how the personal care device is typically used (e.g., how quickly, for how long, etc.).

[0070] In some cases, it may be sufficient to determine the relative orientation of the surface being imaged. For example, if a personal care device functions best in a particular orientation with respect to the surface (e.g., parallel to the surface), knowledge of the relative orientation may be sufficient to adapt the personal care device accordingly (e.g., power off in an inappropriate orientation).

[0071] However, in other cases, it may be preferable to extract feature points (e.g., texture features of teeth, hair roots, etc.) from the captured images, track these features over time, and use structure-from-motion to determine the 3D location of these features. Furthermore, these features may provide more context to the orientation and / or depth measurements. For example, an independent measurement of the depth of the device relative to the surface, combined with knowledge of the 3D location of the feature, may enable determining whether a change in orientation and / or depth is due to movement of the personal care device or movement of the imaged structure.

[0072] FIG. 3 shows a first example of an imaging system.

[0073] The illumination system comprises a light source 120 and a collimating lens 300. The collimating lens 300 produces a parallel light beam, enabling tracking and tilt measurement. The collimating lens can be provided as a piece of simple clear plastic molded window with microlens structures.

[0074] In its most basic implementation, the illumination pattern comprises a pattern of dots (instead of more complex structured light patterns), which simplifies image reconstruction and thereby saves computational power. Any variations in the shape and intensity of the projected dots (e.g., round vs. oval, bright vs. dull) provide information about the 3D topographical profile of the imaged surface.

[0075] Specifically, if the shape of the collimated beam is a circular dot, projection onto a plane perpendicular to the imager will produce a circular feature in the image. In contrast, projection onto a plane oblique to the imager will produce an elliptical feature in the image, as shown in Figure 3.

[0076] The ratio of the major and minor axes of an ellipse is a measure of the angle, and the direction of the axis is a measure of the direction of tilt.

[0077] The position of the projected dot can provide an indication of the depth of the surface because the illumination system is farther from the image capture system, so a greater depth places the dot closer to the center of the image, and a smaller depth places the dot closer to the edge of the image.

[0078] 4 shows a second example where the illumination system includes a light source 120 and a lens system 400 for creating a depth-focused light beam. In this example, a simple transparent plastic molded window with a microlens structure can again be used to generate the depth-focused light beam. This allows for estimation of the distance of the surface being imaged from the imaging device.

[0079] Specifically, the fact that the beam is no longer collimated allows the size or intensity of the spot to be translated into distance from the image sensor. For example, a tissue surface at a distance equal to the focal length of the lens will produce a small spot of high brightness. In contrast, when moving away from the focal point, the spot will have a large size and low intensity. This is depicted in Figure 4.

[0080] If the surface is tilted relative to the imaging sensor, the spot will again be distorted in a similar manner as described above.

[0081] However, in this case, the distorted ellipse forms a narrower section with its major axis closer to the imaging device than away from it. Therefore, as a result of projecting onto a tilted surface, the ellipse will be distorted in an asymmetric manner. Therefore, based on the asymmetric shape of the ellipse, it can be inferred that the surface is tilted. Furthermore, the asymmetric shape can be used to calculate the orientation of the tilted surface.

[0082] In this case, the best measure of the distance to the device is given by the length of the minor axis, since this is the distance to the center of the spot. By analyzing the asymmetry of the ellipse, an indication of the local shape can be obtained.

[0083] One problem with this method is that the spot size will be the same for tissue surfaces at the same distance before and after the focus point. This uncertainty can be addressed by providing a lens system with multiple lenses that create a set of depth-focused light beams.

[0084] Multiple lens structures of the type shown in FIG. 4 can even be combined within a single microlens structure, as shown in FIG.

[0085] There are now three lenses 502, 504, 506 with different focal lengths. This allows the distance from the imaging system device to the surface to be determined more accurately. For example, if there are three different focal planes (as shown in FIG. 5), the tissue distance is closest to the lens focus with the smallest image (the middle lens in this case). Furthermore, the pattern of the three lenses indicates whether the surface being imaged is in front of or behind the lens focus position.

[0086] Figure 5 shows three lenses with different focal lengths. However, multiple lenses can have the same focal length. This can be used to determine the macroscopic topography of the tissue surface by measuring the distance from these two lenses and calculating using triangulation evaluation to obtain the surface orientation.

[0087] For example, if three or four such lenses are applied around the periphery of the illumination area, triangulation can be used to give an indication of the orientation of the device relative to the tissue surface.

[0088] 6 shows another example of an illumination system having a light source 120 and a lens system 600 for creating a pattern that is in focus at all depths of interest but has different patterns at different depths. The lens system may, for example, have an axicon lens for ring illumination, with a constant depth of focus but a variable diameter. The axicon lens has the shape of a conical prism.

[0089] The annular light distribution has a constant ring thickness, and the ring illumination remains focused at all depths regardless of the distance to the surface, but it varies in diameter as shown in Figure 6.

[0090] The observed ring shape provides additional depth cues useful for 3D surface reconstruction. Assuming that locally the perspective projection can be approximated by an orthographic projection, the image of the projected circle becomes an ellipse. The orientation of the larger semi-axis and its ratio to the smaller semi-axis can be used to calculate the surface orientation, similar to the example above. This helps constrain the solution, especially when the method is combined with other 3D sensor modalities such as structure-from-motion (SfM), using local slow changes in the 3D surface shape.

[0091] In another example, the projection of the illumination pattern and image capture occurs at a stationary or slow moment when the image sensor is substantially motionless.

[0092] If the illumination is concentrated when the device is stationary, the image reconstruction process is easier and faster (as a sharper illumination pattern can be used for reconstruction purposes).

[0093] The orientation (and depth, if depth is measured) of the imaged surface is then determined when the movement of the personal care device is relatively small, or when the movement of the personal care device is relatively small relative to the imaged surface.

[0094] In one example, the motion is caused by an actuator of the device (e.g., a motor that generates rotational or vibrational motion of the toothbrush head). The actuator creates motion at an operating frequency.

[0095] The illumination source is strobed at the operating frequency of the actuator.

[0096] 7 shows a toothbrush head angle plot 700 and angular velocity plot 710 in the top pane, the middle plane shows the motor drive signal, and the bottom pane shows the motor drive current.

[0097] FIG. 8 shows timing 800 in plots of angle and angular velocity as an illumination pattern is generated and image capture occurs.

[0098] This provides image stabilization and vibration compensation for the vibrating toothbrush. Timing 800 is generated using a strobe light that flashes at the operating frequency of the toothbrush and corresponds to low velocity regions (or zero brush head movement). This gives a sharper lighting pattern and makes image reconstruction easier.

[0099] In another example, the lighting system further comprises a steering system for steering the lighting pattern over time.

[0100] This approach is of interest, for example, when the device does not have an inherent actuator (e.g., motor, drive train, vacuum pump) that creates vibration or periodic motion. The illumination pattern can be steered to allow optimal spatial sampling (in a statistical sense). In other words, this approach projects an illumination pattern over the entire surface (or most of it) over time.

[0101] If a dot-shaped illumination pattern is always projected in the same direction, the movement of the device moving the image sensor will typically produce a linear locus of points because the device movement is slowly changing (e.g., controlled by hand movements). For a typical image sensor capture frame rate, e.g., 10 Hz, a 2-second capture will result in, e.g., 20 points lying in a straight line, concentrated in a limited portion of the combined image.

[0102] To aid in 3D reconstruction, the dots can be sequentially steered to different locations. For example, a 3x3 LED array can be used, with a pseudo-random selection of which LEDs to light for each frame. This spreads the combined dot pattern across space over the 2-second acquisition. An alternative to using an LED array is to use a fluidic diffractive surface whose orientation can be controlled.

[0103] As explained above, motion sensors can be used to measure the movement of the device. Alternatively, it is possible to predict the movement of the personal care device from image analysis. In particular, the calibrated shape and dimensions of the illumination pattern can be used to train an algorithm used to predict the position of the device.

[0104] By knowing the device's orientation and distance relative to the surface being imaged (e.g., using an IMU and depth sensors to obtain ground truth data), an algorithm can be trained to predict the device's localization (device orientation / position and distance from the tissue surface) without requiring other sensor-based localization techniques during use, i.e., simplifying the system structure of the handheld device.

[0105] 9 illustrates a method for imaging a surface using a personal care device having an illumination system that projects a known illumination pattern onto the surface to be imaged. The method includes receiving an image captured by an image sensor in step 900. In step 902, the illumination pattern is detected in the captured image, and in step 904, the orientation of the imaged surface is derived from the shape of the illumination pattern.

[0106] There are a variety of possible reconstruction algorithms for generating 3D images.

[0107] As mentioned above, structure-from-motion (SfM) can be used to determine the 3D positions of feature points and camera pose over time by using images captured over a time window. This approach assumes a stationary scene for a given time window. The resulting point cloud has unknown up-to-scale parameters. The 3D metric positions of the LED / laser dots can be used to infer the true scale of the scene. For example, consider 100 frames of a moving camera. This gives measurements of 100 points on a 3D surface. Due to device motion, each dot is in a different position relative to the image features. Assigning the 3D position of each point in the 100 images to the spatially closest image feature ties the point cloud to the metric measurements.

[0108] An exemplary feature detector is the Scale Invariant Feature Transform (SIFT), but there are many others. Popular frameworks such as AliceVision or ColMap are examples. For a given part of the human body surface, a dedicated feature detection neural network can be trained, so that this module works optimally for a given scene under given lighting conditions.

[0109] Variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the figures, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality.

[0110] The functions performed by a processor may be performed by a single processor or by multiple individual processing units which together may constitute a “processor.” Such processing units may, in some cases, be remote from each other and in wired or wireless communication with each other.

[0111] The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0112] The computer program may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless communication systems.

[0113] It should be noted that when the term "adapted to" is used in the claims or the specification, it is intended to be equivalent to the term "configured to." It should be noted that when the term "arrangement" is used in the claims or the specification, it is intended to be equivalent to the term "system," and vice versa.

[0114] Any reference signs in the claims should not be construed as limiting the scope of the invention.

Claims

1. 1. A personal care device comprising: a head that provides a personal care function to a tissue surface; an in-head illumination system that projects a known illumination pattern onto the surface being imaged; an in-head image sensor that captures an image of the imaged surface; a processor, the processor comprising: Detecting the illumination pattern in the captured image; A personal care device that derives an orientation of the imaged surface from the shape of the illumination pattern in the captured image.

2. The personal care device of claim 1 , wherein the known illumination pattern is a single unitary shape.

3. The personal care device of claim 1 , wherein the processor further derives a distance between the head and the imaged surface from a position, size, and / or shape of the illumination pattern.

4. The processor further comprises: determining an up-to-scale point cloud of the imaged surface by applying structure-from-motion to the captured images; The personal care device of claim 3 , wherein the distance between the head and the imaged surface is used to scale the up-to-scale point cloud.

5. 5. A personal care device according to any preceding claim, wherein the illumination system comprises a light source and a collimating lens.

6. 5. The personal care device of claim 1, wherein the illumination system comprises a light source and a lens system that creates a depth-focused light beam, and the processor further derives the distance to the imaged surface from the size of the illumination pattern.

7. 7. The personal care device of claim 6, wherein the lens system comprises a plurality of lenses that create a set of depth-focused light beams.

8. A set of different depth-focused light beams depth-focused light beams each with a different focal depth, or 10. The personal care device of claim 7, having a depth-focused light beam with a plurality of different focal depths, the depth-focused light beam comprising a plurality of focused light beams for at least one focal depth.

9. 5. The personal care device of claim 1, wherein the illumination system comprises a light source and a lens system that creates a pattern that is in focus at all depths of interest but has different pattern sizes at different depths.

10. 10. The personal care device of claim 9, wherein the lens system comprises an axicon lens that creates an annular illumination pattern.

11. 5. The personal care device of claim 1, wherein the processor obtains the orientation of the imaged surface when the movement of the personal care device is relatively small or when the movement of the personal care device is relatively small relative to the imaged surface.

12. the head has an actuator that is moved at an operating frequency; 12. The personal care device of claim 11, wherein the lighting system comprises a strobe light source that is strobed at an operating frequency of the actuator.

13. 5. The personal care device of claim 1, wherein the lighting system further comprises a steering system for steering the lighting pattern over time.

14. 1. A method of imaging a surface with a personal care device including a head, an in-head illumination system that projects a known illumination pattern onto the surface to be imaged, and an in-head image sensor, comprising: receiving an image captured by the image sensor; detecting the illumination pattern in the captured image; and deriving an orientation of the imaged surface from the shape of the illumination pattern in the captured image.

15. A computer program having computer program code means adapted to implement the method of claim 14 when said computer program is executed on a computer.

Images