Handheld device

Abstract

Schemes, solutions, concepts, designs, methods, and systems are presented that involve assisting and / or improving image acquisition of a surface portion of an object by a handheld device having a plurality of end effectors for engagement with the surface portion of the object. In particular, deformation of the end effector under load may cause some end effectors to intrude the field of view of image capture of the user surface. Thus, we recognize that the quality of the image can be determined using the load condition of the end effector determined during image capture. Accordingly, transmission of an image may be selectively controlled according to image quality. In this manner, the overall data transfer rate may be reduced by ignoring the poor quality (i.e., highly occluded) image transmissions.

Description

Technical Field

[0001] The present invention relates to the field of handheld personal care devices (e.g., treatment devices or therapeutic devices), and more particularly to handheld personal care devices for capturing images of an object's surface. Background Art

[0002] Handheld personal care devices, such as toothbrushing / combing devices, shaving devices, and breast pumps, are often used. In recent years, cameras and other imaging sensors have been integrated into handheld devices to capture images of the user's surface near the device's end effector (e.g., bristles, filaments, razor caps, etc.).

[0003] Image capture enables image-based remote diagnosis, enhanced position sensing, treatment planning, and / or process monitoring. Nowadays, the process of capturing these images is being gradually and seamlessly integrated into the user's regular personal care devices and daily care, without the use of standalone devices such as smartphones, dental endoscopes, or handheld intraoral scanners. This seamless integration can avoid additional trouble for the user and can immediately provide image-based feedback to the user, as there is no need to use smartphone-based image capture or perform additional personal care daily workflows (such as taking a separate image after brushing).

[0004] However, it is well known that during the use of the device, the end effector typically deforms under the force applied by the user. As a result, the end effector easily encroaches on the camera's image field of view, making image interpretation more difficult (sometimes even impossible). Therefore, a large number of images usually need to be acquired before a useful image can be obtained. This leads to a large amount of data transmission, and most of this data is relatively useless for realizing the advantages of the above-mentioned image capture. Summary of the Invention

[0005] The present invention is defined by the claims.

[0006] According to an example of one aspect of the present invention, there is provided a handheld personal care device, comprising:

[0007] a plurality of end effectors for engaging a portion of an object's surface;

[0008] an image capture device configured to capture an image of the object's surface;

[0009] a sensor unit configured to determine the load state of the plurality of end effectors while the image is being captured; and

[0010] a control unit configured to determine a quality value of the image based on the load state and selectively control the transmission of the image based on the quality value.

[0011] The present disclosure presents solutions, concepts, designs, methods, and systems related to assisting and / or improving image acquisition of a portion of an object's surface by a handheld device having a plurality of end effectors for engaging a portion of the object's surface. Specifically, deformation of the end effectors under a given load state may cause some of the end effectors to encroach on the field of view of the image capture of the user's surface. Thus, the present invention recognizes that the load state of the end effectors determined / estimated / predicted during the image capture process can be utilized to determine the image quality. Accordingly, the transmission of the image can be selectively controlled based on the image quality. In this way, the overall data transmission rate can be reduced by ignoring the transmission of images of poor quality (i.e., highly occluded).

[0012] For example, images of a portion of the user's surface captured by a handheld device having end effectors for engaging the (same) surface of the user are often occluded / encroached upon by the end effectors. This is because when the end effectors contact the user's surface, they experience a load state (i.e., due to force / pressure), causing them to deflect / deform / be forced into the field of view of the image capture device. Of course, in cases of a high degree of end effector encroachment, the user's surface may not even be visible in the image. In addition to the end effectors encroaching on the camera's field of view, the high load state also affects the image capture device itself, thereby degrading the image quality.

[0013] Such images may be of little or no use for further processing. The further processing may include remote image-based diagnosis, enhanced position sensing, treatment planning, and / or process monitoring.

[0014] Accordingly, each image is selectively transmitted. In practice, this means that images of poor quality may not be transmitted (or transmitted at a high compression ratio), while those that provide a clear view of the user's surface are transmitted (or transmitted losslessly or at a low compression ratio). This means that the bandwidth required to transmit the captured images, the storage at the receiver, and the processing required at the receiver are all reduced, while not overly reducing the overall usefulness / quality of the data acquired by the image capture device.

[0015] In addition, the load state can be directly measured / determined, or the likelihood / probability that the end effector is in a loaded state can be estimated. The load state of the end effector provides an efficient (i.e., fast and low computational complexity) method for determining / estimating / predicting the quality of an image. While directly evaluating an image of the user's surface using image processing techniques can provide a highly accurate assessment of image quality, such a solution may be computationally intensive and slow, and this level of accuracy may not be necessary. This may not be ideal for handheld devices with limited size, power, and processing capabilities. Therefore, leveraging the load state as an indirect means of evaluating image quality can facilitate an improved (i.e., more efficient and faster) method for selective image transmission.

[0016] In addition, the present invention proposes selective transmission of images, rather than selective capture of images. Selectively capturing an image in response to a control signal typically requires a complex image capture device, which in itself requires consideration of cost, power, and space. In contrast, image capture devices that simply repeatedly capture images (i.e., at fixed / random time intervals) are widely available, compact, and inexpensive. Therefore, what the present invention proposes is to capture an image and then evaluate its quality (based on the determined / detected / measured load state). This thus helps to reduce the overall cost of handheld devices.

[0017] In some embodiments, the quality value can indicate the degree to which an image of an object surface is occluded by multiple end effectors.

[0018] The quality of an image is related to the extent of the presence of end effectors in the image. Of course, this directly affects the usefulness of the image, as a highly occluded image may not contain sufficient information for further processing. In other words, multiple end effectors present in the image act as noise, and thus the degree to which the end effectors occlude the image determines the degree of noise (i.e., quality) in the image.

[0019] Therefore, defining quality in this way may lead to more efficient selective transmission.

[0020] In an exemplary embodiment, the load state can describe the deformation level of at least one of the multiple end effectors.

[0021] Different load states indicate different degrees / levels / extents of deformation / deflection of the multiple end effectors (thereby indicating the extent to which the end effectors may be deformed into the imaging field of view of the image capture device). The greater the load state applied to the end effector (i.e., the greater the force or pressure), the greater the deformation / deflection of the end effector. Of course, this degree of deformation indicates the likelihood of the presence of the end effector in the field of view of the image capture device.

[0022] Specifically, the load state can also describe at least one of the deformation direction and deformation mode of at least one of the plurality of end effectors.

[0023] Typically, there is more than one direction of deflection and more than one type. Typically, the end effector may deform from rest in any radial direction, or may deform freely (i.e., when in slight contact with the user's surface), in a constrained manner (i.e., when both ends of the end effector are jammed / fixed), in a trapped manner (i.e., the end effector cannot move), or dynamically due to mechanical instability (bistable jump or buckling). All of these factors affect the prediction / determination of the quality of the captured image based on the load state. Thus, this information can provide a more accurate quality value.

[0024] In another embodiment, the sensor unit may include a sensor configured to detect a force applied by at least one end effector on a portion of the object surface. In this case, the load state may be based on the detected force or pressure.

[0025] One way to determine the load state of the end effector is to directly evaluate / detect / measure the force applied by the end effector to the object surface. Since many devices already integrate such sensors (e.g., some electric toothbrushes), this may provide an economical and convenient means for evaluating the load state.

[0026] Alternatively or additionally, in some embodiments, the sensor unit may include at least one deformation sensing element coupled to one of the plurality of end effectors or being one of the plurality of end effectors, and the deformation sensing element is configured to detect the deformation value of the corresponding end effector. Then, the load state may be based on the detected deformation value.

[0027] Another way to determine the load state is by directly measuring the deformation of the end effector. This can be achieved by a device / sensor that changes its properties and / or output signal according to the degree of deformation it experiences. Thus, coupling these sensing elements to the end effector (or adjusting the sensing element to be the end effector itself) can directly determine the load state based on the deformation of the end effector. Thus, this can provide a means for evaluating the load state with high precision.

[0028] Specifically, at least one deformation sensing element may be one of the following: piezoelectric polymer fiber, fiber Bragg grating, Fabry - Perot sensor, or pressure - sensitive material.

[0029] In addition, the sensor unit may further include an optical proximity sensor configured to detect a distance value between a portion of the handheld device adjacent to the end effector and the surface of an object. Accordingly, the load state may be based on the detected distance value.

[0030] Another method by which the load state is determined is based on the distance detected between a portion of the handheld device at the end effector base and the surface of the object. In essence, this means that the distance occupied by the end effector (when in contact with the surface of the object) can be determined. Of course, when this distance decreases to less than the length of the end effector, the load state can be considered to be increasing. An economical, low-power, and convenient means for evaluating the distance is to use an optical proximity sensor, which can evaluate the distance at multiple locations.

[0031] In some embodiments, the handheld device may further include a vibration actuator adapted to vibrate at least one of the plurality of end effectors. In this case, the sensor unit may include a sensor configured to detect a vibration value of at least one of the plurality of end effectors, and the load state may be determined based on the detected vibration value.

[0032] In many cases, the handheld device includes vibrating components to assist with the function of the device (e.g., for assisting with brushing, massaging, or shaving). Accordingly, the degree of vibration can be evaluated by utilizing a sensor such as an accelerometer. In fact, the vibration may affect the load state on the end effector and, more generally, the quality of the image captured by the image capture device (e.g., by blurring or moving the end effector within the field of view).

[0033] In some embodiments, the control unit may be configured to transmit an image in response to a quality value based on a load state corresponding to a predetermined range of acceptable load states.

[0034] It may be the case that the quality of the image for an acceptable load state is known. This range (which may not be continuous) may vary from device to device and can thus be determined during the design, manufacturing, and testing phases. Accordingly, when it is determined that the load state of the end effector during image capture is acceptable, the image is transmitted (and may be stored and subjected to further processing / analysis).

[0035] More specifically, in some embodiments, the predetermined range of acceptable load states may be bounded by a predetermined minimum load state and a predetermined maximum load state, where the predetermined minimum load state may indicate partial contact of at least one of the plurality of end effectors with the surface of the object, and the predetermined maximum load state may indicate a minimum acceptable amount of the unoccluded surface of the object present in the image.

[0036] Thus, the image is transmitted when the end effector is in contact with the surface of the object at the end but not overly occluded (i.e., when the image quality is still high). This can effectively utilize the bandwidth and power of the handheld device for transmitting images, because the images used for processing will be transmitted (while other images may be discarded, sent at a high compression ratio, or stored locally).

[0037] In some embodiments, each of the plurality of end effectors can be an elastically deformable end effector or a rigid end effector mounted on the handheld device through an elastically deformable base.

[0038] There are many different types of end effectors. Two types that are particularly worthy of attention are: flexible / elastically deformable end effectors, and end effectors that are mounted or include flexible / elastically deformable components. Due to the deformable characteristics of these types of end effectors, they may encroach on the field of view of the image capture device under specific load conditions. Therefore, handheld devices equipped with such end effectors may particularly benefit from the embodiments of the present invention.

[0039] In some embodiments, the handheld personal care device can be a processing device or a treatment device.

[0040] That is, the handheld device can perform processing or treatment of personal care functions.

[0041] According to another aspect of the present invention, there is provided a method for selectively controlling the transmission of an image of the surface of an object acquired by a handheld personal care device, the handheld personal care device including a plurality of end effectors for engaging a part of the surface of the object, the method comprising: capturing an image of the surface of the object; determining the load state of the plurality of end effectors while the image is being captured; determining a quality value of the image based on the load state; and selectively controlling the transmission of the image based on the quality value.

[0042] According to yet another aspect of the present invention, there is provided a computer program, including computer program code means, which is adapted to implement the method of the proposed embodiments when the computer program runs on a computer.

[0043] These and other aspects of the present invention will become apparent from the embodiments described hereinafter and will be elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] To better understand the present invention and to more clearly show how to implement the present invention, reference will now be made, by way of example only, to the accompanying drawings, in which:

[0045] Figure 1 Different deformation modes of the end effector in different load states are shown;

[0046] Figure 2 is a graph showing how the proportion of occluded captured images varies with the deformation of the end effector;

[0047] Figure 3 shows a simplified block diagram of a handheld device according to an embodiment of the present invention;

[0048] Figure 4 shows a flowchart of a method for selectively controlling the transmission of an image of an object surface acquired by a handheld device according to another embodiment; and

[0049] Figure 5 provides a simplified block diagram of a computer in which one or more parts of an embodiment can be employed. Detailed Description

[0050] The present invention will be described with reference to the accompanying drawings.

[0051] It should be understood that the drawings are only schematic illustrations and are not drawn to scale. It should also be understood that the same reference numerals are used throughout all the drawings to denote the same or similar components.

[0052] It should also be understood that although the detailed description and specific examples illustrate exemplary embodiments of the apparatus, system, and method, they are for illustrative purposes only and are not intended to limit the scope of the present invention. These and other features, aspects, and advantages of the apparatus, system, and method of the present invention will be better understood through the following description, appended claims, and drawings. The mere fact that certain measures are recited in mutually different dependent claims does not mean that a combination of these measures cannot be advantageously used.

[0053] The present invention presents a concept for assisting and / or improving a handheld device that can capture an image of an object surface when the end effector of the device is in contact with the object surface. Specifically, the load state of the end effector is determined while the image is being captured. Based on this load state, a predicted quality value of the image can be determined, thereby informing the selective transmission of the image. Generally speaking, this means that images of potentially poor quality will not be transmitted, thus reducing the costs and bandwidth required for such unnecessary transmissions.

[0054] In other words, the proposed concept aims to improve the process of image acquisition of a portion of an object's surface by a handheld device having a plurality of end effectors for engaging a portion of the object's surface. Specifically, the deformation of the end effectors under load may cause some of the end effectors to encroach on the field of view of the image capture of the user's surface. Therefore, it has been realized that the load state of the end effectors determined during image capture can be utilized to determine the image quality. Accordingly, the transmission of the image can be selectively controlled based on the image quality. In this way, the overall data transmission rate can be reduced by ignoring the transmission of images of poor quality (i.e., highly occluded).

[0055] To explain, it is well known that end effectors deform under pressure and are prone to encroaching on the image field of view of the image capture device. Of course, when an end effector encroaches on the image field of view of the image capture device while an image is being captured, the resulting image will contain the end effector. Generally, the end effector is not the intended target of the image capture. Instead, ideally, the image would contain the object surface (at least in part). Therefore, the encroachment of the end effector is undesirable.

[0056] As Figure 1 shown, the deformation can exist in different forms / patterns. In the most ideal case, the end effector can deform or move freely (i.e., contact the surface and move freely), as Figure 1 shown. In other cases, the end effector may be restricted, as shown in 20, which means that the end effector may not always be able to move freely relative to the distal end of the handheld device. In other cases, as shown in 30, the end effector may be stuck in a crack in the processing surface, which means that the distal end cannot move and may thus be ineffective. Of course, these different deflection patterns may also affect whether the end effector encroaches on the field of view of the image capture device.

[0057] Such end effectors can be considered upright, elongated members (e.g., bristles, quills, tufts), but are not limited thereto. An end effector refers to any device by which the handheld device contacts the object surface (e.g., teeth, tongue, skin, hair) for, e.g., scrubbing, brushing, shaving, massaging, etc. They are typically elastically deformable / flexible themselves or are mounted on elastically deformable / flexible members such that they deform / deflect under the pressure applied during typical use of the handheld device.

[0058] Thus, deformations of the end effector, such as uncontrolled opening, jamming, static loading / deformation, can occlude the region of interest to be imaged. One solution is to ignore such image (remote) data analysis post facto, but this increases the total amount of data to be wirelessly transmitted to a cloud or other device internal interface. Another solution is to use known image analysis techniques to process the captured images to determine the extent to which the images contain unoccluded / non-occluded regions of interest, but this necessarily consumes a large amount of computational resources and time.

[0059] Accordingly, embodiments of the present invention provide systems and computer-implemented methods (executed by a processor) for selectively transmitting image data for further processing based on a predicted image quality, which is based on the load state of the end effector determined during image capture. When the load state of the end effector (and thus the predicted / determined image quality) is associated with a given range, the image can be transmitted. The range is preferably greater than zero (i.e., the end effector is not in contact with the object surface) and does not exceed the maximum load state (i.e., the end effector is highly deformed and the image quality is low).

[0060] In other words, the embodiments propose to transmit only the captured images associated with low / zero end effector deformation and / or a lower load state within a specific range, where the force and deformation can be directly measured or indirectly evaluated.

[0061] It has been observed that a reasonably designed image capture device can be positioned and configured such that when the end effector is in a specific load state (e.g., pressure, acceleration, force, etc.), the captured image of the object surface is not encroached upon by the bristles. In some cases, the handheld device vibration may also be in some known specific states. It is also recognized that as long as the specific load state (especially the contact pressure of the end effector) remains within a specified value range, the image can remain sufficiently distortion-free. These values are not necessarily continuous, but are typically monotonic.

[0062] Accordingly, it is proposed to selectively transmit the images associated with the determined load state that lies within a specified range (usually lower) of the load state. In fact, this is visible in Figure 2 which shows the cumulative percentage distribution of the relevant parameters. In this case, only the image data associated with the lower 20% deformation (i.e., the lower 20% of the load state) is transmitted, represented by the shaded area. Thus, the overall data transmission rate is reduced by approximately 5 times. Those skilled in the art will appreciate that other selections can be made to reduce or increase the data rate.

[0063] Accordingly, embodiments of the present invention provide the following main elements:

[0064] (i) A handheld personal care device (e.g., a healthcare, therapeutic, or treatment device). For example, the handheld device can be a toothbrush, a hairbrush, a breast pump, a massaging device, a razor, etc. In any case, the handheld device has multiple end effectors. For example, the end effector of a toothbrush is the bristles, the end effector of a razor is the razor head, and the end effector of a hairbrush is the tufts / bristles / upright members. The handheld device also has an (integrated) image capture device that captures an image of a portion of the surface of the object with which the end effector engages / comes into contact.

[0065] (ii) A processor that has a method / algorithm and is connected to the device to directly measure the end effector deformation through the load state or indirectly measure it (i.e., through a proxy from which the deformation is derived). In other words, the load state can be directly measured / determined, or the processor estimates the likelihood / probability that the end effector is in a loaded state.

[0066] (iii) A design value range or threshold for determining the selectively transmitted image (i.e., transmitted to the cloud for further processing) measured from the device of the end effector deformation (or a threshold for predicting the image quality, or the determined load state).

[0067] Next, Figure 3 A simplified block diagram of a handheld device 100 according to an embodiment of the present invention is shown. Specifically, a plurality of end effectors 110, an image capture device 120, a sensor unit 130, and a control unit 140 are depicted in the figure. Alternatively, a vibration device 112, a communication unit 150, and an external / remote processor 160 can also be provided.

[0068] First, it should be noted that although Figure 3 the illustrated embodiment relates to an (electric) toothbrush, the present invention is equally applicable to other devices in which the end effector deformation may obscure the image of an (onboard) image capture device / camera. For example, Figure 3 a hairbrush, a skin cleansing brush, a toothbrushing head, a razor, etc. can also be depicted. Therefore, those skilled in the art can understand that the handheld personal care device can perform processing or therapeutic personal care functions. Similarly, the positions of the said elements do not prescribe their required relative positions.

[0069] The end effector 110 is configured to engage a portion of the surface of the object. In other words, the end effector 110 contacts a portion of the surface of the object.

[0070] The surface can be the teeth, gums, tongue, skin surface, hair surface, etc. of an object. The end effector can refer to a single elongated member that stands upright from the head of a handheld device, such as a brush, a tuft of bristles, or a collection of brushes / bristles. In essence, any deformable end effector (or an end effector mounted on a deformable member) that contacts a part of the object surface can be one of the end effectors 110. In other words, each of the multiple end effectors 110 is an elastically deformable end effector or a rigid end effector mounted on the handheld device through an elastically deformable base.

[0071] The object can be a user of the handheld device 100, and the handheld device 100 contacts the surface of the object with the end effector 110. Alternatively, the object can be a person or an animal that contacts the end effector 110, and the handheld device 100 is operated by another person. In other words, the handheld device 100 can be a toothbrush operated by a veterinarian to brush an animal's teeth, so the object is the animal.

[0072] The image capture device 120 is configured to acquire an image of the object surface. Therefore, the image capture device 120 is designed / located / aligned such that when the end effector 110 contacts a part of the object surface, it has a field of view that includes the surface of the object. In this way, the surface of the object can be imaged for remote analysis (e.g., for diagnosis, advice, etc.).

[0073] The sensor unit 130 is configured to determine the load state of the multiple end effectors 110 while the image is being captured. In fact, the load state can be directly measured / determined, or the likelihood / probability that the end effector is in a loaded state can be estimated.

[0074] The load state can describe at least one of the following: the pressure applied to the end effector 110 by contacting the object surface, the vibration of the handheld device 100, and any other forces that the end effector 110 and / or the image capture device 120 may experience.

[0075] In fact, in some exemplary embodiments, the load state (indirectly) describes the level / degree / extent of deformation of at least one of the multiple end effectors 110. More specifically, the load state can describe at least one of the deformation direction and the deformation mode. In essence, the load state indicates the force that the end effector 110 withstands when contacting the object surface.

[0076] In addition, the sensor unit 130 can be configured to continuously / determine / detect / measure the load state of the end effector 110 during the use of the handheld device 100. The load state can be determined at exactly the same time as when the image is captured, or alternatively / additionally, at a time before / after the image is captured.

[0077] Methods and apparatuses for determining a load state will be described in more detail below.

[0078] The control unit 140 is configured to determine a quality value of an image based on the load state. In other words, the load state of the plurality of end effectors 110 determines the degree of deformation of the end effectors 110, and thus has a direct connection with the end effectors 110 that encroach on the field of view of the image capture device 120. Therefore, at certain load states, the image quality may be low due to the encroachment of the end effectors.

[0079] Of course, this may vary depending on the handheld device 100, and thus the specific details of the connection between the load state and the image quality value may depend on the design of the end effectors 110 and the positioning of the image capture device 120. Generally, the greater the load state, the greater the deformation, and thus the poorer the image quality. However, it can be understood that in some cases, a greater load state may be desirable because the end effectors 110 may deflect / deform out of the field of view of the image capture device 120. The actual implementation details and how the load state relates to the image quality will be apparent to those skilled in the art.

[0080] It should be clarified that the quality value essentially represents the degree to which the image of the object surface is occluded by the plurality of end effectors 110. Thus, a high quality value indicates that the captured image is relatively free of end effectors (i.e., the end effectors are not present in the image), and thus contains a clear view of the object surface. Conversely, a low quality value indicates that the end effectors 110 are present in the image such that the surface of the object may not be visible, or the visibility is not particularly useful for further processing.

[0081] The control unit 140 is also configured to selectively control the transmission of the image based on the quality value. In essence, this means that for some quality values (derived from the load state), the image can be transmitted in a first manner, while for other quality values, the image can be transmitted in a second manner.

[0082] In some cases, this may mean that the quality value determines whether the image is transmitted. In other cases, the parameters / characteristics of the image transmission may vary. For example, low quality images may be transmitted at a high compression rate (strong loss), while high quality images may be transmitted at a low compression rate (lossless or weak loss). Of course, the quality value can determine the image transmission discretely (i.e., in a binary manner) or continuously (i.e., multiple different transmission schemes).

[0083] In any case, images that are useful for further processing / analysis (i.e., for diagnosis, informing treatment, providing advice, etc.) can be transmitted, while less useful images may not be transmitted. As a result, the overall data transmission rate may be reduced. This can be considered a more efficient transmission scheme with a higher ratio between the data utilization per bit transmitted / sent.

[0084] In a particular embodiment, the control unit 140 is configured to transmit an image in response to a quality value based on a load state that corresponds to a predetermined range of acceptable load states. In other words, the image is transmitted in response to the quality value being within an acceptable range of acceptable quality values (derived from the load state). Thus, the range of the quality value / load state (continuous range or segmented range) corresponds to an image that may contain a sufficient number of regions of interest for further processing.

[0085] More specifically, the predetermined range of acceptable load states is (at least) defined by a predetermined minimum load state and a predetermined maximum load state (or minimum and maximum quality values).

[0086] The predetermined minimum load state may indicate that at least one of the plurality of end effectors 110 is in contact with a portion of the object surface. In fact, if the end effector 110 is not in contact with the object surface, the image captured by the image capture device 120 is highly unlikely to contain an image of the object surface (region of interest). The predetermined maximum load state may indicate the minimum acceptable amount of the unobstructed surface of the object in the image.

[0087] In other words, a load state greater than zero but below the upper threshold may be appropriate. The image captured under this condition reflects the moment of contact of the end effector 110 (as opposed to zero pressure when the device is free), but in a manner that minimally interferes with the acquired image. When above the upper threshold, the deformation of the end effector will seriously interfere with the image, making further analysis unnecessary.

[0088] Then, the control unit 140 can be configured to control the communication unit 150 to transmit (or not transmit as the case may be) the image. The image can be transmitted in any suitable manner, such as Bluetooth, Wi-Fi, Zigbee, etc. The image can be transmitted to an external processor 160 (i.e., the cloud, the object's smartphone, etc.) for further processing or analysis. It should be noted that the embodiment is not limited to this arrangement. In fact, the control unit 140 can locally control the transmission of the image for storage and / or future transmission by other means.

[0089] Now, different devices / methods for directly or indirectly determining / measuring / detecting the load state will be discussed. Of course, these embodiments are not exhaustive, and those skilled in the art will understand any suitable additional and alternative methods for determining the load state.

[0090] In a first embodiment, the sensor unit 130 includes a sensor configured to detect / measure the force applied by at least one end effector 110 to a portion of the surface of an object. Thus, the load state is based on the detected force.

[0091] In fact, this embodiment particularly benefits from the implementation of a handheld device 100 that has an integrated force / pressure sensor (such as an inertial measurement unit, IMU). For example, an electric toothbrush typically has a pressure sensor to detect whether the force applied by the user is insufficient to brush properly. However, in this embodiment, the pressure / force sensor can be used to detect the load state of the end effector 110.

[0092] Of course, more than one sensor can be provided for this purpose, enabling an accurate assessment of the force.

[0093] To explain this, the detected force (or pressure) can be utilized as a proxy for the deformation of the end effector (i.e., the load state). To evaluate the contact intensity between the end effector and the object surface, the signals from one or more pressure sensors can be correlated.

[0094] In a simple implementation, only a single sensor is used. In an improved implementation, the signals of a single sensor or multiple sensors can be considered to provide a measurement of a specific load state (i.e., the force applied by the user when the end effector 110 is applied to the object surface). In this way, a more accurate load state assessment can be obtained.

[0095] In a second embodiment, the sensor unit 130 includes at least one deformation sensing element coupled to one of the multiple end effectors 110 (i.e., integrated within or next to one of the end effectors 110). The deformation sensing element is configured to detect the deformation value of the corresponding end effector 110. Alternatively or additionally, the end effector 110 itself can also be a deformation element.

[0096] In this case, the load state is based on the detected deformation value. In other words, the deformation value indicates the pressure borne by the end effector 110, thus indicating the load state.

[0097] For example, the deformation sensing element can be any of the following: a piezoelectric polymer fiber, a fiber Bragg grating, a Fabry - Perot sensor, or a piezoresistive material (e.g., carbon black filled conductive silicone rubber). All of these elements have properties that change with deformation. Thus, when the deformation sensing element is mechanically coupled to the end - effector 110 (or is one of the end - effectors 110), the change in the property can directly measure the deformation.

[0098] In other words, the end - effector deformation can be directly evaluated by measurements taken from the (deformation / strain) sensing element / filament to assess whether the captured image is suitable for transmission for further analysis.

[0099] By using multiple deformation sensing elements, it is possible to distinguish between the regions of the end - effector 110 in contact with an uneven surface (where some end - effectors are slightly deformed while some are more deformed) and the use of the free end - effector 110 (where all end - effectors 110 will deform in a fairly uniform manner as the end - effector 110 travels over the surface).

[0100] In the third embodiment, the sensor unit 130 includes an optical proximity sensor that is configured to detect / measure the distance value between a part of the handheld device 100 adjacent to the end - effector 110 and the object surface. In this case, the load state is based on the detected distance value.

[0101] The distance between the object surface and the part of the handheld device 100 where the end - effector 110 is mounted indicates the space occupied by the end - effector 110. Thus, the load state (and its deformation) of the end - effector 110 can be directly derived from this distance.

[0102] Thus, in this embodiment, the measured deformation of the end - effector 110 is directly utilized to evaluate whether the captured image is suitable for transmission for further analysis. The optical proximity sensor is most suitable for an infrared LED sensor with a relatively small number of pixels on the sensing element, but other optical proximity sensors are equally applicable.

[0103] The concept of considering the deformation differences of the end - effector 110 is also proposed. In other words, the end - effectors 110 at different positions may deform differently. The optical proximity sensor can achieve this in the following ways:

[0104] (i) A scanning method that acquires the distances from one sensor to multiple points on the surface within a given (scanning) time.

[0105] (ii) A volume method, where multiple sensors acquire the distances at key positions of the end - effector.

[0106] In this way, the end effector regions in contact with the uneven surface (i.e., the end effector 110 at a certain position) (some variations of the end effector 110 will be small, while some will be more obvious) can be distinguished from the free movement surface (where when the platen vibrates and the end effector 110 swings freely, all values of the end effector 110 will change in a rather uniform manner).

[0107] Finally, in some embodiments, the handheld device 100 may further include a vibration actuator 112 adapted to vibrate at least one of the plurality of end effectors. Such a vibration actuator 112 is well known and is typically provided to improve the functionality of the handheld device 100. For example, an electric toothbrush may have a platen that vibrates the plurality of end effectors 110 to achieve more effective cleaning.

[0108] When the vibration actuator 112 is provided, the load state may also be indicated by the vibration of the end effector 110. Thus, in this case, the sensor unit 130 includes a sensor configured to detect / measure the vibration value of at least one of the plurality of end effectors 110. Based on this vibration value, the load state can be (at least partially) derived.

[0109] In this case, the sensor may be an accelerometer. For example, the sensor may be part of an IMU that already exists in many handheld devices.

[0110] Figure 4 A flowchart of a method 200 for selectively controlling the image transmission of an object surface acquired by the handheld device as described above is shown. As before, the handheld personal care (e.g., treating and / or processing) device includes a plurality of end effectors for contacting a portion of the object surface and means for capturing an image of the object surface.

[0111] In step 210, an image of the object surface is captured. This image can be captured by a known camera / imaging system.

[0112] In step 220, the load state of the plurality of end effectors is determined. This load state reflects the load state (approximately) simultaneously with the determination that the image is captured. In other words, after the image is captured, the force applied to the plurality of end effectors at approximately the same time (i.e., slightly earlier or slightly later) is determined directly or indirectly.

[0113] The load state can be determined in a variety of different ways. In fact, in optional steps 222-228, the load state can be derived based on a plurality of different detected / measured values. Thus, in some embodiments, the load state is determined based on at least one of force, deformation value, distance value, and / or vibration value. In this way, a comprehensive understanding of the deformation / deformation mode of the end effector can be obtained.

[0114] In step 230, the quality value of the image is determined based on the load state. This quality value can reflect the degree to which the captured image may be blocked by the end effector. Thus, this quality value indicates the possible value / values of the image.

[0115] In step 240, the transmission of the image is selectively controlled. For example, the image can be transmitted losslessly, weakly lossily, strongly lossily, or completely losslessly. This is based on the quality value, where images with higher quality are preferentially used for transmission.

[0116] Overall, the method provides for the selective transmission of the captured image, which reflects the load state of the end effector at that time. In this way, those images that are more likely to be blocked will be preferentially used for transmission, thereby reducing the overall data transmission rate.

[0117] Figure 5 An example of a computer 300 is shown, in which one or more parts of the embodiments can be adopted. The various operations described above can utilize the functions of the computer 300. For example, one or more parts of the system for controlling a handheld device can be incorporated into any of the elements, modules, applications, and / or components discussed herein. In this regard, it should be understood that the system functional blocks can run on a single computer or be distributed across multiple computers and locations (e.g., via an Internet connection), such as a cloud computing-based infrastructure.

[0118] The computer 300 includes, but is not limited to, a PC, a workstation, a laptop, a PDA, a handheld device, a server, a memory, etc. Generally, in terms of the hardware architecture, the computer 300 can include one or more processors 310, a memory 320, and one or more I / O devices 330, which are communicatively connected via a local interface (not shown). The local interface can be (e.g., but not limited to) one or more buses or other wired or wireless connections, as known in the art. The local interface can have other elements, such as controllers, buffers (caches), drivers, repeaters, and receivers, to enable communication. In addition, the local interface can include address, control, and / or data connections to enable proper communication between the above components.

[0119] The processor 310 is a hardware device for executing software that can be stored in the memory 320. The processor 310 can be almost any custom or commercial processor, a central processing unit (CPU), a digital signal processor (DSP), or an auxiliary processor among multiple processors associated with the computer 300, and the processor 310 can be a semiconductor-based microprocessor (in the form of a microchip) or a microprocessor.

[0120] The memory 320 can include any one or a combination of the following: volatile memory elements (e.g., random access memory (RAM), such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.) and non-volatile memory elements (e.g., ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic tape, compact disc read-only memory (CD-ROM), magnetic disk, floppy disk, cassette tape, etc.). In addition, the memory 320 can also incorporate electronic, magnetic, optical, and / or other types of storage media. It should be noted that the memory 320 can adopt a distributed architecture, where the individual components are located far from each other but can be accessed by the processor 310.

[0121] The software in the memory 320 can include one or more individual programs, each program including an ordered list of executable instructions for implementing logical functions. The software in the memory 320 includes a suitable operating system (O / S) 340, a compiler 360, source code 350, and one or more application programs 370 according to the exemplary embodiments. As shown, the application program 370 includes numerous functional components for implementing the features and operations of the exemplary embodiments. The application program 370 of the computer 300 can represent various application programs, computing units, logics, functional units, processes, operations, virtual entities, and / or modules according to the exemplary embodiments, but the application program 370 is not limiting.

[0122] The operating system 340 controls the execution of other computer programs and provides related services such as scheduling, input / output control, file and data management, memory management, and communication control. The inventors envision that the application program 370 for implementing the exemplary embodiments can be applicable to all commercially available operating systems.

[0123] The application program 370 can be a source program, an executable program (object code), a script, or any other entity that contains a set of instructions to be executed. If it is a source program, the program is typically translated by a compiler (such as compiler 360), an assembler, an interpreter, etc. (these programs may or may not be included in the memory 320) to operate properly in conjunction with the operating system 340. Additionally, the application program 370 can be written in an object-oriented programming language (with data and method classes) or a procedural programming language (with routines, subroutines, and / or functions), such as (but not limited to) C, C++, C#, Pascal, BASIC, API calls, HTML, XHTML, XML, ASP scripts, JavaScript, FORTRAN, COBOL, Perl, Java, ADA,.NET, etc.

[0124] The I / O device 330 can include input devices, such as for example but not limited to a mouse, a keyboard, a scanner, a microphone, a camera, etc. Additionally, the I / O device 330 can also include output devices, such as for example but not limited to a printer, a display, etc. Finally, the I / O device 330 can also include devices that perform both input and output communication, such as for example but not limited to a NIC or a modem / demodulator (for accessing remote devices, other files, devices, systems, or networks), a radio frequency (RF) or other transceiver, a telephone interface, a bridge, a router, etc. The I / O device 330 also includes components for communicating via various networks, such as the Internet or an intranet.

[0125] If the computer 300 is a PC, a workstation, a smart device, etc., the software in the memory 320 can also include a basic input / output system (BIOS) (omitted here for simplicity). The BIOS is a set of necessary software routines for initializing and testing the hardware at startup, starting the operating system 340, and supporting data transfer between hardware devices. The BIOS is stored in a type of read-only memory, such as ROM, PROM, EPROM, EEPROM, etc., for execution when the computer 300 starts up.

[0126] When the computer 300 is running, the processor 310 is configured to execute the software stored in the memory 320, transfer data with the memory 320, and overall control the operation of the computer 300 according to the software. The application program 370 and the operating system 340 are read in whole or in part by the processor 310 (possibly cached in the processor 310) and then executed.

[0127] When the application 370 is implemented in software form, it should be noted that the application 370 can be stored on almost any computer-readable medium for use by or in conjunction with any computer-related system or method. In the context of this document, a computer-readable medium can be an electronic, magnetic, optical, or other physical device or apparatus that can contain or store a computer program for use by or in conjunction with a computer-related system or method.

[0128] The application 370 can be embedded in any computer-readable medium for use by or in conjunction with an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other systems that can obtain and execute instructions from an instruction execution system, apparatus, or device. In the context of this document, a "computer-readable medium" can be any device capable of storing, communicating, propagating, or transporting a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be, for example (but not limited to), an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0129] Figure 4 The (multiple) control methods proposed in Figure 3 and the (multiple) systems in

[0130] can be implemented in hardware, software, or a combination of both (e.g., as firmware running on a hardware device). If an embodiment is implemented partially or fully in software, the functional steps shown in the flowchart can be executed by a suitably programmed physical computing device, such as one or more central processing units (CPUs) or graphics processing units (GPUs). Each process and its individual constituent steps shown in the flowchart can be executed by the same or different computing devices. According to an embodiment, a computer-readable storage medium stores a computer program that includes computer program code configured to cause one or more physical computing devices to execute the control method as described above when the program runs on one or more physical computing devices.

[0131] To the extent that an embodiment is implemented partially or fully in hardware form, Figure 3The blocks shown in the block diagrams can be separate physical components, logical subdivisions of a single physical component, or can all be implemented integrally in one physical component. In one embodiment, the functionality of one block shown in the figures can be divided among multiple components, or in one embodiment, the functionality of multiple blocks shown in the figures can be combined into a single component. Hardware components suitable for embodiments of the present invention include, but are not limited to, conventional microprocessors, application specific integrated circuits (ASICs), and field programmable gate arrays (FPGAs). One or more blocks can be implemented as a combination of dedicated hardware for performing some functions; one or more programmed microprocessors and their associated circuitry are used to perform other functions.

[0132] Those skilled in the art, when practicing the claimed invention, can understand and implement various variations of the disclosed embodiments by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" does not exclude a plurality. A single processor or other unit can implement the functions of multiple items recited in the claims. Certain measures are recited in mutually different dependent claims, and this does not mean that the combination of these measures cannot be advantageously utilized. If a computer program is discussed above, the program can be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium provided together with or as part of other hardware, but can also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. If the term "suitable for" is used in the claims or the specification, it should be noted that the meaning of the term "suitable for" is equivalent to "configured to". Any reference signs in the claims should not be construed as limiting the scope.

[0133] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram can represent a module, a segment of a program, or a part of an instruction, which includes one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions shown in the blocks may be different from the order shown in the figures. For example, two consecutive blocks shown may actually be executed substantially simultaneously, or sometimes may be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware system that performs the specified functions or actions or a combination of dedicated hardware and computer instructions.

Claims

1. A handheld personal care device (100), comprising: a plurality of end effectors (110) for engaging a portion of the surface of an object; an image capture device (120) configured to capture an image of the surface of the object; a sensor unit (130) configured to determine a load state of the plurality of end effectors while the image is being captured; and a control unit (140) configured to determine a quality value of the image based on the load state and selectively control transmission of the image based on the quality value.

2. The handheld device according to claim 1, wherein the quality value indicates the degree to which the image of the surface of the object is occluded by the plurality of end effectors (110).

3. The handheld device according to claim 1 or 2, wherein the load state describes a deformation level of at least one of the plurality of end effectors (110).

4. The handheld device according to claim 3, wherein the load state further describes at least one of a deformation direction and a deformation mode of at least one of the plurality of end effectors (110).

5. The handheld device according to any one of claims 1 to 4, wherein the sensor unit (130) includes a sensor configured to detect a force or pressure applied by at least one of the end effectors (110) on a portion of the surface of the object, and wherein the load state is based on the detected force or pressure.

6. The handheld device according to any one of claims 1 to 5, wherein the sensor unit (130) includes at least one deformation sensing element coupled to or as one of the plurality of end effectors (110), the deformation sensing element being configured to detect a deformation value of the corresponding end effector, and wherein the load state is based on the detected deformation value.

7. The handheld device according to claim 6, wherein the at least one deformation sensing element is one of the following: a piezoelectric polymer fiber, a fiber Bragg grating, a Fabry - Perot sensor, or a piezoresistive material.

8. The handheld device according to any one of claims 1 to 7, wherein the sensor unit (130) includes an optical proximity sensor configured to detect a distance value between a portion of the handheld device adjacent the end effector (110) and the surface of the object, and wherein the load state is based on the detected distance value.

9. The handheld device according to any one of claims 1 to 8, wherein the handheld device further comprises a vibration actuator (112) adapted to vibrate at least one of the plurality of end effectors (110), and wherein the sensor unit (130) comprises a sensor configured to detect a vibration value of at least one of the plurality of end effectors, and wherein the load state is based on the detected vibration value.

10. The handheld device according to any one of claims 1 to 9, wherein the control unit (140) is configured to transmit the image in response to the quality value, the quality value being based on a load state corresponding to a predetermined range of acceptable load states.

11. The handheld device according to claim 10, wherein the predetermined range of acceptable load states is defined by a predetermined minimum load state and a predetermined maximum load state, the predetermined minimum load state indicating that at least one of the plurality of end effectors (110) is in contact with a portion of the surface of the object, and the predetermined maximum load state indicating a minimum acceptable amount of the unoccluded surface of the object present in the image.

12. The handheld device according to any one of claims 1 to 11, wherein each of the plurality of end effectors (110) is one of the following: an elastically deformable end effector or a rigid end effector mounted on the handheld device by an elastically deformable base.

13. The handheld device according to any one of claims 1 to 12, wherein the handheld device (100) is a processing device or a treatment device.

14. A method (200) for selectively controlling the transmission of an image of a surface of an object acquired by a handheld personal care device (100), the handheld personal care device comprising a plurality of end effectors (110) for engaging a portion of the surface of the object, the method comprising: capturing (210) an image of the surface of the object; determining (220) the load state of the plurality of end effectors while the image is being captured; determining (230) a quality value of the image based on the load state; and selectively (240) controlling the transmission of the image based on the quality value.

15. A computer program comprising computer program code means adapted to implement the method according to claim 14 when the computer program is run on a computer.

Images