Device and method for operating a personal grooming appliance or a domestic cleaning appliance

Abstract

The present application provides a method for operating a dental appliance, the method comprising providing a dental appliance including at least one motion sensor that provides motion data, the motion sensor being one of an orientation sensor, an acceleration sensor, and an inertial sensor. Additionally, providing at least one camera associated with the dental appliance that provides image data, and using a first machine learning neural network to segment the image data received from the camera to segment surface conditions, thereby generating a surface condition segmentation. Using a second machine learning neural network classifier, classifying the motion data received from the motion sensor and the image data received from the camera to classify the motion and position of the dental appliance relative to the surface of the user's anatomical structure, thereby generating a surface position that includes at least one of a relative motion classification or a relative position classification. Finally, based on the surface conditions and the surface position, generating a user treatment progress status and position, and transmitting the treatment progress status and position to the user.

Description

[0001] Related Applications

[0002] This application is a divisional application of Chinese Patent Application No. 201980079067.X, and claims the priority benefit of U.S. Provisional Patent Application No. 62 / 783,929, filed on December 21, 2018. Background Art

[0003] There is a need for a "smart" grooming or household appliance and associated system that utilizes a combination of internal sensor data and image data to improve the ability to enhance the user experience associated with the grooming or household appliance. Summary of the Invention

[0004] A first aspect of the present disclosure provides a system and method for operating a personal grooming / household appliance, comprising: providing a personal grooming / household appliance comprising (a) an electric and electronically controlled grooming / cleaning tool, and (b) at least one physical sensor selected from the group consisting of: an orientation sensor, an acceleration sensor, an inertial sensor, a global positioning sensor, a pressure sensor, a load sensor, an audio sensor, a magnetic sensor, a humidity sensor, and a temperature sensor; providing a camera associated with the personal grooming / household appliance; using one or more classifiers that classify the physical sensor data and the image data to derive an enhanced classification; and modifying the operation of the grooming / household appliance or tool based on the enhanced classification.

[0005] In one detailed embodiment, the camera is located on the personal grooming / household appliance.

[0006] Alternatively or in addition, the personal grooming / household appliance further comprises a computer network interface for sending and receiving data via a computer network, and the camera is located on a computerized device comprising at least a computer network interface for sending image data via the computer network. In another detailed embodiment, the operation modification step is further based on a processing scheme implemented by a software application operating at least partially on the computerized device. In another detailed embodiment, the processing scheme is customized for the user of the grooming appliance. Alternatively or in addition, the enhanced classification is coordinated according to the processing scheme to determine the processing progress relative to the processing scheme. Alternatively or in addition, the enhanced classification is at least partially used to establish the processing scheme.

[0007] In another detailed embodiment of the first aspect, the step of deriving the enhanced classification is implemented by a single classifier. Alternatively, the method includes classifying sensor data received from a physical sensor using a trained machine learning classifier to generate a physical classification; and classifying image data received from a camera using a trained machine learning classifier to generate an image classification; wherein the step of deriving the enhanced classification is based on a combination of the physical classification and the image classification.

[0008] In another detailed embodiment of the first aspect, the appliance is a dental appliance; the grooming tool is a brush, a fluid nozzle, and / or a dental floss tape; and the enhanced classification relates to the position of the grooming tool relative to the user's oral cavity. In another detailed embodiment, when the enhanced classification indicates that the grooming tool is outside the user's oral cavity, the modification operation step deactivates the grooming tool. Alternatively or in addition, the grooming tool is an electric brush, and the modification operation adjusts the brush speed setting based on the position of the grooming tool relative to the user's oral cavity, as indicated at least in part by the enhanced classification.

[0009] In another detailed embodiment of the first aspect, the enhanced classification relates at least in part to whether a grooming / cleaning tool is being implemented, and the step of the modification operation updates a maintenance setting based on the amount of time that the grooming / cleaning tool is being implemented.

[0010] In another detailed embodiment of the first aspect, the enhanced classification relates at least in part to the position of the grooming tool relative to a user body part, and the step of the modification operation modifies the operation of the grooming tool at least in part based on the position of the grooming tool relative to the user's body part, as indicated at least in part by the enhanced classification. In another detailed embodiment, the grooming tool is a dental appliance, and the grooming tool is an electric brush; and the step of the modification operation adjusts the speed setting of the electric brush based on the position of the grooming tool relative to the user's oral cavity, as indicated at least in part by the enhanced classification. Alternatively, the grooming tool is a shaving appliance, and the grooming tool is a motorized shaving head; and the step of the modification operation adjusts the speed setting of the shaving head based on the position of the grooming tool relative to the user's face, as indicated at least in part by the enhanced classification. Alternatively, the grooming tool is a shaving appliance, and the grooming tool is a razor blade cartridge; and the step of the modification operation adjusts the angle of attack setting of the razor blade cartridge based on the position of the grooming tool relative to the user's face, as indicated at least in part by the enhanced classification. Alternatively, the step of the modification operation adjusts the pressure sensitivity setting of the grooming tool based on the position of the grooming tool relative to the user's body part, as indicated at least in part by the enhanced classification.

[0011] Alternatively, the enhanced classification further includes the surface condition of the user's body part; and the steps of the modifying operation adjust the performance settings of the grooming tool based on the surface condition at the location of the grooming tool relative to the user's face, as indicated at least in part by the enhanced classification. In another detailed embodiment, the grooming appliance is a dental appliance, and the surface condition relates at least in part to the presence of plaque on the user's teeth. Alternatively, the grooming appliance is a shaving appliance, and the surface condition relates at least in part to the presence of whiskers on the user's face.

[0012] In another detailed embodiment of the first aspect, the enhanced classification relates at least in part to the position of the cleaning tool relative to a household target surface, and the steps of the modifying operation modify the operation of the cleaning tool at least in part based on the position of the cleaning tool relative to the household target surface, as indicated at least in part by the enhanced classification.

[0013] In another detailed embodiment of the first aspect, the enhanced classification detects the identity of the user of the grooming / household appliance, and the modifying operation steps apply operation settings customized for the identified user.

[0014] A second aspect of the present disclosure provides a system and method for operating a personal grooming appliance, including: providing a personal grooming / household appliance that includes at least one physical sensor selected from the group consisting of: an orientation sensor, an acceleration sensor, an inertial sensor, a global positioning sensor, a pressure sensor, a load sensor, an audio sensor, a humidity sensor, and a temperature sensor; providing a camera associated with the personal grooming / household appliance; classifying data received from the physical sensor and from the camera using at least one trained machine learning classifier to generate an enhanced classification; and providing user feedback information based on the enhanced classification; wherein the enhanced classification relates to a combination of a first state regarding the position of the grooming / household appliance relative to the user's body part or a household target surface, and also relates to a second state different from the first state. In one embodiment, the second state relates to the identity of the user. In an alternative embodiment, the second state relates to the identity of the grooming appliance.

[0015] In one embodiment, the second state relates to the surface condition of a body part of the user. In another detailed embodiment, the body part of the user is the user's teeth, and the surface condition relates to the presence of dental plaque on the patient's teeth. Alternatively, the surface condition is the presence of whiskers or stubble on the body part of the user. Alternatively or in addition, the first state also relates to the direction of movement of the personal grooming appliance. In another detailed embodiment, the second state is an image classification derived from image data from a camera. In yet another detailed embodiment, the image classification at least partially relates to the identification of the shaving lubricant being used. Alternatively or in addition, the second state is a stroke pressure classification derived from a physical sensor.

[0016] In one embodiment, the second state is an image classification derived from image data from a camera. In another detailed embodiment, the image classification relates to the mood of the user of the grooming appliance. In yet another detailed embodiment, the image classification relates to the negative mood of the user of the grooming appliance, and feedback information provides suggestions for improving the user's experience with the grooming appliance.

[0017] In one embodiment, the image classification relates to at least one of a pre-treatment or post-treatment condition. In another detailed embodiment, the image classification relates to a pre-treatment condition, and the feedback information provides processing instructions based on a combination of the pre-treatment condition and the location of the grooming / home appliance.

[0018] In one embodiment, the image classification relates to the identification of an object used with the grooming / home appliance for grooming / home cleaning. In another detailed embodiment, the feedback information includes marketing information related to the object (e.g., coupons, promotions, advertisements, etc.).

[0019] In one embodiment, the feedback information includes marketing information related to the image classification. In one detailed embodiment, the image classification relates to the skin condition of the user, and the feedback information includes recommended products for treating the skin condition. In another detailed embodiment, the feedback information also includes the product application technique of using the grooming appliance.

[0020] In one embodiment, the image classification relates to the condition of a body part of the user, and the feedback information includes a recommendation for a product to be applied to the body part along with the product application technique of using the grooming appliance.

[0021] In one embodiment, the second state relates to the movement of the grooming / home appliance. Alternatively or in addition, the step of generating the enhanced classification is implemented by a single classifier.

[0022] A third aspect of the present disclosure provides a system and method for operating a personal grooming implement, comprising: providing a personal grooming implement including at least one motion sensor selected from the group consisting of an orientation sensor, an acceleration sensor, and an inertial sensor; providing a camera associated with the personal grooming implement; using at least one trained machine learning classifier to classify data received from the motion sensor and from the camera to generate an enhanced classification; and providing user feedback information based on the enhanced classification; wherein the enhanced classification relates to a combination of a first state regarding the position of the grooming implement relative to a body part of the user, and also relates to a second state different from the first classification. In another detailed embodiment, the second state relates to the identification of the user. Alternatively, the second state relates to the identification of the grooming implement. Alternatively or in addition, the second state relates to the surface condition of the body part of the user. Alternatively or in addition, the first state further relates to the direction of movement of the personal grooming implement.

[0023] In one embodiment, the second state is an image classification derived from image data from the camera. In another detailed embodiment, the image classification relates to the mood of the user of the grooming implement. Alternatively, the image classification relates to at least one of a pre-treatment or post-treatment condition. In another detailed embodiment, the image classification relates to a pre-treatment condition, and the feedback information provides processing instructions based on a combination of the pre-treatment condition and the position of the grooming implement.

[0024] A fourth aspect of the present disclosure provides a system and / or method for operating a personal grooming implement, comprising: providing at least one of a camera associated with the personal grooming implement or a biosensor associated with the personal grooming implement; providing a personal grooming implement having at least one motion sensor such as an orientation sensor, an acceleration sensor, and / or an inertial sensor; using a first learning network classifier to classify at least one of image data received from the camera or biosensor data received from the biosensor to classify the surface condition of the surface of the body structure of the user to generate an initial surface condition classification; generating user processing information based on the initial surface condition classification and transmitting the user processing information to the user; using a second learning network classifier to classify motion data received from the motion sensor to classify the movement of the personal grooming implement relative to the surface of the body structure of the user to generate at least one of a relative motion classification or a relative position classification; generating user processing progress information based on a subsequent surface condition classification and based on the at least one relative motion classification or relative position classification; and transmitting the user processing progress information to the user.

[0025] In one embodiment, the personal grooming appliance further includes a computer network interface for sending and receiving data over a computer network; and a camera is located on a computerized device that includes at least a computer network interface for sending image data over a computer network. In another detailed embodiment, the method further includes the step of modifying the operation of the grooming appliance based on user processing progress information. Alternatively or in addition, the step of generating user processing information based on surface condition classification includes generating a processing plan at least in part based on surface condition information. In another detailed embodiment, the processing plan is implemented by a software application operating at least in part on the computerized device. In another detailed embodiment, the processing plan is customized for a user of the grooming appliance. Alternatively or in addition, user processing progress information is coordinated according to the processing plan to determine the processing progress relative to the processing plan. Alternatively or in addition, the method further includes the step of modifying the processing plan at least in part based on user processing progress information. In one embodiment, the modifying step follows the step of determining that an initial surface condition classification is incorrect. In one embodiment, the method further includes the step of transmitting the modified processing plan to the user.

[0026] A fifth aspect of the present disclosure is to provide a system and / or method for operating a personal grooming appliance, including: providing a computerized device that includes a camera and a network interface that sends image data from the camera over a computer network; providing a personal grooming appliance that includes (a) an orientation sensor, an acceleration sensor, an inertial sensor, a pressure sensor, and / or a load sensor, and (b) a computer network interface for sending and receiving data over a computer network; providing a software application operating at least in part on the computerized device; classifying image data received from the camera using one or more learning network classifiers to generate an image classification; generating a processing plan at least in part based on the image classification; customizing the processing plan based on user information accessible to the software application; implementing at least a portion of the customized processing plan by the software application; using one or more learning network classifiers to classify sensor data received from the at least one sensor to classify the use of the personal grooming appliance relative to the surface of the user's body structure, thereby generating a relational grooming appliance use classification; and generating user processing plan progress information based on the relational grooming appliance use classification and transmitting the user processing plan progress information to the user.

[0027] In another detailed embodiment, the step of classifying the image data includes identifying the user's body structure. Alternatively or in addition, the step of generating a grooming implement usage classification is based on classifying a combination of the image data and the sensor data. Alternatively or in addition, the method further includes the step of modifying the operation of the grooming implement based on the grooming implement usage classification. Alternatively or in addition, the user processing progress information is coordinated according to the treatment plan to determine the treatment progress relative to the treatment plan. Alternatively or in addition, the user information accessible to the software application includes the user profile information collected by the software application. Alternatively or in addition, the user information accessible to the software application includes the information derived from the grooming implement usage classification. Alternatively or in addition, the method further includes the step of training the one or more learning network classifiers based on how the user operates the grooming implement. Alternatively or in addition, the method further includes the step of training the one or more learning network classifiers based on the user information collected by the software application, wherein the user information that can be collected by the software application is at least partially based on the interaction between the user and the software application.

[0028] The sixth aspect of the present disclosure is to provide a method for treating the surface of a user's body part, including: obtaining target surface condition information from the surface of the user's body part using one or more condition sensors such as optical sensors and / or biosensors; classifying the target surface condition information using a machine learning classifier to determine an initial target surface condition classification; obtaining treatment progress information using a combination of the motion sensor data and the surface condition information from the one or more condition sensors; and classifying the treatment progress information using a machine learning classifier to determine a progress classification for treating the initial target surface condition classification.

[0029] In a more detailed embodiment of the sixth aspect, the one or more condition sensors are disposed on at least one of an inspection instrument or a grooming implement. Alternatively or in addition, the method further includes displaying a representation of the treatment progress information. In one embodiment, the displayed representation is a delayed representation; or, in another embodiment, the displayed representation is a real-time representation.

[0030] In a more detailed embodiment of the sixth aspect, the method includes modifying the settings of the treatment system at least partially based on the treatment progress classification. In another detailed embodiment, the modifying step occurs substantially in real time while the treatment system is treating the surface of the user's body part. In another detailed embodiment, the one or more condition sensors are disposed on the processing instrument of the treatment system. In yet another detailed embodiment, the processing instrument is an oral care instrument.

[0031] In a more detailed embodiment of the sixth aspect, the method includes modifying the settings of the processing system based on the classification of the target surface condition. Alternatively or in addition, the method further includes evaluating the change over time of the surface condition of the user's body part based on successive classifications of the target surface condition. Alternatively or in addition, the progress classification indicates that the initial target surface condition classification is incorrect. In this case, the method may further include generating an initial treatment plan based on the initial treatment classification and modifying the initial treatment plan when it is determined that the initial target surface condition classification is incorrect.

[0032] As a non-limiting example, the present application provides the following embodiments:

[0033] 1. A method for operating a personal grooming appliance, comprising:

[0034] Providing a personal grooming appliance, the personal grooming appliance including at least one motion sensor selected from the group consisting of: an orientation sensor, an acceleration sensor, and an inertial sensor;

[0035] Providing a camera associated with the personal grooming appliance;

[0036] Using at least one trained machine learning classifier to classify data received from the motion sensor and from the camera to generate an enhanced classification;

[0037] Providing user feedback information based on the enhanced classification;

[0038] Wherein the enhanced classification relates to a combination of a first state regarding the position of the grooming appliance relative to a body part of the user and also relates to a second state different from the first state.

[0039] 2. The method according to embodiment 1, wherein the second state relates to at least one of an identity identification of the user and an identity identification of the grooming appliance.

[0040] 3. The method according to embodiment 1, wherein the second state relates to the surface condition of the user's body part.

[0041] 4. The method according to any one of the foregoing embodiments, wherein the first state further relates to the moving direction of the personal grooming appliance.

[0042] 5. The method according to embodiment 1, wherein the second state is an image classification derived from image data from the camera.

[0043] 6. The method according to embodiment 5, wherein the image classification relates to the mood of the user of the grooming appliance.

[0044] 7. The method according to embodiment 5, wherein the image classification involves at least one of a pre-processing or a post-processing condition.

[0045] 8. The method according to embodiment 5, wherein the image classification involves the identification of an object used with the grooming implement being groomed.

[0046] 9. The method according to embodiment 5, wherein the image classification involves the skin condition of a user, and the feedback information includes recommended products for treating the skin condition.

[0047] 10. The method according to any one of the foregoing embodiments, wherein the step of generating the enhanced classification is implemented by a single classifier.

[0048] 11. The method according to any one of the foregoing embodiments, further comprising:

[0049] classifying sensor data received from the physical sensor using a trained machine learning classifier to generate a physical classification; and

[0050] classifying image data received from the camera using a trained machine learning classifier to generate an image classification;

[0051] wherein the step of generating the enhanced classification is based on a combination of the physical classification and the image classification.

[0052] 12. A method for operating a personal grooming implement, comprising:

[0053] providing at least one of a camera associated with the personal grooming implement or a biosensor associated with the personal grooming implement;

[0054] providing a personal grooming implement having at least one motion sensor selected from the group consisting of an orientation sensor, an acceleration sensor, and an inertial sensor;

[0055] classifying at least one of image data received from the camera or biosensor data received from the biosensor using a first learning network classifier to classify a surface condition of a surface of a user's body structure, thereby generating an initial surface condition classification;

[0056] generating user treatment information based on the initial surface condition classification and transmitting the user treatment information to the user;

[0057] Classify the motion data received from the motion sensor using a second learning network classifier to classify the motion of the personal grooming appliance relative to the surface of the user's body structure to generate at least one of a relative motion classification or a relative position classification;

[0058] Generate user processing progress information based on subsequent surface condition classifications and based on the at least one relative motion classification or relative position classification; and

[0059] Transmit the user processing progress information to the user.

[0060] 13. The method according to embodiment 12, wherein the camera is positioned on the personal grooming appliance.

[0061] 14. The method according to any of the preceding embodiments, wherein:

[0062] The personal grooming appliance further includes a computer network interface for sending and receiving data over a computer network; and

[0063] The camera is located on a computerized device, the computerized device including a computer network interface for sending image data at least over the computer network.

[0064] 15. The method according to embodiment 14, further comprising the step of modifying the operation of the grooming appliance based on the user processing progress information.

[0065] These and other aspects and objects of the present disclosure will become apparent from the following description, the appended claims, and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In the following, embodiments of the present disclosure are described in more detail in conjunction with the drawings, wherein

[0067] Figure 1 A schematic block diagram of a device according to an embodiment of the present disclosure is shown,

[0068] Figure 2 An example of a target surface to be processed using a movable processing device is shown,

[0069] Figure 3 Another example of a target surface to be processed using a movable processing device is shown,

[0070] Figure 4 A schematic block diagram of a recurrent neural network that can be used in the embodiments disclosed herein is shown,

[0071] Figure 5 A schematic block diagram of a GRU neural network that can be used in the embodiments disclosed herein is shown,

[0072] Figure 6A Shows a schematic block diagram of an LSTM neural network that can be used in the embodiments disclosed herein,

[0073] Figure 6B Shows a schematic block diagram of an LSTM neural network with one layer at different time instances,

[0074] Figure 7 Shows a schematic block diagram of an LSTM neural network with two layers at different time instances,

[0075] Figure 8 Shows a block diagram of a method according to an embodiment of the present disclosure,

[0076] Figure 9 Shows a schematic block diagram of a device according to an embodiment of the present disclosure,

[0077] Figure 10 Shows a schematic block diagram of a device according to another embodiment of the present disclosure,

[0078] Figure 11 Shows a schematic block diagram of a device according to another embodiment of the present disclosure,

[0079] Figure 12 Shows a block diagram of a method according to an embodiment of the present disclosure,

[0080] Figure 13 Is a block diagram representation of a networking system according to an embodiment of the present disclosure,

[0081] Figure 14 Is a flowchart representation of a decision tree according to an embodiment of the present disclosure,

[0082] Figure 15 Is a block diagram representation of a networking system that utilizes enhanced or hybrid machine learning classification according to an embodiment of the present disclosure, and

[0083] Figure 16 Is a block diagram representation of an alternative networking system that utilizes enhanced or hybrid machine learning classification according to an embodiment of the present disclosure. Detailed Description

[0084] Equal or equivalent elements or components having equal or equivalent functions may be represented by equal or equivalent reference numerals in the following description. However, based on this embodiment, similar or equivalent elements may also be represented by different reference numerals.

[0085] In the following text, personal grooming appliances and / or household appliances will be mentioned as non-limiting examples of "portable processing devices". For example, personal grooming appliances may include shaving appliances (manual razors, electric razors, trimmers, hair removal devices, chemical-based hair removal, etc.), dental appliances (manual toothbrushes, electric toothbrushes, polishers, water sprayers, ultrasonic appliances, dental flossers, etc.), exfoliators (exfoliating brushes, etc.), cosmetic application devices, and hair styling appliances (hair brushes, trimmers / cutters, hair dryers, straighteners, curling irons, etc.). Such grooming appliances may also have certain medical and / or dental examination / diagnosis / processing uses, and will be described herein. In these examples, the surface to be processed is a part of the user's body structure, such as the user's teeth, face, legs, etc. Additionally, household appliances may include surface cleaners, polishers, pressure washers, floor cleaners, vacuum cleaners, window cleaners, etc. In these examples, the surface to be processed may be a household surface, such as a floor, wall, countertop, sink, window, mirror, vehicle surface, etc.

[0086] Furthermore, the order of any method steps of a method may be described only as a non-limiting example. Thus, unless explicitly specified to be implemented in a particular order, any method step as described herein may also be implemented in any other order than the described one.

[0087] Although some aspects will be described in the context of a device or apparatus, it should be understood that these aspects also represent a description of the corresponding method, where a block or apparatus corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method or method steps also represent a description of the corresponding block or item or feature of the corresponding device or apparatus.

[0088] Figure 1 An apparatus or system 10 according to an exemplary embodiment is shown. A portable processing device 11 is depicted. The portable processing device 11 may include an inertial sensor 13. Additionally, the portable processing device 11 may be configured to process a target surface 12. As will be described below, a camera 22 may also be associated with the device 11 (as will be described below, the camera is "associated" because the camera may be included as part of the processing device 11, or the camera may be separate from the processing device, such as in a handheld smartphone or in a smart display or mirror - substantially, any networked or wirelessly connected camera).

[0089] As can be seen, the portable processing device 11 may be positioned at certain locations relative to the target surface 12, e.g., within the target surface 12, at the target surface 12, on the target surface 12, or adjacent to the target surface 12. The target surface 12 itself may be divided into one or more zones 211, 212, 213, …, 21 n。The movable processing device 11 can be moved relative to at least one of the regions 211, 212, 213, …, 21 n or positioned at a certain location.

[0090] As Figure 1 depicted in, the device 10 can be configured to implement the positioning of the movable processing device 11 relative to the target surface 12.

[0091] The device 10 can include a motion pattern recognition device 14. The motion pattern recognition device 14 can be configured to distinguish two or more motion patterns included in the set of motion patterns 15 of the movable processing device 11, such as motion patterns 151, 152, 153, …, 15 n . In other words, the movable processing device 11 can be moved, for example, by a user using the movable processing device 11 in different linear and / or rotational directions. Accordingly, each motion of the movable processing device 11 can represent a corresponding or individual motion pattern. The motion pattern recognition device 14 can include a set of different motion patterns 15. The set of motion patterns 15 can include the aforementioned corresponding or individual motion patterns 151, 152, 153, …, 15 n among two or more motion patterns. The motion pattern recognition device 14 can be configured to distinguish these two or more motion patterns 151, 152, 153, …, 15 n . That is, the motion pattern recognition device 14 can be configured to distinguish a first motion pattern 151 from a second motion pattern 152.

[0092] The movement of the movable processing device 11 can be detected by at least one inertial sensor 13. The inertial sensor 13 is an inertia-based sensor and can include at least one of an accelerometer, a gyroscope, and a magnetometer. The inertial sensor 13 can provide sensor data representing at least one of linear velocity, angular velocity, linear acceleration, angular acceleration, and gravity. The inertial sensor 13 can be part of an inertial measurement unit including one or more inertial sensors.

[0093] The device 10 can include an interface 16 for receiving at least one inertial sensor data 171, 172, 173, …, 17 n from the inertial sensor 13 and for providing at least one inertial sensor data 171, 172, 173, …, 17 n to the motion pattern recognition device 14. At least one inertial sensor data 171, 172, 173, …, 17 n represents the movement of the movable processing device 11. In other words, when the movable processing device 11 moves, the inertial sensor 13 senses this motion and generates at least one inertial sensor data 171, 172, 173, …, 17n Accordingly, at least one inertial sensor data 171, 172, 173, …, 17 n represents the corresponding motion of the processing device 11 that is moving.

[0094] The motion pattern recognition device 14 may include a neural network 18. The neural network 18 may be a deep learning network. The neural network 18 may be configured to receive at least one inertial sensor data 171, 172, 173, …, 17 n , and map the at least one inertial sensor data 171, 172, 173, …, 17 n to at least one motion pattern among the motion patterns 151, 152, 153, …, 15 n contained in the motion pattern set 15. This mapping is indicated by the dashed and solid arrows 191, 192, 193, 194 in Figure 1 . The arrow 193 drawn in solid line may exemplarily indicate that the neural network 18 has successfully mapped the at least one inertial sensor data 171, 172, 173, …, 17 n to the third motion pattern 153.

[0095] The different motion patterns 151, 152, 153, …, 15 n contained in the set 15 are exemplarily symbolically represented by different geometric shapes (circles, rectangles, triangles, stars), only for illustrative purposes. Of course, the motion patterns of the movable processing device 11 are not limited to these specific geometric shapes.

[0096] According to the principles of the present invention, the motion patterns 151, 152, 153, …, 15 n are each associated with one or more different regions 211, 212, 213, …, 21 n of the target surface 12. This is indicated by the dashed and solid arrows 201, 202, 203, …, 20 n . As can be seen, the first motion pattern 151 may be associated with the first region 211 of the target surface 12, as indicated by the dashed arrow 201. The second motion pattern 152 may be associated with the second region 212 of the target surface 12, as indicated by the dashed arrow 202. The third motion pattern 153 may be associated with the third region 213 of the target surface 12, as indicated by the arrow 203 drawn in solid line. The fourth motion pattern 154 may be associated with the fourth region 214 of the target surface 12, as indicated by the dashed arrow 204.

[0097] The arrow 203 drawn in solid line may exemplarily indicate that the third motion pattern 153 (at least one inertial sensor data 171, 172, 173, …, 17 nis successfully mapped by the neural network 18 to this motion pattern) is associated with a third region 213 of the target surface 12.

[0098] Accordingly, at least one inertial sensor data 171, 172, 173, …, 17 n is mapped to at least one motion pattern 151, 152, 153, …, 15 n indicating an evaluation of the positioning of the movable processing device 11 relative to one or more regions 21 1 of the target surface 12. 2 、21 3 、21 n 、…、21 n In this example, the mapping of at least one inertial sensor data 171, 172, 173, …, 17

[0099] to the third motion pattern 153 indicates an evaluation of the positioning of the movable processing device 11 relative to the third region 213 of the target surface 12. n In other words, the neural network 18 successfully maps the received at least one inertial sensor data 171, 172, 173, …, 17 n to the third motion pattern 153. According to this example, since the third motion pattern 153 is associated with the third region 213, the device 10 retrieves information that the movable processing device 11 is positioned in the third region 213 or information that the movable processing device 11 is positioned in the third region 213 at least at the time when the at least one inertial sensor data 171, 172, 173, …, 17

[0100] is generated.

[0101] According to one embodiment, the movable processing device 11 can be a personal grooming appliance, and the target surface 12 can be a body part to be processed by the movable processing device 11.

[0102] For example, the movable processing device 11 can be a razor or a comb, which is used to shave or comb a body part of the user's body. In this case, the user's body (or a part of the body) can be the target surface 12. The user's body 12 can be divided into different regions, for example, the left cheek region, the right cheek region, the chin region, etc. By using the razor 11 to execute a predetermined motion pattern, the device 10 can position the razor 11 relative to the user's body. For example, if the razor 11 executes a motion pattern pointing to the upper left corner, where the razor 11 is tilted to the left, the device 10 can position the razor 11, for example, in the left cheek region. Accordingly, the device 10 can simply position itself at the user's face by the motion pattern executed by the razor 11.

[0103] As another example, the movable processing device 11 can be a household appliance, and the target surface 12 can be the surface of a floor, a wall, furniture, etc. For example, the movable processing device 11 can be a vacuum cleaner, and the target surface 12 can be the floor of a room. The room 12 can be divided into different regions, for example, the upper left corner of the room, the lower right corner of the room, the center of the room, under the bed located in the room, etc. By using the vacuum cleaner 11 to execute a predetermined motion pattern, the device 10 can position the vacuum cleaner 11 relative to the floor of the room. For example, if the vacuum cleaner 11 executes a motion pattern that moves only forward and backward when the suction pipe of the vacuum cleaner 11 is lowered close to the ground, the device 10 can position the vacuum cleaner 11, for example, in the "under the bed" region. Accordingly, the device 10 can simply position itself in the room by the motion pattern executed by the vacuum cleaner 11.

[0104] According to another embodiment, the movable processing device 11 can be an oral care device, and the target surface 12 can be the dentition, where the dentition 12 is divided into different tooth regions 211, 212, 213, …, 21 n , where at least one inertial sensor data 171, 172, 173, …, 17 n The mapping with at least one motion pattern 151, 152, 153, …, 15n indicates an evaluation of the positioning of the oral care device 11 relative to one or more tooth regions 211, 212, 213, …, 21 n of the dentition 12.

[0105] The oral care device can be a toothbrush, specifically an electric toothbrush. The oral care device can also be at least one of dental floss, a polisher, a plaque removal device, an ultrasonic device, and a water spraying device. In some embodiments, the oral care device can also be an oral examination device.

[0106] According to this example, by performing a predetermined motion pattern using the oral care device 11, the device 10 can position the oral care device 11 relative to the dentition. For example, if the oral care device 11 performs a motion pattern that only points up and down, and the oral care device 11 is tilted to the left, the device 10 can position the oral care device 11, for example, in the upper left tooth area of the upper jaw. Accordingly, the device 10 can simply position itself relative to the user's dentition based on the motion pattern performed by the oral care device 11.

[0107] According to one embodiment, the dentition can be divided into nine tooth areas, where the first tooth area corresponds to the left buccal surfaces of the upper and lower jaw dentitions, the second tooth area corresponds to the occlusal surfaces of the left and right upper jaw dentitions, the third area corresponds to the occlusal surfaces of the left and right lower jaw dentitions, the fourth tooth area corresponds to the left lingual surfaces of the upper and lower jaw dentitions, the fifth tooth area corresponds to the right buccal surfaces of the upper and lower jaw dentitions, the sixth tooth area corresponds to the right lingual surfaces of the upper and lower jaw dentitions, the seventh tooth area corresponds to the labial surfaces of the upper and lower jaw dentitions, the eighth tooth area corresponds to the palatal surfaces of the upper jaw dentition, and the ninth tooth area corresponds to the oral surfaces of the anterior lower jaw dentition.

[0108] According to another embodiment, at least one predetermined motion pattern 15NB that can be additionally included in the set of motion patterns 15 can be associated with an area 21NB outside the target surface 12 or independent of the target surface 12, where at least one inertial sensor data 171, 172, 173, …, 17 n associated with at least one predetermined motion pattern 15 NB indicates that the movable processing device 11 is positioned in the area 21 NB outside the target surface 12 or independent of the target surface 12.

[0109] In other words, the area 21NB outside the target surface 12 can be an area not directly related to the target surface 12. For example, if the movable processing device 11 can be a toothbrush, the area 21NB outside the target surface 12 can be an area outside the dentition. Accordingly, this area 21NB can indicate that the user is not brushing their teeth. Therefore, this area can also be referred to as the "not brushing teeth" area, abbreviated as "NB". This area 21NB can be at least one area of the target surface 12, or this area 21NB can be an additional area other than one or more areas 211, 212, 213, …, 21 n of the target surface. However, this specific area 21NB outside the target surface 12 is not limited to the above example of brushing teeth.

[0110] Figure 2Illustrated is a dentition 12 for exemplifying the above examples. The dentition 12 may be a target surface. The dentition 12 can be divided into nine tooth regions 1a to 9a. Optionally, a tenth region NB may exist. This tenth region NB is a region outside the dentition 12. Thus, this tenth region NB is not explicitly illustrated in Figure 2 Since this tenth region NB is not related to one of the tooth regions of the dentition 12 and thus is not involved in brushing the teeth of the dentition 12, this tenth region NB may also be referred to as the "not being brushed" region.

[0111] As Figure 2 can be seen, the first tooth region 1a may correspond to the left buccal surfaces of the maxillary and mandibular dentitions 12. The second tooth region 2a may correspond to the occlusal surfaces of the left and right maxillary dentitions 12. The third region 3a may correspond to the occlusal surfaces of the left and right mandibular dentitions 12. The fourth tooth region 4a may correspond to the left lingual surfaces of the maxillary and mandibular dentitions 12. The fifth tooth region 5a may correspond to the right buccal surfaces of the maxillary and mandibular dentitions 12. The sixth tooth region 6a may correspond to the right lingual surfaces of the maxillary and mandibular dentitions 12. The seventh tooth region 7a may correspond to the labial surfaces of the maxillary and mandibular dentitions 12. The eighth tooth region 8a may correspond to the palatal surfaces of the maxillary dentition 12. The ninth tooth region 9a may correspond to the oral surfaces of the anterior mandibular dentition 12.

[0112] Figure 3 Illustrated is a dentition 12 for exemplifying another example. The dentition 12 may be a target surface. The dentition 12 can be divided into sixteen tooth regions 1b to 16b. Optionally, a seventeenth region NB may exist. This seventeenth region NB is a region outside the dentition 12. Thus, this seventeenth region NB is not explicitly illustrated in Figure 3 Since this seventeenth region NB is not related to one of the tooth regions of the dentition 12 and thus is not involved in brushing the teeth of the dentition 12, this seventeenth region NB may also be referred to as the "not being brushed" region.

[0113] As Figure 3As can be seen, the first tooth region 1b may correspond to the left buccal surface of the maxillary dentition 12. The second tooth region 2b may correspond to the occlusal surface of the left maxillary dentition 12. The third tooth region 3b may correspond to the occlusal surface of the left mandibular dentition 12. The fourth tooth region 4b may correspond to the left lingual surfaces of the maxillary and mandibular dentitions 12. The fifth tooth region 5b may correspond to the right buccal surfaces of the maxillary and mandibular dentitions 12. The sixth tooth region 6b may correspond to the occlusal surface of the right maxillary dentition 12. The seventh tooth region 7b may correspond to the occlusal surface of the right mandibular dentition 12. The eighth tooth region 8b may correspond to the palatal surface of the maxillary dentition 12. The ninth tooth region 9b may correspond to the labial surface of the maxillary dentition 12. The tenth tooth region 10b may correspond to the labial surface of the mandibular dentition 12. The eleventh tooth region 11b may correspond to the palatal surface of the maxillary dentition 12. The twelfth tooth region 12b may correspond to the oral surface of the anterior mandibular dentition 12. The thirteenth tooth region 13b may correspond to the left buccal surface of the mandibular dentition 12. The fourteenth tooth region 14b may correspond to the left lingual surface of the mandibular dentition 12. The fifteenth tooth region 15b may correspond to the right buccal surface of the mandibular dentition 12. The sixteenth tooth region 16b may correspond to the right lingual surface of the mandibular dentition 12.

[0114] Figure 2 and Figure 3 is described only as a non - limiting example. The target surface 12 may also include more or fewer than the exemplary nine or sixteen tooth regions. Additionally, the tenth / seventeenth tooth region NB outside the target surface 12 is optional. The exact distribution of one or more tooth regions of the dentition 12 may vary from the above examples.

[0115] Regarding the positioning of the movable processing device relative to the target surface 12, the device 10 can be self - learning. The device 10 can utilize artificial intelligence, for example, by employing a deep - learning network. For example, the device 10 can utilize a classifier developed by artificial intelligence and / or learning networks; in addition, the device 10 (and the systems described herein) can also teach such classifiers. Accordingly, the device 10 for implementing the positioning of the movable processing device 11 relative to the target surface 12 can enhance its performance over time by using the neural network 18.

[0116] According to one embodiment, the neural network 18 can be a recurrent neural network (RNN).

[0117] For example, the neural network can be a long short - term memory (LSTM) network or a gated recurrent unit (GRU) network.

[0118] RNNs may suffer from the so - called vanishing gradient problem, where the gradient rapidly vanishes as the number of layers increases. The vanishing gradient can lead to a rather slow training speed. Therefore, LSTM networks and / or GRU networks can be used to avoid the vanishing gradient problem.

[0119] The LSTM network is an artificial neural network that contains LSTM blocks in addition to regular network units. The LSTM blocks contain gates that determine when the input is significant enough to remember, when to continue remembering or forget the value, and when to output the value.

[0120] Figure 4 An example of an RNN in its most general form is shown. The input 41 can be fed into the neuron 40 at a certain time point t. The input 41 can be a single value or a vector containing two or more values. The input 41 at a certain time point t can also be symbolically represented by X. t for symbolic representation.

[0121] The neuron 40 may also optionally include another input 42. This other input 42 can be provided from a neuron (not shown here) at the previous time point t - 1.

[0122] The neuron 40 can contain at least one gate 43, which can provide a mathematical operation. In this example, the gate 43 is a single tanh gate.

[0123] The neuron 40 can include at least one output 46. The output 46 can contain the operation result of the tanh gate 43 that has been fed with the input 41 and optionally another input 42. The output 46 can result in a hidden state 45, which will be explained later.

[0124] The neuron 40 can optionally contain another output branch 46, which branches out from the operation output result of the tanh gate 43 that has been fed with the input 41 and optionally another input 42 as described above.

[0125] In Figure 4 , each drawn line can carry a complete vector from the output of one node to the input of other nodes. Line merging (e.g., at 47) represents association, while line forking (e.g., at 48) means that its components are copied and the copies travel to different locations. This also applies to other neural networks that will be described below with reference to the following figures.

[0126] Figure 5 An example of a GRU network is shown. The GRU network includes a neuron 50. In addition to the above - mentioned RNN neuron 40, the GRU neuron 50 can include two additional gates, namely the first sigmoid gate 53 and the second sigmoid gate 54. In addition, the GRU neuron 50 can contain dot operations 55, 56, 57, 58, 59, for example, such as vector addition 58.

[0127] Figure 6AAn example of an LSTM network is shown, which can be used as the neural network 18 of device 10. The LSTM may include a neural unit 60, which may also be referred to as an LSTM block in the context of the LSTM network. In addition to the above neural units 40, 50, the neural unit 60 of the depicted LSTM network may include a cell state, which is a horizontal line 61 running through the top of the neural unit 60. The neural unit 60 may receive a cell state input 62 and may generate a cell state output 66.

[0128] The neural unit 60 may also include four gates 43, 53, 54, 63. For example, compared with the above GRU network, it may include another sigmoid gate 63. With the help of these gates 43, 53, 54, 63, information can be removed from or added to the cell state (horizontal line 61).

[0129] Figure 6B Another example is shown in which the previous state and the subsequent state (relative to time point t) of the neural unit are depicted. Specifically, the neural unit 60t at time point t is depicted. In addition, another neural unit 60t - 1 at the previous time point t - 1 is depicted. Further still, another neural unit 60t + 1 at the subsequent time point t + 1 is depicted. The depicted neural units 60t - 1, 60t, 60t + 1 may represent the same neural unit at different time points, that is, at time point t, at the previous time point t - 1, and at the subsequent time point t + 1.

[0130] The above input 41, also symbolized by the letter X, may include at least one sensor data 171, 172, 173, …, 17 from the inertial sensor 13 n . The input X can be time - dependent, so X = X(t). Specifically, the depicted input Xt may include the sensor data 172 acquired during the considered time point t, the depicted input Xt - 1 may include the sensor data 171 acquired during the previous time point t - 1, and the depicted input Xt + 1 may include the sensor data 173 acquired during the subsequent time point t + 1.

[0131] As can also be seen from Figure 6B it, the neural units 60t - 1, 60t, 60t + 1 may provide corresponding output values yt - 1, yt, yt + 1 at their respective depicted time points t - 1, t, t + 1, for example, through prediction. The output value y(t) can be a single value or a vector including one or more vector elements.

[0132] The output value y(t) can be calculated as:

[0133] yt = softmax ( Why·ht + b)

[0134] The output value y(t) may include, for example, a probability value, as will be explained in more detail with reference to Figure 7 below. For example, the output value y(t) may be a vector including one or more vector elements, where each vector element may represent one of the motion patterns 151, 152, 153, …, 15 n or, more specifically, where each vector element may represent a probability value that indicates how likely the input X(t) (i.e., the inertial sensor data 171, 172, 173, …, 17 n ) may correspond to one of the motion patterns 151, 152, 153, …, 15 n .

[0135] Furthermore, the depicted neural units 60t-1, 60t, 60t+1 may be arranged in the same layer, namely the first layer. Some examples may include one or more other layers, where each layer may include its own neural units. Such examples will be referred to later, for example, with reference to Figure 7 below. However, examples and embodiments having at least the first layer will be further referred to with reference to Figure 6B below.

[0136] According to this embodiment, the neural network 18 may include a first layer, where the first layer includes the neural unit 60t, and where, at a first time point t, at least one inertial sensor data Xt 172 is input into the neural unit 60t of the first layer. At a subsequent second time point t+1, second inertial sensor data Xt+1 173 and at least one output ht46 of the neural unit 60t at the previous first time point t are input into the neural unit 60t+1 of the first layer.

[0137] Figure 7 shows another example, where the neural network 18 may include at least two layers, namely the first layer 71 and the second layer 72. The first layer 71 includes at least a first neural unit 60t, and the second layer 72 includes at least a second neural unit 70 t .

[0138] As can be seen, the sensor data 171, 172, 173 acquired during different time instances t-1, t, t+1 may be fed as inputs Xt-1, Xt, Xt+1 into the corresponding neural units 60t-1, 60t, 60t+1 of the first layer 71.

[0139] The outputs 46t-1, 46t, 46t+1 of each neural unit 60t-1, 60t, 60t+1 of the first layer 71 may be fed as inputs into the corresponding neural units 70t-1, 70t, 70t+1 of the second layer 72.

[0140] The neural units 60t-1, 60t, 60t+1 of the first layer 71 and the neural units 70t-1, 70t, 70t+1 of the second layer 72 can be the same. Alternatively, the internal structures of the neural units 60t-1, 60t, 60t+1 of the first layer 71 and the neural units 70t-1, 70t, 70t+1 of the second layer 72 can be different from each other.

[0141] According to Figure 7 In the embodiment shown, the neural network 18 can include at least a first layer 71 and a second layer 72, where the first layer 71 can include a first neural unit 60t and where the second layer 72 can include a second neural unit 70t, where at a first time point t, at least one inertial sensor data Xt172 is input into the first neural unit 60t of the first layer 71, and where the output ht46 of the first neural unit 60t is input into the neural unit 70t of the second layer 72.

[0142] So far, the signal path in the vertical direction has been described, that is, from the first layer 71 at the bottom to the second layer 72 at the top. However, in Figure 7 the embodiment, the signal path in the horizontal direction is also shown.

[0143] As can be seen, the cell state output Ct66 of the first neural unit 60t at the first time point t and / or the output ht46 of the first neural unit 60t at the first time point t can be fed back as inputs into the first neural unit 60 again, that is, into the first neural unit 60t+1 at a subsequent time point t+1. As described above, the neural unit 60 itself can be the same neural unit, but for the sake of simplicity in illustrating the states of the neural unit 60 at different time instances t-1, t, t+1, it can only be described as a plurality of series-connected neural units 60t-1, 60t, 60t+1 in the drawings. In other words, the horizontal signal path can describe the signal paths of the neural unit 60 at different subsequent time instances t-1, t, t+1. This also applies to the second layer 72 and any other layer.

[0144] Accordingly, the depicted subsequent time instances t-1, t, t+1 can represent a length 77 during which the neural network 18 can sample and process the acquired sensor data 171, 172, 173, …, 17 n . The length 77 can thus be referred to as the running length, sample length, or sampling period. For example, the sampling length 77 can correspond to one second, where the time instances t-1, 5t, t+1 can be fractions of the one second. For example, the sampling period 77 can have a length of fifty samples, that is, fifty time instances. The neural network 18 can run once within the sampling period, or the neural network 18 can keep running within two or more sampling periods.

[0145] Thus, according to another embodiment, the neural network 18 may include at least a first layer 71 and a second layer 72, wherein the first layer 71 may include a first neural unit 60t and wherein the second layer 72 may include a second neural unit 70t, wherein at a first time point t, at least one inertial sensor data Xt 172 may be input into the first neural unit 60t of the first layer 71, and wherein at least one output ht 46 of the first neural unit 60t may be input into the neural unit 70t of the second layer 72. So far, it may be the same as described above. However, in addition, at a subsequent second time point t+1, second inertial sensor data Xt+1 173 and at least one output ht 46 of the first neural unit 60t at the first time point t are input into the first neural unit 60t+1 at the subsequent second time point t+1.

[0146] As described above, several mathematical operations may be performed in the doors 43, 53, 54, 63 by the neural network 18. In Figure 7 the example shown, the following mathematical operations may be performed at different stages:

[0147]

[0148] where

[0149] · i(t) is the input gate activation vector

[0150] · f(t) is the forget gate activation vector

[0151] · o(t) is the output gate activation vector

[0152] · c(t) is the cell state vector

[0153] · h(t) is the output vector of the LSTM section or neural unit 60, 70

[0154] According to this example, the input sensor data Xt 172 may be an element vector For example, it may be an input tensor [Ax, Ay, Az, Gx, Gy, Gz] T

[0155] In addition, the weights W(t) and the bias values b(t) are depicted in Figure 7 where, in this example:

[0156] · weights

[0157] · bias

[0158] In addition, the output vector y(t) may be calculated as:

[0159] y_t = softmax(W_hy·h_t + b)

[0160] The depicted hidden state h(t) can also be an element vector, for example, an element vector containing 256 elements of the element vector.

[0161] In addition, the depicted hidden state C(t) can also be an element vector, for example, an element vector containing 256 elements of the element vector.

[0162] As described above, the input inertial sensor data X_t172 can be an element vector containing six vector elements For example, the input tensor [A_x, A_y, A_z, G_x, G_y, G_z] T . These vector elements [A_x, A_y, A_z, G_x, G_y, G_z]^T can also be referred to as the inertial sensor data part.

[0163] According to one embodiment, at least one inertial sensor data 171 can include at least three inertial sensor data parts of a group, the group including linear velocities in the x, y, and z directions, angular velocities with respect to the x, y, and z axes, linear accelerations in the x, y, and z directions, and angular accelerations with respect to the x, y, and z axes.

[0164] In other words, the inertial sensor 13 can provide inertial sensor data 171, 172, 173,..., 17 at one or more time instances t - 1, t, t + 1 n , where the inertial sensor data 171, 172, 173,..., 17 n can depend on the current orientation and motion of the movable processing device 11 at an observable time instance t - 1, t, t + 1. Each inertial sensor data 171, 172, 173,..., 17 n can be a vector including at least three, or in other examples at least six vector elements, where the vector elements represent the above-mentioned inertial sensor data parts, and at least one of the inertial sensor data parts can be zero.

[0165] Accordingly, the inertial sensor data 171, 172, 173,..., 17 n (vector) (specifically the sensor data part (vector element)) can represent the current motion mode of the movable processing device 11 as sampled within a sampling period 77 including one or more subsequent time instances t - 1, t, t + 1.

[0166] According to Figure 7In the embodiments depicted, at least one inertial sensor data 172 (vector) may include one or more inertial sensor data portions (vector elements), where the input to the neural unit 60t at the first time point t is the corresponding inertial sensor data 172 that includes one or more inertial sensor data portions retrieved during the first time point t. At least one inertial sensor data 171, 172, 173, …, 17 n may be sampled during the sampling time 77.

[0167] The neural network 18 may map at least one sampled inertial sensor data 171, 172, 173, …, 17 that has been sampled during the sampling time 77 n to at least one motion pattern 151, 152, 153, …, 15 included in the set of motion patterns 15 n , as originally referenced Figure 1 described. After the mapping, the selected one motion pattern may be referred to as the mapped motion pattern.

[0168] In other words, the neural network 18 may receive the inertial sensor data 171, 172, 173, …, 17 n as the input x(t), and it may output one or more probability values as the output y(t). As described above, in Figure 7 the example shown, the output value y(t) may also be an element vector that includes, for example, at least three, or at least six, or at least twelve vector elements. Each vector element of the output vector y(t) may represent a motion pattern 151, 152, 153, …, 15 n that may be associated with a class or region 211, 212, 213, …, 21 n with a probability value. In some embodiments, the output value y(t) may be an element vector that includes, for example, at least two to as many desired classes or regions as possible, such as nine regions, twelve regions, or sixteen regions.

[0169] Accordingly, the output vector y(t) may represent different regions 211, 212, 213, …, 21 of the target surface 12 n . For example, if the target surface 12 may include twelve regions (e.g., eleven tooth regions and a twelfth “not brushing” region ‘NB’), then the element output vector y(t) may include twelve vector elements, such as Figure 7 shown in the example of Accordingly, each vector element may represent one of the different regions 211, 212, 213, …, 21 of the target surface 12 n .

[0170] As previously mentioned, the vector elements can represent probability values. These probability values can represent the probability values of different regions 211, 212, 213, …, 21 n in the target surface 12. In other words, the neural network 18 can receive at least one inertial sensor data 171, 172, 173, …, 17 n and map the at least one inertial sensor data 171, 172, 173, …, 17 n to at least one motion pattern 151, 152, 153, …, 15 n , and since the motion patterns 151, 152, 153, …, 15 n can each be associated with one or more different regions 211, 212, 213, …, 21 n in the target surface 12, the probability values can indicate the likelihood that the at least one inertial sensor data 171, 172, 173, …, 17 n acquired can correspond to one of the different regions 211, 212, 213, …, 21 n in the target surface 12. This is referred to as the mapping of the at least one inertial sensor data 171, 172, 173, …, 17 n to at least one of the motion patterns 151, 152, 153, …, 15 n .

[0171] Since each motion pattern 151, 152, 153, …, 15 n can be associated with one or more different regions 211, 212, 213, …, 21 n in the target surface 12, the mapping of the at least one inertial sensor data 171, 172, 173, …, 17 n to the at least one motion pattern 151, 152, 153, …, 15 n indicates an evaluation of the positioning of the movable processing device 11 relative to one or more regions 211, 212, 213, …, 21 n in the target surface 12. For example, the positioning of the processing device 11 can be evaluated because, compared with the absolute value geographic data from GPS, the positioning detection of the present invention can be based on the probability values mentioned above.

[0172] In other words, the device 10 can simply receive the sensor data 171, 172, 173, …, 17 n and map the sensor data 171, 172, 173, …, 17 n to one or more regions 211, 212, 213, …, 21 nAssociated motion patterns 151, 152, 153, …, 15 n , which region 211, 212, 213, …, 21 within the target surface 12 the movable processing device 11 is located in is derived from the neural network 18 n for evaluation.

[0173] Thus, according to one embodiment, the output y(t) of the neural network 18 may include one or more probability values, which are used for evaluating the positioning of the movable processing device 11 relative to one or more regions 211, 212, 213, …, 21 of the target surface 12 n for evaluation.

[0174] According to another embodiment, the motion pattern recognition device 14 may be configured to determine the relative movement between the movable processing device 11 and the target surface 12 from at least one inertial sensor data 171, 172, 173, …, 17 n and remove the determined movement of the target surface 12 from the determined movement of the movable processing device 11.

[0175] For example, the movable processing device 11 may be a toothbrush, and the target surface 12 may be the user's dentition. The user may turn their head while brushing their teeth. In this case, the inertial sensor 13 will sense the relative movement between the user's head and the toothbrush because the toothbrush moves with the head. This may result in incorrect motion detection, thus incorrect mapping, and ultimately incorrect positioning based on the mapping.

[0176] However, according to the above embodiment, the sensed or determined movement of the user's head (target surface) 12 can be removed from the sensed relative movement between the head and the toothbrush. Therefore, only the desired movement of the toothbrush (processing device) 11 is retained. As will be further detailed below, this movement of the user's head (target surface) 12 can be detected by a camera associated with the device 10, where the image and / or video output of the camera can be classified by the neural network 18 (or by a separate learning network).

[0177] Figure 8 A block diagram showing an example of a method for implementing the positioning of the movable processing device 11 relative to the target surface 12 is shown, where the movable processing device 11 includes an inertial sensor 13 and where the movable processing device 11 is configured to process the target surface 12.

[0178] In block 801, the method includes the step of differentiating between two or more motion patterns 151, 152, 153, …, 15 included in the set of motion patterns 15 of the movable processing device 11 n for the step.

[0179] In block 802, the method includes receiving at least one inertial sensor data 171, 172, 173, …, 17 from the inertial sensor 13 n where the at least one inertial sensor data 171, 172, 173, …, 17 n represents the movement of the movable processing device 11.

[0180] In block 803, the method includes receiving and processing the at least one inertial sensor data 171, 172, 173, …, 17 by means of the neural network 18 n and mapping / classifying the at least one inertial sensor data 171, 172, 173, …, 17 n to / into at least one motion pattern 151, 152, 153, …, 15 included in the set of motion patterns 15 n where the motion patterns 151, 152, 153, …, 15 included in the set of motion patterns 15 n are each associated with one or more different regions 211, 212, 213, …, 21 of the target surface 12 n such that the mapping / classification of the at least one inertial sensor data 171, 172, 173, …, 17 n to / into at least one motion pattern 151, 152, 153, …, 15 n indicates an assessment of the positioning of the movable processing device 11 relative to one or more regions 211, 212, 213, …, 21 of the target surface 12. n

[0181] Figure 9 FIG. shows another exemplary device 100 according to the present disclosure. The device 100 may be similar to the above device 10. In addition, all the above features of the device 10 are combined with the following device 100, and vice versa.

[0182] The device 100 may be different from the device 10 (see Figure 1 ), where the motion patterns 151, 152, 153, …, 15 n may be mapped / classified into one or more class members 101A, 101B, …, 104A, 104B of different classes 101, 102, 103, 104 instead of different regions 211, 212, 213, …, 21 of the target surface 12 n .

[0183] ​Thus, the device 100 is configured to classify the motion of a movable personal equipment 11 (also referred to herein as a movable processing device 11), which includes an inertial sensor 13. The device 100 includes a motion pattern recognition device 14, which is configured to distinguish between two or more motion patterns 151, 152, 153, ..., 15n contained in a motion pattern set 15 of the movable personal equipment 11.

[0184] Furthermore, the device 100 comprises an interface 16 for transmitting at least one inertial sensor data 171, 172, 173, ..., 17 from the inertial sensor 13 to the user. n is provided to the motion pattern recognition device 14, wherein at least one inertial sensor data 171, 172, 173, ..., 17 n Indicates the movement of the movable personal appliance 11.

[0185] The motion pattern recognition device 14 includes a neural network 18, which is configured to receive at least one inertial sensor data 171, 172, 173, ..., 17 n and at least one inertial sensor data 171, 172, 173, ..., 17 n Mapped to / classified as at least one motion mode 151, 152, 153, ..., 154 included in the motion mode set 15 n , wherein at least one of the mapped motion modes 151, 152, 153, ..., 15 n At least one class member 101A, 101B, 102A, 102B, 103A, 103B, 104A, 104B of one or more classes 101, 102, 103, 104 is associated such that at least one class member 101A, 101B, ..., 104A, 104B is selected based on the movement of the movable personal appliance 11.

[0186] In other words, the neural network 18 may be, for example, as described above with reference to Figures 1 to 8 The method converts at least one inertial sensor data 171, 172, 173, ..., 17 n Mapped to / classified as at least one motion mode 151, 152, 153, ..., 15 n Due to the mapped motion patterns 151, 152, 153, ..., 15 n Each of the movable personal device 11 may be associated with at least one class member 101A, 101B, ..., 104A, 104B of one or more classes 101, 102, 103, 104, so that at least one class member 101A, 101B, ..., 104A, 104B may be associated with at least one class member 101A, 101B, ..., 104A, 104B based on at least one mapped motion pattern 151, 152, 153, ..., 154An , i.e., selected based on the movement of the movable personal appliance 11.

[0187] Figure 9 Non-limiting examples of Figure 9 show four classes 101, 102, 103, 104, where each class includes two class members 101A, 101B, …, nA, nB. However, there may be at least one class, and each class may include at least two class members. There may also be more than two classes or even more than the exemplary four classes depicted.

[0188] As can be seen in the example of Figure 9 , the first mapped / classified motion pattern 151 may be associated with the class member 101A of the first class 101. The nth mapped / classified motion pattern 154 may be associated with the class member nB of the fourth class 104. The second mapped / classified motion pattern 152 may be associated with two class members of different classes, for example, associated with the class member 101B of the first class 101 and associated with the class member 102A of the second class 102. The third mapped / classified motion pattern 153 may be associated with two class members of the same class, for example, associated with the two class members 103A, 103B of the third class.

[0189] Generally, at least one mapped / classified motion pattern 151, 152, 153, …, 15 n may be associated with at least one class member 101A, 101B, 102A, 102B, 103A, 103B, 104A, 104B of one or more classes 101, 102, 103, 104.

[0190] Some examples of classes and class members will be described below.

[0191] According to one embodiment, at least one class 101 of one or more classes 101, 102, 103, 104 may include at least one class member 101A, where the one class 101 may represent a user group, and where the at least one class member 101A may represent at least one user of the user group, where at least one mapped motion pattern 151, 152, 153, …, 15 n may be associated with at least one class member 101A for identifying the at least one user based on the movement of the movable personal appliance 11.

[0192] In other words, one of the classes 101, 102, 103, 104 may represent a user group, i.e., a group of users who use the movable personal appliance 11. The corresponding class may include at least one class member, and the at least one class member may represent a specific user of the user group. For example, the first class 101 may represent a user group, and the user group may be a single family. In this example, the user group 101 may contain only one class member 101A, i.e., one person. The device 100 of the present invention may be configured to simply identify the at least one user 101A based on the movement of the movable personal appliance 11. Thus, the device 100 of the present invention may personalize any action or interaction with respect to the one identified user 101A, as will be described using some examples hereinafter.

[0193] According to another embodiment, at least one class 101 of one or more of the classes 101, 102, 103, 104 may include at least two class members 101A, 101B, where the one class 101 may represent a user group, and where the at least two class members 101A, 101B may represent at least two users of the user group, where at least one of the mapped motion patterns 151, 152, 153, …, 15 n may be associated with one of the at least two class members 101A, 101B for identifying at least one user in the user group based on the movement of the movable personal appliance 11.

[0194] In other words, one of the classes 101, 102, 103, 104 may represent a user group, i.e., a group of users who use the movable personal appliance 11. The corresponding class may include at least one class member, and the at least one class member may represent a specific user of the user group. For example, the first class 101 may represent a user group, and the user group may be a family. The class members 101A, 101B of the class 101 may represent family members. For example, the user group 101 may include one or more family members, such as where the first class member 101A may represent the mother in the family and the second class member 101B may represent the child in the family.

[0195] The device 100 may be configured to simply identify at least one user based on the movement of the movable personal appliance 11. This can be achieved if each user uses the 1 movable personal appliance 11 in a different or individual way.

[0196] For example, in an embodiment where the movable personal appliance 11 may be a movable oral care device such as a toothbrush, specifically an electric toothbrush. The movable oral care device may also be at least one of dental floss, a plaque removal device, an ultrasonic device, and a water spraying device.

[0197] Taking the above example, mother 101A may use the toothbrush 11 in a different way from child 101B. The inertial sensor 13 of the toothbrush 11 may provide its inertial sensor data 171, 172, 173, …, 17 n to a motion pattern recognition device 14 including a neural network 18. The neural network 18 may map / classify the inertial sensor data 171, 172, 173, …, 17 n to / into at least one motion pattern 151, 152, 153, …, 15 n .

[0198] For example, as Figure 9 shown, the mother may have a brushing style corresponding to the first motion pattern 151. This motion pattern 151 may be associated with the class member 101A representing the mother. While the child may have a brushing style different from that of the mother, for example, a brushing style corresponding to the second motion pattern 152. This motion pattern 152 may be associated with the class member 101B representing the child.

[0199] Therefore, the device 100 may simply identify the user in the user group based on the motion of the movable personal appliance 11. As described above, the device 100 of the present invention may personalize any action or interaction with respect to the identified user.

[0200] According to one embodiment, the motion pattern recognition device 14 may be configured to select a user-specific set of motion pattern presets 115 based on the step of identifying the at least one user 101A, the set of motion pattern presets including two or more user-specific motion patterns 1151, 1152, 1153, …, 115 n of the at least one identified user 101A.

[0201] Such an example is shown in Figure 10 . This embodiment may also be referred to as a two-step method. In the first step 121, the user is identified. The identified user may have a user-specific set of motion pattern presets 115 that has been separately trained by the neural network 18. In the second step 122, the neural network 18 uses the user-specific motion patterns 1151, 1152, 1153, …, 115 n from the user-specific set of motion pattern presets 115. Thus, the device 100 of the present invention may then act separately and interact with each identified user.

[0202] In Figure 10 is shown the first step 121, in which the neural network 18 receives at least one inertial sensor data 171, 172, 173, …, 17 nAnd map the at least one inertial sensor data to at least one of the motion patterns 151, 152, 153, …, 15 included in the set of motion patterns 15 n in the set. The at least one mapped motion pattern, for example, the nth motion pattern 154, may be associated with a class member 101B of the first class 101. This process may correspond to the process described above with reference to Figure 9 the process described above.

[0203] The class 101 may be a user group, and the class member 101B may be a user of the user group. Taking the above example, the identified user 101B may be a child in the family. The device 100 may have stored user-specific motion patterns. That is, the identified user, i.e., the child 101B, may have a preset set 115 of its own individual user-specific motion patterns 1151, 1152, 1153, …, 115 n For any other action after the identification in the first step 121, the motion pattern recognition device 14, and specifically the neural network 18, may use these user-specific motion patterns 1151, 1152, 1153, …, 115 n .

[0204] Therefore, after the step 121 of identifying the at least one user 101B, the neural network 18 may select at least one preset set 115 of user-specific motion patterns, which includes two or more user-specific motion patterns 1151, 1152, 1153, …, 115 n that are characteristic of the at least one identified user 101B.

[0205] Accordingly, in the second step 122 after the first step 121 of identifying the user, the neural network 18 may use the user-specific preset set 115 of user-specific motion patterns 1151, 1152, 1153, …, 115 n instead of the set 15 of motion patterns 151, 152, 153, …, 15 n That is, all the actions that can be performed by the devices 10, 100 as described herein by utilizing the set 15 of motion patterns 151, 152, 153, …, 15 n can also be performed by the devices 10, 100 individually or personally for each identified user by utilizing the preset set 115 of user-specific motion patterns 1151, 1152, 1153, …, 115 n instead of the set 15 of motion patterns 151, 152, 153, …, 15 n .

[0206] Thus, according to one embodiment, after the first step 121 of identifying the at least one user 101B, the neural network 18 may be configured to replace the set of motion patterns 15 with a selected user-specific set of preset motion patterns 115, and replace two or more motion patterns 151, 152, 153, …, 15 included in the set of motion patterns 15 n with two or more user-specific motion patterns 1151, 1152, 1153, …, 115 included in the user-specific set of preset motion patterns 115 n .

[0207] Additionally or alternatively, the device 100 may include at least two neural networks. Figure 11 An example of such is shown.

[0208] Figure 11 An example of the device 100 of may substantially correspond to Figure 10 the device 100 shown in the example of. Figure 11 The device of is different from the device of Figure 10 in that Figure 11 the device of may include a second neural network 182.

[0209] As can be seen in Figure 11 , in the first step 121, the first neural network 181 may perform the above actions, e.g., identifying the user 101B in the user group 101. However, in the second step 122, the inertial sensor data 171, 172, 173, …, 17 n may be fed into the second neural network 182. The second neural network 182 may use the user-specific set of preset motion patterns 1151, 1152, 1153, …, 115 n as described above.

[0210] In other words, after the first step 121 of identifying the at least one user 101B, the motion pattern recognition device 14 may use the second neural network 182, wherein the second neural network 182 may be configured to receive at least one inertial sensor data 171, 172, 173, …, 17 n and map the at least one inertial sensor data 171, 172, 173, …, 17 n to at least one user-specific motion pattern 1151, 1152, 1153, …, 115 included in the user-specific set of preset motion patterns 115 n , wherein the user-specific motion patterns 1151, 1152, 1153, …, 115 nEach is associated with at least one class member 102A, 102B of one or more classes 101, …, 104 such that at least one class member 102A, 102B is selected based on the movement of the movable personal appliance 11. In other words, the neural network 18 can be a neural network trained specific to the user.

[0211] Accordingly, the motion pattern recognition device 14 can be configured to use a user-specific set of motion patterns 1151, 1152, 1153, …, 115 n of a user-specific preset set 115, and classify the movement of the personal appliance 11 in a user-specific manner by means of at least one inertial sensor data 171, 172, 173, …, 17 n

[0212] As Figure 10 and Figure 11 shown in the example of, the device 100 can include at least one class 102 for classifying the target object in the second step 122. However, in the second step 122, the device 100 can include more than one class, as Figure 9 shown in the example of.

[0213] In the second step 122, for example, after the specific user has been identified in the first step 121, different actions can be performed by the personal appliance 11. For example, the personal appliance 11 can change its operation mode based on the identified user. For example, the personal appliance 11 can be electrically driven and can include a motor (see Figure 15 and Figure 16 ), where the personal appliance 11 can change one or more motor-specific characteristics, such as frequency, amplitude, or pulsation, based on the identified user. Additionally or alternatively, the personal appliance 11 can include one or more elements for communicating with the user or providing feedback to the user, for example, a visual element such as a light, for example an LED, or a tactile element such as a vibration motor. For example, by changing the operation mode of the element for communication, for example by changing the LED light to a different color based on the identified user or by providing different pulse feedback using the vibration motor, the personal appliance 11 can change the user experience based on the identified user.

[0214] ​Additionally or alternatively, in order to identify a specific user within a group of users, such as family members in a household, the device 100 may be configured to identify a specific user type. For example, if the personal appliance 11 is a toothbrush, some people start brushing from their front teeth or incisors, while others start from their back teeth or molars. In another example, if the personal appliance is a razor, some people may shave with the grain, while others may shave against the grain. The summarized user type can be the type of user who uses the personal appliance 11 in a specific manner. There may be two or more users who can be classified into the user type group. The foregoing examples of user identification instead individually identify each user.

[0215] According to one embodiment for identifying user types, at least one class 104 among one or more classes 101, 102, 103, 104 may include at least two class members nA, nB, where the one class 104 may represent the user type of the movable personal appliance 11, where the first class member nA may represent the first user type of the movable personal appliance 11 and where the second class member nB may represent the second user type of the movable personal appliance 11, where at least one mapped motion pattern 151, 152, 153, …, 15 n may be associated with the first class member nA or the second class member nB for identifying the user type of the movable personal appliance 11 based on the motion of the movable personal appliance 11.

[0216] According to another embodiment, the motion pattern recognition device 14 may be configured to select a user type-specific preset set 115 of motion patterns after the step of identifying the user type, the preset set including two or more user type-specific motion patterns 115 1 、115 2 、115 3 、…、115 n of the movable personal appliance 11, which are characteristic of the identified user type, and where the neural network 18 may be configured to replace the set 15 of motion patterns with the selected user type-specific preset set 115 of motion patterns and replace two or more motion patterns 151, 152, 153, …, 15n included in the set 15 of motion patterns with two or more user type-specific motion patterns 1151, 1152, 1153, …, 115 n .

[0217] Regarding the user-specific motion patterns 1151, 1152, 1153, …, 115 nEverything explained by the user - specific preset set 115 also applies to the user - type - specific motion patterns 1151, 1152, 1153, …, 115 n of the user - type - specific preset set 115.

[0218] As described above, the identified user types can be classified into one or a group of user types. Thus, the device 100 can perform a clustering analysis, where the user can use the personal appliance 11 a predetermined number of times before this user is classified into a specific user - type group. For example, the user can use his razor five times within five consecutive days. Four out of the five days the user shaves against the grain. Thus, after the fifth day, the device 100 can classify this user into a user - type group in which all users who shave against the grain are aggregated.

[0219] The clustering analysis can also be performed at shorter time intervals, i.e., the toothbrush 11 can be turned on and off directly in succession. For example, the user can start his electric toothbrush 11 for the first time, turn it off, and then turn it on a second time to restart the toothbrush 11 again. When the toothbrush 11 is restarted, the device 100 of the present invention, specifically, the neural network 18, can also be restarted. When the toothbrush 11 is started, it can collect information for the clustering analysis. However, at least the neural network 18 should be restarted each time before the information for the clustering analysis is collected. In summary, the device 100 can repeatedly (e.g., five times) perform the clustering analysis before finally classifying the user into a specific user - type group.

[0220] After the user has been classified into a specific user - type - specific group, the neural network 18 can use the user - type - specific preset set 115 of the associated user - type - specific motion patterns 1151, 1152, 1153, …, 115 n of the user - type - specific preset set 115.

[0221] According to such an embodiment, the motion pattern recognition device 14 can be configured to repeatedly perform the clustering analysis a predetermined number of times, where in each of said clustering analyses, the neural network 18 can be configured to restart and, after restarting, perform the steps of receiving at least one inertial sensor data 171, 172, 173, …, 17 n and mapping at least one inertial sensor data 171, 172, 173, …, 17 n to at least one motion pattern 151, 152, 153, …, 15 included in the motion pattern set 15 n and where the neural network 18 can be configured to select the user - type - specific motion pattern preset set 115 after performing the clustering analysis a predetermined number of times.

[0222] The device 100 of the present invention can provide even more solutions for classifying the movement of the movable personal appliance 11. Therefore, reference should be made again to Figure 9 .

[0223] According to one embodiment, at least one class 102 among one or more classes 101, 102, 103, 104 may include at least two class members 102A, 102B, wherein the one class 102 may represent a disposal evaluation of the movable personal appliance 11, wherein the first class member 102A may represent the correct disposal of the movable personal appliance 11, and wherein the second class member 102B may represent the incorrect disposal of the movable personal appliance 11, wherein at least one mapped movement pattern 151, 152, 153, …, 15 n can be associated with the first class member 102A or the second class member 102B for evaluating the disposal of the movable personal appliance 11 based on the movement of the movable personal appliance 11.

[0224] In other words, the device 100 can be configured to check whether the user of the movable personal appliance 11 can use the movable personal appliance 11 correctly. Of course, in the above Figure 10 and Figure 11 two-step process, for example, after identifying the user and / or the user type, the one class 102 representing the disposal evaluation can also be used as a class in the second step 122.

[0225] According to another embodiment, at least one class 103 among one or more classes 101, 102, 103, 104 may include at least two class members 103A, 103B, wherein the one class 103 may represent the movement execution quality of the movable personal appliance 11, wherein the first class member 103A may represent good movement execution of the movable personal appliance 11, and wherein the second class member 103B may represent poor movement execution of the movable personal appliance 11, wherein at least one mapped movement pattern 151, 152, 153, …, 15n can be associated with the first class member 103A or the second class member 103B for evaluating the movement execution quality of the movable personal appliance 11 based on the movement of the movable personal appliance 11.

[0226] In other words, the device 100 can be configured to check whether the user of the movable personal appliance 11 can use the movable personal appliance 11 in a good or poor manner. A good manner can be a manner of implementing the movement of the movable personal appliance as expected, while a poor manner can be a manner of not implementing the movement of the movable personal appliance 11 as expected. For example, if the personal appliance 11 is a toothbrush, the device can check whether the user can have good or poor brushing techniques.

[0227] Of course, in the above Figure 10 andFigure 11 In the two-step process, for example, after identifying the user and / or user type, a class 103 representing the quality of exercise execution can also be used as a class in the second step 122.

[0228] Another embodiment of device 100 can be similar to device 10 as described with reference to Figures 1 to 8 the device 10.

[0229] According to such an embodiment, at least one class 104 among one or more classes 101, 102, 103, 104 can include at least two class members nA, nB, where the one class 104 can represent the positioning of the movable personal appliance 11 relative to the target surface 12, where the first class member nA can represent a first positioning area 211 of the movable personal appliance 11 relative to the target surface 12, and where the second class member nB can represent a second positioning area 212 of the movable personal appliance 11 relative to the target surface 12, and where at least one of the mapped motion patterns 151, 152, 153, …, 15n can be associated with at least one of the first class member nA and the second class member nB for positioning the movable personal appliance 11 in at least one of the first positioning area 211 and the second positioning area 212 based on the movement of the movable personal appliance 11.

[0230] In other words, a class 104 can represent the target surface 12. The class members nA, nB of the one class 104 can represent different areas 211, 212 of the target surface 12. Accordingly, the positioning of the movable personal appliance 11 relative to the target surface 12 can be performed by device 10 in the same or at least a similar manner as described above with reference to Figures 1 to 8 device 10.

[0231] Of course, in the above Figure 10 and Figure 11 two-step process, for example, after identifying the user and / or user type, a class 104 representing the positioning of the movable personal appliance 11 relative to the target surface 12 can also be used as a class in the second step 122.

[0232] According to an embodiment, at least one class 101 among one or more classes 101, 102, 103, 104 can include at least one class member 101A, where the one class 101 can represent the appliance type (e.g., shaving appliance, dental appliance, broom), and where the at least one class member 101A can represent at least one specific appliance in the group of appliance types, and where at least one of the mapped motion patterns 151, 152, 153, …, 15 n can be associated with the at least one class member 101A for identifying the at least one specific appliance based on the movement of the movable personal appliance 11.

[0233] The neural network 18 of device 100 may include features that are the same as or similar to those of the neural network 18 of device 10 already described with reference to Figures 4 to 7 . Therefore, reference will be briefly made again to Figure 7 .

[0234] According to one embodiment, the neural network 18 may include at least a first layer 71 and a second layer 72, where each layer may include neural units 60, 70, where at a first time point t, at least one inertial sensor data Xt 172 may be input into the neural unit 60 of the first layer 71, and where at a subsequent second time point t+1, second inertial sensor data Xt+1 173 and at least one output ht 46 from the previous first time point t may be input into the neural unit 60 of the first layer 71, and / or where at a subsequent second time point t+1, at least one output ht 46 from the first time point t may be input into the neural unit 71 of the second layer 72.

[0235] All that has been described above regarding any features of the neural network 18 of device 10 as shown in Figures 4 to 7 also applies to the neural network 18 of device 100 as described with reference to Figures 9 to 11 .

[0236] Figure 12 A block diagram showing the method of the present invention for classifying the motion of a movable personal appliance 11 including an inertial sensor 13 is shown.

[0237] In block 1201, the method includes the step of distinguishing between two or more motion patterns 151, 152, 153, …, 15 included in the set of motion patterns 15 of the movable personal appliance 11 n .

[0238] In block 1202, the method includes the step of providing at least one inertial sensor data 171, 172, 173, …, 17 from the inertial sensor 13 n to the motion pattern recognition device 14, where the at least one inertial sensor data 171, 172, 173, …, 17 n represents the motion of the movable personal appliance 11.

[0239] In block 1203, the method includes receiving and processing at least one inertial sensor data 171, 172, 173, …, 17 by means of the neural network 18 n and mapping the at least one inertial sensor data 171, 172, 173, …, 17 n to at least one motion pattern 151, 152, 153, …, 15 included in the set of motion patterns 15 nsteps, where at least one of the mapped motion patterns 151, 152, 153, …, 15 n is associated with at least one class member 101A, 101B, 102A, 102B, …, nA, nB of at least one of the classes 101, 102, 103, 104 such that at least one of the class members 101A, 101B, 102A, 102B, …, nA, nB is selected based on the motion of the movable personal appliance 11.

[0240] According to yet another example of the devices 10, 100, the movable processing device 11 can be a personal grooming appliance, and the target surface 12 can be a body part to be processed by the movable processing device 11.

[0241] According to yet another example of the device 10, 100 according to the present invention, the movable processing device 11 or the movable personal appliance 11 can include a pressure sensor for sensing the pressure applied to the target area by the personal appliance and / or a load sensor for sensing the motor load of the motor that can drive the personal appliance.

[0242] In addition to at least one inertial sensor data 171, 172, 173, …, 17 n alternatively or as an alternative, the corresponding sensor data of the pressure sensor and / or the load sensor can also be fed as input into the neural unit 18.

[0243] According to yet another example of the device 10 according to the present invention, the device 10 can include an output interface for outputting to the user one or more areas 211, 212, 213, …, 21 of the target surface 12 in which the movable processing device 11 is located. n 。

[0244] According to yet another example of the device 100 according to the present invention, the device 100 can include an output interface for outputting information to the user, the information being related to one or more of the classes 101, 102, 103, 104 and / or related to one or more class members 101A, 101B, …, nA, nB of one or more of the classes 101, 102, 103, 104.

[0245] In each of the embodiments described herein, the sensor data 171, 172, 173, …, 17 n can be stored on the movable personal appliance or processing device 11 and can be fed into the devices 10, 100 in the above manner at a later time. This stored sensor data 171, 172, 173, …, 17 nAny post - processing into different regions or classes can be used to show consumers or users on a dashboard how well they are covered or which regions are covered, what they have missed, and what is within and outside the target. This data can be shown at each use or aggregated over time after use (i.e., provide a simple dashboard to consumers or users showing how they brushed their teeth over a week).

[0246] The following features may also be included:

[0247] · Attention mechanism (added to RNN)

[0248] · Pre - filtering operation

[0249] · Avoid head - position dependence (look at linear acceleration)

[0250] · Dynamic time warping for user ID (fingerprint)

[0251] · Local high - frequency sampling and 8 - bit FFT to distinguish the tongue and buccal surfaces (buccal damping based on the signal - this will be done through a simple on - device classifier, followed by the raw signal + device classifier into the RNN)

[0252] · Train not only the position predictor but also "whether brushing is correct"

[0253] · Conduct clustering analysis (let users brush their teeth 1 - 5 times before grouping users) to place users in a defined space with an RNN trained custom - tailored for that type of user

[0254] Figure 13 An exemplary networked appliance system 1000 in accordance with the present disclosure is disclosed. The networked appliance system includes a grooming appliance 1003, which in this example is shown as a razor appliance. However, the appliance can be any grooming appliance, household appliance, or processing device 11 disclosed herein. In the current example, the razor appliance 1003 includes grooming tools such as a removable razor blade cartridge 1006, a razor handle 1002, an internal power source 1118, an optional multi - color LED display 1050, and an optional camera 1082.

[0255] As described above and herein, the razor implement 1003 can include a plurality of internal sensors, such as motion sensors, orientation sensors, cartridge ejection sensors, new cartridge detection sensors, and / or pressure sensors associated with the handle 1002 and / or the razor cartridge 1006. The shaving implement 1003 can also include implement circuitry 1052 that is connected to receive (via a data connection) sensor signals from the plurality of sensors included within the razor implement 1003. In the present embodiment, the networked implement system 1000 further includes a base station 1301, where the base station includes a cradle 1056 for receiving and engaging the handle 1002 of the razor implement 1003. In the present embodiment, the base station 1301 can be powered by electricity via a wire 1058 that can be inserted into a standard power outlet. The cradle 1056 can include electrodes (also not shown) adapted to engage and / or mate with corresponding electrodes (not shown) on the razor implement handle 1002. Through such electrodes, the base station 1301 can provide power to charge a power source (such as a rechargeable battery) 1118 within the razor implement 1003 and / or can provide an electrical connection to transfer data signals from the sensor circuitry 1052 within the razor handle 1002 to base station circuitry 1060 resident within the base station 1301. Also within the scope of the present disclosure is that power can be provided from the base station 1052 to the power source 1118 of the razor by non-connected capacitive coupling as known in the art or any other wireless mechanism known for wirelessly / non-contactlessly transferring power from a first power source to a rechargeable power source. Also within the scope of the present disclosure is that the power source 1118 can be removable, such as a disposable battery and / or a rechargeable battery that is charged by something other than the base station 1301. Additionally, within the scope of the present disclosure, data sent / received between the razor 1003 and the base station 1301 can be via a wireless data connection, such as a Bluetooth connection and the like. Also within the scope of the present disclosure is that some or all of the mechanisms, circuitry, and / or functions of the base station 1301 as described herein can reside within the razor 1003. It should be understood that although the base station 1301 is described in this example as being associated with the razor 1003, similar base stations and base station functions can also be associated with other implements disclosed herein.

[0256] In this embodiment, base station 1301 includes base station circuitry 1060 that includes a processor and corresponding circuitry for receiving sensor signals (and / or information derived from sensor signals) and converting the sensor signals / information into associated analysis / mapping / classification information, as described herein. In this embodiment, base station circuitry 1060 also includes network circuitry for wireless data communication (e.g., such as cellular and / or WiFi connections) with a computer network 1062 such as a cellular network and / or an Internet network. Base station 1301 may also include a visual display 1064, such as an LCD display and / or a similar text or image display device known to those of ordinary skill in the art, where such display device 1064 may be controlled by base station circuitry 1060. Base station 1301 may also include a sound actuator 1066 that is also controlled by base station circuitry 1060, where sound actuator 1066 may include a speaker or similar sound-producing component.

[0257] The networked shaving appliance system 1000 also includes a computerized and networked user interface device 1080. The computerized and networked user interface device 1080 may be in the form of a smart phone, a tablet computer, a personal assistant device, a laptop or desktop computer, a smart display, a smart mirror, a computerized wearable appliance such as a smart watch or smart glasses, etc. The computerized and networked user interface device 1080 may include a display 1066 and user input devices such as a cursor control device 1068 (or a touch screen or voice activation control, or a motion sensor, or an eye movement sensor, etc. readily available in the art), a camera 1070, and associated processing circuitry 1072. The computerized and networked user interface device 1080 may be used to implement various software applications, such as computerized tools, which may be in the form of a personal application 1073 associated with the appliance 11 (see Figure 15 and Figure 16 ), as will be discussed in further detail herein. In this embodiment, application 1073 is a personal shaving application and may include a graphical user interface 1074 that may be displayed on the display screen 1066 and may be controlled and / or receive user input from user input devices such as cursor control device 1068 and / or a touch screen. User device circuitry 1072 may include network circuitry for wireless connection with computer network 1062 for receiving and / or sending data through computer network 1062.

[0258] Similarly, as Figure 13As shown, the computer network 1062 can have various computer servers and / or distributed computing devices (collectively 1076) that can access it, and can additionally include various data storage devices 1077 operatively coupled to it via a data connection. For example, the software application 1073 can include operations implemented on one or more of the computer servers / devices 1076 and / or on the device circuitry 1072. Similarly, the data storage devices associated with the software application 1073 can be within one or more of the data storage devices 1077 and / or on the device circuitry 1072.

[0259] At a very high level, one or more of the appliance circuitry 1052, the base station circuitry 1060, the user device circuitry 1072, and / or the processors associated with the distributed computing environment 1076 include sensor circuitry for receiving sensor signals from the razor appliance 1003 and for analyzing / mapping / classifying the sensor signals, as described herein. Similarly, at a very high level, one or more of the appliance circuitry 1052, the base station circuitry 1060, the user device circuitry 1072, and / or the processors associated with the distributed computing environment 1076 include image processing circuitry for receiving image data from the camera 1082 and / or 1070 and for analyzing / mapping / classifying the image data, as described herein. This analyzed, mapped, and / or classified information will also be transmitted via the computer network 1062 such that a computerized tool in the form of a software application 1073 operating on the networked user interface device 1080 can receive the analyzed, mapped, and / or classified information (or at least portions thereof) associated with the user of the computerized device 1080 from the network 1062. The computerized tool in the form of the software application 1073 can also be configured to receive user profile data information from the user via the graphical user interface 1074 provided by the software application 1073. Additionally, the software application 1073 can utilize the user profile data provided by the user via the software application to process the analyzed, mapped, and / or classified information received from the computer network 1062 to generate user feedback information associated with the user's experience with the appliance 11 (in this example, the razor 1003), as described herein; finally, this user feedback information is transmitted to the user via the graphical user interface 1074 provided by the computerized tool as also described herein and / or via the LED 1050 on the razor 1003 and / or via the visual display 1064 on the base station, and / or via the sound actuator 1066 on the base station.

[0260] As Figure 14As shown, specific examples of the measurement information or shaving event information of the razor 1003 may include (but are not limited to) the razor movement information 1102 of the acceleration in the X, Y, and Z directions derived from the sensor data received from the three-axis accelerometer 1110; the razor orientation information 1104 of the angular information derived from the sensor signals received from the three-axis gyroscope 1130; the razor forward direction information 1106 based on the relationship with the magnetic north derived from the sensor signals received from the three-axis magnetometer 1210; the cartridge pivot movement information 1108 (including also the cartridge presence, cartridge contacts, and / or trimmer contacts) of the relationship between the magnet and the pivot plunger derived from the sensor signals received from the three-axis magnetometer 1160; the in-hand razor information of the air pressure (information corresponding to the user gripping the handle 1002) 1110 derived from the sensor signals received from the capacitive sensor 1420; and the razor attitude information 1112 derived from the sensor signals received from the air pressure sensor 1440.

[0261] Also as Figure 14 shown, the razor attitude information 1114 can be derived from a combination of the razor movement information 1102, the razor orientation information 1104, and the razor forward direction information 1106. The cartridge contact information 1116 can be derived from the pivot movement information 1108. The stroke event information can be derived from a combination of the razor attitude information 1114, the razor contact information 1116, the in-hand razor information 1110, and the razor attitude information 1112.

[0262] As Figure 14 Further shown, the measurement and shaving event information may also include image information provided by the camera and associated analysis, mapping, and / or classification. For example, as will be described in further detail below, the hair growth direction information 1120 can be provided by the image information received through the camera 1082 / 1070 and through the stubble analysis 1122 performed on the image information using an appropriate computer learning or statistical analyzer, mapper, and / or classifier as described herein. Thus, the relative stroke direction information 1124 (which determines whether the stroke direction is related to the hair growth direction on the user's face) can be derived from a combination of the razor attitude information 1114, the stroke event information 1118, and the hair growth direction information 1120 provided by the image analysis. Similarly, the over-stroke information or over-stroke of the with / against-the-grain beard can be determined based on a combination of the sensor readings obtained from multiple identical sensors and the image information for the shaving direction information and / or relative shaving direction information.

[0263] As described herein, additional sensors may include a thermistor for sensing the handle operating temperature and / or the temperature inside the handle; a capacitive sensor for sensing the in-hand razor; a multi-capacitive sensor for sensing the gripping position; a clock for sensing time; an acoustic sensor for sensing shaving performance (such as with / against-the-grain beard), etc.

[0264] Another aspect of the present disclosure is that the shaving event information can be cumulative shaving event information starting from when the system senses or is informed that a new shaving cartridge 1006 is attached to the razor 1003. Determination of a new cartridge can be provided by receiving a sensor signal associated with the cartridge ejection button on the razor appliance 1003. Similarly, new cartridge determination information can be provided by activating a new cartridge sensor when cartridge ejection occurs (such as a mechanical switch being set to activate when the cartridge is ejected), where the new cartridge sensor can then be actuated when a new cartridge is inserted. New cartridge information can also be manually indicated by the user, such as by a software application 1073 or by the user pressing a reset button (etc.) on, for example, the base station 1301. Additionally, the new cartridge information can be detected by the razor appliance 1003 by detecting the unique ID of each shaving cartridge attached to the handle 1002. For example, the unique ID can be a bar code on the cartridge sensed by an associated bar code reader on the handle; it can be an RFID tag on the cartridge sensed by an associated RFID reader on the handle; it can be an ID on the cartridge that communicates with the handle via magnetic, electrical, or capacitive data communication; it can be a physical ID, such as a physical key on the cartridge 1006 sensed by the handle 1002; etc. Basically, any known way for the appliance 1003 or the system 1000 to detect or be informed when a new shaving cartridge 1006 is coupled to the handle 1002 (new cartridge event) will be the collection point for starting the accumulation of shaving event data, where this cumulative shaving event data will thereafter be associated with the life of the new shaving cartridge 1006. This cumulative shaving event information can be used to calculate or estimate, for example, the sharpness of the associated blades contained within the cartridge 1006.

[0265] The systems and methods of the present disclosure may include training one or more convolutional neural networks (“CNNs”) for determining a method of treating a target surface 12. The CNN may be used to identify the target surface 12, treatment tool 11, and practitioner information associated with the treatment method determined by the CNN. The CNN may utilize training images and / or audio data to train the convolutional neural network and may receive one or more training images or audio files for use by the CNN to determine elements defining the surface type, treatment tool, and practitioner. Once the CNN is trained, a camera 22 may capture images (e.g., digital images) of the target surface, tool, and practitioner for analysis by the CNN. The camera 22 may then provide image data 23 for analysis, mapping, and / or classification, as described herein. Analysis of the captured image data 23 may include determining the target surface 12 type, target surface 12 condition, target surface diagnosis, tool 11 type, user information, and treatment method, additional associated treatment product, and treatment scenario information. The CNN and RNN architectures may be used sequentially or in parallel to evaluate data and determine the surface.

[0266] The image capture logic of the camera 22 and software tools in the form of computer applications 1073 may include and / or utilize software components, hardware circuits, firmware, and / or other computing infrastructure, as described herein. As described in more detail below, the image capture logic may facilitate the capture, storage, preprocessing, analysis, transmission, and / or performance of other functions on digital image data 23. The application 1073 may be configured to provide one or more user interfaces 1074 to a user, the one or more user interfaces may include questions, options, and the like.

[0267] Features detected by the analysis / mapping / classification system may include edges, shapes, colors, which may be used to identify age, gender, mood, skin type, hair type, floor type, fabric type, tooth color, skin color, acne / acne vulgaris, redness, skin and hair luster. Products / devices - toothbrush, comb, hairbrush, ProX, razor, grooming device, Swiffer, beauty / cosmetic appliance. Accordingly, the remote processing server and / or computer 1076 includes a memory component 1077 that stores training logic and analysis logic. The training logic may facilitate the creation and / or training of the CNN and may thus facilitate the creation and / or operation of the convolutional neural network. The analysis logic may cause the processing server and / or computer 1076 to receive data from a mobile computing device 1080 (or other computing device) and process the received data via the user interface 1074 for providing treatment product recommendations and the like.

[0268] The training computer or server 1076 may be coupled to the network 1062 to facilitate the training of the CNN. For example, a trainer may provide one or more images to the CNN via the training computer or server 1076. The trainer may also provide information and other instructions to inform the CNN which evaluations are correct and which are incorrect. Based on the input from the trainer, the CNN may adapt automatically, as described in more detail below.

[0269] It should also be understood that although the training computer or server 1076 is described as implementing convolutional neural network processing, this is merely an example. An RNN or a multi-layer perceptron (MLP) may be used as an alternative network architecture and applied to video or other digital data including audio data. Any of these networks may be used because they are capable of analyzing, mapping, and / or classifying video and / or sensor information. The convolutional neural network processing may be performed by any suitable computing device as needed.

[0270] The system of the present invention may include a convolutional neural network (“CNN”) that serves as a surface treatment expert system. For example, the CNN may be stored as logic in the memory component of a computing device. The CNN may be configured to receive training images (or multiple training images) and take the raw image pixels from the training images as input and automatically learn a feature extractor related to determining surface, tool, and practitioner types from the captured digital images. Recent advances in machine learning techniques known as deep learning have led to breakthrough performance in the field of neural networks, as described in U.S. Patent No. 8,582,807. Deep learning-type neural networks utilize multiple layers inspired by the human visual cortex.

[0271] Predefined features and / or automatically learned features may be used to train the CNN. After training the CNN, it may be used to determine surface treatment options from the captured images of the user by the learned features. In some cases, the CNN may learn to identify important features in the images through a process called supervised learning. Supervised learning generally means training the CNN by analyzing examples of images in which the surface treatment options have been predefined. Depending on the desired accuracy, the number of training images may vary from a small number of images to continuous input images to provide continuous training. In any case, after training, the CNN learns key features for accurately predicting the processing methods for various surface types.

[0272] The CNN may include multiple stages. The first stage may include preprocessing, and the second stage may include convolutional neural network training. During preprocessing, one or more features common to most scenes and users ("anchor features") may be detected in the received image. The detection may be performed based on edge detection, shape detection, and / or similar detection mechanisms, as known. Based on the positions of the one or more anchor features, the image may be scaled and rotated to make the image substantially horizontal, and the one or more anchor features are arranged at predetermined positions in the final image. By doing so, the training images may be consistently aligned, thus providing more consistent results. The image may then be cropped to a predetermined area of pixels as input for further processing.

[0273] During preprocessing, data augmentation may also be performed to create additional samples from the training images. For example, the input images may be randomly enlarged and / or shrunk, randomly rotated in the clockwise and / or counterclockwise directions, randomly trimmed, and / or randomly varied with respect to saturation and / or exposure. In some cases, the input images may be subjected to random vertical readout, which randomly reads out columns of pixels (feature maps) of the image. The higher the layer, the larger the area of elements covered by the readout. By reading out entire columns of pixels in the input image, the CNN can learn to rely on multiple features for surface treatment evaluation rather than a specific feature. Random vertical readout can also prevent overtraining of the CNN, thus maintaining the desired accuracy level. Regardless of the specific implementation techniques, data augmentation allows the CNN to become more robust to variations in the input images. In this way, the CNN learns to extract important features under the expected environmental variations caused by the way people take images, the conditions under which the images are taken, and the hardware used to take the images.

[0274] Preprocessing may also include normalization. For example, global contrast normalization may be utilized to standardize the training images (and / or user images). Similarly, the images may be masked with a fixed (or predetermined) size oval mask to minimize the influence of other features. This also forces the CNN to learn and not rely solely on information in the more fixed spatial positions of the image.

[0275] During training, the embodiments described herein may utilize mini-batch stochastic gradient descent (SGD) with Nesterov momentum (and / or other algorithms) to train the CNN. An example of using stochastic gradient descent is disclosed in US 8,582,807. The objective function may include mean squared error. In some embodiments, approximately 10% of the training objects may be reserved. The training error and validation error regarding the reserved set may be monitored for the training progress.

[0276] Once the CNN is trained, one or more of the CNN parameters can be fixed. As described in more detail below, the captured image can be propagated forward through the CNN to obtain a determined surface treatment solution, which can optionally be displayed to the user on, for example, a mobile computing device.

[0277] The CNN can include an input image, one or more convolutional layers C1, C2, one or more subsampling layers S1 and S2, a fully integrated layer, and an output. To begin analyzing or training the CNN, an image (e.g., a user image) is input into the CNN. The CNN can sample one or more portions of the image to create one or more feature maps in the first convolutional layer C1. For example, the CNN can sample six portions of the image to create six feature maps in the first convolutional layer C1. Next, the CNN can subsample one or more portions of the feature maps in the first convolutional layer C1 to create the first subsampling layer S1. In some cases, the subsampled portion of the feature map can be a half-region of the feature map. For example, if the feature map includes a sampled region of 28x28 pixels from the image, the subsampled region can be 14x14 pixels. The CNN can implement one or more additional levels of sampling and subsampling to provide a second convolutional layer C2 and a second subsampling layer S2. It should be understood that the CNN can include any number of convolutional layers and subsampling layers as needed. After completing the final subsampling layer, the CNN generates a fully connected layer F1, where each neuron is connected to every other neuron. From the fully connected layer F1, the CNN can generate an output, such as a predicted age or a heat map.

[0278] In some cases, at least some of the images and other data described herein can be stored as historical data for later use. For example, user progress tracking can be determined based on this historical data. According to an embodiment, other analyses can also be performed on this historical data.

[0279] In one embodiment, a CNN-based model is used to detect and track the grooming tool 11 in consumer videos. The model utilizes multiple CNNs and other neural network components (such as fully connected networks or RNNs) to accomplish this task. Image data 23 in the form of consumer videos is fed into the model as a series of image frames. Each image frame is first processed by a CNN to extract a set of feature maps (high-level features of the image). A second CNN, the region proposal network, is used to propose a series of possible regions in the feature maps that may contain the grooming tool. Then the feature maps within the proposed regions are extracted to further suggest to a fully connected network to determine whether the proposed region contains the grooming tool, refine the location of the proposed region, and map the coordinates of the proposed region to the original image. The end result is that for each image frame, the model is able to determine whether the grooming tool is present, and if so, to determine the location of the grooming tool within the image. At the same time, the consumer's face can also be located using various face recognition algorithms including CNNs or any other face detector algorithms. The face can also be taken as part of the objects detected in the region proposal network. Recurrent neural networks can also be overlapped to capture the temporal information of the video. By combining the location information of the grooming tool and the consumer's face, the tool can respond accordingly to provide an optimal grooming experience. In one embodiment, the operating parameters of the tool can be changed according to the way the user shaves or, in other words, grooms themselves or others. In one embodiment, the system can provide the user with information related to the current and historical use of the grooming tool and the target surface.

[0280] One or more images, as well as one or more outputs from the neural network, can be passed to a database and aggregated with similar data from other users of the method. The aggregated data can be evaluated and classified into clusters using known clustering methods. Then, based on the user, surface, and tool data, the instant user, surface, and tool can be associated with one or more pre-defined aggregated populations. Then, the association with a particular aggregated population can result in providing cluster-specific information to the user as part of the method. For example, users can be classified according to age, race, and gender, and the results of comparing the user's data with the data of aggregated populations having the same gender, age, and race can provide usage insights to practitioners when they are provided as part of the method.

[0281] In one embodiment, a method for treating a target surface 12 includes the steps of automatically evaluating digital image data 23 of the target surface 12. The digital image data 23, which may be a set of images, can be provided to a machine learning classifier for evaluation. The set of images may also include additional data associated with the content or context of the images. Data including audio, temperature, humidity, or other environmental data captured simultaneously with the images can also be provided to the classifier. The classifier may have been previously trained to identify the nature of the target surface 12 by presenting it with training data that includes images of representative target surfaces, either alone or together with other data as described above. The nature of the target surface 12 may include the following attributes: the classification of the surface, such as skin, facial skin, teeth, fabric, leather, plastic, wood, glass, ceramic, stone, or other hard or soft surfaces, and the current condition of the surface 12. The presence of facial hair, dental plaque, dirt, stains, and combinations thereof on the target surface can be determined by analyzing the surface via an image of the surface. Surface roughness or surface finish can also be determined.

[0282] The analysis of the at least one image may also identify or determine at least one available treatment tool. In one embodiment, the determination may include determining the presence of a handheld treatment tool. In one embodiment, the determination can be made by matching the content in the current image with images of suitable tools present in a training dataset. In one embodiment, the determination can be made by inference, where specific tools are associated with certain surfaces according to previously defined rules available for analysis. In this embodiment, a toothbrush may be associated with teeth, a razor and tools for skin-body hair, a stiff brush for hard surfaces, etc.

[0283] Further analysis of the at least one image and the additional data may determine at least one surface treatment associated with the identified target surface, either alone or in combination with the identified treatment tool. This determination can be made using the determination of the nature of the surface, treatment tool, practitioner, or combinations thereof. For example, a grooming protocol suitable for a grooming tool and the combination of skin and hair can be determined. Then, the analysis can determine the use of the identified tool in completing the identified surface treatment.

[0284] After determination, information similar to that determined to be used can be provided to the user via a display system. Such information can include specific instructions regarding: the handling and use of tools during treatment, the possible outcomes of treatment, the progress of treatment evaluated by the method during a series of procedures, the condition of the tools related to the execution of the procedure, etc. The information can be provided via one or more digital displays, auditory cues from the tool or from different speakers, or visual cues such as indicator lights or other lighting changes in the treatment environment. In one embodiment, the step of providing information includes providing a cue similar to the spatial interaction between the determined tool and the determined surface. In one embodiment, the step of providing information includes providing information similar to the temporal interaction between the determined tool and the determined surface. In one embodiment, the step of providing information includes providing information by changing the characteristics of the determined tool.

[0285] In one embodiment, the information to be presented can be stored in a database and called in response to the output of the CNN. The presented information can be real-time information collected during the process, information from the database, and a hybrid combination of the two. For example, display templates of the upper and lower teeth can be presented to the user overlapped with real-time data, thus showing which parts of the user's teeth have and have not been brushed during the current session. Data showing the user's toothbrushing history trend from the database can be presented.

[0286] In one embodiment, the step of providing information can include providing information associated with the determined use, product, or tool and information associated with the user's social network. Social network information accessed using the account information provided by the user enables the presentation of information about similar procedures performed by members of the user's social network, including the similarities and differences between the procedures performed by the user and those performed by other members of the user's social network. Social network information can also be used as an indication of which social influencers are most likely to have an impact on the user. This information can be used to select guidance content of celebrities or social influencers to be presented to the user, as well as product reviews and testimonial information from the identified influencers or the closest analogs of the identified influencers.

[0287] In one embodiment, the method further includes the step of providing information about a processing tool related to the determined surface or surface treatment, where the tool is not detected in the data analysis. For example, the analysis of the data can indicate the use of a grooming tool but not the use of a complementary product that can improve or enhance the processing activity. In this example, information about the missing product can be provided to the user.

[0288] In one embodiment, the step of providing information may include gamification aspects. The information to be provided may be presented to the user in the form of a game. The game may include aspects such as point scoring, competing with others, and game rules. For example, using an oral care tool such as a toothbrush may involve presenting information related to the time spent brushing, and the areas of the oral cavity I that have been treated so far (including tooth surfaces, tongue, and gums) and the remaining portions to be treated, which may be presented in the form of a game, where the user must move the tool in a way that clears objects from the display as a timer counts up or down. In this embodiment, the graphical elements presented for removal may coincide with the surfaces to be cleaned and may only be removed from the display after the user has spent sufficient time in treating / cleaning those surfaces.

[0289] In one embodiment, the method may further include the step of determining one or more characteristics of the treating practitioner based on an evaluation of at least one image. Characteristics including practitioner gender, dominant hand, skin condition, beard condition, etc. may be determined by analyzing the data and the context of the data along with other information about the user and the usage environment. The determined characteristics of the practitioner may be used as an input in determining what information to provide when evaluating treatment activity. Information specifically applicable to the gender, dominant hand, skin condition, beard condition, and combinations thereof of the user may be provided.

[0290] In one embodiment, information about the user may be combined with information about the product (including brand, package quantity, and quantity used per treatment) to calculate the remaining quantity of the product, so as to provide the user with an indication of when the current product may run out and an indication of when to replace or reorder the product using the typical way the user obtains the product.

[0291] In one embodiment, the method further includes the step of determining one or more environmental characteristics based on the evaluation of the one or more images together with at least one additional data source. For example, the method may determine the location of the practitioner and the surface, the time of day, the lighting available at the location, and other characteristics of the local or external environment. The determined environmental characteristics may be used as an input in determining what information to provide to the user (as part of the method).

[0292] In one embodiment, the method may further include the steps of: tracking an initially determined treatment of a determined target surface; providing information similar to the determined treatment, tracking and evaluating subsequent treatments of the target surface; and changing the information provided subsequently based on a machine learning evaluation of the tracked initial treatment and subsequent determined treatments and previously provided information. For example, a user may use the method to evaluate their shaving experience. Information may be provided to the user to enhance their shaving experience. Subsequent evaluations may indicate that a portion of the previously provided information has been successfully followed or incorporated into the shaving activity, while other portions have not been successfully added. After this determination, the information provided may be customized to include only information related to the portions that have not been successfully added to the treatment activity (shaving in this example). The types of information include shaving or treatment trends, ongoing treatment results (how well the user has shaved, what opportunities remain to improve their experience, and diagnostic information related to the user's grooming tool and their shaving activity).

[0293] In one embodiment, the method may further include the steps of: tracking an initially determined treatment of a determined target surface; tracking at least one subsequent determined treatment of the same determined treatment surface; using machine learning in evaluating a combination of the tracked determined treatments of the determined target surface; and providing information similar to the determined treatment of the determined target surface based on an evaluation of the combination of the tracked determined treatments. The information provided may include an indication of improvements to the grooming activity and superior opportunities for further improvement based on the progress of the grooming results.

[0294] In this embodiment, the machine learning step in evaluating a combination of the tracked determined treatments of the determined target surface may include evaluating a practitioner in the combination using the environmental context of the treatment along with any information provided by the user.

[0295] In this embodiment, the machine learning step in evaluating a combination of the tracked determined treatments of the determined target surface may include evaluating a tool in the combination, where the tool may be evaluated based on the manufacturer and tool model and the operating conditions of the tool considered in terms of the tool's performance in completing the surface treatment. For example, as the operating conditions of the tool degrade, the work required to complete the task will also change.

[0296] In this embodiment, the machine learning step in evaluating a combination of the tracked determined treatments of the determined target surface may include evaluating a surface in the combination. The properties of the surface may be evaluated to provide input for determining the information to be provided. An evaluation of the user's face may indicate light or heavy hair growth, resulting in different information being provided based on the facial hair present during the treatment.

[0297] In one embodiment, the method further includes the step of changing a performance characteristic of the tool. In this embodiment, the drive frequency of the tool can be changed to change the performance or to provide an auditory cue to the practitioner regarding the treatment of the surface with the tool.

[0298] A system for implementing the method can include a network, which can be embodied as a wide area network (such as a mobile phone network, a public switched telephone network, a satellite network, the Internet, etc.), a local area network (such as Wi-Fi, Wi-Max, ZigBee TM Bluetooth TM etc.) and / or other forms of networking capabilities. Coupled to the network are a computing device, a remote computing device, a kiosk computing device, and a training computing device.

[0299] The computing device can be a mobile phone, a tablet computer, a laptop computer, a personal digital assistant, a dashboard mirror or a smart mirror, and / or other computing devices configured to capture, store, and / or transmit images such as digital photos and videos. Thus, the mobile computing device can include an image capture device, such as a digital camera, including a depth sensing camera and / or can be configured to receive images from other devices. The mobile computing device can include a memory component that stores image capture logic and interface logic. The memory component can include random access memory (such as SRAM, DRAM, etc.), read-only memory (ROM), registers, and / or other forms of computing storage hardware.

[0300] Recent advances in machine learning techniques known as deep learning have led to breakthrough performance in the field of neural networks. Examples of deep learning neural networks include convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0301] CNNs utilize multiple layers inspired by the human visual cortex. A CNN consists of an input layer and an output layer, as well as multiple hidden layers. The hidden layers of a CNN typically consist of convolutional layers, pooling layers, fully connected layers, and normalization layers. They have a wide range of applications in image and video applications, such as image classification, object detection, localization, segmentation, etc.

[0302] A CNN can be trained using predefined features and / or automatically learned features in a process called supervised learning. Supervised learning generally means training a CNN by analyzing examples of images in which image classification / localization / detection / segmentation, etc. have been predefined. Depending on the desired accuracy, the number of training images can vary from a small number of images to continuously input images to provide continuous training. In any case, after training, the CNN learns the key features for performing image-related tasks.

[0303] After the CNN is trained, it can be used to generate image classification, localization, detection, and / or segmentation related to the operation of a personal grooming appliance. In some cases, the CNN can learn to discriminate facial / oral feature structures, identify and localize grooming appliance devices, discriminate treatment areas and treatment options, and evaluate treatment results.

[0304] A recurrent neural network (RNN) is a class of deep learning neural networks where the connections between nodes form a directed graph along a sequence. This allows it to exhibit temporal dynamic behavior within a time series. RNNs use their internal state (memory) to process sequences of input data. This makes them suitable for tasks such as machine translation, speech recognition, video analysis, sound detection, and motion tracking, etc.

[0305] Similar to the CNN, supervised learning can be used to train the RNN. Supervised learning of the RNN generally means training the RNN by analyzing examples of sequence data, such as text, speech, sound, video, sensor data streams, where the meanings of the translated words, sounds, actions of the video, and corresponding physical measurements have been predefined. Depending on the desired accuracy, the number of training samples can vary from several short data stream segments to continuous input of the data stream to provide continuous training. In any case, after training, the RNN learns the key features for performing tasks involving sequence data.

[0306] After the RNN is trained, it can be used to analyze data streams from cameras or physical sensors and provide additional information related to the operation of a personal grooming appliance. In some cases, the RNN can learn to locate the grooming application device, discriminate the movement patterns of the grooming application device, and / or evaluate the use of the application device.

[0307] Multiple types of deep learning neural networks are typically used simultaneously to enhance each other in order to achieve higher performance. In some cases, the CNN and RNN can be independently used to analyze the same or different data streams, and the outputs from different neural networks are jointly considered to drive user feedback. In other cases, a hybrid neural network architecture can be adopted - a hybrid neural network consisting of both CNN and RNN branches or layers. In one type of hybrid network, the intermediate results of the CNN and RNN are jointly fed into an additional layer of the neural network to produce the final output. In other types of hybrid networks, the output of one network (e.g., the CNN) is fed into an additional layer of the network (e.g., the RNN) for further processing before obtaining the final result.

[0308] In one embodiment, a CNN and an RNN are used to analyze images from an external camera and motion and pressure sensors from an electric shaver. The CNN first identifies processing options, such as recommended shaving techniques, based on the facial area and the natural condition of the facial hair. The RNN then provides real-time tracking of the toothbrush movement to ensure that the consumer follows the recommendation, and the CNN is used to finally provide a post-shave assessment.

[0309] In another embodiment, a hybrid CNN / RNN model is used to provide highly accurate tooth positioning during toothbrushing. The toothbrush is equipped with an intraoral camera and a motion sensor and feeds both a video stream and a motion sensor stream to the hybrid network. The CNN and RNN components of the hybrid network analyze the video stream and the motion sensor stream, respectively, to provide intermediate results of the positioning of the brush head within the mouth. The intermediate positioning results are further processed by an additional layer of the neural network to produce an enhanced positioning of the brush head as part of the feedback to the user for better brushing results.

[0310] Thus, it should be understood that the internal sensor data 171, 172, 173, …, 17 n is not necessarily limited to motion sensor data, and the corresponding classifications 151, 152, 153, …, 15 n is not necessarily limited to motion patterns. For example, the internal sensor data 171, 172, 173, …, 17 n can include data from one or more pressure sensors, load sensors, temperature sensors, audio sensors / receivers, battery usage sensors, humidity sensors, biosensors, etc. (such internal sensors of the appliance can also be referred to as “physical sensors”). Similarly, the corresponding classifications 151, 152, 153, …, 15 n can be the result of classifying data from one or more of these sensors or a combination of two or more of these sensors. Further, as will be discussed in more detail below, within the scope of the present disclosure, the classifications 151, 152, 153, …, 15 n can include hybrid or enhanced classifications based on a combination of the internal sensor data 171, 172, 173, …, 17 n and the image data 22.

[0311] Figure 15 A schematic block diagram representation of an exemplary system that utilizes such hybrid or enhanced classifications in accordance with an embodiment of the present disclosure is provided. As Figure 15As shown, a camera 1082 / 1070 is provided, which generates image data 1502 that is fed into a classifier 1504. Similarly, one or more sensors 1506 provide sensor data 1508 that is fed into the same classifier 1504. The classifier 1504 has access to a knowledge base 1510 that has been trained (and / or is being trained) based on various machine learning techniques described herein. The trained knowledge base 1510 is accessed by the classifier 1504 to classify a combination of the image data 1502 and the sensor data 1508, thereby providing a hybrid classification 1512 that can be accessed by a software application 1073. The classifier 1504 can be any type of machine learning classifier and can utilize neural networks described herein (such as CNNs and / or RNNs), and the knowledge base 1510 includes a set of trained (or training) potential classifications 151, 152, 153, …, 15 n and / or class members 101A, 101B, 102A, 102B, …, nA, nB for matching / classifying based on the input to the classifier.

[0312] For example, the classifier 1504 can be used to analyze images from a camera 1070 associated with a smart phone or a smart mirror, and the sensor data 1508 can include pressure sensor data from an electric razor. This hybrid classification 1512 generated by the classifier 1504 can provide a combination of location and pressure information for use by the application 1073 for real-time tracking of razor movement and pressure to ensure, for example, that the consumer follows a recommended shaving / processing procedure. The classifier 1504 receives the image data 1502 and also receives a combination of the image data 1502 and the sensor data 1508, and can also provide a post-shave assessment based on a combination of tracking classification and post-shave image classification. As another example, the classifier 1504 can be used to provide highly accurate tooth positioning during toothbrushing. The toothbrush can be equipped with both an intraoral camera 1082 and a motion sensor 1506, and the classifier 1504 can receive a combination of the camera data 1502 and the motion sensor data 1508 to provide a hybrid classification 1512, where the hybrid classification can use a combination of the motion sensor and intraoral image information to provide an accurate positioning of the toothbrush head within the mouth. The software application 1073 can utilize this hybrid classification to generate feedback to the user for better brushing results.

[0313] This feedback can be provided to, for example, display devices 1064 / 1066, to a sound actuator device 1067, and / or to one or more LEDs 1050. The software application 1073 can also utilize the hybrid classification 1512 to adjust or modify the operation of the appliance. For example, the software application 1073 can adjust the operation of a motor 1514 present in a grooming appliance. As described herein, modifying the operation of the motor 1514 can be used, for example, to change the speed of an electric toothbrush operation, to change the speed of an electric shaving appliance operation, to change the angle of attack on a razor blade cartridge, to modify the operating speed or frequency of a motor that controls the rotation or vibration of a brush or the like of a household appliance, and the like. Similarly, the software application can utilize the hybrid classification information 1512 to change various settings 1516 for operating the grooming appliance and / or for operating the software application 1073. For example, depending on the hybrid classification information 1512, device warning or notification settings can be changed (e.g., an overpressure warning setting can be set at different pressures depending on the position of the shaving appliance relative to the user's face or body part).

[0314] Similarly as Figure 15 shown, the software application can utilize the hybrid classification information 1512 as part of a training process to further train the knowledge base 1510. For example, based on how the user interacts with the software application 1073, the training process 1520 can use this user interaction information 1518 to further train the knowledge base.

[0315] Figure 16 A schematic block diagram representation of an exemplary system that utilizes a hybrid or enhanced classification generated in different ways in accordance with embodiments of the present disclosure is provided. Now referring to Figure 16 , a schematic block diagram representation of an alternative system is provided. In this exemplary system, sensor data 1508 from one or more sensors 1506 is fed into a sensor classifier 1602, while image data 1502 received from cameras 1082 / 1070 is fed into a separate image classifier 1604. The sensor classifier classifies the sensor data based on access to a trained knowledge base 1606, while the image classifier classifies the image data 1502 by accessing a training knowledge base 1608. The classifiers 1602 / 1604 can be any type of machine learning classifier and can utilize neural networks (such as CNNs and / or RNNs) described herein, and the knowledge bases 1606 / 1608 include trained (or training) sets of potential classifications (such as potential classifications 151, 152, 153, …, 15 n and / or class members 101A, 101B, 102A, 102B, …, nA, nB) to perform matching / classification based on the inputs to the classifiers 1602 / 1604.

[0316] The sensor classifier 1602 generates one or more sensor classifications 1610 that are fed into the mediator 1612, while the image classifier 1604 generates one or more image classifications 1614 that are also sent to the mediator 1612. The mediator 1612 receives the sensor classifications 1610 and the image classifications 1614 (and in some embodiments, receives associated confidence values), and generates a hybrid classification 1616 based on the combination of the sensor classifications 1610 and the image classifications 1614. The mediator 1612 can classify using any neural network described herein (such as CNNs and / or RNNs), or can use other forms of classification, such as using forms of statistical classification, such as multinomial logistic regression, methods (or alternative classification methods) known to one of ordinary skill in the art. Then the software application 1073 can access the hybrid classification 1616 information for operations as described herein. Similarly, the knowledge base 1606 can be further trained by the training module 1618, and the knowledge base 1608 can be further trained by the training module 1620. These training modules 1618 / 1620 can further train the respective knowledge bases 1606 / 1608 based on user interaction information received from the software application 1073.

[0317] The mediator 1612 can be a separate module or can be incorporated into the software application 1073. Additionally, in all embodiments, there may be no mediator. For example, the sensor classifications 1610 and the image classifications 1614 can be provided to the software application 1073 separately, where the software application 1073 may not necessarily generate a hybrid classification 1616 based on the combination of the sensor classification and the image classification. For example, as described in the various use cases discussed herein, the software application 1073 can initially use the image classification to identify the surface condition of the user's body part (such as detecting the presence of plaque on the user's teeth or detecting whiskers on the user's face or leg), and then develop a treatment schedule based on the image classification, where the treatment schedule can be provided to the user via the software application 1073. After the development of the treatment procedure (such as advice on how to brush teeth, how to apply cosmetics, or how to shave), the software application can use the sensor classification information 1610 to follow the progress of the user's treatment of the surface condition (such as brushing teeth, applying cosmetics, shaving the face or another body part). Then, the software application 1073 can be used to convey progress information to the user based on the received sensor classification.

[0318] Also within the scope of the present disclosure is that the treatment solution can be based solely on the image classification 1614, while the progress information can be based on the hybrid classification 1616. For example, the image classification 1614 can be used to determine the surface condition, and based on this surface condition, the software application can establish a treatment solution. Thereafter, the software application can use the hybrid classification 1616 (which utilizes a combination of the subsequent sensor classification 1610 and the subsequent image classification 1614) to follow the treatment progress of the surface condition. Thereafter, based on how the user is progressing relative to the initially established treatment solution, the software application 1073 can communicate to the user how the treatment solution is being implemented, can modify the treatment solution, or can correct the initial image classification (indicating that the initial image classification may have detected an incorrect surface condition) and develop a new treatment solution based on the modified indication.

[0319] Figure 15 and Figure 16 The exemplary systems shown can be used in many use cases, including but not limited to grooming appliance and home appliance use cases. Only a number of potential use cases will be discussed herein, but it should be understood that more scenarios can be envisioned and these scenarios are within the scope of the present disclosure.

[0320] In a first use case, the image data 1502 can be used to indicate the lubricating product being used (i.e., shaving gel) (such as by brand or type); and can also indicate the facial hair area being shaved, the direction of shaving, and / or the delay time between the application of the shaving lubricant and the shaving action. In the same example, the sensor data 1508 can be MEMS (microelectromechanical systems) motion information, speed information, pressure information of the razor on the face, and / or position information. In this example, the hybrid classification 1512 / 1616 can be used by the software application 1073 for various benefits. For example, when the razor is worn, the efficacy decreases and consumers typically accelerate the strokes and apply additional pressure to compensate. The software application 1073 can detect subtle changes in the user's routine (shaving metrics) and changes in the shaving components (such as shaving gel type) to recommend changes to the user's routine and / or components to ensure a successful shaving experience. Alternatively, the software application can use the hybrid classification 1512 / 1616 to detect the need for different angles of attack on the shaving cartridge or different resistance levels in the pivoting of the shaving cartridge, and accordingly modify the operation of the component 1514 and / or modify the settings 1516 of the appliance.

[0321] In another exemplary use case, the image data 1502 may be an image of a user's face, and the sensor data 1508 may include MEMS sensor information regarding the position of the grooming appliance relative to the user's face. The software application 1073 may analyze the image classification 1614 or the hybrid classification 1512 / 1616 to determine the user's emotional response when using a grooming appliance, such as a shaving appliance, a dental appliance, or a cosmetic application device; and similarly, use the position information present in the sensor classification 1610 or the hybrid classification 1512 / 1616 to determine the position at which the grooming appliance was located (and / or how the grooming appliance was used) when the emotional response was experienced. The application 1073 may then use this information combination to provide feedback to the user or modify the operation of the component 1514 and / or modify the settings 1516 of the appliance. For example, if the user shows a negative emotion while shaving his neck and the hybrid classification 1512 / 1616 indicates a certain pressure and shaving direction were being used at that time, the application 1073 may suggest a different way for the user to shave his neck the next time.

[0322] In another exemplary use case, the image data 1502 may be used to provide pre-shave and / or post-shave information, and the sensor data 1508 may include position information, movement speed information, and / or pressure information of the shaving device. Using this combination of information as described above, the application 1073 may analyze the image before shaving to determine the direction of the whiskers and may analyze the best processing method to obtain a close shave. Thereafter, data from the sensor information 1508 and / or the hybrid classification 1512 / 1616 may indicate how well the shave was performed compared to the recommendation. The post-shave image information may then be analyzed, and the application may refine its guidance for the user to perform a better shave the next time the shaving appliance is used. For example, if the post-shave image shows irritation and the guidance was followed, the application may provide post-shave options (such as a shaving ball) or recommend a pre-shave routine (such as using a warm towel, a new blade, and / or a different shaving foam) to help minimize the problem. The application 1073 may also label the consumer for future product development follow-up (e.g., if the consumer tends to get ingrown hairs after shaving, the application 1073 may provide marketing information to the user that can be used to avoid future ingrown hairs).

[0323] In the next exemplary use case, a combination of image classification and sensor classification can be used in a marketing environment. For example, the image data 1502 can provide information about the products used by the user during brushing (electric toothbrush vs. manual, toothbrush, toothpaste type, mouthwash, dental floss, etc.), while the sensor data 1508 can provide location, speed, and pressure information during the brushing activity. An application 1073 that utilizes hybrid classification 1512 / 1616 and context data (such as the user's identity, the user's age, the user's ethnicity, user habits, products present, etc.) can cross-sell consumers different products. For example, a consumer who brushes with a manual toothbrush and uses sensitive toothpaste can receive messages, coupons, and samples for an electric toothbrush with soft bristles and / or a different toothpaste for minimizing future sensitivity issues.

[0324] In another use case, the image data 1502 can be used by the software application 1073 to classify skin age analysis before product use and can also be used for location information during use. The sensor data 1508 can be various performance conditions of the beauty product application device being used, such as use speed, pressure, etc. Then, based on this information combination, the application 1073 can recommend beauty and application techniques to the user through the application 1073 to maximize performance. The application 1073 can utilize data from the sensors of the application device to understand how the product (pressure and movement) is applied, tapped, rubbed, or dabbed. The application 1073 can track and coordinate techniques to encourage the use of the application to comply with the guidance.

[0325] In another use case example, a camera 1082 located at the front end of a dental appliance can be used to identify the presence and location of dental plaque on the user's teeth. Based on this identification of the location and presence of dental plaque on the user's teeth, the software application 1073 can then generate a treatment plan to be transmitted to the user. The treatment plan can provide advice on how to brush the user's teeth using a manual or electric toothbrush and can also guide the user in real time to various tooth locations during brushing (such as via a display 1066 on a smart mirror device). For example, when the user brushes their teeth in front of the smart mirror device, an animated display can be present in a corner of the device that shows the user the locations that have been brushed and the locations that have not been brushed as well as the locations that still indicate the presence of dental plaque, so that the user will know which parts of their mouth have been brushed before completing the brushing activity. Thus, during brushing, the hybrid classification 1616 / 1512 can be a hybrid combination of location data from the sensor data 1508 and the image data 1502 to provide the application 1073 with progress information on how the user's brushing activity is going.

[0326] In a similar example, an external camera 1070 located on a computerized smart phone or smart mirror 1080 can be used to analyze a target, such as a room, floor, window, etc., and an application 1073 can be used to determine the state of the target. Using this information, the software application 1073 can set a target or a processing scenario for using a grooming appliance or a household appliance to process the target. Thereafter, once implemented, the software application 1073 can utilize sensor information 1602 (which may include motion sensor data) or hybrid classification 1512 / 1616 to monitor the use of the appliance relative to the target to determine whether the goal of processing the target state has been achieved.

[0327] In another example, an image classifier 1604 can be used to determine the identification of a target and the appliance being used relative to that target (the user's face and a shaving appliance). Thereafter, sensor classification 1610 or hybrid classification 1512 / 1616 can be used to determine how the grooming appliance engages with the target. The software application 1073 can utilize the hybrid classification 1512 / 1616 to obtain a refined determination of how the tool and the target interact, and based on this hybrid classification information, provide user feedback in real time or after use.

[0328] As some additional examples, when operating as a toothbrush, the application can utilize the hybrid classification 1512 / 1616 to determine when the toothbrush is outside of the user's mouth, and based on that determination, the device can be disabled or turned off, or at least the motor 1514 that operates the electric toothbrush can be disabled or turned off. As another example, based on this hybrid classification information 1512 / 1616, the software application 1073 can change the color of the multi-color LED 1050 (such as the smart ring) based on the software application's utilization of the hybrid classification 1512 / 1616 to determine the user's identity or other characteristics. In another example, the hybrid classification 1512 / 1616 can be used to detect the type of the brush head of a grooming appliance, and then used to determine whether to change the brush speed setting 1516 (detecting soft bristles can result in setting the default speed to "gentle"). As another example, the hybrid classification 1512 / 1616 can be used to detect the brushing position relative to the user, and then the software application 1073 can automatically adjust the brushing speed by modifying the operation of the motor 1514 or by modifying the speed setting 1516 (for example, when the hybrid classification indicates that the brush is positioned on the user's tongue, modifying the operation of the device based on the tongue cleaning setting). As another example, the hybrid classification 1512 / 1616 can be used to determine the position of a shaving appliance relative to the user's skin, and then adjust the pressure warning setting according to the area of the grooming device relative to the user (for example, if the grooming device is in a less likely to be sensitive position on the skin, the setting 1516 can be modified such that if the hybrid classification indicates that the shaving device is in a highly sensitive area, the pressure warning generated by the speaker 1066 will only be activated at a higher level of pressure compared to a lower level). As another example, the hybrid classification 1512 / 1616 can be used to determine how long the grooming appliance has been used since a self-grooming tool such as a razor blade cartridge or a toothbrush head has been changed. Thus, if the appliance has been used longer than the recommended replacement schedule, the application 1073 can use this information to inform the user that it is time to replace the brush head on the toothbrush or the razor blade cartridge on the razor.

[0329] As described above, the software application 1073 can use the hybrid classification 1512 / 1616 to adjust the operation of the grooming appliance or the household appliance in many different ways. For example, the software application 1073 can turn on or off the grooming appliance or tool included in the appliance; can adjust the speed of the tool of the grooming / household appliance; can adjust the allowed pressure setting before warning the grooming / household appliance; can activate lights, LEDs, colors, etc. based on the operation of the grooming appliance / household appliance; can adjust the operation of the appliance for maintenance-related issues such as compensating for the lack of maintenance (old shaving foils, old brushes, etc.); can provide feedback to the user to recommend replacing the tool, such as recommending replacing a used brush or a razor blade cartridge, etc.; can adjust the operation setting based on the facial position, such as the approach angle of the shaving device; and can adjust the stiffness or maneuverability of the appliance tool (such as the pivot stiffness of the razor blade cartridge).

[0330] As described above, the application 1073 can provide various forms of feedback to the user of the appliance. Such feedback can be visual (such as provided through a networked user interface device 1080 such as a smart phone, tablet, personal assistant, or smart mirror device), audio (voice, sound, etc.) from the appliance itself or from some other device such as a personal assistant device, tactile feedback from the appliance or from some other source, and so on. Of course, the feedback can be any combination of visual, auditory, and tactile. For example, the feedback can be in the form of an animated video presented on the device 1080 before, during, and / or after using the appliance.

[0331] The training module 1520 / 1618 / 1620 can train the classifier based on the interaction with the user of the application 1073, as described above. But the training can also be based on an individual's use of the appliance (e.g., left-handed use versus right-handed use). The training can also be population-based training, where multiple users train the classifier based on general habits and usage.

[0332] The application 1073 can also train the classifier based on how the user follows (or does not follow) the treatment recommendations or other feedback.

[0333] The embodiments disclosed herein can use enhanced or hybrid classification 1512 / 1616 to determine the relative orientation / position of the target surface and the nature of the target surface of the subject. One embodiment provides an appliance including an appliance equipped with sensors for collecting and transmitting sensor data 1508 and for receiving and processing the sensor data to determine the relative position of the target surface and the nature of the target surface.

[0334] The assessment of the target surface includes collecting and processing at least one digital image 1502 of the target surface and its nature, which consists of information on oral conditions, problems, and diseases, including but not limited to dental plaque, stain, tartar, discoloration, early and late stages of dental caries, white spot lesions, fluorosis, demineralization, gingivitis, bleeding, gingival recession, periodontitis, fistulas, gingival abrasion, aphthae, other lesions of the mucosa, and the structure and cleanliness of the tongue. The assessment of the target surface also includes determining sound teeth, gums, mucosa, and tongue, and determining missing teeth, tooth alignment, and artificial materials such as dental implants, dentures, crowns, inlays, fillings, brackets, and other tooth position correction appliances. The assessment of the target surface can also be part of determining an oral health index, which is generated via an intraoral camera and an intelligent analysis system based on but not limited to machine learning, deep learning, and artificial intelligence.

[0335] Evaluations (data) from location, target surface, and its properties over time help drive the robustness of the analysis of endpoints such as plaque, gingivitis, and other endpoints listed above. To collect data, a sensor-equipped device can use sensors such as, but not limited to, optical sensors, cameras, biosensors, and neurometric units. To collect data, a sensor-equipped device can use an additional light source to allow detection of endpoints (e.g., but not limited to plaque) and optimize the environmental conditions of the sensor. A sensor-equipped device can use preprocessing or filtering methods to alter the sensor data before transmission. To process the acquired location, target surface, and properties of the target surface (data), software within the system can use mathematical methods such as, but not limited to, statistical analysis, machine learning, deep learning, artificial intelligence, etc., such as those described herein.

[0336] In one embodiment, the device can include software for operating the device and for processing and displaying the location, target surface, and properties of the target surface (data). In one embodiment, the location of the target surface, the target surface, and the properties are displayed in real time during data collection or after data processing. In this setup, the target surface and target surface property data can be displayed separately or can be displayed in combination with the location data to be projected onto a real or abstract model.

[0337] Relative to Figure 15 and Figure 16 In an embodiment, a biosensor can be used in place of camera 1082 / 1070 such that enhanced or hybrid classification 1512 / 1616 can be based on a classification that combines biosensor data and internal sensor data 1508. In one example, the system can detect gingivitis in the upper buccal tooth region. This information is transmitted to the oral care cleaning system so that when brushing in the designated area, application 1073 changes the brushing mode setting 1516 to "sensitive", and the "pressure threshold" setting 1516 will be set lower.

[0338] As another modified use case, the diagnosis of a medical condition combined with the location of the condition can also result in a modification of the operation of the appliance at that location. For example, when hybrid classification 1512 / 1616 determines that the toothbrush is in a location in the mouth where gingivitis has been (or is being) diagnosed, the operation of the electric toothbrush can be automatically adjusted to a "sensitive" operation mode.

[0339] It will be apparent that changes or modifications can be made to the exemplary embodiments without departing from the scope as claimed below. Additionally, it is not necessary that any of the objectives or advantages discussed herein fall within that scope, as there can be many advantages that need not be disclosed herein.

[0340] The dimensions and values disclosed herein should not be construed as being strictly limited to the exact numerical values recited. Instead, each such dimension is intended to represent the recited value and a functionally equivalent range around that value, unless otherwise indicated. For example, a dimension disclosed as "40 mm" is intended to represent "about 40 mm".

[0341] Each document cited herein, including any cross-referenced or related patent or patent application publication, is hereby incorporated by reference in its entirety, unless expressly excluded or otherwise limited. The citation of any document is not an admission that it is prior art to any of the embodiments disclosed herein or claimed, or that it independently or in any combination with any other one or more references, teaches, suggests or discloses any such embodiment. Further, when any meaning or definition of a term in this invention conflicts with any meaning or definition of the same term in a document incorporated by reference, the meaning or definition assigned to the term in this invention shall govern.

[0342] The dimensions and values disclosed herein should not be construed as being strictly limited to the exact numerical values recited. Instead, each such dimension is intended to represent the recited value and a functionally equivalent range around that value, unless otherwise indicated. For example, a dimension disclosed as "40 mm" is intended to represent "about 40 mm".

Claims

1. A method for operating a dental appliance, the method comprising: Providing a dental appliance comprising at least one motion sensor that provides motion data, the motion sensor selected from an orientation sensor, an acceleration sensor, and an inertial sensor; Providing at least one intraoral camera associated with the dental appliance that provides image data; Using a first machine learning neural network to segment the image data received from the intraoral camera to segment surface conditions, thereby generating a surface condition segmentation; Using a second machine learning neural network classifier to classify the motion data received from the motion sensor and the image data received from the intraoral camera to classify the motion and position of the dental appliance relative to the surface of the user's anatomy, thereby generating a surface position that includes at least one of a relative motion classification or a relative position classification; And Based on the surface condition and the surface position, generating a user treatment progress status and position and transmitting the treatment progress status and position to the user.

2. The method according to claim 1, wherein the image segmentation relates to the surface condition of the user's oral cavity.

3. The method according to claim 1, wherein the first and second machine learning neural networks are convolutional neural networks or Transformer neural networks.

4. The method according to claim 1, wherein the surface condition and the surface position modify the operation of the dental appliance.

5. The method according to claim 1, wherein the surface of the user's anatomy is the user's oral cavity and the oral cavity is divided into treatment areas.

6. The method according to claim 1, wherein the treatment progress status and position include feedback on the amount of operation time of the dental appliance in each treatment area.

7. The method according to claim 1, wherein the treatment progress status and position include feedback on the position of the dental appliance in each treatment area.

8. The method according to claim 1, wherein the treatment progress status and position include feedback and recommendations based only on the surface condition.

9. The method according to claim 1, wherein: The dental appliance further comprises a computer network interface for sending and receiving data via a computer network; and The camera is located on a computerized device that includes a computer network interface for sending image data at least via the computer network.

Images