Robotic system for product application and learning process

Abstract

Robotic system for product application and learning method. Robotic system (1) for applying a product to a part of the human body, in particular a face, comprising: A robotic arm (10), an applicator (11) mounted at the end of the arm, at least one force sensor for measuring the force exerted by the applicator (11) during application, at least one optical sensor (13), an arm control system, comprising: a robot operating system (ROS) (100), and an AI system (310) interacting in a closed loop with the robot operating system (100), this AI system having as inputs the outputs of the robot operating system and having been trained on at least one digital twin (320). Figure for the abstract: Fig. 1

Description

Title of the invention: Robotic system for the application of a product and learning process. Technical field

[0001] The present invention relates to systems for the automated application of a product to a surface, and more particularly but not exclusively to systems for the application of a makeup product to the face. Previous technique

[0002] The automated application of a makeup composition can be useful to help people with visual or motor disabilities to apply makeup.

[0003] This can also make it possible to reproduce elaborate makeup looks or looks that are difficult to achieve manually.

[0004] There may also be an advantage in being able to carry out application tests of a composition on an artificial surface such as a mannequin head, in order to perform performance tests in the laboratory. These tests should be as representative as possible of real application conditions.

[0005] Robotic systems are known in which the arm carries an end effector and makes it follow a predefined trajectory to perform a task. Such systems require a significant amount of prior trajectory programming and are poorly suited to processing a moving surface such as a person's face during a makeup session.

[0006] Devices that fit in a hand and are intended to be moved by a person on the face have also been proposed, comprising an inkjet print head which is automatically driven as it moves over the skin so as to apply a makeup composition according to a predefined pattern.

[0007] The design of a robotic system for performing makeup must meet certain conditions.

[0008] First, in the case of applying makeup to a real person, the system must be able to perform the makeup despite the inevitable movements of the person's face, and without exerting uncomfortable pressure on the person.

[0009] Furthermore, in the context of use in point of sale or as an aid or development tool, the system must be able to adapt quickly to various face shapes and to the execution of various types of makeup.

[0010] Finally, the makeup must be able to be carried out relatively quickly, safely for the user, and with a repeatability at least as good as if the makeup were carried out by an experienced person. Description of the invention

[0011] The invention aims to provide a robotic system for satisfactorily applying makeup to a real face or test surfaces of various shapes. Summary of the invention

[0012] The invention thus relates to a robotic system for applying a product to a part of the human body, in particular a face, comprising: - A robotic arm that can move with several degrees of freedom, including six degrees of freedom, - an applicator mounted at the end of the arm, - at least one force sensor to measure the force exerted, particularly in a direction normal to the surface to be treated, by the applicator on the surface to be treated during application, - at least one optical sensor to image the surface to be treated, - a control system for the arm, comprising: • a robot operating system, connected to the robotic arm, the force sensor, and the optical sensor, having as inputs actions to modify the arm's positioning, including actions to be performed on the robot's actuators, and as outputs the arm's states, the force measured by the force sensor, and the images from the optical sensor, and • an AI system interacting in a closed loop with the operating system, this AI system having as inputs the outputs of the robot's operating system and having been trained at least on a digital twin of at least one part of the human body, including a face, and of the robotic system with a deep reinforcement learning model, including DQN type, to generate as outputs said actions of modifying the positioning of the arm.

[0013] Preferably, the deep reinforcement learning model is of the DQN type.

[0014] A Deep Q-Network (DQN) is a type of neural network used in deep reinforcement learning. It combines deep neural networks with a Q-leaming-type algorithm to enable a machine to learn to make optimal decisions in a given environment. Q-leaming is a reinforcement learning method that aims to learn an action value function, called Q, which indicates the quality of an action in a given state. DQN learning models use neural networks to approximating the Q function, which allows managing environments with a large number of possible states.

[0015] The robotic system according to the invention is capable of self-learning in a virtual environment, thanks to the digital twin, and of generalizing what it has learned in the virtual environment to apply it to the real world.

[0016] The system is thus capable of autonomously applying a cosmetic composition to perform makeup on a predefined area of ​​the face, the application being carried out with a force low enough not to cause discomfort but high enough nevertheless for the composition to transfer onto the skin or lips.

[0017] The self-learning capability on a digital twin also facilitates the use of the robotic system on new media and / or with new applicators or compositions.

[0018] Learning in a virtual environment further reduces the time required for human programming, as learning to perform a new makeup application does not require complex manual programming of a new arm trajectory, and increases safety because the learning does not generate any real movement of the arm.

[0019] The system is finally able to easily adapt to a new face and to the person's movements during makeup application.

[0020] Preferably, the optical sensor is mounted on the arm. This improves the accuracy of image capture of the surface to be processed.

[0021] The optical sensor is preferably a camera that generates depth information between the optical sensor and the surface to be processed. This depth information can be useful for controlling the distance between the application and the surface to be processed.

[0022] The system can be arranged to apply a product to the face and include a position stabilization module to bring the optical sensor, when worn by the arm, to a predefined position relative to the face in the optical sensor's reference frame, by acting on the robot arm control.

[0023] The robotic system can thus include a module enabling compensation for the movements of the person's face, in particular aimed at following the movement of the face so as to maintain it with a predefined position, in particular centered, in the camera's reference frame, taking into account the movements of the arm, this servoing being implemented in the control loop of the robot arm.

[0024] The robotic system may include a force stabilization module, receiving a setpoint value from the AI ​​system, and comprising a loop of regulation, for example of the PID type, to act on the control of the arm in order to respect this setpoint value as closely as possible.

[0025] The system may include an artificial head defining the surface to be treated. This may allow for application tests of a cosmetic composition to be carried out on this head. The latter may also be used for training the robotic system.

[0026] The robotic system may include an interface allowing the user to select an area to be treated and a product to be applied.

[0027] The system may include a mirror, the arm control system being arranged to hold the mirror in front of the user, particularly after applying makeup, to allow them to assess the result. The mirror may be arranged so as to be grasped by the arm instead of the applicator.

[0028] The product may be a makeup composition, in particular chosen from lip and skin makeup products, in particular from a lipstick, a foundation and an eyeliner.

[0029] The invention further relates to a method of learning a robotic system according to the invention, as defined above, in which the AI ​​system is trained using the digital twin of the surface to be treated and the robotic system.

[0030] The AI ​​system can be trained initially on the digital twin and then subsequently on an artificial head.

[0031] Training can be carried out by successive instances, by generating the ST +ide states of the next instance from the ST states of the current instance, so as to try to increase an RT reward.

[0032] This reward can be increased when at least one of the following conditions is met: - Complete the application over the entire area to be treated, - complete the application in less time, - complete the application with consistent force, - avoid going beyond the area to be treated.

[0033] The ST states of the current instance can correspond to: - The position of the arm holding the applicator, - the configuration of the surface to be treated, - the value of the applied force, - the result of the current application

[0034] The ST+ide states of the following instance can be generated in response to actions generated by the system aimed at improving the reward, these actions being able to include the desired position of the end of the arm carrying the applicator for continue the application and the desired application force on the surface to be treated to continue the application.

[0035] The robot's operating system is preferably configured to process inputs from the digital twin in the same way as those from the real environment. Thus, the digital twin may include a virtual robotic arm having the same dimensions and control system as the real robotic arm, the same optical sensor, a head mesh with at least one local physical property, and an application rendering module. This last module generates the resulting image of the virtual application of the makeup composition. Brief description of the drawings

[0036] The invention will be better understood upon reading the detailed description that follows, a non-limiting example of its implementation, and upon examination of the accompanying drawing, in which:

[0037] [Fig-1] Fig. 1 schematically represents an example of a system robotized according to the invention,

[0038] [Fig.2] [Fig.2] represents the robotic arm in isolation,

[0039] [Fig.3] Fig.3 schematically illustrates the control of the robotic arm,

[0040] [Fig.4] [Fig.4] is an example of a graphical representation generated from the digital twin of the robotic system,

[0041] [Fig.5] [Fig.5] illustrates the location of reference points that can be used to track the face during the treatment of the surface to be treated,

[0042] [Fig.6] [Fig.6] is an example of vision simulation by the on-board optical sensor of a virtual face to be made up,

[0043] [Fig.7] [Fig.7] is an example of lip topography modeling,

[0044] [Fig.8] [Fig.8] illustrates different steps of an example makeup process using the robotic system according to the invention, and

[0045] [Fig.9] [Fig.9] is an image representing an example of virtual face makeup. Detailed description

[0046] Figure 1 shows an example of a robotic system 1 according to the invention.

[0047] This system 1 comprises a robotic arm 10, shown in isolation in [Fig.2], this arm being supported by a frame 15, which can house the power supply of the arm and its control interface.

[0048] The arm 10 is for example a six-degree-of-freedom arm, known in itself, comprising several motorized segments rotating, under the effect of motorized actuators, around respective axes of rotation.

[0049] The arm 10 can carry any type of gripper, for example a two-finger gripper 11, as illustrated.

[0050] The arm 10 incorporates a force sensor housed inside one or more elements 12 of the arm, which measures the force exerted in the axis of the clamp 11, on it, this axis preferably coinciding with the normal to the surface to be treated.

[0051] The arm 10 carries a color camera 13, which is preferably a depth-sensing camera, meaning that it not only delivers an image but also measures in real time the distance between the camera and the imaged surface. This camera can be carried by the last segment of the arm, as illustrated, in order to have a view along the axis of the gripper 11.

[0052] The system 1 may include a support 15a which allows one or more makeup product applicators, for example a lipstick 20 and an eyeliner 30 in the illustrated example, to be pre-positioned on the frame 15. The frame 15 may also allow for the pre-positioning of a mirror suitable for being grasped by the arm.

[0053] An artificial head 40 can be positioned in front of the arm 10 using a tripod or any other suitable support.

[0054] The robotic system 1 includes a computer system for controlling the arm which receives information from the camera 13, the force sensor 12 and the arm 10. This computer system may include a microcomputer or any other computing means.

[0055] The software component executed by the computer system can be broken down into a component running on the robot operating system environment 100, referred to as the ROS (Robot Operating System), and a component based on an artificial intelligence system 110, as illustrated in [Fig. 3]. This artificial intelligence system (also called the "engine") can be based on the GYM software architecture from OPEN AI.

[0056] Communication between the ROS 100 environment and the artificial intelligence engine can be carried out in real time via a UDP communication protocol.

[0057] The ROS 100 environment can feature a so-called "Sim-to-real" software architecture to manage two operating modes: simulated operation, where the artificial intelligence engine's learning can take place virtually, and real operation, where the artificial intelligence engine is used to control the arm in the real world. An example of a "Sim-to-real" software architecture is described in the publication Zhao, W., Queralta, JP and Westerlund, T., 2020, December. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In 2020 IEEE Symposium Series on Computational Intelligence (SSCI) (pp. 737-744). IEEE.

[0058] The simulated operation is based on a digital twin of the robotic arm and its environment, as illustrated in [Fig.4].

[0059] This digital twin simulates the behavior of a robotic arm 10' which is a replica of the arm 10. The virtual arm 10' thus reproduces the dimensions of the real arm, its degrees of freedom, its actuation speeds, its constraints and physical collisions.

[0060] This arm 10' thus carries a virtual gripper 11' and a virtual camera 13' which takes up the parameters of the real camera and the digital twin makes it possible to simulate the image which would be delivered by such a camera, which is similar to that delivered by the camera 13 carried by the arm 10, with the same field of view and point of observation from the arm.

[0061] The digital twin also includes a 20' virtual applicator and one or more 40' and 50' virtual faces.

[0062] These virtual 40' or 50' faces are created, for example, with a mesh that reproduces the relief of a real face, as illustrated in [Fig. 5], and a texture model can be associated with this mesh that simulates the appearance and at least one physical property of the surface, such as local hardness. This texture makes it possible to generate force feedback information as a function of the position of the applicator's application surface relative to the resting skin surface. Thus, the force exerted on the applicator can be modeled as a function of its degree of penetration into the skin.

[0063] The virtual face can be an already available model or be created from a real face. An example of reconstructing a virtual face from a real face is described in the publication O'Toole, AJ, Price, T., Vetter, T., Bartlett, JC and Blanz, V., 1999. 3D shape and 2D surface textures of human faces: The role of “averages” in attractiveness and age. Image and Vision Computing, 18(1), pp.9-19.

[0064] The simulation of makeup application, also called MVTO (“Make-up Virtual Try On”), can be done using a technique called deep graphics inversion, as described in the article Kips, Robin, et al. "Realtime Virtual Try On from a Single Example Image through Deep Inverse Graphics and Learned Differentiable Renderers." Computer Graphics Forum. Vol. 4L No. 2. 2022.

[0065] The robotic system 1 is arranged in such a way that regardless of the operating mode of the arm, real or simulated, the states of the arm are generated in the same format and the control of the arm is carried out in the same way.

[0066] The robotic system 1 is arranged to control the arm in real time to perform makeup on a given area of ​​the face thanks to prior learning with the digital twin.

[0067] System 1 can use a so-called DQN artificial intelligence engine, whose learning is deep reinforcement learning, in which the artificial intelligence engine receives a reward when the task is performed correctly. At each iteration, depending on the input states, the artificial intelligence engine tries to apply an action to increase the reward, and the next action is generated.

[0068] Reinforcement learning is a machine learning technique where an agent learns to act optimally based on rewards received from the environment. The agent acts according to the state of the environment ST. The environment provides a reward R^ to the agent based on the effectiveness of the agent's actions in completing the task.

[0069] In one example of an implementation of the invention, the agent's task is to apply makeup to a target surface, which could be a face, an application surface in a product performance evaluation context, etc. The agent is then the robotic arm applying the makeup, and the environment is the computer system with the camera and the application surface to which the product is to be applied. The environment provides rewards based on the task's completion status.

[0070] The action model of the robotic arm, called police, is a neural network a(Xp F\ST) which, given the current state of the environment Sp, predicts the trajectory Xp that the robotic arm must travel to apply the makeup to the application surface, as well as the values ​​of the force sensor during the application FT.

[0071] The state of the environment is composed of the position of the robotic arm, the values ​​of the force sensor, the current configuration of the application surface, and the target configuration of the application surface.

[0072] The predicted trajectory is represented by a set of consecutive points. In terms of neural network architecture, a(Xp, FlSp) is a deep neural network composed of three subnetworks. The first subnetwork takes as input the current configuration of the application surface and the target configuration of the application surface and produces a vector representation. The second subnetwork takes as input the position of the robotic arm, as well as the force sensor values, and produces a vector representation. A third subnetwork combines the two vector representations to predict the trajectory that the robotic arm should follow, as well as the force sensor values ​​during the trajectory.

[0073] The RT reward is a signal returned by the environment to the robot regarding its action. The reward is a scalar value that is higher the more similar the configuration of the application surface is to the configuration of the target surface when the makeup is applied correctly. Furthermore, the reward is higher the more perfectly the task is completed within a short time.

[0074] The agent's learning consists of finding the right parameters, called weights in the context of a neural network. During learning, the agent will apply the makeup, and step by step, improve its action model based on the rewards received. The higher the rewards received, the better the font. For learning the agent's font, an algorithm called actor-critical can be used, as described in the publication Konda, Vijay, and John Tsitsiklis. "Actor-critical algorithms." Advances in neural information processing Systems 12 (1999), which consists of introducing an additional neural network called a value network, which is an approximation of the value function v[sj] consisting of the expected cumulative rewards at the end of the task, knowing that at step t the environment was in state St.In terms of architecture, the value network can have the same structure as the neural network representing the agent's police force, the only difference being that the value network predicts the value of the environment state received as input.

[0075] The network representing the agent's policy and the value function network are jointly trained. Using the policy, the agent generates actions, and the rewards received are used to update the value function network. The value function network is then used to update the policy and favor actions leading to states associated with strong value functions.

[0076] An example of an artificial intelligence engine suitable for implementing the invention is described in the publication Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, L, Wierstra, D. and Riedmiller, M., 2013, Playing atari with deep reinforcement learning, arXiv preprint arXiv: 1312.5602.

[0077] The ROS 100 environment can generate the following 300 states as outputs at each iteration, as illustrated in [Fig.3]: - 301 arm position states (i.e., for example, Cartesian coordinates of the gripper carried by the arm and its orientation, - Camera data 302, namely the camera image with associated depth data, - Force data 303 (i.e. the value of force exerted on the applicator in reaction to its contact with the surface to be treated, in particular the normal component of this force).

[0078] From the data from camera 302, the system can calculate the configuration of the application surface, and the current result of the application.

[0079] The artificial intelligence engine 310 treats indifferently the states 311 coming from the simulation and those coming from the real world, the same formats of data being used to describe real or virtual states in the digital twin 320.

[0080] The ROS 100 environment also provides the AI ​​engine 310 with rewards 312 for the completion of certain tasks.

[0081] For example, rewards are assigned for the successful completion of the following tasks: - Complete the application over the entire area to be treated, - to complete the application in less time than before, - complete the application with even force, - avoid going beyond the area to be treated.

[0082] System 1 may include an interface allowing the type of makeup to be applied to be defined, and where appropriate the type of applicator or makeup composition to be selected.

[0083] System 1 can be arranged to perform pre-processing in order to recognize characteristic points on the image from the real camera and calculate the coordinates and orientation of the face in the arm's reference frame.

[0084] The characteristic points of the face are for example those referenced 60 and illustrated in [Fig.5]. These characteristic points are for example located at the outer corners of the eyes, at the top of the nose, at the corners of the lips and at the top of the chin.

[0085] System 1 is also arranged to simulate the detection of face topology from data from the virtual camera 13' of the digital twin.

[0086] An example of simulation of the image observed by the virtual camera 13' is given in [Fig.6], for two viewing angles.

[0087] An example of determining the topology of the lips of a face is illustrated in [Fig.7]. This determination can be carried out in a preliminary step of recognizing the surface to be treated, real or virtual, by the robotic system.

[0088] The system 1 may include two stabilization modules 316 and 317, one for the position of the face, the other for the force exerted by the applicator.

[0089] The 316 module for stabilizing the force applied by the applicator aims to apply a constant pressure during the application of the composition using the applicator. It can implement a PID control loop.

[0090] At each instant T, the DQN model can calculate a desired force FT for continuing the makeup application, and the force stabilization module 316 takes this value as its setpoint. A feedback loop is implemented to control the force during application.

[0091] The face stabilization module 317, or visual servoing, aims to track facial movement, taking into account arm movements. This servoing is implemented in the robot arm control loop. An example of such visual servoing is described in the publication by Chaumette, F. and Hutchinson, S., 2006. Visual servo control. I. Basic approaches. IEEE Robotics & Automation Magazine, 13(4), pp.82-90.

[0092] The AI ​​engine is trained using the digital twin by successive instances, the model taking as inputs the states of the system at instance T, and generating the actions to be performed at instance T+1 in order to maximize the reward.

[0093] The actions 314 are, for example, the target coordinates and orientations of the end of the arm carrying the applicator, according to a six-degree-of-freedom model, as well as the force to be applied.

[0094] The ROS environment can convert this information in step 315 into orientation values ​​to be taken around each of the articulation axes of the arm, using an inverse kinematic matrix, thus allowing precise control of the arm movements.

[0095] An example of makeup 41' of a virtual face 40' is illustrated in [Fig.9].

[0096] Once the training has been completed on the digital twin, the system is ready to be used in the real world.

[0097] We will describe, with reference to [Fig. 8], an example of a makeup process using the robotic system according to the invention. Makeup using the robotic system 1 can be performed on an artificial head 40 as illustrated in [Fig. 1] or on a person.

[0098] In step 210, the mannequin is placed in front of the robotic arm 10, ensuring that no obstacle hinders the arm's movements.

[0099] In step 211, the camera 13 is used to recognize characteristic points on the face and measure the distance to the face and guide the arm 10 towards the face. The system 1 keeps the face centered in the field of view of the camera 13.

[0100] In step 212, the system reconstructs a digital twin of the face using the images acquired by the camera 13. This digital twin includes the topology of the surface to be treated, for example the lips as illustrated in [Fig.7].

[0101] This digital twin of the face can be used in step 213 to allow a user to select, via the interface, a type of makeup, for example the application of lipstick.

[0102] The makeup rendering is obtained using the MVTO module detailed above, and can be displayed on a screen with the digital twin, as illustrated in [Fig.9].

[0103] In step 214, the system executes a closed loop using the previously trained artificial intelligence engine, where the input is given by the states and the output by the actions that modify the states for the next iteration. This loop determines the trajectory of the arm and the force to be exerted during application, the position of the arm being corrected in real time during the execution of this loop. Simultaneously, at In step 215, the system tracks the face in order to stabilize the camera's vision and regulates the applied force, thanks to the aforementioned face position stabilization modules in the camera's reference frame and applied force.

[0104] Once the makeup has been applied, at step 216, the system can control the arm to reposition the applicator.

[0105] When the made-up face is not that of a mannequin but of a person, the system can be arranged at this stage to bring the arm to grasp a mirror in order to present it to the person who has just been made up.

[0106] Of course, the invention is not limited to the example just described. The invention covers the application of any cosmetic, makeup or skincare composition to human keratinous materials.

[0107] Where appropriate, the learning carried out using the digital twin can be supplemented by subsequent learning carried out in the real world, for example on the mannequin.

Claims

Demands

1. A robotic system (1) for applying a product to a part of the human body, in particular a face, comprising: - A robotic arm (10) having several degrees of freedom, in particular six degrees of freedom, - an applicator (11) mounted at the end of the arm, - at least one force sensor for measuring the force exerted, in particular in a direction normal to the surface to be treated, by the applicator (11) on the surface to be treated during application, - at least one optical sensor (13) for imaging the surface to be treated, - an arm control system, comprising: • a robot operation system (100), connected to the robotic arm, the force sensor and the optical sensor, having as inputs actions for modifying the positioning of the arm and as outputs the states of the arm, the force measured by the force sensor and the images from the optical sensor,and • an AI system (310) interacting in a closed loop with the robot's (100) operating system, this AI system having as inputs the outputs of the robot's operating system and having been trained at least on a digital twin (320) of at least one part of the human body, in particular a face, and of the robotic system with a deep reinforcement learning model, in particular of the DQN type, to generate as outputs (314) said actions of modifying the positioning of the arm.

2. System according to claim 1, the optical sensor (13) being carried by the arm.

3. System according to any one of the preceding claims, the optical sensor (13) being a camera generating depth information between the optical sensor and the surface to be processed.

4. A system according to any one of the preceding claims, including a connection to claim 2, being arranged to apply a product to the face and comprising a face stabilization module enabling compensation of facial movements, from position to bring the optical sensor to a predefined position relative to the face in the optical sensor's frame of reference, by acting on the control of the robot arm.

5. A system according to any one of the preceding claims, comprising a force stabilization module, receiving a setpoint value from the AI ​​system, and comprising a control loop, in particular of the PID type, to act on the control of the arm in order to respect this setpoint value as closely as possible.

6. System according to any one of the preceding claims, comprising an artificial head (40) defining the surface to be treated.

7. System according to any one of the preceding claims, the product being a makeup composition, in particular selected from lip and skin makeup products, in particular lipstick, foundation and eyeliner.

8. A method for learning a robotic system (1) according to any one of the preceding claims, wherein the AI ​​system is trained using a digital twin (320) of the surface to be treated and the robotic system.

9. Method according to claim 8, wherein the AI ​​system trained on the digital twin (320) is then trained on an artificial head (40).

10. A method according to any one of claims 8 and 9, the training being carried out by successive instances, by generating the ST+ide states of the next instance from the ST states of the current instance, so as to try to increase an RT reward.

11. Method according to claim 10, the reward being increased when at least one of the following conditions is met: - Complete the application over the entire area to be treated, - complete the application in less time, - complete the application with a regular force, - avoid going beyond the area to be treated.

12. A method according to claim 10 or 11, the ST states of the current instance corresponding to: - The position of the arm carrying the applicator, - the configuration of the surface to be treated, - the force value, - the result of the application in progress.

13. A method according to any one of claims 10 to 12, wherein the ST+ide states of the following instance are generated in response to actions generated by the system to enhance the reward, the actions comprising the desired position of the end of the arm carrying the applicator to continue the application and the desired application force on the surface to be treated to continue the application.

14. A method according to any one of claims 8 to 13, the operating system being arranged to treat inputs from the digital twin in the same way as those from the real environment.

15. A method according to any one of claims 8 to 14, the digital twin comprising a virtual robotic arm having the same dimensions and the same control system as the real robotic arm, the same optical sensor, a head mesh with at least one local physical property, an application rendering module.

Images