DEVICE AND METHOD FOR AUTOMATIC APPLICATION OF COSMETICS TO A USER - Patent application

Abstract

Apparatus and method for automatically applying cosmetics to a user This patent of invention relates to the technical field of a method or apparatus for reading and individualizing patterns for viewing the preparation and application of cosmetics. More specifically, it shows an apparatus and method for automatic application of cosmetics to a user, which is composed of a portable device (1) for automatic application of cosmetics, which comprises a casing (4) and a base support (5) for receiving and enclosing a movement mechanism (3) by computer numerical commands (CNC) through a list of movements based on a series of pre-programmed commands in G-code.

Description

[Technical field]

[0001] This patent of invention relates to the technical field of a method or device for pattern reading and individual recognition for viewing the preparation and application of cosmetics. More specifically, it shows a device and method for automatically applying cosmetics to a user, which consists of a portable device with a camera coupled to recognize lips and generate G-code (a computer language responsible for controlling the movement of machines) to draw a path and manipulate computer numerical commands (CNC) to control the movement of the mechanism for moving cosmetics such as lipstick on the user. [Background technology]

[0002] Representing a long-felt and highly sought-after need in the art, various appliances and methods for automatically applying cosmetic products to a user have been disclosed in the patent literature, for example, in published applications PI1107004-8; US10479109; CN103885461, US9811717, and EP3384802B1.

[0003] Patent document EP3384802B1 further provides regarding the use of computer numerical command (CNC) processing for controlling the movement of an automatic make-up mechanism by operating movement actuators in the X, Y and Z directions. Furthermore, patent document WO13040865 also explicitly indicates the use of CNC mechanisms for movement on coordinate axes, which are also supported in the main structural text.

[0004] Evaluating the use of Code G technology for converting, controlling and operating a step motor to orient the applicator is provided in document CN113071097 in this particular case, 3D printing.

[0005] In particular, patent document BR102020022162-0, entitled "Method for Detecting and Segmenting Lip Areas", owned by the same applicant as the present invention, already provides for the use of deep learning algorithms, machine learning and artificial intelligence to detect and segment the lip area in an accurate manner to enable the automatic application of lip products. Summary of the Invention [Problem to be solved by the invention]

[0006] Although the art provides for the existence of various tools and methods for automatically applying cosmetics to a user or even the use of CNC driven mechanisms or Code G technology already known in 3D printing for driving / moving stepper motors, said documents do not provide in that regard current devices that allow for greater compactness and portability of the arrangement of the moving mechanisms and the arrangement of the cameras.

[0007] Furthermore, the art also exists in need of a unique solution associated with compactness and portability with the desired characteristics of greater speed and precision in acquiring and processing images, generating G-code and driving CNC mechanisms, and there is no mention of the use of G-code, which is essential for special division with better recognition and certain assertion of the lip area, as seen for example in the device of patent document EP 3 384 802 B1, which moreover requires a complex arrangement on the workbench, and for considering greater speed and precision in the movement mechanisms involved in applying cosmetics to the user. [Means for solving the problem]

[0008] The present invention provides various solutions to problems found in the art and essentially provides a compact and portable device for automatically applying cosmetics such as lipstick to a user with fast, practical and effective results by using a camera coupled to identify a user image, e.g., the user's lips, and generating a code G file to plot a path and manipulate computer numerical commands (CNC) to control the movement mechanism involved in automatically applying the user's cosmetics such as lipstick.

[0009] Furthermore, in addition to using the algorithm for detecting and segmenting the lip region as taught in document BR102020022162-0, the present invention also provides, through code G, a post-processing approach for the segmentation of an image of, for example, human lips, in order to assist in mapping the movement path and pass this information to a CNC-based accurate movement process for automatically applying cosmetics such as lipstick to a user. The following description includes, but is not limited to, the elements shown and cited in the figures, the following descriptions of which are referred to below by reference. [Brief description of the drawings]

[0010] [Figure 1A] FIG. 1A shows a portable device for automatic application of cosmetic products to a user, according to the present invention. [Figure 1B] FIG. 1B shows a casing for enclosing an electromechanical moving device. [Figure 1C] FIG. 1C shows an electromechanical movement device attached to a base support. [Figure 2A] FIG. 2A shows a simplified block diagram referencing the steps of recognizing an image of a user by a camera coupled to a portable device for automatic application of cosmetics, generating a code G file, and driving a CNC electromechanical movement device. [Figure 2B] FIG. 2B shows a block diagram of the electronic drivers and controls for the automated application of cosmetic product to a user. [Figure 2C]FIG. 2C shows a block diagram of the complete architecture of the system of electronic drivers and controls for the automated application of cosmetic product to a user. [Figure 2D] FIG. 2D shows a flow diagram of how the modules of the system can be used in a reduced size computer part on a single board such as that organized on a Raspberry Pi board. [Diagram 3] FIG. 3 shows a diagram of various states of a portable device for the automated application of cosmetic products to a user. [Figure 4A] FIG. 4A shows a schematic diagram of the various electronic and electromechanical drive and control components present in a portable device for the automated application of cosmetic products to a user. [Figure 4B-4C] 4B and 4C show examples of electronic elements present in a portable device for the automated application of cosmetic products to a user. [Figure 5A-5D] 5A through 5D show details of the pre-assembled components of the X-axis translation module of the electromechanical drive device. [Figure 6A-6C] 6A to 6C show details of the components of the Y-axis movement module of the electric equipment driving device before assembly. [Figure 6D-6E] 6D and 6E show details of the components of the Y-axis movement module of the electric equipment driving device before assembly. [Figure 7A-7C] 7A to 7C show details of the pre-assembled components of the Z-axis movement module of the electrical equipment drive device. [Fig. 7D-7F] 7D through 7F show details of the Z-axis movement module of the electrical equipment driving device before assembly. [Figure 8A-8B] 8A and 8B show details of the integration of parts between the X-axis translation module, the Y-axis translation module, and the Z-axis translation module prior to assembly. [Fig. 8C-8E] 8C through 8E show details of the integration of pre-assembly components between the X-axis translation module, the Y-axis translation module, and the Z-axis translation module. [Figure 9A-9B] 9A and 9B show details of the camera position on the X-axis translation module. [Figure 10A-10B] 10A and 10B show details of the location of the electronic circuit board components and reduced size computer components on a single board in the X-axis translation module. [Figure 11A-11B] 11A and 11B show perspective views of the top and bottom of a portable device for automatic application of cosmetics in an essential embodiment of the invention. [Figure 12A-12B] 12A and 12B show top and bottom perspective views of a portable device for automatic cosmetic application in a second embodiment of the invention. [Figure 13A-13B] 13A and 13B show top and bottom perspective views of a portable device for automatic cosmetic application in a third embodiment of the invention. [Figure 14A-14B] 14A and 14B show details of the fixation of the CNC translation mechanism to the base support. [Figure 15A-15B] 15A and 15B show details of the fastening of the base support to the mannequin's head support in one embodiment. [Figures 16A-16C] 16A to 16C show details of the casing of a portable device for automated cosmetic application. [Figure 17] FIG. 17 shows a simplified diagram of the utilization of computer vision models. [Figures 18A-18C] 18A to 18C show images resulting from various steps of the lip image segmentation process used in the method for automated cosmetic application of the present invention. [Fig. 18D-18H] 18D through 18H show images resulting from various steps of the lip image segmentation process used in the method for automated cosmetic application of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The present invention shows a portable device (1) for automatic application of cosmetics to a user, such as lipstick, comprising a camera (2) coupled to capture an image of the user, such as the lips of the user, and generating a code G for plotting a computer numerical command (CNC) path and drive to control a CNC movement mechanism (3) for automatic application of cosmetics on the user, such as lipstick, as illustrated in Figures 1 and 2 in a substantial embodiment of the invention. The portable device (1) for automatic application of cosmetics to a user further comprises a casing (4) and base support (5) for receiving and enclosing said CNC movement mechanism (3) and other electronic drives, and for controlling the automatic application of cosmetics on the user.

[0012] A simplified block diagram is shown in FIG. 2A, referring to the steps of recognizing an image of a user by a camera coupled to a portable device for automatic application of cosmetics, generating a code G file for operating a CNC and controlling a movement mechanism for automatic application of cosmetics to the user.

[0013] In particular, the Code G file is generated by a process that maps the portions of the image captured by the camera and defines coordinate values ​​of the X, Y and Z axis to ensure improved quality due to the high precision required in performing the application of the cosmetic product, by encoding the movements of all mechanical moving devices (3) by CNC during the process of applying the cosmetic product, such as lipstick, to the user.

[0014] More specifically, FIG. 2B shows a block diagram of an electronic drive and control component for automated application of cosmetic products to a user in accordance with the present invention, including a reduced size computer component in a single board such as a Raspberry Pi4 board, an electronic hardware platform such as an Arduino MEGA board, and corresponding peripherals thereof such as LEDs, buttons, and buzzers.

[0015] According to the present invention, the complete architecture of the system of electronic drivers and control of automatic application of cosmetics to the user is depicted in FIG. 2C. As can be seen in FIG. 2C, the system works within the Raspberry Pi and performs the operation of the hardware components, i.e., the LEDs acting as feedback of the system, for example, switching on and off the device for automatic application of cosmetics to the user, the buzzer, which is a sound driver used in combination with the LEDs to create feedback of the system, the camera flash, which in this case is the LED acting as a flash that aids in the process of image capture, and finally, the camera in the Raspberry Pi, which is used to capture a photo of the lips and to create the process of lip segmentation through a model for detecting the image in an efficient way in the generation of a high quality segmentation mask, for example through the CNNMask-R model. Furthermore, the system also has two ways, i.e., through an external programming interface that allows the use of CNNMask-R, or carrying a local model (inside the Raspberry Pi) when the programming interface is not available. This architecture will then have access to an Aeduino board that will operate the CNC, the mechanical movement device that actually applies the cosmetics like lipstick.

[0016] The flow diagram shown in FIG. 2D clearly shows how the modules of the system used in the Raspberry Pi are organized.

[0017] Various statuses of the portable device for automatic application of cosmetics according to the present invention can be shown during the progress of its execution method as illustrated in the state diagram shown in Fig. 3, initially the ON-button is pressed, then the system starts, the user activates the button to capture an image of the lips, once the image is captured, the computer vision model segments the lips, after the segmentation the system transmits coordinates based on the segmentation to drive the CNC and mechanical movement devices, and at the end of the movement and at the end of the pigmentation of the lips the button can be toggled to turn off the system. Obviously, the present invention uses actuation by a button, but other actuation modes, such as inter alia automatic actuation and constant actuation, can be understood and used by those skilled in the art without departing from the scope of the invention.

[0018] In a non-limiting manner, according to the present invention, various specific indicators of the operational status of a portable device for automatic application of cosmetics can be determined for perception by a user, for example, according to Table 1. [Table 1]

[0019] As can be seen in FIG. 4A, a schematic diagram shows the various electronic and electromechanical drive and control components of the control components for the automatic application of cosmetic to a user that are present in a portable device in accordance with the present invention as previously detailed.

[0020] According to the present invention, between the electronic driving part and the control of the automatic application of cosmetics to the user, a computer part (6) of miniaturized size on a single board such as a Raspberry Pi4 board is the central component of the system. The miniaturized computer part (6) on a single board has the main function of managing the other components of the system to ensure that they operate according to the routine of the system and method of automatically applying cosmetics to the user. Some functions performed during the operation are as follows: It sends an electronic drive signal to an instantaneous luminous intensity generator (7) (e.g., an LED flash). Create an image capture. To create inferences, the cloud programming interface is accessed, and electronic drive signals are sent to the user interface units such as the light signal generator (8) (LED) and the buzzer (9). Send operational commands to a hardware electronic platform (10), such as an Arduino MEGA board.

[0021] For operation of the miniaturized computer part (6) on a single board it is necessary to use a power supply such as, for example, 5V / 3A with a USB Type-C connector; however, in an unrestricted manner it is possible to use an independent power supply, e.g. rechargeable.

[0022] Next, a hardware electronic platform such as an Arduino MEGA board (10) functions to control sensors and actuators related to the driving of the CNC and mechanical movement device (3), driving step motors (11) in three axes (X, Y and Z) according to end stroke switches (12). For the control, the driving and movement of the mechanical movement device (3) is performed using embedded computer logic such as firmware Marlin, which is widely used to control CNC.

[0023] The portable device (1) for the automatic application of cosmetic products to a user further comprises an electronic circuit board (13), e.g., a shielded RAMPS, along with auxiliary connections for creating the necessary electrical interface for the connection between a hardware electronic platform (10) (e.g., Arduino MEGA) and the electronic components contained in the CNC mechanical movement device (3), such as actuation drivers, step motors (11) and end stroke switches (12), during its electronic drive and control.

[0024] A driver (15) (e.g. A4988) appears as another component in said portable device (1) for automatic application of cosmetics to a user, in order to perform an electrical interface between the electronic circuit board (13), i.e. for example the shield RAMPS, and the step motor (11) used for the movement of the CNC machine movement device (3). The main function of the driver (15) is to condition the electrical signals coming from a hardware electronic platform (10), such as an Arduino MEGA board, so that they can be used to control the movement of the step motor (11).

[0025] The step motors (11) are used to move the CNC machine movement device (3) along the three axes of freedom of the system. To create the movement in the Y-axis, for example, two step motors (11A) of type NEMA-17 can be used, one for each axis. For the movement in the Y-axis, a step motor (11B) of type 28BYJ-48 can be used, for example.

[0026] Furthermore, the end stroke switch (12) also has the purpose of providing a reference position with reference to the CNC machine movement device (3) such that when the portable device (1) for the automatic application of cosmetic to a user is used, three end stroke switches are used, one of each being positioned at the origin of the Cartesian axes (X, Y and Z) of said CNC machine movement device (3).

[0027] In addressing components that indicate the operational status of the portable device (1) for the automated application of cosmetic products to a user, it includes a buzzer (9) that provides an audible indication to the user while the system is running.

[0028] Complementing the buzzer (9), considering greater accessibility to the user, including accessing users who are deficient or have any hearing impairment, or considering the use of the portable device for automatic application of cosmetics to said user in noisy places, a light signal component (8) such as an LED for interface to indicate the operational status of the portable device for automatic application of cosmetics to said user (1) is added, enabling communication with the user in an efficient manner.

[0029] The portable device (1) for the automatic application of cosmetic products to a user further includes an actuator (16), such as a user button, the function of which is to initialize the tasks of the system (processing of image capture, image segmentation (e.g. lips) and pigment application).

[0030] Finally, looking at the complete implementation of the image capture stage, a camera and an instantaneous luminosity generator (7) (e.g., LED flash) are also placed in the portable device (1) for automatic application of cosmetics to the user. In particular, the purpose of the camera (2), connected with a flat cable to a camera module port (17) present in a reduced-size computer part (6) on a single board (Raspberry Pi), is to capture an image of the user, such as lips, to be segmented by a computer vision model. Then, the function of the instantaneous luminosity generator (7) is to provide brightness so that the quality of the image captured by the camera (2) can be the best for the computer vision model, so that, for example, a LED cob 12V can be used.

[0031] The portable device (1) for automatic application of cosmetics can include two power supplies, one with 5V 3A output on USB-C to power a reduced-size computer part (6) on a single board (Raspberry Pi), and the second power supply provides a voltage of 12V 5A output on P4, which is used to power the instantaneous luminosity generator part (7) and the electronic circuit board (13) as well as the respective CNC movement mechanism (3).

[0032] In particular, the essential element for the implementation of the present invention, i.e. Computer Numerical Command (CNC), is a system that allows automatic control of machines with simultaneous movement control of various axes through a move list based on a series of pre-programmed commands in G-code. CNC is used in industry to control the movement of tools in various types of machines. The system is also often used for movement control in 3D printer mechanisms.

[0033] In an embodiment of the present invention, the CNC concept is used for the implementation of the movement control of a CNC mechanical movement device (3) which performs computerized numerical control of three step motors (11) responsible for linear movement in three orthogonal directional axes called X, Y and Z, where X represents the horizontal movement, Y the vertical movement and Z the depth movement.

[0034] Concerning the application of cosmetics, the portable device (1) for the automatic application of cosmetics includes a cosmetics application support (18) on a CNC machine moving device (3), as can be seen in Figures 1A and 1C. In the case of applying lipstick to a user, a lipstick pencil is chosen. In this case, in particular, the cosmetics application support (18) can comprise a cylinder into which the pencil lipstick is inserted with a resilient element, such as a spring, which is responsible for adjusting the pressure exerted by the pencil lipstick on the user by means of the preload of the resilient element, as well as for compensating for slight changes in the size of the pencil by friction. In an alternative embodiment, without departing from the scope of the invention, the application support (18) can include a set of two cylinders, the first of which includes a resilient element for receiving a pencil lipstick that can be screwed onto the second cylinder of the application support itself.

[0035] Regarding the selection of components of the CNC machine movement device (3), the basic design concept of the machine chosen for use in automated cosmetic application such as lipstick is based on using step motors (11) for three-dimensional movement utilizing computer numerical commands (CNC), since this allows obtaining optimal three-dimensional movement resolution.

[0036] As already explained and seen in FIG. 4, the X-axis and the Y-axis use step motors of type NEMA-17 (11A), one for each axis. The step motor (11A) has a step angle of 1.8°, and 200 steps are required for a full rotation (360°) of the motor shaft, ensuring good resolution. The torque of this step motor (11A) is 4.8 kgf.cm, which is sufficient for the present application. For both axes, the step motor (11A) is coupled with a type TR8 fuse through the use of a flexible coupling. Threaded against the type TR8 threads are four-start lead screw nuts compatible with the threads detailed above, two screw nuts are required, one for the X-axis and one for the Y-axis.

[0037] The combination between the screw and the screw nut is a system responsible for converting the motion of the rotary part of the screw coupled with the step motor (11A) to a linear motion of the CNC machine moving device (3). For each full rotation of the step motor (11A) and screw combination, the screw nut moves linearly 8 mm. This relationship ensures good precision of the linear motion of the CNC machine moving device (3), since the step motor (11A) has a step angle of 1.8°, resulting in a linear motion of 0.04 mm for each step.

[0038] The Z-axis, however, utilizes a type 28BYJ-48 stepper motor (11B). For simplicity and reduction in weight and size, instead of a screw and screw-nut combination, the Z-axis uses a pinion and toothed rod combination to convert linear motion to rotation. The resolution of the stepper motor (11B) is 64 steps per full revolution, i.e., each step has a step angle of 5.625°. For a full revolution of the stepper motor (11B), there is a linear displacement corresponding to 45.43 mm. If the step angle is 5.625°, the resulting displacement is approximately 0.71 mm for each step.

[0039] In particular, the CNC mechanical movement device (3), according to the present invention, as can be seen in the component details shown in Figures 5A to 5D, 6A to 6E, 7A to 7F, 8A to 8E and 9A to 10B, is composed of an X-axis movement module, a Y-axis movement module and a Z-axis movement module, all integrated with each other and with electronic drive and control components for the automatic application of cosmetic products to a user.

[0040] As seen in FIG. 5A, the X-axis translation module (19) includes an X-axis structural base (19A) that is axially aligned and inserted into an opening (19C) in a first wall segment (19D) and a second wall segment (19E) and has a pair of bearings (19B). FIG. 5B shows details of the components of a first step motor (11A) that receives a flexible coupling (19F) with a type TR8 thread (19G) on its rotation axis. FIG. 5C then shows the positioning of the step motor (11A) and the pre-slot of the flexible coupling (19F) with the thread (19G) on the X-axis structural base (19A). Finally, as can be seen in FIG. 5D, the components of the X-axis movement module (19) are put together with the insertion of an end stroke sensor (19H) positioned on the inner surface (19I) of the first wall segment (19D), the displacement of a flexible coupling (19F) having a threaded portion (19G) through an opening (19C) in the first wall segment (19D), the displacement of the threaded portion (19G) through a bearing (19B) in the second wall segment (19E), and the positioning of a step motor (11A) on the outer surface (19J) of the first wall segment (19D).

[0041] As seen in FIG. 6A, the Y-axis moving module (20) initially includes a Y-axis platform (20A) with a pair of bearings (20B) aligned axially and inserted into openings (20C) present in the first wall segment (20D) and the second wall segment (20E). The axis platform (20A) further includes a threaded nut (20F) inserted into an opening (20G) in one side (20H) with an orthogonal axis to the axis aligned between the openings (20C) as seen in FIG. 6B. FIG. 6C shows the details of the components of the second step motor (11A) that receives a flexible coupling (20I) with a type TR8 thread (20J) for its axis. Next, FIG. 6D shows the pre-slot positioning and slot combination of the step motor (11A) and the flexible coupling (20I) with the thread (20J) for the Y-axis platform (20A). Finally, as can be seen in FIG. 6E, the components of the Y-axis movement module (20) are assembled with the insertion of an end stroke sensor (20K) positioned on the upper outer surface (20L) of the first wall segment (20D), the insertion of an end stroke sensor (20K) positioned on the inner surface (20M) of the first wall segment (20D), the displacement of a flexible coupling (20I) having a threaded portion (20J) through an opening (20C) in the first wall segment (20D), the displacement of the threaded portion (20J) through a bearing (20B) of the second wall segment (20E), and the positioning of a step motor (11A) on the outer surface (20N) of the first wall segment (20D).

[0042] As can be seen in Fig. 7A, the Z-axis movement module (21) first includes a Z-axis structural base (21A) containing a first receiving and receiving facility (21B) for a step motor (11B), i.e., type 28BYJ-48, and a second receiving and receiving facility (21C) for a screw nut (21D). Fig. 7B shows the details of the positioning of the step motor (11B) in the first facility (21B) of the Z-axis structural base (21A). In particular, as can be seen in Figures 7C to 7E, a guide portion (21E) is positioned on the first facility (21B) and on each step motor (11B) and includes a passage opening (21F) for the shaft of said step motor (11B) and a female guide recess (21G) for receiving a male guide protrusion (21H) from a support portion (21I) of the cosmetic application support (18), and a gear (21J) is positioned on the shaft of said step motor (11B) between said guide portion (21E) and said support portion (21I), in particular said gear (21J) is fitted into the seat of said support portion (21I) of said cosmetic application support (18). Alternatively, an additional fixed lid (21K) is screwed onto said cosmetic application support (18), as shown in Figure 7F.

[0043] The assembled components between the X-axis translation module (19), the Y-axis translation module (20), and the Z-axis translation module (21) in the CNC machine translation device (3) are shown in Figures 8A to 8E.

[0044] As can be seen in FIG. 8A, the two smooth slide bars (22) of the Z-axis translation module are inserted into the Y-axis translation module (20) and intersect with the Z-axis translation module (21), thereby allowing the Z-axis translation module (21) to move back and forth.

[0045] FIG. 8B shows the positioning of the mating slot of the stepper motor (11A), flexible coupling (20H) with threaded portion (20I) relative to the Y axis structural base (20A) as already described, however with the presence of a Z axis translation module (21) slidably positioned on a smooth bar (22) that intersects the Y axis translation module (20).

[0046] As can be seen in Figures 8C and 8D, the two smooth slide bars (23) of the Y-axis translation module (20) and, consequently, the Z-axis translation module (21) are inserted into the X-axis translation module (19) and cross the Y-axis translation module (20), thereby allowing reciprocating movement of the Y-axis translation module (20) in the same manner as in the Z-axis translation module (21).

[0047] FIG. 8E shows the positioning of the mating slot of the stepper motor (11A), flexible coupling (19F) with threaded portion (19G) to the X-axis structural base (19A) as already described, however with the presence of a Y-axis translation module (20) together with a Z-axis translation module (21) slidably positioned on a smooth bar (23) that crosses the X-axis translation module (19).

[0048] The CNC translation mechanism (3) includes a camera (2) that is positioned on the inner surface of the X-axis structural base (19A) of the X-axis translation module (19), as illustrated in Figures 9A and 9B.

[0049] Finally, according to the present invention, as shown in FIG. 10A, with respect to the components of the CNC movement mechanism (3), the electronic circuit board component (13), e.g., ShieldRAMPS, and the hardware electronic platform (10), e.g., in an Arduino Mega, reduced size computer part (6) in a single board such as a Raspberry Pi board, are located on the outer surface of the X-axis structural base (19A) of the X-axis movement module (19), resulting in a maximal combination of the CNC movement mechanism (3), as shown in FIG. 10B.

[0050] Besides the CNC movement mechanism (3), the portable device (1) for automated cosmetic application includes, in a non-limiting manner, in an essential embodiment of the invention, a casing (4) that receives and encases said movement mechanism (3) and a base support (5), as can be seen in Figures 11A and 11B, although other embodiments with auxiliary supports are nevertheless made possible by the invention.

[0051] In an alternative embodiment of the present invention, a portable device (1) for automatic application of cosmetics, which allows for greater stability in image capture and application of cosmetics to a user and prevents undesired movement of the user's head, comprises a user head support (24) attached to one end of a base support (5), as seen in Figures 12A and 12B. The user head support (24) also includes a recess (25) for accommodating the user's chin, providing greater support for the user's head.

[0052] Furthermore, in another alternative embodiment of the present invention, the portable device (1) for automatic cosmetic application can be oriented, for example, for test use or demonstration of automatic cosmetic application. In this case, the use of a mannequin's head is essential, and consequently, said portable device (1) for automatic cosmetic application can include a mannequin's head support (26) attached to one end of the base support (5), as seen in Figures 13A and 13B. The mannequin's head support (26) can then also constitute a vertical structure (27) that aids in the fixation of the mannequin's head.

[0053] Regarding the fixation of the CNC moving mechanism (3) to the base support (5), this can be done using the locks (28) present in said base support, as can be seen in Figures 14A and 14B.

[0054] Next, with regard to the fixation of the base support (5) to the user's head support (24) or to the mannequin's head support (26) (this embodiment illustrated in Figures 15A and 15B), said fixation exists using a box between the wings (29) of the support (24 or 26) and the openings (30) of the base support (5) or another fixation type.

[0055] Finally, as regards the casing (4), as can be seen in Figures 16A to 16C, it can be made up of one or more parts (which in this case can be fixed by slots or by screws) that are fixed by means of slots or other fixing means to a base support (5) that receives and encases the CNC movement mechanism (3), thus making it possible, according to the present invention, to create a portable device (1) for automatic cosmetic application.

[0056] Therefore, in light of all the above, the present invention is based on the control of a CNC structure using the output generated from a computer vision model that performs the process of segmenting an area such as lips, starting from image capture. Generally speaking, the structure of the solution including the use of a computer vision model is shown in a simplified diagram in FIG. 17. In particular, the first point of the system refers to a camera (2) responsible for the capture of an image that will be provided as input for an existing computer vision model in a computer vision processing module. The computer vision model will then be responsible for segmenting the lips and providing this information to a logic recognition component of the image spatial mapping logic, translating and generating a list of commands of the code G type for controlling the CNC movement mechanism (3). With the present invention, an integration is possible between the computer vision component and the CNC movement mechanism (3), thereby enabling an end-to-end operation from image capture to image segmentation such as lips, mapping and translation of areas for applying cosmetic materials, and the movement of the movement mechanism (3) for pigmentation of areas such as lips.

[0057] With respect to computer vision processing, the present invention also provides a post-processing approach for segmenting an image of human lips with the goal of conveying this information to a user for the mobile process of automated cosmetic application.

[0058] In a post-processing approach to lip segmentation, by means of a lip image segmentation model (MaskR-CNN), it is possible to detect and segment human lips with the purpose of conveying this information to the cosmetic application process by a CNC moving mechanism (3).

[0059] The process of lip image segmentation is visually demonstrated through the images detailed in Figures 18A through 18H.

[0060] FIG. 18A shows an image of lips at the first stage of computer vision processing, where a computer vision model is trained to detect and segment the image of lips.

[0061] Meanwhile, Fig. 18B shows an image of lips through Mask R-CNN segmentation to demonstrate the results of the prediction of the model of the deep learning architecture applied in the above Mask R-CNN segmentation for lip detection and segmentation. As shown in this image in Fig. 18B, the segmentation is made into two classes, upper lip and lower lip, consecutively. This computer vision processing step also considers an upper limit of accuracy of at least 90% for each class, because below this value important information in the lips, such as Cupid's bow, is lost.

[0062] After the lip segmentation step, a binary image is generated with values ​​of 0 and 1, where value 0 represents all information not referring to the background and lips, and value 1 includes information at the lips. An example of this result of lip segmentation through the Mask R-CNN segmentation method is shown in Figure 18C.

[0063] Given that the apparatus and method for automated application of cosmetic products to a user in accordance with the present invention constitutes an integration between computer vision, hardware and CNC motion mechanisms, a post-processing step must be used to apply computer vision filters to enhance the application of lipstick, as described in detail above.

[0064] A post-processing step is performed to remove possible jagged edges from the lip segmentation step, resulting in a smoother contour. With a more complex contour, the upper lip image segmentation is more difficult to process and more image transformations are required. Then the lower lip image segmentation is easier to process, so the process is also simplified.

[0065] The post-processing of the image segmentation of the upper lip is carried out according to the following steps: first, the points are defined on the horizontal axis where the image segmentation starts and ends, and the position of the point of high relief of the Cupid's bow is calculated and determined. The position of each of the reliefs of the Cupid's bow of the image segmentation of the lips is then used as a base for determining the position of the maximum descent point, from which point the image segmentation of the upper lip is divided into two parts on the vertical axis, the reason for using the Cupid's bow as a metric is that this relief is unnecessarily filled. Each one of the parts of the image segmentation of the lips is processed separately using polygons and taking advantage of the convex casing to generate smoother contours that are filled. The choice to use polygons is due to the fact that the convex casing generates the main polygonal contour in and of itself, so it is not a problem to use polygons to fill in all the contours. Finally the two parts of the image segmentation of the upper lip are combined to form, resulting in the expected image result. 18D and 18E show images of the upper lip segmentation with and without the application of a post-processing step, respectively.

[0066] Furthermore, the post-processing step of the lower lip image segmentation occurs in a similar manner to that of the upper lip image segmentation, but in a simplified manner, since this part of the lower lip image does not show the main complexities of relief, it is not necessary to separate in any axis and the post-processing is performed on the complete segmentation. In a first step, a dilation of the lower lip image is performed to fill in potential defects that would affect the next step. Then, similar to the post-processing of the upper lip image segmentation described above, a convex casing is generated to soften the contours and clean up potential defects and cracks, a fill using polygons is performed, and erosion is applied to return to the pre-dilator size and generate the final expected image result. Figures 18F and 18G show images of the lower lip segmentation with and without the application of the post-processing step, respectively.

[0067] Finally, having completed the post-processing step of upper and lower lip image segmentation, the removal of possible sawtooth from the lip image segmentation step is perceptible as can be seen in FIG. 18H, which shows the resulting image of lip segmentation through the application of the post-processing step in addition to the Mask R-CNN segmentation technique.

[0068] Therefore, with the improved image processing techniques as described above, which are crucial for the device and method of automatic application of cosmetics to a user, an integrated synergistic technology effect is enabled between computer vision, hardware components, and CNC motion mechanism, allowing the computer vision model of image segmentation to perform image spatial mapping logic and provide improved information for performing conversion and generation of code G files for describing paths and CNC drives, so that a complete motion is enabled for the CNC motion mechanism according to the present invention.

Claims

1. 1. An apparatus for automatically applying cosmetics to a user, comprising a portable device (1) for automatic application of cosmetics, comprising a casing (4) for receiving and enclosing a movement mechanism (3) by computer numerical commands (CNC) through a list of movements based on a pre-programmed sequence of commands in G-code, and a base support (5), The CNC moving mechanism (3) A reduced size computer unit (6) on a single board such as a Rasberry Pi4 board for sending electronic driving signals to an instantaneous luminosity generating unit (7), for a light signal unit (8), and for a buzzer (9), for image capture processing, for accessing a cloud programming interface, and for sending movement commands for a hardware electronic platform (10) such as an Arduino MEGA board; said hardware electronic platform (10) controlling the drives of the CNC and step motors (11) of the mechanical movement device (3) for three axes (X, Y and Z) and associated sensors and actuators according to an end stroke switch (12); an electronic circuit board (13) such as a ShieldRAMPS with auxiliary connections to perform the electronic connection interface between the hardware electronic platform (10) and electronic components including the CNC machine movement device (3) such as actuation drivers, stepper motors (11) and end stroke switches (12); For an electronic interface between the electronic circuit board (13) and the step motor (11) used for the movement of the CNC machine movement device (3), a driver (15) such as A4988 that regulates the emission of electronic signals of the hardware electronic platform (10) so that they can be used to control the movement of the step motor (11); said step motors (11) for the movement of said CNC mechanical moving device (3) along three axes of degrees of freedom X, Y and Z, such as two step motors (11A) of type NEMA-17, one on the X-axis and one on the Y-axis, and a step motor (11B) of type 28BYJ-48 for movement on the Z-axis; the end stroke switch (12) positioned at the origin of the Cartesian axes X, Y and Z of the CNC machine movement device (3); said buzzer (9) for sounding to indicate to a user the operating status of said portable device (1) for automatic application of said cosmetic product; said light signalling arrangement (8), such as an LED of an interface, indicating the operational status of the portable device (1) for automatic application of said cosmetic product to said user; A driver (16) such as a user button for initiating the image capture process, image segmentation such as lips, and pigment application; An instantaneous luminosity generating unit (7) such as an LED flash; A camera (2) for capturing user images to serve as an input for a computer vision model such as lips, coupled with a reduced size computer part (6) on a single board and the CNC movement mechanism (3); a computer vision processing module including said computer vision model for image segmentation, image spatial mapping logic, and components for performing conversion and generation of G-code files to describe paths and CNC drives for controlling said moving mechanism (3); 1. An apparatus for automatically applying cosmetic to a user, comprising:

2. The CNC movement mechanism includes: an X-axis structural base (19A) having a pair of bearings (19B) aligned axially and inserted into openings (19C) present in a first wall segment (19D) and a second wall segment (19E); a first step motor (11A) receiving on its shaft a flexible coupling (19F) having a thread (19G) of type TR8; an end stroke sensor (19H) positioned on an inner surface (19I) of the first wall segment (19D); Displacing the flexible coupling (19F) having the threaded portion (19G) through the opening (19C) of the first wall segment (19D), displacing the threaded portion (19G) through the bearing (19B) of the second wall segment (19E) and displacing the step motor (11A) on the outer surface (19J) of the first wall segment (19D). an X-axis movement module (19) for positioning the a Y-axis structural base (20A) having a pair of bearings (20B) aligned axially and inserted into openings (20C) present in the first wall segment (20D) and the second wall segment (20E); a threaded nut (20F) inserted into an opening (20G) in a side surface (20H) of the Y-axis structure base (20A) having an axis perpendicular to the alignment axis between the openings (20C); a second step motor (11A) receiving on its shaft a flexible coupling (20I) having a thread (20J) of type TR8; the end stroke sensor (20K) positioned on an upper outer surface (20L) of the first wall segment (20D); the end stroke sensor (20K) positioned on an inner surface (20M) of the first wall segment (20D); displacing a flexible coupling (20I) having said threaded portion (20J) through an opening (20C) in said first wall segment (20D); displacing said threaded portion (20J) through a bearing (20B) of a second wall segment (20E); positioning the step motor (11A) on an outer surface (20N) of the first wall segment (20D); A Y-axis moving module (20) including: A Z-axis structural base (21A) including a first receiving and receiving device (21B) for a step motor (11B), i.e., type 28BYJ-48, and a second receiving and receiving device (21C) for a screw nut (21D); a guide portion (21E) positioned on the first facility (21B) and each step motor (11B), including a passage opening (21F) for the shaft of said step motor (11B), and including a female guide recess (21G) for receiving a male guide protrusion (21H) from a support portion (21I) of the cosmetic application support (18); a gear (21J) positioned on the shaft of the step motor (11B) between the guide part (21E) and the support part (21I), the gear (21J) fitting into the toothed rod of the support part (21I) of the cosmetic application support (18); A Z-axis movement module (21) including: the X-axis translation module (19), the Y-axis translation module (20) and the Z-axis translation module (21) integrated by two smooth sliding bars (22) of the Z-axis translation module (21) inserted into the Y-axis translation module (20) through the Z-axis translation module (21); A slot in combination with the step motor (11A) and a flexible coupling (20H) having a threaded portion (20I) for the Y-axis structural base (20A) with the Z-axis movement module (21) slidably positioned on a smooth bar (22) passing through the Y-axis movement module (20); At least two smooth slidable bars (23) of the Y-axis translation module (20) and, consequently, of the Z-axis translation module (21) inserted through the Y-axis translation module (20) into the X-axis translation module (19); a slot in combination with the step motor (11A) and a flexible coupling (19F) having the threaded portion (19G) to the X-axis structural base (19A) with the presence of the Y-axis translation module (20) having the Z-axis translation module (21) slidably positioned on a smooth bar (23) passing through the X-axis translation module (19); Characterized in that it comprises 13. An apparatus for automatically applying the cosmetic product of claim 1 to a user.

3. characterised in that it comprises an additional fixed lid (21K) attached to the cosmetic application support (18), 3. An apparatus for automatically applying the cosmetic product of claim 2 to a user.

4. the CNC movement mechanism (3) includes a camera (2) positioned on an inner surface of the X-axis structural base (19A) of the X-axis movement module (19), 3. An apparatus for automatically applying the cosmetic product of claim 2 to a user.

5. the electronic circuit board (13), the hardware electronic platform (10) and the reduced-size computer part (6) on a single board are located on an outer surface of the X-axis structural base (19A) of the X-axis movement module (19).

3. An apparatus for automatically applying the cosmetic product of claim 2 to a user.

6. characterised in that the fixing of the CNC movement mechanism (3) to the base support (5) is performed by means of a lock (28) present on the base support.

13. An apparatus for automatically applying the cosmetic product of claim 1 to a user.

7. The portable device (1) for automatic application of cosmetics is characterized in that it comprises a user's head support (24) fixed on one end of the base support (5), 13. An apparatus for automatically applying the cosmetic product of claim 1 to a user.

8. The user's head support portion (24) includes a recess (25) for accommodating the user's chin.

8. An apparatus for automatically applying the cosmetic product of claim 7 to a user.

9. The portable device (1) for automatic application of cosmetics includes a mannequin head support (26) fixed at one end of the base support (5), The mannequin head support (26) includes a vertical structure (27) for fixing the mannequin's head.

13. An apparatus for automatically applying the cosmetic product of claim 1 to a user.

10. the fixing of the base support (5) to the user's head support (24) or the mannequin's head support (26) is by means of a case between the wings (29) of the support (24 or 26) and the opening (30) of the base support (5); 10. A device for automatically applying the cosmetic product of claim 7 or 9 to a user.

11. moving the motion mechanism (3) using computer numerical commands (CNC) along three axes of freedom X, Y and Z through a motion list based on a series of pre-programmed commands in G-code; Using a reduced-size computer unit (6) on a single board, such as a Rasberry Pi4 board, to send electronic drive signals to an instantaneous luminosity generator (7), a light signal unit (8) and a buzzer (9), process image capture, access a cloud programming interface and send movement commands to a hardware electronic platform (10), such as an Arduino MEGA board; using said hardware electronic platform (10) such as an Arduino MEGA board to control sensors and actuators involved in driving the CNC and mechanical movement devices (3) and drive step motors (11) for three X, Y and Z axes according to end stroke switches (12); creating the electronic interface required for the connection between the hardware electronic platform (10) and the electronic components contained in the CNC mechanical movement device (3) such as actuation drivers, the stepper motor (11) and the end stroke switch (12) using an electronic circuit board (13) such as ShieldRAMPS; creating an electronic interface between said electronic circuit board (13), e.g. ShieldRAMPS, and said step motor (11) used for the movement of a CNC machine movement device (3) and coordinating electronic signal transmissions from said hardware electronic platform (10) using a driver (15) that can be used to control the movement of said step motor (11); using said end stroke switches (12) to provide positioning references for said CNC machine movement device (3) each one located at the origin of Cartesian axes X, Y and Z of said CNC machine movement device (3); using said buzzer (9) to generate a sound indicating an operational status of a portable device for automatic application of cosmetic products to said user; creating a light using a light signaling component (8), such as an LED, to indicate an operational status of the portable device (1) for automatic application of cosmetic products to the user; initializing the image capture process, image segmentation and dye application functions using an actuator (16) such as a user button; generating a luminous intensity using an instantaneous luminous intensity generator (7), such as an LED flash; generating an image capture of a user, such as lips, using a camera (2) coupled with a reduced size computer part (6) on a single board and a CNC movement mechanism (3) to provide said image of the user for a computer vision model; in a computer vision processing module, using said computer vision model to create image segmentation and provide this information to components to create image spatial mapping logic, and for conversion and generation of G-code files to describe CNC paths and drives to control said moving mechanism (3); A method for automatically applying a cosmetic product to a user, comprising:

12. The segmentation process using the computer vision model includes: Detecting an image of lips and applying inference of a deep learning architecture model (deep learning) in a method of Mask-CNN segmentation for lip segmentation, The segmentation continues with a first class of upper lips and a second class of lower lips, with an upper limit of at least 90% in accuracy for each class being considered in this computer vision processing step, generating a binary image with values ​​of 0 and 1, where value 0 represents all information not referring to the lips and value 1 includes information on the lips, Implementing post-processing of the upper lip image segmentation to determine at what points on the horizontal axis the image segmentation begins and ends, calculating and determining where the high relief points of Cupid's bow are; using the location of the respective high relief of the Cupid's bow of the lip image segmentation as a basis for determining where the maximum descent point is located, and from this point, dividing the upper lip image segmentation into two parts on a vertical axis, implementing a process of separating each one of the two parts of the upper lip image segmentation to soften the filled contours using polygons and to utilize a convex casing, implementing the junction of the two parts of the upper lip image segmentation, and generating a final image of the upper lip; Implementing post-processing of the image part of the lower lip to generate an extension of the image of the lower lip to fill in possible defects, generating a convex casing to soften the contours to trim defects and gaps, hole filling implemented to leverage polygons, creating an erosion to return the image to the size before the extension of the image of the lower lip, and generating a final image of the lower lip; Characterized in that it comprises 12. A method for automatically applying the cosmetic product of claim 11 to a user.

Images