Robot control method, device, dexterous hand, robot, storage medium and program product

Abstract

This application discloses a control method, device, dexterous hand, makeup robot, storage medium, and program product for a makeup robot, belonging to the technical fields of makeup robots and dexterous hands. The makeup robot includes an execution component; the execution component includes a robotic arm and a dexterous hand disposed at the end of the robotic arm, the execution component being used to grip matching makeup tools; the method includes: determining matching makeup tools based on current makeup requirement instructions, and obtaining gripping parameters corresponding to the matching makeup tools; controlling the execution component to grip the matching makeup tools based on the gripping parameters; determining dipping parameters corresponding to the matching makeup tools; controlling the execution component to perform interactive actions with other makeup products while gripping the matching makeup tools, to dip the matching makeup tools into the required makeup materials; and controlling the execution component to interact with the user's face. The control method for the makeup robot in this application improves the quality of automated makeup operations and the user experience.

Description

Technical Field

[0001] This application belongs to the field of robotics and dexterous hand technology, and particularly relates to a robot control method, device, dexterous hand, robot, storage medium and program product. Background Technology

[0002] With the advancement of service robot technology, automated makeup application has gradually become a research hotspot. However, in related technologies, the automatic gripper-changing function of some high-end models is limited to changing the physical shape of the end effector, without adjusting the underlying control strategy according to the operational characteristics of different tools. Furthermore, they generally lack control logic for complex interactive actions such as cosmetic application and tool cleaning. These control modes in related technologies result in a low overall level of automation, with fragmented operational processes, affecting the automation level and efficiency of makeup robots in performing automated makeup tasks. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a control method, device, dexterous hand, makeup robot, storage medium, and program product for a makeup robot, establishing a closed-loop operation process from tool matching, automated grasping, precise application to facial interaction, thereby improving the makeup quality and user experience of automated makeup operations.

[0004] In a first aspect, this application provides a control method for a makeup robot. The makeup robot includes an execution component, which includes a robotic arm and a dexterous hand disposed at the end of the robotic arm. The execution component is used to grip a matching makeup tool. The method includes: determining the matching makeup tool based on a current makeup requirement instruction and obtaining gripping parameters corresponding to the matching makeup tool; controlling the execution component to grip the matching makeup tool based on the gripping parameters; determining dipping parameters corresponding to the matching makeup tool; the dipping parameters include dipping force and rotation trajectory; controlling the execution component to perform interactive actions with other makeup products while gripping the matching makeup tool, so as to dip the matching makeup tool into the makeup materials required by the matching makeup tool; and controlling the execution component to perform interactive actions with the user's face.

[0005] According to the control method of the beauty robot in this application, the robotic arm and dexterous hand are controlled to grasp the matching makeup tools according to the grasping parameters determined by the current makeup demand instruction, realizing automated tool scheduling and grasping, and improving the stability and safety of the grasping action; at the same time, the robotic arm and dexterous hand are controlled to pick up makeup materials based on the dipping parameters, realizing the automatic dipping of makeup materials in an appropriate amount and evenly; the method establishes a closed-loop operation process from tool matching, automated grasping, precise dipping to facial interaction, which helps to realize intelligent and continuous automatic makeup operation of beauty robots, and improves the makeup quality and user experience of automatic makeup operation.

[0006] According to one embodiment of this application, determining the dipping parameters corresponding to the matched makeup tool includes: acquiring the texture characteristics of other cosmetic products that perform interactive actions with the matched makeup tool held by the robotic arm; and determining the dipping parameters based on the texture characteristics.

[0007] According to one embodiment of this application, determining the dipping parameters based on the texture characteristics includes: when the texture characteristic is a liquid texture, determining the dipping force in the dipping parameters as a first dipping force; when the texture characteristic is a powder texture, determining the dipping force in the dipping parameters as a second dipping force; and when the texture characteristic is a paste texture, determining the dipping force in the dipping parameters as a third dipping force; wherein the first dipping force is less than the third dipping force, and the second dipping force is less than the third dipping force.

[0008] According to one embodiment of this application, the rotation trajectory includes a rotation speed and a number of rotations; determining the dipping parameters based on the texture characteristics includes: when the texture characteristics are liquid, determining the rotation speed as a first rotation speed and the number of rotations as a first number of rotations; when the texture characteristics are powdery, determining the rotation speed as a second rotation speed and the number of rotations as a second number of rotations; when the texture characteristics are paste-like, determining the rotation speed as a third rotation speed and the number of rotations as a third number of rotations; wherein, the first rotation speed is less than the third rotation speed, the third rotation speed is less than the second rotation speed; the first number of rotations is less than the second number of rotations, and the second number of rotations is less than the third number of rotations.

[0009] According to one embodiment of this application, the grasping parameters include at least one of the following: hand-eye calibration matrix, target angle sequence of each joint of the dexterous hand, and grasping force threshold.

[0010] According to one embodiment of this application, the step of controlling the execution component to grasp the matching makeup tool based on the grasping parameters includes: recognizing the tool image corresponding to the matching makeup tool to obtain the initial position information of the matching makeup tool; converting the initial position information into target position information in the base coordinate system of the robotic arm based on the hand-eye calibration matrix; controlling the robotic arm to move to the position of the matching makeup tool based on the target position information; controlling the joints of the dexterous hand to adjust their posture and close based on the target angle sequence to cover and grasp the matching makeup tool, and obtaining a force feedback value; and determining that the grasping action is completed when the force feedback value reaches the grasping force threshold.

[0011] According to one embodiment of this application, controlling the execution component to perform interactive actions with the matching makeup tool and other cosmetic products based on the dipping parameters includes: obtaining the dipping force and rotation trajectory based on the dipping parameters; when the robotic arm moves above the other cosmetic products, controlling the execution component to drive the matching makeup tool to move along a first direction toward the surface of the other cosmetic products, and performing smooth interactive control in the first direction until the actual interaction force between the matching makeup tool and the surface of the other cosmetic products reaches the dipping force; wherein, the first direction is a direction perpendicular to the surface of the other cosmetic products; and controlling the execution component to drive the matching makeup tool to move on a target plane based on the dipping force and the rotation trajectory; the target plane is a plane formed by a second direction and a third direction on the surface of the other cosmetic products; wherein, the second direction intersects the third direction.

[0012] According to one embodiment of this application, after controlling the execution component to perform interactive actions with the matching makeup tool and other makeup products based on the dipping parameters, the method includes: if the dipping amount of the other makeup products does not meet the makeup requirements, controlling the execution component to repeatedly perform the interactive actions with the other makeup products.

[0013] Secondly, this application provides a control device for a makeup robot, the makeup robot including an execution component, the execution component including a robotic arm and a dexterous hand disposed at the end of the robotic arm, the execution component being used to grip matching makeup tools; the device includes: The first processing module is used to determine the matching makeup tool based on the current makeup requirement instruction and obtain the grasping parameters corresponding to the matching makeup tool; the second processing module is used to control the execution component to grasp the matching makeup tool based on the grasping parameters; the third processing module is used to determine the dipping parameters corresponding to the matching makeup tool; the dipping parameters include: dipping force and rotation trajectory; the fourth processing module is used to control the execution component to perform interactive actions with other makeup products based on the dipping parameters, so as to dip the matching makeup tool into the makeup materials required by the matching makeup tool; the fifth processing module is used to control the execution component to perform interactive actions with the user's face.

[0014] According to the control method of the beauty robot in this application, the robotic arm and dexterous hand are controlled to grasp the matching makeup tools according to the grasping parameters determined by the current makeup demand instruction, realizing automated tool scheduling and grasping, and improving the stability and safety of the grasping action; at the same time, the robotic arm and dexterous hand are controlled to pick up makeup materials based on the dipping parameters, realizing the automatic dipping of makeup materials in an appropriate amount and evenly; the method establishes a closed-loop operation process from tool matching, automated grasping, precise dipping to facial interaction, which helps to realize intelligent and continuous automatic makeup operation of beauty robots, and improves the makeup quality and user experience of automatic makeup operation.

[0015] Thirdly, this application provides a dexterous hand, which is configured as an end effector of a beauty robot and installed at the end of the robot's robotic arm for gripping matching makeup tools; The dexterous hand operates based on the control method for the beauty robot as described in the first aspect above.

[0016] Fourthly, this application provides a beauty robot, comprising: a dexterous hand; the dexterous hand for holding matching makeup tools; a control device for the beauty robot as described in the second aspect; the control device for the beauty robot being connected to the dexterous hand.

[0017] According to one embodiment of this application, the beauty robot includes: a robotic arm; a dexterous hand is mounted at the end of the robotic arm; and the robotic arm is electrically connected to the control device of the beauty robot.

[0018] Fifthly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the control method for the beauty robot as described in the first aspect above.

[0019] Sixthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the control method for the beauty robot as described in the first aspect above.

[0020] The above-described one or more technical solutions in the embodiments of this application have at least one of the following technical effects: Based on the grasping parameters determined by the current makeup requirements, the robotic arm and dexterous hand are controlled to grasp the matching makeup tools, realizing automated tool scheduling and grasping, and improving the stability and safety of the grasping action. At the same time, based on the dipping parameters, the robotic arm and dexterous hand are controlled to dip the makeup materials, realizing the automatic dipping of makeup materials in appropriate and even amounts. This method establishes a closed-loop operation process from tool matching, automated grasping, precise dipping to facial interaction, which helps to realize intelligent and continuous automatic makeup operation by beauty robots, and improves the makeup quality and user experience of automatic makeup operation.

[0021] Furthermore, by guiding the robotic arm through visual positioning and controlling the dexterous hand's posture using a target angle sequence, the robotic arm and dexterous hand collaboratively grasp makeup tools, thus highly replicating the natural posture of a human holding tools. Simultaneously, by comparing force feedback values ​​with grasping force thresholds to control the grasping force, the stability and safety of the grasp are improved. This method achieves automated management of the entire process from tool recognition and strategy scheduling to precise grasping, and improves the makeup quality and user experience of automated makeup operations.

[0022] Furthermore, by adaptively matching the dispensing force and rotation trajectory based on the texture characteristics of cosmetics, a reliable control benchmark is provided for subsequent adaptive and precise material dispensing. This method employs multi-dimensional parameter configurations for different textures such as powders, creams, and liquids. While improving dispensing accuracy, it effectively reduces the risk of damaging powders or splashing liquids, meets the friction softening conditions required for creams, enhances the adaptability and smoothness of makeup tools when interacting with various forms of cosmetic materials, and improves the makeup quality and user experience of automated makeup operations.

[0023] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0024] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is one of the flowcharts illustrating the control method for the beauty robot provided in this application embodiment; Figure 2This is a second schematic flowchart of the control method for the beauty robot provided in the embodiments of this application; Figure 3 This is the third flowchart illustrating the control method for the beauty robot provided in this application embodiment; Figure 4 This is the fourth flowchart illustrating the control method for the beauty robot provided in this application embodiment; Figure 5 This is the fifth flowchart illustrating the control method for the beauty robot provided in the embodiments of this application; Figure 6 This is the sixth flowchart illustrating the control method for the beauty robot provided in this application embodiment; Figure 7 This is the seventh flowchart illustrating the control method for the beauty robot provided in this application embodiment; Figure 8 This is the eighth flowchart illustrating the control method for the beauty robot provided in this application embodiment; Figure 9 This is a schematic diagram of the control device for the beauty robot provided in the embodiments of this application; Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0026] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0027] The control method, device, dexterous hand, beauty robot, storage medium, and program product of the present application embodiment will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0028] The control method for a beauty robot provided in this application embodiment can be executed by a beauty robot or functional modules or entities within the beauty robot that can implement the control method, such as a task state machine, makeup quality assessment module, safety monitoring module, visual processing thread, local trajectory planner, or central control algorithm main process. Alternatively, it can be a server that communicates and connects with the beauty robot. The following description uses a beauty robot as the executing entity to illustrate the control method for a beauty robot provided in this application embodiment.

[0029] In some embodiments, the makeup robot includes an execution component. In this embodiment, the execution component includes a robotic arm and an end effector disposed at the end of the robotic arm; the end effector is used to grip, mount, or connect matching makeup tools. It is understood that, to adapt to makeup operations using different materials and processes, the end effector can be a gripping mechanism (such as a dexterous hand, a multi-finger mechanical gripper, a flexible clamp, etc.) for gripping makeup tools.

[0030] The following description uses robotic arms and dexterous hands as representative execution components, in conjunction with the beauty robot control method of this application. In some embodiments, the matched makeup tool refers to an end-effector with specific physical form and material properties that matches the current makeup requirements. In actual execution, the specific type of the makeup tool is determined by the type of makeup technique in the current makeup requirement instruction, the characteristics of the target facial area, and the expected physical interaction constraints (e.g., a fine-tipped brush suitable for local line repair, a filling brush or foundation brush suitable for large-area initial coloring or uneven color filling, and a blending stick or sponge beauty blender suitable for edge blurring).

[0031] During the research and development process, the inventors discovered that mainstream architectures in related technologies often employ general-purpose robotic arms equipped with end-effector makeup tools to automate makeup application for users. However, the human face is a complex, dynamic, non-rigid, irregular surface with minute deformations, and the physical hardness varies significantly between different areas such as the forehead, cheekbones, and cheeks. When makeup tools come into contact with hard areas of the face, the control parameters may not be able to adapt to the local bone hardness, leading to excessive force from the robotic arm or even vibration. Conversely, when contacting soft areas, insufficient adaptability can cause interruptions in force application or application, making it difficult to maintain a consistent and smooth dynamic fit. This can result in inconsistent application pressure, uneven makeup, and compromises on safety and user experience. Furthermore, complete automated makeup application involves multiple steps, including foundation application, eyeshadow blending, and lipstick application. The materials of the tools used in different steps (such as soft sponges, flexible brushes, and hard silicone pens) and the required interaction techniques vary. The single, fixed control logic in related technologies cannot meet these diverse makeup effect requirements, affecting the successful implementation of complex, multi-step makeup looks. To address the issue of low makeup quality after automated makeup application, the inventors, through in-depth research, designed a control method for a makeup robot. The makeup robot includes an execution component, comprising a robotic arm and a dexterous hand at the end of the robotic arm. The execution component is used to hold matching makeup tools. The method includes: acquiring a current makeup requirement instruction while the execution component is holding the matching makeup tools to apply makeup to a user; configuring target compliant interaction parameters based on the current makeup requirement instruction; the target compliant interaction parameters include: target motion inertia parameters, target motion resistance parameters, and target contact elasticity parameters; and controlling the movement of the execution component based on the target compliant interaction parameters. According to the control method for the makeup robot provided in this application, by acquiring the current makeup requirement instruction and configuring target compliant interaction parameters including target motion inertia parameters, target motion resistance parameters, and target contact elasticity parameters, the movement of the robotic arm and dexterous hand can be controlled. This allows for adaptation to different underlying mechanical logic based on different makeup requirements, helping the makeup tools to better conform to facial contours to alleviate stiffness in movement, while maintaining relative stability in the overall trajectory, thus better meeting the differentiated requirements of interaction intensity for different makeup tasks. The above methods facilitate smooth interaction between makeup tools and the user's face, and improve the makeup quality and user experience of automated makeup operations.

[0032] like Figure 1 As shown, the control method of the beauty robot includes steps 110, 120 and 130.

[0033] Step 110: When the user is applying makeup using the matching makeup tools held by the component, obtain the current makeup requirement instruction. In this step, it can be understood that when performing a complete makeup process, multiple makeup steps are involved. The user's facial area, the makeup tools used to apply makeup to the facial area, and the mechanical interaction characteristics between the makeup tools and the corresponding facial area may all be different under different steps.

[0034] The current makeup requirement instruction is generated based on the current makeup needs. The makeup needs characterize the mechanical interaction characteristics of the corresponding makeup procedures and related areas at the current makeup moment. The current makeup requirement instruction can be continuously and dynamically updated based on the makeup actions. The current makeup requirement instruction is a comprehensive set of information including at least one of the following: the specific makeup task the makeup robot is currently performing, the physical interaction constraints within the makeup task, the expected presentation characteristics of the target makeup effect, and the user's personalized attributes. The current makeup requirement instruction may include, but is not limited to: specific makeup task-related instructions: makeup technique type (e.g., foundation application, eyeshadow blending); makeup tools and their material properties (e.g., soft makeup sponge, elastic brush); makeup area and target facial region characteristics (e.g., harder or softer facial regions; where harder facial regions include the forehead area with a higher proportion of bone; softer facial regions include the facial region with a higher proportion of muscle). Physical interaction constraint-related instructions: specific interaction techniques matched to the current procedure (e.g., different mechanical behaviors such as rubbing, light stroking, and dotting). The expected presentation of features and personalized attributes related instructions includes: target makeup thickness, desired color saturation, a safe threshold range for force application based on the user's skin type (e.g., sensitive skin), and timeliness requirements (e.g., detailed slow makeup or quick simple makeup). It is understood that the feature parameters of various dimensions involved in the actual operation of the makeup robot, such as process execution details, physical environmental constraints, equipment operating status, and dynamic user interaction feedback, can all serve as components of the current makeup requirement instruction. During actual execution, the current makeup requirement instruction can be automatically generated based on a pre-built standard makeup process database, the user's personalized digital makeup profile, or historical usage records; it can also be dynamically configured based on personalized setting parameters input by the user through a human-computer interaction interface (e.g., touch terminal, mobile application, voice interaction module, etc.); in some embodiments, it can also be derived through adaptive planning by combining user facial state information collected by visual sensors. This application does not strictly limit the specific source and triggering method of the current makeup requirement instruction. In some embodiments, the current makeup requirement instruction may also include extended evaluation parameters such as target makeup thickness, desired color saturation, preset safe threshold range for force application, or evenness index of local application, used for subsequent quality analysis and makeup repair.

[0035] Step 120: Based on the current makeup requirement command, configure the target compliant interaction parameters. The target compliant interaction parameters are adaptively generated according to the current makeup requirement command, and are kinematic characteristic parameters such as force, elasticity, and speed related to the mechanical interaction characteristics required by the current makeup process and area, used to guide the movement of the robotic arm. The target compliant interaction parameters include at least one of the following: target motion inertia parameters, target motion stagnation parameters, and target contact elasticity parameters.

[0036] The target compliant interaction parameters are primarily used to construct a compliant control dynamics model (e.g., a second-order dynamics model) for the interaction between the robotic arm's end effector and the user's face. By dynamically adjusting the parameters in the target compliant interaction parameters, the beauty robot can simulate the robotic arm's end effector as a compliant control end with specific physical characteristics, enabling the robotic arm to adaptively switch between position control and force control. This allows the end effector carrying makeup tools to have a compliant feel similar to that of a professional makeup artist, maintaining the accuracy of the makeup trajectory while improving the safety and comfort of the interaction between the robotic arm and the face.

[0037] It should be noted that although the embodiments of this application are illustrated using a standard second-order dynamic model as an example, in practical applications, this dynamic model is not limited to a standard linear second-order model. It can also be a variable-parameter dynamic model that introduces nonlinear terms, a higher-order dynamic model that includes higher-order derivatives of motion (such as jerk), or a control model based on fractional calculus. Furthermore, other control logics that achieve compliant interaction by adjusting inertia, energy dissipation, and elastic recovery characteristics also fall within the scope of protection of this invention.

[0038] The target motion inertia parameter, measured in kilograms (kg), is used to simulate the virtual inertia of the robotic arm's end effector, determining its response speed and inertial retention capacity to external forces. In practical applications, a smaller target motion inertia parameter indicates less inertia at the robotic arm's end effector, resulting in a more sensitive response to force changes. Conversely, a larger target motion inertia parameter indicates greater inertia at the end effector, smoother movement, and a relatively increased response delay. The target motion stagnation parameter, measured in Newton-seconds per meter (N·s / m), simulates the robotic arm's resistance to motion stagnation, similar to the viscous resistance experienced by an object moving in a viscous medium. By adjusting the target motion stagnation parameter, the degree to which the robotic arm absorbs collision energy at its end effector can be controlled, thereby suppressing large-amplitude oscillations generated at the moment of force application or contact. In actual execution, a larger target motion stagnation parameter makes it easier for the robotic arm's end effector to brake, resulting in more stable and smooth motion. A smaller target motion stagnation parameter indicates weaker motion resistance at the end effector, facilitating faster response and more flexible and efficient spatial transfer. The target contact elasticity parameter, measured in Newtons per meter (N / m), simulates the elastic recovery capability of the robotic arm's end effector, similar to the stiffness coefficient of a spring, determining the degree to which the robotic arm resists deformation from external forces. In practical applications, the smaller the target contact elasticity parameter, the easier it is for the robotic arm to adapt to the guidance of external forces and move accordingly; the larger the target contact elasticity parameter, the stronger the robotic arm's position holding and trajectory tracking capabilities. For example, if the current makeup requirement command corresponds to the blush "base application" step, a lower contact elasticity parameter and a higher speed can be set in the target smoothness interaction parameters to achieve rapid color application; if the current makeup requirement command corresponds to the blush "blending" step, the speed can be reduced and the contact elasticity parameter fine-tuned to increase the contact time and simulate a manual blending technique.

[0039] In some embodiments, step 120 includes: determining the makeup operation parameters for the current makeup stage, the contact state between the matched makeup tool and the user's face, the physiological characteristics of the user's face, and the characteristics of the matched makeup tool based on the current makeup requirement instruction. Based on the makeup operation parameters, contact state, and physiological characteristics, and in conjunction with the characteristics of the makeup tool, the target smooth interaction parameters are configured. The makeup operation parameters are used to determine the scale and technique attributes of the makeup operation in the current makeup stage, such as large-area routine application during initial makeup application, localized fine touch-ups during retouching, or the size of the makeup area. Analyzing the makeup operation parameters can clarify the control objective of the current makeup stage and determine the target motion inertial parameters, enabling the robotic arm to switch corresponding dynamic behaviors when performing different makeup tasks.

[0040] Contact status is used to determine whether the makeup tool held by the robotic arm is currently physically interacting with the user's face. This helps determine whether the current control objective is to improve interaction compliance or motion flexibility, thus determining the target motion resistance parameters. By detecting this contact status, it can be determined whether the robotic arm needs to absorb contact impact or reduce motion resistance. For example, if the contact status indicates contact, the current control objective is to improve interaction compliance; if the contact status indicates no contact, the current control objective is to improve motion flexibility. The physiological characteristics of the user's face are used to characterize the geometry and biomechanical properties of the current makeup area. This clarifies whether the control objective for the current stage is to maintain makeup trajectory accuracy or achieve soft contact compliance, thus determining the target contact elasticity parameters. For example, by combining the surface hardness and curvature of the current facial area (such as the soft, undulating cheek area or the relatively hard and flat forehead bone protrusion area), the difference in the degree of resistance to external force deformation in different areas can be perceived, allowing the robotic arm to maintain makeup trajectory accuracy in harder areas and achieve soft contact compliance in softer areas. The characteristics of makeup tools primarily characterize the physical and mechanical properties of the selected tools, such as the material's hardness (e.g., a fluffy soft-bristled brush, a dense sponge beauty blender, or a hard, thin eyebrow pencil), overall weight, and center of gravity distribution. By combining these characteristics, the values ​​of target motion inertia parameters, target motion resistance parameters, and target contact elasticity parameters can be fine-tuned, allowing the robotic arm's force control performance to adapt to the actual force differences experienced by different physical tools. In actual execution, the characteristics of makeup tools and makeup operation parameters can be acquired during the initialization phase of the beauty robot or before applying makeup to the user. For example, the specific characteristic parameters of different tools can be accessed by reading a pre-built digital makeup tool library, or preset through user-defined input, experimental calibration data, and historical experience. In other actual execution processes, the beauty robot can also adaptively adjust the aforementioned parameters based on visual feedback data and force sensor data during the makeup process. For example, a beauty robot can fine-tune and iteratively optimize the values ​​of target smooth interaction parameters, the movement speed of makeup tools, or the control strength of makeup tools in the current makeup stage based on the actual force curve recorded after the makeup tool dips into the makeup product or applies it to the user's face, or based on the makeup quality inspection score generated after each makeup session.

[0041] In some embodiments, makeup operation parameters include: makeup technique type, makeup area, and makeup tool characteristics; contact status includes: whether the matched makeup tool has made contact with or not with the user's face; physiological characteristics include the surface hardness and curvature of the current makeup area. In actual execution, to ensure that the force control performance of the robotic arm and dexterous hand can accurately adapt to complex makeup conditions, a multi-dimensional parameter mapping logic can be constructed to establish a correspondence between the different categories of input information and the corresponding target compliant interaction parameters.

[0042] In some embodiments, the target compliant interaction parameters are configured based on makeup operation parameters, contact state, and physiological characteristics, combined with the characteristics of makeup tools. This includes: determining the target motion inertia parameters based on the makeup process type, makeup area, and makeup tool characteristics in the makeup operation parameters; determining the target motion resistance parameters based on the contact state and makeup tool characteristics; and determining the target contact elasticity parameters based on surface hardness, curvature, and makeup tool characteristics.

[0043] Based on the physiological characteristic of the cheek as the application site and the relatively soft nature of the chosen tool, a blush brush, a low target contact elasticity parameter is calculated and configured. This allows the robotic arm to achieve a comfortable, soft contact that conforms to the natural curves of the face, much like a soft spring. By dynamically configuring a low target contact elasticity parameter to suit the tool characteristics of the blush brush and the soft skin of the cheek area, the robotic arm exhibits extremely high compliance, conforming to the undulations of the face like a soft spring for a comfortable, soft contact. When the blush brush is detected to be in contact with the face, an appropriate target motion resistance parameter is configured to effectively absorb high-frequency vibrations during movement, resulting in a smoother and more fluid application process. Depending on the process attributes of different makeup stages, such as initial coloring (large-area base application) or deep blending (local fine-tuning), the target motion inertia parameter can be adaptively adjusted to meet the dynamic inertial requirements of long-trajectory stability or short-trajectory sensitivity. During blush application, the actual force fed back by the force sensor can be compared with the preset ideal force (e.g., 0.6N) to further adjust the force control performance of the robotic arm and dexterous hand, ensuring that the force between the makeup tool and the user's face remains constant. In actual execution, a lower contact elasticity parameter and a higher speed are set during the blush "base" stage to achieve rapid color application; during the "blending" stage, the speed is automatically reduced and the contact elasticity parameter is finely adjusted to increase the contact time and simulate a manual blending technique.

[0044] According to the control method for the beauty robot provided in this application, when configuring target smooth interaction parameters, the target motion inertia parameters are determined based on the type and area of ​​the makeup process to adapt to the operational inertia; the target motion retardation parameters are determined based on the contact state to smooth out sudden force changes and vibrations; and the target contact elasticity parameters are determined based on surface hardness and facial curvature to conform to the softness and hardness of the facial contours. This multi-dimensional parameter configuration strategy facilitates smooth interaction between makeup tools and the user's face, and improves the makeup quality and user experience of automated makeup operations.

[0045] In some embodiments, determining a target motion inertia parameter based on the makeup technique type, makeup area, and makeup tool characteristics in the makeup operation parameters includes: if the goal of the current makeup stage is to improve the stability of the makeup process based on the makeup technique type and makeup area, then determining a first target motion inertia parameter in conjunction with the makeup tool characteristics; if the goal of the current makeup stage is to improve the sensitivity of the makeup process based on the makeup technique type and makeup area, then determining a second target motion inertia parameter in conjunction with the makeup tool characteristics. In this embodiment, the first target motion inertia parameter is greater than the second target motion inertia parameter.

[0046] In actual execution, based on the type of makeup technique and the area covered, when the goal of the current makeup stage is to improve the stability of the makeup process (e.g., applying makeup to a large area of ​​the face), a first target motion inertia parameter (e.g., 2.0-5.0 kg) is configured, taking into account the characteristics of makeup tools such as foundation brushes, blush brushes, or beauty sponges, to filter high-frequency vibrations during movement and improve the stability of long-trajectory movements. When the goal of the current makeup stage is to improve the sensitivity of the makeup process (e.g., making small-area fine touch-ups for local blemishes), a relatively small second target motion inertia parameter (e.g., 0.5-1.0 kg) is configured, taking into account the characteristics of makeup tools such as fine-tipped brushes, filler brushes, or eyeliners, to reduce the virtual inertia at the end of the robotic arm. This allows the robotic arm to quickly follow minute changes in force feedback, thereby achieving rapid and precise local following response. According to the control method of the beauty robot provided in this application embodiment, when determining the target motion inertia parameter, the magnitude of the motion inertia parameter can be adaptively adjusted according to different operational goals corresponding to the type of makeup technique and the area covered. When the goal of the current makeup stage is to improve process smoothness, a larger primary target motion inertia parameter is configured to increase virtual inertia. This helps to smooth out external disturbances during the interaction process and maintain smooth and continuous large-area or basic application movements. When the goal is to improve process sensitivity, a smaller secondary target motion inertia parameter is configured to reduce virtual inertia, enabling more agile responses to minute changes in force and meeting the higher requirements for force control response speed in detailed local processes. This dynamic configuration strategy, which balances macroscopic motion smoothness and microscopic response sensitivity, facilitates smooth interaction between makeup tools and the user's face, improving the makeup quality and user experience of automated makeup operations.

[0047] In some embodiments, determining the target motion resistance parameter based on the contact state and the characteristics of the makeup tool includes: if, based on the contact state, the goal of the current makeup stage is to improve the smoothness of makeup interaction, then, in conjunction with the characteristics of the makeup tool, determining the target motion resistance parameter as a first target motion resistance parameter; if, based on the contact state, the goal of the current makeup stage is to improve the flexibility of makeup movements, then, in conjunction with the characteristics of the makeup tool, determining the target motion resistance parameter as a second target motion resistance parameter. The first target motion resistance parameter is greater than the second target motion resistance parameter. Based on the contact state, when the goal of the current makeup stage is to improve the smoothness of the makeup interaction, such as when the contact state indicates that the makeup tool has made contact with the user's face or when a large area of ​​the face is being applied, the first target motion resistance parameter (e.g., 100-200 N·s / m) is configured for makeup tools with smooth physical properties, such as blush brushes, foundation brushes, or blending sticks, to make the application process smoother and safer. When the goal is to improve the flexibility of makeup movements, such as when the contact state indicates that the makeup tool is moving freely in space and has not made contact with the face, a relatively small second target motion resistance parameter (e.g., 50-100 N·s / m) is configured to reduce the sense of motion resistance of the robotic arm and improve the efficiency of motion transfer.

[0048] In some embodiments, the target motion resistance parameter can also be determined based on the type of makeup process. In this embodiment, the makeup process types include: routine application and fine repair. In actual execution, when performing a large-area routine application, an appropriate target motion resistance parameter (e.g., 50-100 N·s / m) is configured to achieve a balance between the stability and response speed of the robotic arm, resulting in a smooth application action. When performing fine repair on local blemishes, the value of the target motion resistance parameter is increased (e.g., 100-200 N·s / m). By enhancing the motion resistance effect of the robotic arm and dexterous hand, the jitter and overshoot that may occur during local fine repair are effectively suppressed, thereby improving the accuracy of the local repair action.

[0049] According to the control method of the beauty robot provided in this application embodiment, a larger target motion resistance parameter is configured when contacting the face or performing local fine-tuning to absorb contact impact and effectively suppress end-effector jitter, thereby improving the smoothness of the fit and the accuracy of local operations. Conversely, during free movement without contact with the face or in the large-area routine application stage, a smaller or moderate target motion resistance parameter is configured to reduce the feeling of motion resistance and better balance the movement stability and response speed of the robotic arm. This strategy enables the robotic arm to adapt to the resistance requirements of different operation stages, facilitating smooth interaction between the makeup tools and the user's face, and improving the makeup quality and user experience of automated makeup operations.

[0050] In some embodiments, step 120 includes: when the goal of the current makeup stage is determined to be maintaining the accuracy of the makeup trajectory based on surface hardness and curvature, a target contact elasticity parameter is determined as a first target contact elasticity parameter, taking into account the characteristics of the makeup tools; when the goal of the current makeup stage is determined to be achieving soft compliance based on surface hardness and curvature, a target contact elasticity parameter is determined as a second target contact elasticity parameter, taking into account the characteristics of the makeup tools. The first target contact elasticity parameter is greater than the second target contact elasticity parameter.

[0051] Based on surface hardness and curvature, when the robot senses that the current makeup area is a hard and flat region (e.g., the forehead) and determines that the goal of the current makeup stage is to maintain the accuracy of the makeup trajectory, a first target contact elasticity parameter (e.g., 30-50 N / m) is configured in conjunction with the tool characteristics. This allows the robotic arm to achieve hard positioning in the hard region, thereby improving trajectory maintenance accuracy. When the current makeup area is sensed to be a soft and undulating region (e.g., the cheek) and the goal is to achieve soft contact compliance, a relatively small second target contact elasticity parameter (e.g., 10-20 N / m) is configured. This allows the robotic arm to achieve elastic yielding in the soft makeup region, thereby achieving soft contact compliance.

[0052] In some embodiments, the target contact elasticity parameter can also be determined based on the type of makeup process. In this embodiment, the makeup process types include: routine application and fine touch-up. In actual execution, when performing a large-area routine application, a target contact elasticity parameter of, for example, 30-50 N / m is configured based on the characteristics of the makeup tool. This allows the robotic arm to maintain the stability of the makeup trajectory while conforming to the natural curves of the face. When performing fine touch-up on local blemishes, the value of the target contact elasticity parameter is reduced (e.g., 10-20 N / m) based on the characteristics of the makeup tool to lower the control contact elasticity parameter at the end of the robotic arm. This allows the robotic arm's interaction to more delicately conform to the subtle undulations of the skin surface, achieving a more precise local coloring effect.

[0053] Step 130: Control the movement of the execution components based on the target compliant interaction parameters. In this step, control commands are generated based on the target compliant interaction parameters to enable the robotic arm and dexterous hand to move according to the kinematic characteristics determined by the target compliant interaction parameters. Controlling the movement of the robotic arm and dexterous hand according to these control commands allows them to perform tasks according to certain dynamic characteristics such as interaction force, movement speed, spatial displacement, and response compliance, providing a stable and smooth makeup application for the user. In actual execution, to achieve rapid application of blush "base" in the initial coloring stage, a lower contact elasticity parameter and a higher movement speed are configured, allowing the robotic arm and dexterous hand to efficiently complete the initial coloring along the planned trajectory. In the blending stage, the contact elasticity parameter at the end of the robotic arm is slightly adjusted to reduce the makeup speed and increase the contact time to simulate manual blending techniques and achieve natural color fusion.

[0054] In some embodiments, step 130 includes: controlling the movement of the actuator based on target compliant interaction parameters, user facial image information, and the actual force between the matching makeup tool and the user's face. For the force control inner loop, the actual force fed back by the force sensor at the end of the robotic arm is collected, and the force error between it and the target force is calculated; the force error is input into a second-order differential dynamics model constructed based on the target compliant interaction parameters (target motion inertia parameters, target motion stagnation parameters, and target contact elasticity parameters) to solve for the virtual acceleration, virtual velocity, and virtual displacement at the end of the robotic arm, and the virtual displacement is used as force control displacement compensation. For the visual outer loop, the latest dynamic pose in the facial image information is extracted, and the positional offset caused by the user's head micro-movement is calculated to obtain visual displacement compensation. The preset reference trajectory, force control virtual displacement compensation, and visual displacement compensation are superimposed and synthesized to obtain the target pose control command, and the target pose control command is converted into a joint angle command through inverse kinematics calculation and sent to the driver to control the robotic arm and dexterous hand to perform the makeup action. Makeup robots can perform multi-layered application with varying strategies: during the first layer of initial coloring (base layer), they move along a trajectory with a faster speed and lower target contact pressure; during the second or third layer of deeper blending, the robotic arm returns to the facial work area, adaptively reduces the speed, and fine-tunes the target contact elasticity parameters to a lower range, thereby increasing the effective contact time between the makeup tools and the skin, allowing the makeup color to blend naturally with the skin.

[0055] In some embodiments, step 130 includes: acquiring initial compliant interaction parameters, makeup quality inspection model parameters, and characteristic parameters of each makeup tool; establishing a work coordinate system reference; acquiring an initial facial image of the user's face; calculating the multi-dimensional spatial pose of the user's face and mapping it to the work space of the robotic arm for trajectory tracking after receiving a makeup start command; performing multi-source displacement compensation calculation based on the work coordinate system reference; generating joint control commands to control the makeup robot to apply makeup to the user. The makeup quality inspection model parameters are used to perform quality analysis of the makeup after the current makeup stage ends. During the initialization and multi-dimensional parameter loading stages in the actual execution process, the central control algorithm main process inside the makeup robot acts as the execution subject, triggering corresponding initialization operations upon power-on reset or upon receiving a makeup start command. Regarding the establishment of the work coordinate system reference, by reading a preset configuration file and hand-eye calibration matrix, the mapping relationship between the camera visual space and the physical work space of the robotic arm is established, and the initialization of sensors and visual models is completed simultaneously. Regarding the acquisition of initial compliant interaction parameters and characteristic parameters of makeup tools, by calling a preset digital tool attribute library, the basic mechanical parameters and target forces specific to different tools are read. Furthermore, pre-load quantitative evaluation criteria for subsequent closed-loop quality analysis, including but not limited to: the standard color hue-saturation-brightness (HSV) range of the target makeup, the minimum coverage threshold (e.g., 95%), the quality pass score (e.g., 85 points), and the maximum number of retry attempts (e.g., 3 times).

[0056] According to the control method for the beauty robot provided in the embodiments of this application, in the initialization stage before controlling the movement of the execution components, not only are the basic mechanical parameters and makeup tool characteristics required for the underlying dynamic control loaded, but also relevant quantitative evaluation standards are established in advance for makeup quality analysis. At the same time, a coordinate system benchmark for mapping the visual space and the physical operation space is constructed, which provides a complete and effective reference for subsequent calculation of facial multi-dimensional spatial posture, execution of dynamic trajectory tracking, and multi-source displacement compensation calculation. This helps to achieve control accuracy and overall operation consistency in complex beauty operations, and improves the makeup quality and user experience of automatic makeup operations.

[0057] In some embodiments, the movement of the actuator is controlled based on target compliant interaction parameters, the user's facial image information, and the actual force between the matching makeup tool and the user's face. This includes: performing displacement compensation calculation on the facial image information to determine the spatial posture changes of the user's face and obtaining visual displacement compensation; determining the force response changes at the end of the robotic arm based on the actual force and target compliant interaction parameters and obtaining force-controlled displacement compensation; and correcting the motion path of the robotic arm based on visual displacement compensation and force-controlled displacement compensation. During actual execution, a dual closed-loop trajectory correction mechanism is constructed using both visual (i.e., the user's facial image information) and force control (i.e., target compliant interaction parameters and actual force) methods to output joint commands to control the robotic arm and dexterous hand to complete the makeup application. The actual force is collected from the end-effector force sensor, and the force error between it and the target force is calculated. This force error is input into a second-order dynamic model constructed based on the target compliant interaction parameters (target motion inertia parameters, target motion stagnation parameters, and target contact elasticity parameters) to obtain the virtual acceleration, virtual velocity, and virtual displacement of the end of the robotic arm. The virtual displacement is then used as force-controlled displacement compensation.

[0058] In this second-order differential dynamics model, the specific physical meanings of each calculated parameter are as follows: Force error is the difference between the actual force between the matched makeup tool and the user's face and the target force, primarily used to characterize the required response and elimination of external force offset under the current contact state. Virtual acceleration is the rate of change of virtual velocity over time, primarily used to characterize the dynamic response degree of the second-order differential dynamics under the current force error. Virtual velocity is the rate of change of virtual displacement over time, primarily used to characterize the dynamic compensation rate generated by the response force error at the end of the robotic arm. Virtual displacement is the virtual position offset of the end of the robotic arm caused by the force error. This parameter is superimposed as force-controlled displacement compensation into the control commands of the robotic arm and dexterous hand, enabling the robotic arm and dexterous hand to adaptively adjust their positions according to the actual force conditions, thereby achieving a dynamic and compliant interaction effect with specific impedance characteristics. For the visual outer loop, the dynamic pose in the facial image information is extracted, and the position offset caused by the user's head micro-movements is calculated to obtain visual displacement compensation.

[0059] The preset baseline trajectory, force-controlled virtual displacement compensation, and visual displacement compensation are superimposed and synthesized to obtain the target pose control command. The target pose control command is then converted into joint angle command through inverse kinematics calculation and sent to the actuator, thereby controlling the robotic arm and dexterous hand to perform the makeup application action.

[0060] During the research and development process, the inventors also discovered that with the advancement of service robot technology, automated makeup has gradually become a research hotspot. Most current makeup robots use visual sensors to extract key facial points to plan motion paths and generate preset trajectory parameters, controlling the robotic arm and dexterous hand to perform makeup actions according to these parameters. In real makeup scenarios, users' faces inevitably produce micro-expressions or slight movements. Related technologies typically rely heavily on initial static calibration; static preset planning parameters cannot track and compensate for these dynamic facial changes in real time. This results in makeup robots having weak anti-interference capabilities during makeup operations, easily leading to makeup position deviations and affecting the overall makeup quality. A single position control mode lacks the ability to adapt to actual forces. If the user's face deviates from the initial calibration position, it can easily cause abnormal changes in force, increasing the safety hazard of the robotic arm's end effector puncturing the user. To address the issue of low makeup quality after automated makeup application, the inventors, through in-depth research, designed a control method for a makeup robot. The makeup robot includes an execution component, comprising a robotic arm and a dexterous hand at the end of the robotic arm, used to hold matching makeup tools. The method includes: acquiring target compliant interaction parameters; these target compliant interaction parameters are parameters generated based on the makeup action characteristics corresponding to the current makeup needs, used to characterize the motion features of the execution component; obtaining visual displacement compensation based on the user's facial image information; obtaining force-controlled displacement compensation based on the target compliant interaction parameters and the actual force between the matching makeup tools and the user's face; correcting preset trajectory parameters based on visual displacement compensation and force-controlled displacement compensation; and controlling the movement of the execution component based on the corrected preset trajectory parameters. According to the control method for the makeup robot provided in this application, by acquiring target compliant interaction parameters matching the current makeup needs and combining visual displacement compensation based on facial image information and force-controlled displacement compensation based on actual force, the preset trajectory parameters are dynamically corrected. This allows the method to adapt to spatial position changes caused by slight movements of the user during the makeup process, enabling the makeup trajectory to better conform to the target facial area. Simultaneously, by combining force-controlled displacement compensation obtained from the target compliant interaction parameters, the compliance of the robotic arm and dexterous hand can be adjusted according to different makeup movement characteristics. This method facilitates compliant interaction between the robotic arm and dexterous hand and the user's face, improving the makeup quality and user experience of automated makeup operations.

[0061] like Figure 2 As shown, the control method of the beauty robot includes steps 210, 220, 230, 240 and 250.

[0062] Step 210: Obtain the target compliant interaction parameters. In this step, the target compliant interaction parameters are parameters generated based on the makeup action characteristics corresponding to the current makeup requirements, used to characterize the motion characteristics of the execution component. These motion characteristics essentially refer to the force and motion state of the robotic arm's end effector (such as a dexterous hand or the makeup tool it holds) in the operating space. The specific method for obtaining the target compliant interaction parameters has been explained in detail above and will not be repeated here.

[0063] Step 220: Obtain visual displacement compensation based on the user's facial image information. Facial image information can be obtained through an image acquisition device installed on the makeup robot (or an image acquisition terminal or server connected to the makeup robot), used to characterize the user's facial features (such as the spatial distribution and relative positions of facial features), facial bone contours and surface undulations, skin texture, and makeup effects during and after makeup application. Visual displacement compensation refers to the position correction amount superimposed on a preset reference trajectory to maintain the relative pose relationship between the robotic arm's end effector and the user's face. The preset reference trajectory can be a dynamically changing trajectory. In some embodiments, the preset reference trajectory is generated based on the user's initial static facial 3D reconstruction model in the initial state. Alternatively, in other embodiments, the preset reference trajectory can also be pre-recorded and stored based on manually taught data.

[0064] In some embodiments, after the preset reference trajectory is compensated and corrected through visual displacement compensation, the corrected preset reference trajectory can be used as the preset reference trajectory for the next compensation. The preset trajectory parameters are parameterized descriptions of the preset reference trajectory of the robotic arm's end effector in a spatial coordinate system, used to instruct the robotic arm and dexterous hand to perform corresponding makeup actions. It should be noted that, in the embodiments of this application, the preset trajectory parameters specifically represent the target pose of the robotic arm's end effector (e.g., the dexterous hand mounted on the end effector or the makeup tool held by it) in the facial work space. Reference Figure 7 During the actual execution process, such as applying blush, the user's facial information is collected by image acquisition devices such as cameras or webcams for dynamic pose calculation. The visual processing thread (Visual Thread) acts as the execution subject to plan and generate a preset baseline trajectory. When the user's facial pose changes due to slight head movements, visual displacement compensation calculated from the visual pose change is superimposed on the preset makeup trajectory, so that the dexterous hand at the end of the robotic arm and / or the makeup tools it holds dynamically follow the facial movement.

[0065] In some embodiments, step 220 includes: performing multidimensional reconstruction on the user's facial image information to obtain the multi-degree-of-freedom pose matrix of the user's face; mapping the multi-degree-of-freedom pose matrix to the base coordinate system of the robotic arm and calculating the inter-frame displacement change; and smoothing the inter-frame displacement change to generate visual displacement compensation.

[0066] In actual execution, high-definition image information of the user's face is continuously acquired through a visual sensor (e.g., an RGB-D camera image stream), and a 3D face reconstruction algorithm (e.g., the 3DDFA_V2 deep learning model) is used to decompose the high-definition image to obtain the six-degree-of-freedom pose information of the current face in the camera coordinate system. A multi-degree-of-freedom pose matrix (e.g., a 4×4 homogeneous transformation matrix) containing the rotation and translation information of the face relative to the camera is output. The calculation and updating process of the multi-degree-of-freedom pose matrix can be triggered by a timer or the arrival of a new image frame. In the dynamic preset reference trajectory generation stage, based on the transformation matrix from the camera to the robotic arm base, the aforementioned multi-degree-of-freedom pose matrix of the face in the camera coordinate system is mapped to the base coordinate system of the robotic arm. The transformation matrix can be obtained in advance through hand-eye calibration. In some embodiments, the calculation formula for the above mapping transformation process can be expressed as: = × .in, This represents the face pose matrix transformed to the robot arm base coordinate system at the current time t. This represents the transformation matrix from the camera to the robotic arm base, obtained through pre-calibration. This represents the multi-degree-of-freedom pose matrix of the face in the camera coordinate system output at the current time t.

[0067] After completing the coordinate system transformation, save the multi-degree-of-freedom pose matrix of the face in the base coordinate system at the previous moment. The translation components in the pose matrices corresponding to the current and previous moments are extracted, and the 3D position change relative to the previous moment is obtained through differential calculation, i.e., the inter-frame displacement change. To reduce the instability of compensation caused by the jitter of the visual recognition algorithm, the inter-frame displacement change can be smoothed by low-pass filtering to remove high-frequency noise and retain a smooth and continuous facial motion trajectory. Furthermore, the smoothed 3D displacement vector can be directionally gained and limited based on the current makeup scene. For example, in the direction perpendicular to the face (such as the X-axis), compliance compensation mainly relies on the force control inner loop, and a small gain coefficient can be set for visual compensation in this direction; simultaneously, a maximum safety threshold is set for the overall compensation to reduce large jumps in the position of the robotic arm's end effector caused by visual misrecognition. After the above smoothing and limiting processing, visual displacement compensation is generated.

[0068] In actual execution, the change in three-dimensional position (i.e., inter-frame displacement change) between the current moment and the previous moment can be calculated using the following formula, and visual displacement compensation can be generated based on this change: 。 Wherein, P() is the translation component extraction function, which is used to extract the three-dimensional translation components (i.e., the X, Y, Z coordinate values ​​in the corresponding coordinate system) in the corresponding pose matrix. This represents the visual displacement compensation calculated at the current time t. and Representing the current time t and the previous time respectively. The three-dimensional position components of the face in the base coordinate system are used to superimpose them onto the preset reference trajectory in the subsequent instruction synthesis stage. Even if the face moves, the robotic arm end effector can automatically follow and maintain a constant relative position. The calculation formula for this instruction superimposition synthesis process can be expressed as: P_compensated = P_ref + ΔP_vision. Where, P_compensated represents the final target position coordinates of the robotic arm end effector after compensation superimposition, P_ref represents the preset original reference trajectory position coordinates, and ΔP_vision represents the visual displacement compensation obtained after the above calculation and processing.

[0069] In some embodiments, visual pose compensation ΔR_vision(t) can be synchronously calculated and generated based on the inter-frame variation of the rotation angle in the multi-degree-of-freedom pose matrix. Superimposing the visual pose compensation onto the preset reference trajectory at the end of the robotic arm enables the makeup tool to maintain a relatively fixed spatial angle to fit the user's face.

[0070] Step 230: Based on the target compliant interaction parameters and the actual force between the matching makeup tool and the user's face, force-controlled displacement compensation is obtained; the actual force can be collected by the mechanical sensor set at the end of the robotic arm.

[0071] Force-controlled displacement compensation is calculated and generated in response to changes in actual force, and is superimposed on the dynamic position correction amount in the control command. It is used to enable the end of the robotic arm (i.e., the dexterous hand and the makeup tools it holds) to have preset physical impedance characteristics when subjected to force interaction, so as to maintain a constant and smooth contact state when dealing with minor facial undulations.

[0072] In some embodiments, after a preset reference trajectory is compensated and corrected through force-controlled displacement compensation, the corrected preset reference trajectory can be used as the preset reference trajectory for the next compensation. (Reference) Figure 6Upon detecting contact between the makeup tool and the user's face and the application process, the force control loop thread is activated and continuously executes in a preset high-frequency control cycle (e.g., a control frequency of no less than 500Hz). (Continue to refer to...) Figure 6 In the force sensing data acquisition stage, a six-dimensional force sensor can be used to collect the actual force between the makeup tool and the user's face at the current moment, and calculate the force error between this actual force and the target force. (Continue to refer to...) Figure 6 In the compliant interaction model calculation stage, the force error is input into the second-order differential dynamic model constructed by the target compliant interaction parameters for calculation and solution. This allows the dynamic response of the robotic arm end to the current force state to be calculated, thereby obtaining the corresponding virtual displacement, which is then used as the force-controlled displacement compensation output in the current control cycle.

[0073] In some embodiments, step 230 includes: calculating the difference between the actual force and the target force to obtain the force error; inputting the force error into the target dynamics model for solution to obtain force-controlled displacement compensation. In this embodiment, the target dynamics model is constructed based on target compliant interaction parameters; the target compliant interaction parameters include: target motion inertia parameters, target motion stagnation parameters, and target contact elasticity parameters. The target force refers to the force that should be maintained between the makeup tool held by the actuator and the user's facial skin, so that when the makeup tool conforms to the undulations of the facial surface, it can automatically perform backward force relief when encountering protruding areas and automatically perform forward compensation when encountering concave areas, thereby achieving constant and safe compliant interaction between the end effector of the robotic arm and the face. During the data acquisition phase of the actual execution process, the actual force fed back by the end effector force sensor at the current time t is obtained. and the latest visual pose This is used for calculating subsequent force errors and correcting preset reference trajectories. In some embodiments, the difference between the actual force and the target force is calculated to obtain the force error, which can be expressed by the formula: . Indicates the error in force application. Indicates the actual force. Indicates a constant target force.

[0074] The force deviation calculated above Substitute the values ​​into the target dynamics model and perform compliant interaction calculations. Based on the target compliant interaction parameters configured by the currently matched makeup tool (i.e., the target motion inertia parameter M, the target motion stagnation parameter B, and the target contact elasticity parameter K), solve the discrete form of the second-order differential equation and calculate the force-controlled virtual displacement at the end of the robotic arm, which serves as the force-controlled displacement compensation output at the current moment.

[0075] In some embodiments, the force error is input into the target dynamics model for solution to obtain force-controlled displacement compensation. This includes: substituting the target motion inertia parameters, target motion stagnation parameters, and target contact elastic parameters into the target dynamics model for solution to obtain virtual displacement; and outputting the virtual displacement as force-controlled displacement compensation. In actual execution, the force deviation is input into a preset continuous domain target dynamics model (i.e., the MBK model). The second-order differential equation of the target dynamics model can be expressed as: in, Represents the target's inertial parameters. Indicates the target motion retardation parameters. Indicates the target contact elasticity parameter. For virtual acceleration, For virtual speed, For virtual displacement, This represents the error in the applied force.

[0076] The above continuous dynamics model is discretized and approximated. The virtual velocity and virtual acceleration at the current moment are derived through numerical integration, and the virtual displacement at the current moment is calculated accordingly. The discretized solution formula can be expressed as: .in, and Each represents the current time. Compared to the previous moment The calculated force-controlled virtual displacement, The virtual displacement obtained by discretizing the discretized data represents the time interval between two adjacent moments (i.e., the control period). It is output as force-controlled displacement compensation to achieve dynamic correction of the trajectory of the robotic arm end effector.

[0077] In actual implementation, combined with Figure 6 The dual closed-loop architecture and Figure 7 The judgment process, trajectory planning and generation control logic, is divided into two branches based on whether the user's face has shifted: normal path and abnormal correction. like Figure 7 As shown, in cases where a branch is determined to be on a normal path (e.g., the user's face does not shift or the shift is within the allowable error range), there is no need to compensate for the preset reference trajectory. Instead, commands are synthesized directly based on the preset reference trajectory, and the robotic arm and dexterous hand are controlled to execute along the preset reference trajectory. (Continue to refer to...) Figure 7 In cases where a branch is determined to be an abnormal correction (e.g., displacement of the user's face is detected), such as Figure 6As shown, the visual displacement compensation calculated by the visual closed loop and the force control displacement compensation output by the force control closed loop will be merged and enter the virtual displacement correction stage. The preset reference trajectory will be synchronously superimposed with the above two displacement compensations. After the trajectory correction is completed, the final control command will be generated through the command synthesis stage and sent to the robotic arm and dexterous hand for execution.

[0078] According to the control method of the beauty robot provided in the embodiments of this application, visual displacement compensation is used to provide spatial pose correction data to reduce trajectory errors caused by user micro-movements and improve the accuracy of spatial positioning. At the same time, combined with force-controlled displacement compensation based on target dynamics model calculation, compliant feedback data of physical interaction is provided to absorb sudden resistance and maintain stable contact with complex facial surfaces. The above method is conducive to realizing smooth interaction between makeup tools and user face, and improves the makeup quality and user experience of automatic makeup operation.

[0079] Step 240: Based on visual displacement compensation and force control displacement compensation, correct the preset trajectory parameters; the preset trajectory parameters are the parameterized description of the preset reference trajectory of the robotic arm end in the spatial coordinate system, which is used to instruct the robotic arm and dexterous hand to perform the corresponding makeup actions.

[0080] In some embodiments, after the preset reference trajectory is compensated and corrected by at least one of visual displacement compensation and force-controlled displacement compensation, the corrected preset reference trajectory can be used as the preset reference trajectory corresponding to the next compensation.

[0081] In some embodiments, step 240 includes: synchronously superimposing visual displacement compensation and force control displacement compensation onto the coordinate dimension corresponding to the preset trajectory parameters based on a unified time reference, so as to correct the preset trajectory parameters. Based on safety boundary parameters, performing a facial safety boundary constraint check on the superimposed preset trajectory parameters, and correcting the preset trajectory parameters based on the check results. In this embodiment, the unified time reference refers to a common time reference system established to resolve the difference in frequency between the output of the vision system and the output of the force control system.

[0082] Safety boundary parameters refer to the physical protection thresholds configured to address the issue of makeup tools held by the actuators injuring users. These parameters may include, but are not limited to, the maximum permissible virtual penetration depth and the maximum permissible force of the robotic arm end-effector into the facial surface. They can be determined based on user-defined parameters or experimental data.

[0083] In actual execution, because the control frequency of the force control system (usually above 500Hz) is significantly higher than the sampling frequency of the vision system (usually 30-60Hz), direct superposition would lead to abrupt changes in the control quantity. Therefore, time-series alignment processing is required for the visual displacement compensation. In some embodiments, algorithms such as linear interpolation, cubic spline interpolation, or Kalman filtering can be used for time-series alignment processing. The low-frequency visual displacement compensation is smoothly interpolated and resampled within each high-frequency force control cycle, thereby matching its timing with the high-frequency control requirements and achieving smooth trajectory superposition. In actual execution, the process of correcting the preset trajectory parameters can be expressed by the following displacement superposition formula: . and Each represents the current time. Force-controlled displacement compensation and visual displacement compensation, This represents the uncorrected preset trajectory parameters at the current time t (corresponding to pre-planned standard makeup trajectory points such as circles and S-shapes). This represents the control command (i.e., the corrected preset trajectory parameters) output after superposition and calculation at the current time t.

[0084] According to the control method of the beauty robot provided in the embodiments of this application, by establishing a unified time reference to perform time-series alignment processing on visual displacement compensation, the smooth synchronous superposition of visual and force control compensation components of different frequencies on preset trajectory parameters is achieved. This method can better adapt to the slight relative position changes that occur during the makeup application process, making the makeup application fit the target facial area better. This facilitates smooth interaction between the makeup tools and the user's face, and improves the makeup quality and user experience of the automatic makeup application.

[0085] In some embodiments, based on a unified time reference, visual displacement compensation and force-controlled displacement compensation are synchronously superimposed onto the corresponding coordinate dimension of the preset trajectory parameters, including: performing time-series alignment processing on visual displacement compensation and force-controlled displacement compensation to obtain each displacement compensation after time synchronization; mapping each displacement compensation after time synchronization to a unified coordinate system and superimposing it onto the coordinate dimension corresponding to the preset trajectory parameters to obtain the superimposed preset trajectory parameters.

[0086] In actual execution, visual displacement compensation and force-controlled displacement compensation are time-aligned to obtain each displacement compensation after time synchronization. A high-precision time synchronization mechanism (such as the timestamp alignment mechanism based on the robot operating system ROS) can be used to ensure that the preset trajectory parameters, force-controlled displacement compensation and visual displacement compensation all have the same timestamp t, so that multi-source data can be accurately and synchronously superimposed and calculated within the same control cycle.

[0087] In some embodiments, attitude compensation can also be introduced to correct the preset trajectory parameters. The synthesis of control commands adopts a multi-axis independent linear superposition method, and its core superposition logic can be expressed as: final command pose = preset trajectory parameters + force control displacement compensation + visual displacement compensation. The following is a detailed explanation of correcting and obtaining the preset trajectory parameters at the current moment.

[0088] In actual execution, a basic motion trajectory is generated based on the specific makeup task (e.g., blush application) and extracted facial key points. The preset trajectory parameters for each moment are... This includes three-dimensional position coordinates (x, y, z) and attitude angles (rx, ry, rz). The force control system calculates the mechanical deviation between the actual force and the target force, and uses the target dynamics model to calculate the force-controlled displacement compensation at the current time t. The force-controlled displacement compensation is directly superimposed onto the corresponding force axis of the preset trajectory parameters (usually a coordinate axis perpendicular to the facial direction, such as the X-axis). For example, if the reference coordinate of the preset trajectory parameters on the X-axis is 0.200m, and the calculated force-controlled displacement compensation is +0.005m, then the superimposed X-axis coordinate is corrected to 0.205m. The vision system tracks the six-dimensional pose of the face, calculates the change in position of the current frame relative to the standard reference, and obtains the visual displacement compensation at the current time t. The change is directly superimposed onto the corresponding coordinate dimension of the preset trajectory parameters. For example, if a tiny movement of 0.003m to the right of the user's head is detected, the overall coordinate of the preset trajectory on the Y-axis increases by 0.003m. In actual execution, the position compensation components in the X, Y, and Z axes are calculated and superimposed independently. The compensation for the attitude angles (rx, ry, rz) also follows the principle of independent superposition of each dimension. It should be noted that since the interaction between the robotic arm's end effector and the face is usually mainly based on position compliance, attitude compensation is usually provided primarily by the vision system. Assuming the position coordinates of the preset trajectory point at time t are (0.200, 0.100, 0.150) and the attitude angles are (0.1, 0.2, 0.3), force control calculations show that a 0.005m retraction is needed in the X direction (i.e., ...). The value was -0.005), and the visual detection showed that the face had moved 0.003m to the right (i.e., If the value is +0.003, the final synthesized pose is: position (0.195, 0.103, 0.150), pose maintained (0.1, 0.2, 0.3). This method achieves dynamic and precise correction of the preset makeup trajectory.

[0089] In some embodiments, based on the inspection results, the preset trajectory parameters are corrected, including: if the inspection result fails, truncating the superimposed preset trajectory parameters to obtain corrected preset trajectory parameters and outputting a warning message; if the inspection result passes, the superimposed preset trajectory parameters are determined as the corrected preset trajectory parameters. In actual execution, if the inspection result fails, parameters exceeding the limits are automatically truncated, forcibly restricted to within a safe threshold, thereby obtaining corrected preset trajectory parameters, and a warning message is simultaneously output to the user. If the inspection result passes (i.e., the superimposed trajectory is within a safe range), the superimposed preset trajectory parameters are directly determined as the corrected preset trajectory parameters for subsequent execution stages.

[0090] In some embodiments, to achieve a natural blended boundary makeup effect, a gradient extension displacement component can be automatically superimposed on the edge area of ​​a preset trajectory parameter. By generating a smooth, outward-expanding transition path at the end of the trajectory, the discontinuity phenomenon at the makeup boundary can be reduced.

[0091] According to the control method for the beauty robot provided in this application, when the trajectory exceeds the physical movement range of the robotic arm's end effector or the preset facial safety boundary due to the superposition of multi-source displacement compensation, the system can automatically truncate the parameters to force them within the safety threshold, thereby helping to reduce the risk of mechanical interference or excessive contact caused by commands exceeding the limit. Simultaneously, the synchronous output of warning information further enhances the system's ability to warn and perceive abnormal states. This method helps to achieve safe and reliable human-computer interaction in automated beauty processes, and also improves the makeup quality and user experience of automated makeup operations.

[0092] Step 250: Based on the corrected preset trajectory parameters, control the movement of the actuator. Combined with... Figure 6 and Figure 7 The control flow shown enters the instruction synthesis stage after completing the aforementioned trajectory planning and anomaly correction. Based on the corrected preset trajectory parameters obtained in the above steps, the final control instructions are generated through the instruction synthesis stage and sent to the robotic arm and dexterous hand to drive the makeup tools to the specified spatial pose, thereby performing the corresponding makeup actions.

[0093] In some embodiments, step 250 includes: parsing target pose control commands and end effector commands based on corrected preset trajectory parameters; performing inverse kinematics calculation on the target pose control commands to obtain joint control commands for the robotic arm; and sending the joint control commands and end effector commands to their respective drivers to control the robotic arm and dexterous hand to collaboratively execute corresponding makeup actions. In this embodiment, inverse kinematics calculation is the process of converting the target pose of the robotic arm's end effector in Cartesian space into the corresponding angles of each joint of the arm.

[0094] Obtain the target pose control command generated in the preceding steps. This command corresponds to the preset trajectory parameters corrected at the current time t. Corrected preset trajectory parameters This includes three-dimensional position coordinates (x, y, z) and three-dimensional pose (rx, ry, rz) represented in Euler angles. Quaternions are used to represent the target pose. For example, the input Euler angles need to be converted into four quaternion components (w, x, y, z) in the ZYX rotation order beforehand. Compared with the traditional Euler angle representation, using quaternions can not only effectively avoid kinematic singularities (such as gimbal lock) but also facilitate smooth pose interpolation calculations later. The rotation matrix constructed based on these quaternion components and the above three-dimensional position coordinates are combined to obtain a 4×4 homogeneous transformation matrix corresponding to the target pose. In this homogeneous transformation matrix, the upper left 3×3 submatrix is ​​the rotation matrix converted from quaternions, the upper right 3×1 column vector is the translation vector corresponding to the three-dimensional position coordinates, and the last row of the matrix is ​​fixed at [0, 0, 0, 1]. The homogeneous transformation matrix is ​​input into the inverse kinematics solution interface provided by the robotic arm manufacturer (e.g., calling the `rm_kinematics_inverse` function), and inverse kinematics calculation is performed based on the robotic arm's DH parameter model. To address the solution multiplicity of a six-DOF robotic arm's inverse kinematics solution (e.g., 8 possible joint angle combinations), the optimal solution needs to be selected based on preset evaluation criteria (e.g., the "shortest path" principle, the "configuration consistency" principle, and the "avoiding singularities" principle). That is, under the premise of satisfying the physical limit boundaries and obstacle avoidance requirements of each joint of the robotic arm, the solution with the smallest change in joint angles compared to the current actual joint angle is selected as the candidate control command. After generating the candidate command, to reduce abrupt changes or stuttering in joint motion during the interaction process, boundary constraint planning for velocity and acceleration is required. In actual execution, trapezoidal or S-shaped velocity curves can be used to smooth the joint motion, thereby generating the final joint control command. This includes joint control commands containing the target angles (θ1~θ6) for each joint. The commands are transmitted to the robotic arm's actuator via an industrial communication bus (such as CAN bus or Ethernet). Simultaneously, the parsed end-effector commands are sent to the dexterous hand's controller. Upon receiving the commands, the robotic arm's actuator controls the servo motors of each joint to move the robotic arm's end effector to the target pose. The dexterous hand then synchronously executes corresponding gripping or posture fine-tuning actions based on the end-effector commands, thereby driving both to collaboratively perform the corresponding makeup application.

[0095] In some embodiments, after controlling the robotic arm and dexterous hand to collaboratively perform the corresponding makeup actions, a closed-loop verification mechanism for the actual end-effector pose can also be executed. After the robotic arm performs the action, the actual joint angle feedback is read through the underlying encoder, and the actual arrival pose of the robotic arm end-effector is obtained using forward kinematics calculation. This actual pose is then compared with the target pose control command. If the pose tracking error exceeds the preset tolerance threshold, an additional error compensation strategy or an alarm message will be triggered to further improve the safety and accuracy of the beauty operation task.

[0096] After receiving the current makeup request instruction, the robotic arm, carrying the dexterous hand and makeup tools attached to its end effector, is first moved to a safe observation position in front of the face. A visual sensor then calculates the 6D pose of the user's face. Based on this pose information, target feature points related to the makeup task (such as the starting point of the cheekbone and the ending point along the hairline) are extracted to generate an initial basic application path. This basic application path is parameterized to generate preset trajectory parameters characterizing the target pose of the robotic arm's end effector (i.e., the dexterous hand and the blush brush it holds) in Cartesian space. Based on these preset trajectory parameters, the robotic arm drives the dexterous hand to move smoothly from the safe observation position to the facial work space until the blush brush precisely contacts the starting point of the cheekbone with a preset initial contact force, thus formally establishing human-computer physical interaction. During the process of establishing physical contact and sliding along the path, to cope with complex variables in an unstructured environment, the system not only issues target pose control instructions but also simultaneously loads corresponding target smooth interaction parameters. During the dynamic interaction of actual application, multi-dimensional trajectory replanning is performed to cope with user micro-movements and changes in facial firmness. Among them, the tool continuously monitors the user's facial state to calculate the visual displacement compensation amount and maintains the tool's dynamic tracking of feature points; at the same time, based on the configured target compliant interaction parameters and force feedback, the corresponding force-controlled virtual displacement compensation amount is calculated and superimposed on the preset trajectory parameters, thereby achieving compliant unloading and compensation of force when contacting hard or soft areas.

[0097] In some embodiments, in conjunction with specific interaction requirements, an extension offset can be automatically superimposed on the trajectory edge to simulate a natural blending effect. By fusing preset trajectory parameters, visual tracking compensation, force-controlled flexibility compensation, and extension offset, a composite motion trajectory of the robotic arm's end effector is dynamically generated. This allows the dexterous hand and its held makeup tools to accurately follow facial feature points in space while maintaining flexibility during force interaction, thus balancing trajectory accuracy and force flexibility in the makeup process. It is understood that the target smooth interaction parameters described in this application are dynamically updated based on the current makeup requirements during the makeup process; visual displacement compensation and force-controlled displacement compensation are dynamically updated based on changes in the user's posture during the makeup process, and the two can be executed sequentially or simultaneously.

[0098] According to the beauty robot control method provided in this application, the target smooth interaction parameters are flexibly configured by combining makeup operation parameters, makeup tool characteristics, and facial physical features. This enables the system to flexibly adapt to different underlying mechanical response logics for different makeup procedures, effectively improving the stiffness of the robotic arm's movements and the smoothness of the initial contact. Based on this, force-controlled displacement compensation is generated according to the target smooth interaction parameters, and combined with visual displacement compensation, dynamic compensation is performed on the user's pose changes. This not only effectively addresses spatial trajectory deviations caused by the user's micro-movements and maintains accurate dynamic tracking, but also actively unloads abnormal forces when contacting areas of different hardness on the face to smooth out sudden force changes. This achieves smooth interaction between the robotic arm and the user's face, and improves the makeup quality and user experience of automated makeup operations.

[0099] During the research and development process, the inventors also discovered that when devices in related technologies achieve automatic end-effector replacement, they are often limited to changing the physical shape of the end effector. They fail to adapt the underlying control strategies to the specific operational characteristics of various makeup tools and generally lack control logic for complex physical interactions such as makeup application and tool cleaning. These limitations in control dimensions, along with the lack of a refined state management mechanism in the overall workflow, make it difficult for makeup robots in related technologies to maintain fully automated continuous operation. In multi-tool collaborative scenarios, these technologies often rely on manual intervention to replace specific workpieces or can only perform fixed switching based on preset sequences, making it difficult to dynamically schedule tool combinations and workflows based on the current makeup state. To address the issue of low makeup quality after automated makeup application, the inventors, through in-depth research, designed a control method for a makeup robot. The makeup robot includes an execution component, which comprises a robotic arm and a dexterous hand at the end of the robotic arm. The execution component is used to grip matching makeup tools. The method includes: determining matching makeup tools based on the current makeup requirement command and obtaining the corresponding gripping parameters for the matching makeup tools; controlling the execution component to grip the matching makeup tools based on the gripping parameters; determining the corresponding dipping parameters for the matching makeup tools; the dipping parameters include dipping force and rotation trajectory; controlling the execution component to perform interactive actions with other makeup products while gripping the matching makeup tools, in order to dip the matching makeup tools into the required makeup materials; and controlling the execution component to interact with the user's face. According to the control method for the beauty robot provided in this application, the robotic arm and dexterous hand are controlled to grasp the matching makeup tools based on the grasping parameters determined by the current makeup demand instruction. This achieves automated tool scheduling and grasping, improving the stability and safety of the grasping action. At the same time, the robotic arm and dexterous hand are controlled to pick up makeup materials based on the dipping parameters, achieving automatic dipping of makeup materials in an appropriate amount and evenly. This method establishes a closed-loop operation process from tool matching, automated grasping, precise dipping to facial interaction, which helps to realize intelligent and continuous automatic makeup operation of the beauty robot, and improves the makeup quality and user experience of the automatic makeup operation.

[0100] like Figure 3 As shown, the control method of the beauty robot includes steps 310, 320, 330, 340 and 350.

[0101] Step 310: Based on the current makeup requirement instruction, determine the matching makeup tool and obtain the corresponding grasping parameters for the matching makeup tool. In this step, the current makeup requirement instruction is a comprehensive set of information including at least one of the following: the specific makeup task to be performed by the beauty robot, the physical interaction constraints in the makeup task, the expected presentation characteristics of the target makeup effect, and the user's personalized attributes. The current makeup requirement instruction has been explained in detail above and will not be repeated here. The grasping parameters are used to control the robotic arm to grasp the matching makeup tool.

[0102] In some embodiments, the grasping parameters include at least one of: a hand-eye calibration matrix, a target angle sequence of each joint of the dexterous hand, and a grasping force threshold. In this embodiment, the hand-eye calibration matrix is ​​used to characterize the spatial transformation relationship between the vision module coordinate system and the robotic arm base coordinate system, so as to convert the identified tool image position into physical space coordinates executable by the robotic arm. The target angle sequence of each joint of the dexterous hand is used to control the dexterous hand to perform specific spatial pose changes to adapt to the humanoid grasping gestures of different makeup tools. The grasping force threshold serves as a force control safety benchmark during the grasping process. Grasping stability can be achieved by monitoring the force between the dexterous hand and the makeup tool during the closing phase and ensuring that it reaches and does not exceed this grasping force threshold. In actual execution, different makeup tools can be mapped and associated with corresponding grasping parameters through a pre-established digital makeup tool library.

[0103] Step 320: Based on the grasping parameters, control the execution components to grasp the matching makeup tools. In actual execution, this step is executed by the task state machine, which uniformly schedules the robotic arm, dexterous hand, and tool holder according to the received makeup requirement instructions. For example, by accessing a preset digital beauty tool library, specific grasping parameters are retrieved and loaded for the specific makeup tool to be grasped. Subsequently, the robotic arm and dexterous hand are controlled to move to the location of the makeup tool, and the dexterous hand joints are driven to deflect to a specific angle according to the target angle sequence in the grasping parameters, completing the pre-adjustment of the human-like gesture to simulate the grasping posture of a human hand holding a pen or squeezing a sponge, so as to achieve a stable grasp of the makeup tool. After grasping, the robotic arm and dexterous hand can be controlled to hold the matching makeup tools and interact with other corresponding makeup products to perform the task of picking up makeup materials. For example, if the tool being held is an eyeshadow brush, the tool can be controlled to move towards the target eyeshadow palette to pick up eyeshadow powder; if the tool is a blush brush, the tool can be controlled to pick up blush powder or cream blush; and if the tool is a beauty blender or powder puff, the tool can be controlled to press the cushion or pick up foundation.

[0104] In some embodiments, step 320 includes: recognizing the tool image corresponding to the matched makeup tool to obtain the initial position information of the matched makeup tool; converting the initial position information into target position information in the base coordinate system of the robotic arm based on the hand-eye calibration matrix; controlling the robotic arm to move to the position of the matched makeup tool based on the target position information; controlling the joints of the dexterous hand to adjust their pose and close based on the target angle sequence to cover and grasp the matched makeup tool, and obtaining force feedback values; determining that the grasping action is completed when the force feedback value reaches the grasping force threshold. In actual execution, the work surface is scanned by the vision module to identify the feature marks on the matched makeup tool (e.g., blush brush handle), and the initial position and spatial pose of the makeup tool in the camera coordinate system are calculated. Then, the hand-eye calibration matrix is ​​used for spatial mapping to accurately convert the image coordinates detected by vision into physical spatial target position information that the robotic arm can directly execute, so that the robotic arm can locate the position of the matched makeup tool; at the same time, the target angle sequence of each joint of the dexterous hand and the grasping force threshold matched with the makeup tool are retrieved from the built-in digital beauty tool library. The robotic arm, carrying a dexterous hand, precisely moves to a safe point above a preset coordinate on the tool holder. Based on the retrieved target angle sequence, it drives the joints of the dexterous hand to deflect along a specific temporal path, continuously executing a series of actions: finger opening, wrist posture fine-tuning, fingertip approaching the target, multi-finger structure wrapping around the tool handle, and applying a locking force. This completes a stable grip on the matching cosmetic tool, accurately replicating professional gripping postures such as holding a pen or squeezing a sponge. During the gripping action, the force feedback value between the dexterous hand and the cosmetic tool is monitored during the closing process and compared with a gripping force threshold. The gripping force threshold serves as a control benchmark for determining whether the gripping state is stable and safe. When the monitored force feedback value reaches this threshold, the gripping action is considered successfully completed.

[0105] According to the control method for a beauty robot provided in this application, the robotic arm is guided by visual positioning, and the dexterous hand is controlled to adjust its posture by combining the target angle sequence. This enables the robotic arm and dexterous hand to collaboratively grasp makeup tools, thereby highly replicating the natural posture of a human holding tools. Simultaneously, the grasping force is controlled by comparing the force feedback value with the grasping force threshold, improving the stability and safety of the grasp. This method achieves automated management of the entire process from tool recognition and strategy scheduling to precise grasping, and improves the makeup quality and user experience of automated makeup operations. Step 330: Determine the dipping parameters corresponding to the matched makeup tools; In some embodiments, the picking parameters include: picking force and rotation trajectory. Picking force refers to the desired contact pressure applied and maintained by the makeup tool in a direction perpendicular to the surface of the cosmetic. Based on this picking force, the makeup tool held by the robotic arm and dexterous hand is controlled to maintain appropriate contact with the cosmetic to obtain an appropriate amount of powder. Rotation trajectory refers to the corresponding movement path (such as circular or sweeping) performed by the makeup tool held by the robotic arm and dexterous hand in a plane parallel to the surface of the cosmetic while maintaining the aforementioned picking force, so that the powder can be evenly adhered to the powder-picking surface of the makeup tool.

[0106] In some embodiments, the rotation trajectory is determined based on at least one of parameters such as rotation speed, number of rotations, and rotation radius. In this embodiment, rotation speed is used to control the speed at which the tool moves on the cosmetic surface to reduce powder splashing caused by excessive rotation speed; the number of rotations is used to quantitatively control the total amount of powder acquired in a single operation; and the rotation radius can be adapted to cosmetic containers of different sizes, ensuring that the powder-grabbing action is always confined to the effective area of ​​the powder compact.

[0107] In some embodiments, step 330 includes: acquiring the texture characteristics of other cosmetic products that perform interactive actions with the matching makeup tool held by the robotic arm; and determining the pick-up parameters based on the texture characteristics. In this embodiment, the texture characteristics of other cosmetic products include, but are not limited to: powder, cream, liquid, cushion / honeycomb, gel, block solid, and spray / gaseous, etc., and this application does not specifically limit them.

[0108] In some embodiments, powder textures may include pressed powder or loose powder; cream textures may include foundation cream, concealer, contour cream, or blush cream; and liquid textures may include liquid foundation, liquid highlighter, or lip gloss. This application does not specifically limit the specific cosmetic types covered by a particular texture category or the classification criteria for subdivision. In actual implementation, the picking parameters are determined based on the texture characteristics. For example, a larger picking force and a specific circular rotation trajectory can be configured for powder textures; while for liquid textures, the picking force is reduced and the rotation action is reduced or eliminated to improve the rationality of the picking process and the accuracy of the powder amount. It should be noted that the aforementioned cosmetics mainly refer to separate materials that require tools for powder picking; in fact, the cosmetic products involved also include integrated makeup tools such as eyebrow pencils, lipsticks, or concealer sticks that do not require picking and can be applied directly. For this type of directly applied cosmetic product, the corresponding dipping parameters can be converted into usage preparation parameters (e.g., the pulling threshold for controlling the robotic arm to pull out the pen cap, the torque and rotation angle for unscrewing the pen tip, etc.), or the system can directly skip the dipping step and enter the makeup process. This application does not make any specific limitations here.

[0109] In some embodiments, determining the dipping parameters based on texture characteristics includes: when the texture characteristic is liquid, determining the dipping force in the dipping parameters as a first dipping force; when the texture characteristic is powdery, determining the dipping force in the dipping parameters as a second dipping force; when the texture characteristic is pastey, determining the dipping force in the dipping parameters as a third dipping force; wherein the first dipping force is less than the third dipping force, and the second dipping force is less than the third dipping force. In actual implementation, the appropriate dipping force is quantified and dynamically matched based on the specific texture of other cosmetic products identified: when the texture is liquid, due to its high fluidity and easy adhesion, the dipping force is determined to be a smaller first dipping force, with a value range of, for example, 0.2N to 0.4N; when the texture is powder (such as loose powder or pressed powder), the dipping force is determined to be a second dipping force, with a value range of, for example, 0.3N to 0.5N, to grab an appropriate amount of powder while reducing the risk of crushing the powder structure; when the texture is cream, due to the high viscosity resistance and density of cream materials, the dipping force is determined to be a larger third dipping force, with a value range of, for example, 0.5N to 0.8N.

[0110] In some embodiments, the rotation trajectory includes a rotation speed and a number of rotations; determining the pickling parameters based on texture characteristics includes: when the texture characteristic is liquid, determining the rotation speed as a first rotation speed and the number of rotations as a first number of rotations; when the texture characteristic is powdery, determining the rotation speed as a second rotation speed and the number of rotations as a second number of rotations; when the texture characteristic is pastey, determining the rotation speed as a third rotation speed and the number of rotations as a third number of rotations; in this embodiment, the first rotation speed is less than the third rotation speed, the third rotation speed is less than the second rotation speed, the first number of rotations is less than the second number of rotations, and the second number of rotations is less than the third number of rotations.

[0111] In actual application, parameters can be adapted for different textures of cosmetics to determine the appropriate application force and rotation trajectory. For liquid textures, due to their high fluidity and adhesion, a first rotation speed (or even a slight movement at zero speed) and a first number of rotations (e.g., no rotation or only tapping) are used to prevent liquid splashing or excessive application in a single squeeze. For powder textures, a faster second rotation speed is used to avoid damaging the powder's surface structure, allowing for rapid sweeping. Simultaneously, to reduce waste from excessive powder pickup, a moderate second number of rotations is used. For cream textures, since creams typically have high viscosity and require frictional heat to soften (or emulsify) the cosmetic material, a moderate third rotation speed is used to maintain stable friction. Simultaneously, a third number of rotations is used to increase the number of rotations, ensuring the tool picks up a sufficient amount of cream.

[0112] According to the control method for a beauty robot provided in this application, the dispensing force and rotation trajectory are adaptively matched based on the texture characteristics of the cosmetics, providing a reliable control benchmark for subsequent adaptive and precise material dispensing. This method employs multi-dimensional parameter configurations for different textures such as powder, cream, and liquid, improving dispensing accuracy while effectively reducing the risk of damaging powder or splashing liquid. It also meets the friction softening conditions required for creams, enhancing the adaptability and smoothness of the makeup tools when interacting with various forms of makeup materials, and improving the makeup quality and user experience of automated makeup operations.

[0113] Step 340: Based on the dipping parameters, control the execution component to hold the matching makeup tool and other makeup products to perform interactive actions, so as to pick up the makeup materials required by the matching makeup tool; In actual operation, based on the dispensing parameters, the robotic arm and dexterous hand are controlled to perform interactive actions with the matching makeup tools and other cosmetic products, primarily through state machine-based process control. The robotic arm, carrying the matching makeup tools held by the dexterous hand, is moved to a preset safe position above other cosmetic products (such as powder containers containing makeup materials). Subsequently, a compliant interaction mode is switched in the direction perpendicular to the cosmetic product surface (i.e., the Z-axis). By introducing a compliant interaction model that includes motion inertia parameters, motion resistance parameters, and contact elasticity parameters, soft contact between the robotic arm's end effector and the cosmetic product surface is achieved. In this compliant control mode, the dispensing force in the dispensing parameters is invoked. (For example, 0.5N) causes the matching makeup tool to move downwards and smoothly conform to the surface of the makeup material based on this dipping force. While maintaining this dipping force, the rotation trajectory in the dipping parameters is further invoked to drive the matching makeup tool to perform interactive actions in a plane parallel to the surface of the makeup product in order to pick up the required makeup material.

[0114] In some embodiments, the rotational trajectory can be configured as a helical trajectory, whose planar parametric equation can be expressed as: Once the required makeup materials have been applied and the state machine's termination condition is met, the robotic arm, carrying the dexterous hand and the matching makeup tool it holds, is quickly lifted to a safe release height. Simultaneously, the state machine automatically transitions to the "makeup application execution" state. This tool-grabbing and application process constitutes the preliminary stage of a complete makeup application operation. After this stage is successfully closed, the robotic arm and dexterous hand, carrying the matching makeup tool with the applied makeup materials, can then perform the makeup application interaction with the user's face.

[0115] In some embodiments, step 340 includes: acquiring the dipping force and rotation trajectory based on the dipping parameters; when the robotic arm moves above other cosmetic products, controlling the execution component to drive the matching makeup tool to move along a first direction toward the surface of the other cosmetic product, and performing compliant interactive control in the first direction until the actual force between the matching makeup tool and the surface of the other cosmetic product reaches the dipping force; and controlling the execution component to drive the matching makeup tool to move on a target plane based on the dipping force and rotation trajectory. In this embodiment, the first direction is a direction perpendicular to the surface of the other cosmetic product; the second direction intersects with a third direction. The target plane is a plane formed by the second direction and the third direction on the surface of the other cosmetic product.

[0116] First, the robotic arm and dexterous hand, carrying the matching makeup tool, are moved to a safe position directly above other cosmetic products (e.g., 2-3 cm above their surfaces). Then, based on the acquired application force, the makeup tool is moved in a direction perpendicular to the surfaces of the other cosmetic products. During this movement along the first direction, compliant interaction is performed, causing the makeup tool to slowly approach downwards until the actual force exerted on the surfaces of the other cosmetic products reaches a preset application force (e.g., 0.5 N). During the application phase, when the actual force reaches the application force, based on the determined application force and rotational trajectory, the robotic arm and dexterous hand are controlled to move on a target plane to obtain the desired makeup material. The target plane lies on the surface of the other cosmetic products and is a plane formed by a second and a third direction (typically two mutually orthogonal horizontal directions). During this phase, compliant force control is continuously performed in the first direction to maintain a consistent application force, while a rotational trajectory (e.g., a plane parametric equation) is executed on the target plane. The defined spiral pattern simulates the action of a real person using a tool to pick up powder in a powder box by making circular motions. The appropriate picking force and rotation trajectory are dynamically adjusted according to the different textures of the cosmetics. For example, a smaller picking force (0.3-0.5N) and a faster, fewer rotation trajectory are used for loose powder; a moderate picking force (0.5-0.8N) and a moderate number of rotations are used for creams; and a smaller picking force (0.2-0.4N) and correspondingly reduced or eliminated rotational movement on the target plane are used for liquids to solve the problem of excessive liquid extrusion.

[0117] In some embodiments, step 340 further includes: if the actual force exerted by the matched makeup tool on the surface of other cosmetic products in the first direction is greater than or equal to a force threshold, controlling the robotic arm and dexterous hand to stop moving towards other cosmetic products, and controlling the matched makeup tool to move in the opposite direction of the first direction. In actual execution, after the dipping action is completed, the robotic arm and dexterous hand are first lifted vertically a short distance along the first direction at a relatively slow speed. After the force sensor confirms that the actual force has returned to zero and the tool has completely detached from the surface, the horizontal movement is then performed, thereby effectively reducing powder splashing.

[0118] In some embodiments, step 340 further includes: continuously monitoring the actual force during the process of controlling the robotic arm and dexterous hand to press down in the first direction; and stopping the movement and urgently lifting the robotic arm when the actual force is detected to increase sharply or be overloaded, so as to reduce the risk of equipment damage caused by object placement deviation or tool jamming.

[0119] In some embodiments, after step 340, the method includes: if the amount of other cosmetic products picked up is insufficient to meet makeup requirements, controlling the execution component to repeatedly perform interactive actions on other cosmetic products. For the closed-loop control of the repeated picking action, a Task State Machine can act as the global execution entity, responsible for coordinating and scheduling the robotic arm, end effector, tool holder, and cosmetic containers. When the Task State Machine receives a "powder quantity insufficient" signal from the evaluation module, it determines that the current single powder quantity cannot meet subsequent makeup application needs. At this time, the Task State Machine will automatically reuse the previously determined picking parameters (including picking force and rotation trajectory) to control the robotic arm and end effector to repeatedly perform picking interactive actions on other cosmetic products until a sufficient amount of makeup material is obtained.

[0120] An adaptive optimization mechanism can also be built based on real-world operational data. For example, during each interactive action (i.e., the powder-picking action), the actual force data between the matching makeup tool and other cosmetic products is collected and recorded (e.g., generating a force feedback curve that changes over time). By extracting the mechanical characteristics from the actual force data, the preset application force (i.e., the target force) is iteratively updated and dynamically optimized accordingly.

[0121] Step 350: Control the execution component to interact with the user's face.

[0122] In actual execution, multiple application strategies can be implemented to achieve a more refined makeup effect. During the first application (i.e., the base stage), a faster movement speed and lower target contact pressure are configured to drive the robotic arm and dexterous hand to hold the makeup tool and perform the initial application along a preset facial trajectory. After the first application, if it is determined that makeup material needs to be replenished, the powder replenishment logic is automatically triggered. At this time, the robotic arm and dexterous hand hold the makeup tool and temporarily leave the facial work area, moving it to the corresponding makeup product (such as a blush powder compact). Subsequently, the aforementioned smooth interaction logic and pick-up parameters are reused to control the robotic arm and dexterous hand to perform vertical downward pressure and slight friction movements within the target plane (i.e., the aforementioned rotation trajectory), so that the powder-picking surface of the makeup tool can reabsorb sufficient makeup material. After the automatic powder replenishment is completed, the robotic arm and dexterous hand are driven to carry the makeup tool back to the facial work area to enter the second or third deep blending stage. During this stage, the movement speed of the robotic arm and dexterous hand is adaptively reduced, and the target contact elasticity parameter in the compliant interaction model is fine-tuned to a lower numerical range. This increases the smoothness of the makeup tools' fit to the facial skin and the effective contact time, enabling a natural transition and blending of makeup colors with the skin. It can be understood that, for example, in scenarios requiring a color gradient effect, the robotic arm and dexterous hand can carry multiple different makeup tools within the same target facial area, coordinating interactive actions according to a preset execution sequence and target contact pressure.

[0123] During the research and development process, the inventors also discovered that with the development of artificial intelligence and service robot technology, automated makeup has gradually become a research hotspot. In actual makeup application, complex factors such as changes in lighting conditions, differences in skin texture, uneven powder application, or changes in user posture can easily cause makeup deviations such as uneven coloring, too light a color, or patchiness in the target area. When the makeup result deviates from expectations, the relevant technology struggles to automatically correct the deviations, resulting in a low tolerance for error and a final effect that is difficult to achieve the desired outcome, often requiring manual intervention for secondary touch-ups. To address the issue of low makeup quality after automated makeup application, the inventors, through in-depth research, designed a control method for a makeup robot. The makeup robot includes an execution component, comprising a robotic arm and a dexterous hand at the end of the robotic arm, used to hold matching makeup tools. The method includes: upon receiving a makeup analysis command, analyzing the acquired facial image information of the user's face to obtain a makeup quality score; if the makeup quality score is less than a preset makeup quality threshold, determining a makeup repair strategy matching the defect type based on the defect type corresponding to the target sub-facial region of the user's facial area; and controlling the execution component to operate based on the makeup repair strategy. According to the control method of the beauty robot provided in the embodiments of this application, the user's facial image information is analyzed to obtain a makeup quality score. When the score is lower than a preset threshold, the corresponding makeup repair strategy is adaptively matched according to the specific defect type of the target sub-facial area, thereby driving the execution component in the beauty robot to perform precise makeup repair. By introducing a closed-loop quality detection and dynamic error correction mechanism based on visual feedback, the color deviation caused by external interference such as changes in light or differences in skin texture can be effectively compensated, thereby improving the makeup quality and user experience of the automatic makeup operation.

[0124] like Figure 4 As shown, the control method of the beauty robot includes steps 410, 420 and 430.

[0125] Step 410: Upon receiving a makeup analysis instruction, analyze the acquired facial image information of the user's face to obtain a makeup quality score. In this step, the makeup analysis instruction can be triggered based on various working conditions, including but not limited to: stage-by-stage triggering during the makeup operation (e.g., based on a preset time interval or timer), automatic triggering after the completion of a single or overall makeup task, such as triggering once after completing eye makeup and once after completing base makeup; or triggering after completing the entire face makeup; and manual instruction triggering based on the user interface.

[0126] For reference Figure 6 or Figure 7The makeup robot's makeup process, based on steps 110-130, 210-250, and 310-350 as described above, involves controlling a robotic arm and dexterous hand to hold matching makeup tools and apply makeup to the user. Afterward, the user enters the makeup quality assessment module to evaluate the current quality of the user's makeup; then, it continues to refer to... Figure 7 If the makeup quality is acceptable, the current makeup task ends; if the makeup quality is unacceptable, defect detection is performed, and a repair trajectory is planned and generated based on a local trajectory planner to obtain a repair strategy corresponding to the defective parts of the makeup.

[0127] In some embodiments, prior to step 410, the method further includes: obtaining a preset standard color range and a minimum coverage threshold. The standard color range is characterized based on the HSV (Hue, Saturation, Value) color space. The standard color range refers to the reasonable range of values ​​corresponding to the target makeup color in the HSV color space. During image analysis, if the HSV value of a pixel in the target facial region falls within the standard color range, the pixel is determined to be a qualified pixel (i.e., a covered pixel). The minimum coverage threshold is used to characterize the minimum lower limit of the proportion of qualified pixels in the makeup area to the total number of pixels in the entire target facial region. For example, when set to 95%, it means that at least 95% of the pixels in the target facial region must be determined to be qualified pixels in order to be considered as having satisfactory coverage.

[0128] In some embodiments, the minimum coverage threshold can be based on an overall threshold value or a regional dynamic threshold setting.

[0129] In this embodiment, the overall threshold value can be set within the range of 85% to 95%. Regional thresholds refer to setting corresponding coverage requirements for different areas of the same makeup task. For example, in blush application, the minimum coverage threshold for the core blush area can be set to a higher standard (e.g., 95%), while the threshold for the edge transition area can be appropriately relaxed (e.g., 85%) to more accurately simulate the natural blending and transition effects of the makeup edges.

[0130] In some embodiments, the standard color range is determined based on the HSV color space representation (including H hue, S saturation, and V brightness range), the type of cosmetic (such as blush, eyeshadow, and lipstick), and the specific color depth of the cosmetic product; the minimum coverage threshold parameter is determined based on the makeup integrity requirements of the target makeup area. For example, for blush, in order to achieve a healthy, natural, and warm red tone that suits a specific skin tone, its standard color range can be determined as follows: H hue range of 0 to 10 degrees, S saturation range of 0.3 to 0.7, and V brightness range of 0.4 to 0.8. Accordingly, its minimum coverage threshold can be determined as 90%, which means that at least 90% of the area (or pixels) of the target facial region needs to reach the above standard color range to be considered acceptable.

[0131] In some embodiments, the standard color range can be obtained through standard color charts, sample analysis, expert evaluation, and custom input.

[0132] In practice, standard color chart calibration involves extracting HSV baseline values ​​from color chart images under standard lighting conditions; sample analysis involves statistically analyzing a large number of ideal makeup images to define color value ranges that conform to aesthetic preferences; expert evaluation refers to inviting professional makeup artists to score the samples and selecting the HSV features of high-scoring samples as baseline values; and custom input refers to dynamically adjusting relevant parameters according to the cosmetic category and user skin tone characteristics, and storing them in the form of configuration files to achieve adaptation to different beauty products and application scenarios.

[0133] In some embodiments, the minimum coverage threshold parameter is based on experimental verification, dynamic thresholds, and user preference settings. In actual implementation, experimental verification refers to determining the minimum coverage threshold by statistically analyzing the visual effects and repair success rates of makeup under different thresholds through extensive actual makeup tests. Dynamic thresholds refer to the ability to dynamically adjust the threshold based on factors such as lighting conditions and skin tone; user preferences allow users to adjust the threshold according to their personal preferences. In actual implementation, the makeup quality assessment module and the decision state machine work together as the execution entities to collaboratively execute the automated assessment process. First, preset quality standards such as the standard HSV color range and minimum coverage threshold are loaded from the configuration file or digital tool library. After each application action (including initial application or repair application), the robotic arm and dexterous hand are moved to a safe observation position, and the vision module is instructed to acquire a high-resolution image of the current target makeup area (e.g., the blush area). Subsequently, multi-dimensional feature analysis is performed on the acquired image to calculate the proportion of pixels within the standard HSV color range that meet the standards, obtaining color uniformity and coverage integrity, and a comprehensive makeup quality score (denoted as Scoretotal) is calculated.

[0134] refer to Figure 7, then perform a qualification judgment process: compare the calculated total score (Scoretotal) with a preset quality threshold (Threshold). If Scoretotal ≥ Threshold: refer to Figure 7 , then determine that the current makeup quality is qualified, mark the completion of the makeup task in this area, and control the robotic arm and the dexterous hand to perform the corresponding tool return actions. If Scoretotal < Threshold: refer to Figure 7 , then determine that the current makeup is unqualified and enter the defect detection process. In the defect detection process, by analyzing the spatial distribution characteristics of the non-compliant pixels and using the heat map mapping technology, determine the defective sub-region with the largest color difference or coverage missing, and extract the center coordinates (xdefect, ydefect) and the influence range Rdefect of this defective sub-region based on this, and use them as the precise repair parameters for generating the adaptive repair strategy subsequently. Further, input the above-extracted precise repair parameters into the local trajectory planner. Based on the specific spatial coordinates and range of the defect, the local trajectory planner constructs a targeted local repair path and feeds back the local path information to the trajectory planning and generation module. In the trajectory planning and generation module, combine the dynamic pose calculation data to globally fuse and reconstruct the local repair path, and then output updated motion instructions to the downstream instruction synthesis module to drive the robotic arm and the dexterous hand to hold the matching makeup tool to perform the makeup repair operation. In some embodiments, the detection data can also be continuously recorded for later iterative optimization of the standard color range and coverage threshold parameters.

[0135] In some embodiments, step 410 includes: performing semantic segmentation processing on the facial image information to obtain the color uniformity and coverage integrity of the target sub-facial area; performing weighted calculation on the color uniformity and coverage integrity to obtain the makeup quality score. The color uniformity refers to the degree of consistency between the color characteristics of each pixel point in the target sub-facial area and the preset standard color, and is used to characterize the flatness and color deviation of the cosmetic application; the coverage integrity refers to the proportion of the number of compliant pixels in the target sub-facial area to the total number of pixels in this area, and is used to quantitatively evaluate the filling degree of the cosmetic to the target area. In the actual execution process, a semantic segmentation network (such as DeepLabV3+) can be used to achieve pixel-level extraction of the makeup area. Compared with the conventional segmentation technology that locks the overall region of interest (ROI) of the human face, the semantic segmentation network can identify different semantic categories such as the blush area, the foundation area, and the eyeshadow area, making the quality assessment more targeted at the actual makeup range, thereby reducing the interference of the natural colors of non-makeup parts such as the eyes and mouth and improving the objectivity of the scoring.

[0136] In some embodiments, semantic segmentation networks can also be used to locate makeup boundaries and defect areas. In actual execution, for tasks with high boundary control requirements, such as lipstick application, semantic segmentation networks can be used to identify the physical boundaries of target organs, and the boundary clarity can be analyzed by comparing the overlap between the makeup area and the target anatomical area. When the makeup quality score is below standard, the semantic segmentation results are analyzed in conjunction with heatmap methods to identify pixel clusters with large color differences or missing coverage, and the center coordinates and range of the defect area are output, providing guiding data for subsequent local repair. It is understood that a single makeup task typically involves multiple target areas (e.g., bilateral cheek areas), and semantic segmentation networks can simultaneously and in parallel identify and track multiple independent target areas, thereby achieving classification and evaluation of the makeup quality of each sub-region and generating corresponding adaptive repair decisions.

[0137] In some embodiments, the semantic segmentation network can be trained on a labeled dataset to learn and extract makeup area features under different skin tones and complex lighting conditions, thereby improving the system's robustness in diverse environments. In this embodiment, for different types of cosmetics (such as blush, eyeshadow, or lipstick), differentiated dedicated segmentation models can be trained and deployed separately, which can further improve the semantic segmentation network's recognition accuracy and segmentation detail for specific makeup styles, thus providing a data foundation for high-quality makeup evaluation.

[0138] According to the control method of the beauty robot provided in the embodiments of this application, by acquiring facial image information and combining it with high-precision visual processing algorithms such as semantic segmentation networks for multi-dimensional feature analysis, the color uniformity and area coverage integrity of the target sub-facial region in the standard color space can be accurately extracted, and then a comprehensive makeup quality score can be calculated by weighting. This not only provides a quantitative judgment benchmark for the generation of subsequent automated defect location and repair strategies, but also enables the beauty robot system to have a closed-loop detection capability for autonomous quantitative evaluation of the work results, thereby improving the makeup quality and user experience of automated makeup operations.

[0139] In some embodiments, semantic segmentation processing is performed on facial image information to obtain the color uniformity and coverage integrity of the target sub-facial region. This includes: extracting color feature parameters of the target sub-facial region and obtaining color uniformity based on the degree of difference between the color feature parameters and a preset standard color; counting the number of defective pixels in the target sub-facial region that are not colored or have excessive color difference, and calculating the coverage rate based on the number of defective pixels and the total number of pixels in the target sub-facial region to obtain coverage integrity. For ease of explanation, this embodiment uniformly quantifies the color uniformity used to evaluate color consistency as a color uniformity score; and uniformly quantifies the coverage integrity used to evaluate pixel fill ratio as a coverage integrity score. In actual execution, after the control robotic arm and dexterous hand hold the matching makeup tools and complete the application action, they move to a preset safe observation position. At the same time, an RGB-D camera is used to acquire image information of the facial makeup area in high-resolution mode; in such cases... Figure 6The makeup quality assessment module, as shown, processes the acquired images using a semantic segmentation network (such as DeepLabV3+) to extract pixel-level masks for specific makeup areas, such as blush or eyeshadow, thereby reducing interference from non-target areas like the eyes and mouth in subsequent analysis. Once the module identifies substandard blemishes through feature analysis, it further calculates and outputs the specific blemish location information, forming defect coordinate feedback. This feedback is then fed back to the front end of the visual loop to guide the next high-definition image acquisition (such as close-up acquisition of blemish areas) or directly participate in subsequent dynamic baseline trajectory generation. Color feature parameters of the target sub-facial region are extracted, and color uniformity is obtained based on the degree of difference between these parameters and a preset standard color. This can be achieved by converting the extracted makeup area image from the RGB (Red, Green, Blue) color space to the HSV color space, which includes hue, saturation, and brightness dimensions. The decoupling characteristics of the HSV space for brightness and chromaticity information enhance the robustness of color analysis. The color features of each pixel within the target sub-face region are compared with preset standard color chart parameters. The degree of color deviation is quantified by calculating the Euclidean distance between each pixel and the standard target color in HSV 3D space. The obtained Euclidean distance values ​​are normalized and mapped to a preset score range (e.g., 0-100 points). In this embodiment, the smaller the calculated Euclidean distance, the closer the pixel is to the standard color, resulting in a higher color uniformity score for the region. In actual execution, the number of uncolored or excessively color-difference defective pixels within the target sub-face region is counted. The coverage rate is calculated based on the number of defective pixels and the total number of pixels in the target sub-face region to obtain the coverage integrity. This can be achieved as follows: Based on a preset color deviation threshold, quality analysis is performed on each pixel within the target sub-face region. Pixels that are uncolored or have excessively large color differences are identified as defective pixels. The calculation process of coverage can be expressed by the formula: Coverage rate = (Total number of pixels in the makeup area - Number of defective pixels) / Total number of pixels in the makeup area × 100%. The obtained coverage rate value is directly mapped to the corresponding coverage integrity score. For example, if the coverage rate of the target sub-facial area is calculated to be 95%, the corresponding coverage integrity score is determined to be 95 points.

[0140] In some embodiments, a weighted calculation of color uniformity and coverage integrity is performed to obtain a makeup quality score. This includes: determining a weight sequence based on the cosmetic category corresponding to the makeup in the target sub-facial region; and performing a weighted calculation of color uniformity and coverage integrity based on the weight sequence to obtain the makeup quality score. A weighted average method can be used to calculate the makeup quality score, expressed by the formula: Makeup Quality Score = Weight 1 × Color Uniformity Score + Weight 2 × Coverage Integrity Score. Weight 1 corresponds to the color uniformity score, and Weight 2 corresponds to the coverage integrity score.

[0141] In some embodiments, the weight sequence can be determined based on the actual makeup effect requirements of different cosmetics. For example, for blush cosmetics, when determining the weight sequence, the proportion of color uniformity can be increased (e.g., weight 1 is set to 0.6 and weight 2 to 0.4).

[0142] Step 420: If the makeup quality score is less than the preset makeup quality threshold, determine a makeup repair strategy that matches the defect type based on the defect type corresponding to the target sub-facial region of the user's facial area. In this step, the target sub-facial region refers to each area of ​​the user's face, such as the eyes, mouth, or eyebrows.

[0143] In actual execution, the calculated comprehensive score (i.e., makeup quality score) is compared with a preset makeup quality threshold (e.g., a passing score of 85). If the makeup quality score is not lower than the preset threshold, the makeup in the current target area is deemed acceptable, and the current makeup task or inspection process ends. If the makeup quality score is lower than the preset threshold, the current makeup is deemed unacceptable, and the process automatically enters the defect location and repair process. In subsequent processes, further analysis is performed based on the specific defect type corresponding to the target sub-facial area within the user's facial region. This allows for the determination of a makeup repair strategy that strictly matches the defect type, enabling the beauty robot to achieve automated closed-loop error correction and further improve the final makeup effect.

[0144] In some embodiments, step 420 includes: performing defect analysis on the target sub-facial region in the facial image information to determine at least one defect type among broken lines, uneven color, and blurred boundaries; and determining a makeup repair strategy based on at least one defect type among broken lines, uneven color, and blurred boundaries.

[0145] refer to Figure 8In actual execution, the beauty robot initially operates in standby mode. Upon timer triggering or the arrival of a new image frame, the visual processing thread begins image scanning. When a quality inspection request is received from a preceding workflow (such as the completion of an application process), it adaptively switches to quality inspection mode and drives the vision module to acquire facial image streams at high resolution. In this mode, the core data processing path is as follows: the acquired image stream is input into the relevant network architecture for pose calculation and semantic segmentation. Subsequently, the output head spatial pose data, along with the pixel-level mask and color distribution histogram of the target region (such as the blush area), are written into shared memory for efficient use by subsequent business logic. When an anomaly is detected during the image scanning stage (i.e., the calculated makeup quality score is less than a preset makeup quality threshold), a defect identification process is triggered. In the defect identification process, firstly, based on the basic data in shared memory, deep multidimensional feature extraction is performed on the target sub-facial region in the facial image information (e.g., RGB-D camera image stream). Firstly, features are extracted from the target sub-facial region in the facial image information (e.g., RGB-D camera image stream). Based on a three-dimensional facial alignment model (e.g., 3DDFA_V2 deep learning model) and a semantic segmentation network (e.g., DeepLabV3+), the defect type is analyzed. For example, by combining quantitative information such as the image's edge contour features and color distribution gradient, local blemishes are classified, thereby determining that the anomaly belongs to at least one defect type among broken lines, uneven color, and blurred boundaries. Based on the identified specific defect types such as broken lines, uneven color, and / or blurred boundaries, a makeup repair strategy matching the defect type is further determined. In some embodiments, the defect type can also be customized based on specific makeup areas or the material properties of cosmetics (e.g., powder, liquid, etc.), the user's personalized facial features (e.g., skin texture, skin tone differences, etc.), and makeup style (e.g., everyday natural makeup, stage makeup, etc.) or determined based on historical experimental data; this application does not impose any limitations on this.

[0146] According to the control method of the beauty robot provided in this application embodiment, a defect recognition process is triggered when the makeup quality score fails to meet the standard. This process can perform feature analysis on facial image information to locate local blemish areas and identify defect types such as broken lines, uneven color, or blurred boundaries. Based on the above analysis results, a local repair trajectory and operation strategy matching the specific defect type can be automatically planned. This drives the robotic arm and dexterous hand to grasp the matching makeup tools and perform targeted repairs on the target sub-facial area. This achieves automated error correction of local makeup blemishes and improves the makeup quality and user experience of automated makeup operations.

[0147] In some embodiments, a makeup repair strategy is determined based on at least one defect type among broken lines, uneven color, and blurred boundaries. This includes: when the defect type is broken lines, determining the makeup repair strategy as a fine-line drawing mode, corresponding to a first repair trajectory parameter and a first force control interaction parameter; when the defect type is uneven color, determining the makeup repair strategy as a fill mode, corresponding to a second repair trajectory parameter and a second force control interaction parameter; and when the defect type is blurred boundaries, determining the makeup repair strategy as a blending mode, corresponding to a third repair trajectory parameter and a third force control interaction parameter. In this embodiment, the values ​​of each repair trajectory parameter and each force control interaction parameter can be the same or different, and their specific values ​​can be customized based on the actual makeup process or determined based on historical experimental data. For defects such as missing or broken lines, the fine-line drawing mode can be switched to and the fine-line drawing brush can be used for makeup repair; for defects such as mottled color patches or uneven color, the fill mode can be switched to and the fill brush can be used for large-area color filling; and for defects such as blurred boundaries, the blending mode can be switched to and the blending stick can be used for soft edge transition processing. Understandably, when multiple defect types exist simultaneously in the target sub-facial region, the optimal tool switching sequence can be autonomously planned and generated by combining the spatial distribution characteristics of each defect with the physical properties of the corresponding repair tools.

[0148] According to the control method of the beauty robot provided in the embodiments of this application, based on specific defect types such as broken lines, uneven color, or blurred boundaries, the corresponding makeup repair strategy is adaptively determined, and matching repair tools and underlying control parameters are called. Differentiated and precise repair actions can be performed for different blemish features, thereby realizing automated error correction of local makeup blemishes, effectively improving the makeup quality, error tolerance rate, and user experience of automated makeup operations.

[0149] Step 430: Control the operation of the execution components based on the makeup repair strategy. In this step, when the received current makeup requirement instruction is "makeup repair" (e.g., the received makeup trajectory is "repair trajectory"), the current makeup parameters are switched to fine repair parameters, and the operation of the robotic arm and dexterous hand is controlled based on the fine repair parameters to reduce the phenomenon of local makeup material accumulation after applying makeup to local blemishes using conventional makeup parameters.

[0150] In some embodiments, step 430 includes: acquiring repair trajectory parameters and force control interaction parameters corresponding to the makeup repair strategy; controlling the execution component to reduce motion inertia parameters and contact elasticity parameters, increase motion resistance parameters, and reduce target force according to the force control interaction parameters; the target force is the force between the matching makeup tool held by the execution component and the user's face; controlling the execution component to reduce movement speed, reduce trajectory point spacing, and cover and repair the target sub-facial area corresponding to the defect type along the target trajectory according to the repair trajectory parameters; the target trajectory is a locally reciprocating or spiral trajectory. In this embodiment, the target trajectory is a locally reciprocating or spiral trajectory. The target force is the force between the matching makeup tool held by the robotic arm and dexterous hand and the user's face. Compared to conventional large-area application operations, the underlying smooth interaction model can be dynamically adjusted according to the force control interaction parameters in the force control interaction dimension.

[0151] For example, the motion inertia parameter can be reduced from a relatively large value (e.g., 2.0-5.0 kg) to a lower value (e.g., 0.5-1.0 kg) during conventional application to improve dynamic response speed. This allows the matching makeup tools held by the robotic arm and dexterous hand to more sensitively follow subtle mechanical changes on the face. The motion resistance parameter can be increased from a moderate value (e.g., 50-100 N·s / m) to a higher value (e.g., 100-200 N·s / m) to effectively suppress high-frequency jitter or force control overshoot during fine local movements by enhancing the resistance effect. The contact elasticity parameter can be reduced from a conventional value (e.g., 30-50 N / m) to a lower level (e.g., 10-20 N / m) to give the end a smoother interactive characteristic, thus more delicately conforming to the microscopic curves of the face. While adjusting the above model parameters, the target force can be reduced from the moderate value used in conventional coloring (e.g., 0.5-1.0 N) to 0.2-0.4 N to reduce the risk of harsh makeup or overly dark color caused by excessive pressure on defective areas with existing base color. In terms of motion trajectory planning, the motion logic can be adjusted according to the repair trajectory parameters. The movement speed of the robotic arm end effector can be reduced from the relatively high speed of conventional operations (e.g., 0.02-0.05 m / s) to a fine operation speed (e.g., 0.005-0.01 m / s) to improve the accuracy and safety of local repairs through physical deceleration. The spacing between trajectory waypoints can be reduced from the larger distance of conventional coverage (e.g., 2-3 mm) to a denser spacing (e.g., 0.5-1 mm) to generate a high-density local repair path, ensuring that evenly covered sub-areas with fine defects are fully and uniformly covered by the paint.

[0152] In some embodiments, unlike the large-scale circular or S-shaped trajectories used in the conventional application stage, the target trajectory in the fine-touch stage can be primarily limited to local reciprocating motions (e.g., small-scale figure-eight reciprocating motions) or spiral shapes (e.g., small-radius spirals). This allows the robotic arm and dexterous hand to control the matching makeup tool, which is then held along the target trajectory, to precisely cover and repair the target sub-facial area corresponding to a specific defect type. In actual execution, the defect coordinates output from the aforementioned steps are converted into input parameters, which directly drive the robotic arm and dexterous hand to perform subsequent actions. (Reference) Figure 8 In the local trajectory planning stage, high-density local repair trajectories can be generated with the center of the located defect sub-region as the origin (e.g., using a small-radius spiral or a small-range figure-eight reciprocating motion). Simultaneously, the effective coverage area of ​​this trajectory can be set slightly larger than the calculated defect influence range to achieve full envelopment of the flawed area. Combined with the currently matched repair tool and working mode, compliant interaction (MBK) parameters specifically designed for "fine repair" scenarios are invoked, such as loading the aforementioned low contact elasticity parameters and low movement speed configurations, to improve the compliant contact characteristics and motion accuracy during local repair operations. (Continue to refer to...) Figure 8 The generated local path can be fed back to the trajectory planning and generation module for path fusion, and joint commands can be output downstream. This then calls the underlying interfaces corresponding to the aforementioned vision and force control architecture to drive the robotic arm and the matching makeup tool held by the dexterous hand to complete the closed-loop repair execution along the planned trajectory. (Continue to refer to...) Figure 8 After completing the closed-loop repair process described above, the process enters the secondary quality inspection stage. In this stage, a visual processing thread re-evaluates the makeup quality of the target sub-facial area after local repair. If the assessment determines that the repaired makeup quality meets the preset standard (i.e., is deemed acceptable), the repair task for the current target sub-facial area is marked as complete, and the robotic arm and dexterous hand are controlled to perform the corresponding tool return action or move to the next area. If the assessment determines that the repaired makeup quality does not meet the preset standard (i.e., is still deemed unacceptable), a dynamic feedback mechanism is triggered, tracing the control commands back along the closed-loop path to the aforementioned tool selection logic node. Based on the current local image features, the remaining defect type is re-analyzed, and the repair strategy is adaptively adjusted or the corresponding repair mode (such as detailed drawing mode, fill mode, or blending mode) is switched accordingly, initiating a new round of repair iterations until the makeup quality of the area passes the quality inspection.

[0153] According to the control method of the beauty robot provided in the embodiments of this application, a matching local repair trajectory and strategy are automatically planned based on the defect analysis results. During the repair process, by dynamically adjusting the parameters of the compliant interaction model and the motion planning parameters, the matching makeup tools held by the robotic arm and the dexterous hand can delicately conform to the facial curves with more sensitive and compliant interaction characteristics. This effectively suppresses high-frequency vibrations of fine movements and solves the problem of stiff makeup effects caused by excessive pressure, thereby effectively improving the makeup quality, error tolerance, and user experience of automated makeup operations.

[0154] In some embodiments, after step 430, the method further includes: updating the makeup quality score; based on the updated makeup quality score, repeatedly executing the step of "determining a repair strategy matching the defect type based on the defect category corresponding to the target sub-facial region of the user's face when the makeup quality score is less than a preset quality threshold", and obtaining the number of repeated executions; when the number of repeated executions reaches a preset retry threshold and the updated makeup quality score is less than the preset makeup quality threshold, controlling the execution component to stop the makeup task and output a warning message.

[0155] In actual execution, after each local repair action is completed, the makeup quality detection can be re-executed to obtain the updated makeup quality score. The retry counter is incremented to obtain the current number of repetitions, and the loop feedback logic is entered based on the updated data: if the number of repetitions is less than the preset retry threshold and the updated makeup quality score is still less than the preset makeup quality threshold, a repair trigger command and the corresponding defect coordinates are output to drive the robotic arm and dexterous hand to repeat the aforementioned makeup repair process; if the number of repetitions reaches the preset retry threshold and the updated makeup quality score is still less than the preset makeup quality threshold, a task termination command is output to control the robotic arm and dexterous hand to stop the current makeup task, and an early warning message is output to prompt human intervention.

[0156] In actual execution, the closed-loop repair status can be monitored throughout the process. When the local makeup quality is reassessed and determined to meet the qualified standard, or when the local repair process has been repeated a maximum number of times even though the assessment is not qualified, the process termination command is output to control the robotic arm and dexterous hand to perform the corresponding tool return action and end the makeup operation.

[0157] In some embodiments, step 430 includes: during the process of controlling the execution component to perform makeup repair on the target sub-facial area based on the makeup repair strategy, acquiring the actual force between the matched makeup tool and the user's face; and if the actual force exceeds a preset force threshold, controlling the execution component to trigger an emergency stop operation. In actual execution, throughout the entire process of controlling the robotic arm and dexterous hand to perform closed-loop makeup repair on the target sub-facial area based on the makeup repair strategy, underlying physical safety and control logic monitoring is run synchronously. For example, acquiring the actual force between the currently matched makeup tool and the user's face; once the actual force is detected to exceed the preset force threshold (i.e., the set safety force limit), immediately controlling the robotic arm and dexterous hand to trigger an emergency stop operation to reduce potential collision risks during human-computer physical interaction.

[0158] In actual execution, the safety monitoring module continuously acquires force data between the currently matched makeup tool and the user's face to make mechanical safety judgments regarding the task queue flow and physical contact status during the closed-loop repair process. Once the absolute value of the actual force is detected to be greater than the preset safety force threshold, a high-priority intervention command is immediately issued to control the robotic arm and dexterous hand to trigger an emergency stop operation, thereby effectively reducing the potential collision risk during human-machine physical interaction.

[0159] In the actual execution of this blush application task, the task state machine, as the core execution entity, is responsible for the overall management of tool scheduling and the closed-loop makeup process. The task state machine predefines various execution states for tool switching and the makeup process, including but not limited to: idle state (i.e., system standby), initialization state, tool recognition state (e.g., scanning and locating the blush brush), gesture adjustment state, tool grasping state, dipping state (e.g., taking powder from the powder box), makeup execution state (i.e., performing the aforementioned double-loop application), quality inspection state, local repair state, tool return state, completion state, and abnormal state. The task state machine can control the continuous state transition of the entire process according to preset entry and exit conditions. For example, after the quality inspection state ends, the task state machine can automatically determine branches based on various comprehensive score results: if the score is qualified, the process flows to the tool return state; if the score is unqualified and the preset maximum retry limit has not been reached, the process flows backward to the local repair state; if the score is unqualified and the retry limit has been reached, the process is forced to flow to the abnormal state and subsequent actions are blocked.

[0160] For complex physical interactions (such as the "approach-press-rotate-lift" action in the blush application process mentioned above), the task state machine can call nested sub-state machines for fine-grained management. Once the sub-state machine process is closed, control is returned to the main process. Regarding global safety management, the safety monitoring state machine can operate independently in parallel, continuously monitoring the mechanical data at the robotic arm's end effector. If an emergency stop condition is triggered, the current makeup task is immediately interrupted. Furthermore, the task state machine is responsible for state persistence and recovery. By recording the current running state and key contextual data (such as the number of repairs completed and the currently used tools), the overall control flow can accurately resume execution from the breakpoint after a brief failure or communication interruption, significantly improving the fault tolerance and reliability of the makeup robot system's fully automated operation.

[0161] This application uses the example of a robot performing a "blush application" task to illustrate the detailed process of the entire operation. This process can connect technologies such as fully automatic switching of multiple tools, vision-force control dynamic replanning, MBK flexible interaction, and visual closed-loop quality inspection to achieve fully automated operation from tool recognition, grasping, dabbing to final closed-loop quality inspection.

[0162] In the intelligent tool recognition and humanoid grasping stage, the vision module performs image scanning operations. By recognizing the feature markings on the blush brush handle on the workbench, it calculates the tool's precise 3D position and posture in the robot's base coordinate system. Subsequently, it retrieves a dedicated parameter package from its built-in digital tool library. This package includes a hand-eye calibration matrix, target angle sequences for each joint of the dexterous hand, and grasping force thresholds. The robotic arm is controlled to move to a safe observation point above the matched makeup tool, and the dexterous hand is controlled to perform continuous movements according to the grasping parameters, replicating the grasping posture of a human holding a slender-handled tool. After grasping, force feedback confirms the tool's stability. In the facial key point capture and dynamic trajectory generation stage, the robotic arm and dexterous hand can move the blush brush to a safe observation position in front of the face. The vision module, as the execution entity, can acquire images of the user's face and use a 3D face reconstruction algorithm to calculate the six degrees of freedom of the face's posture, thereby accurately locating key anatomical points such as the start point, intermediate transition point, and end point of the blush area. Based on this, a basic application path can be generated and dual-compensation trajectory planning can be performed: monitoring the user's head micro-movements and, upon detecting changes in facial posture, superimposing visual displacement compensation on the original trajectory, allowing the makeup tool to dynamically follow facial movements; combining the expected force error to calculate force-controlled displacement compensation and superimposing it on the trajectory. Furthermore, an extension offset can be automatically added to the edges of the basic path to enhance the naturalness of the edge blending effect. In the adaptive smooth application and automatic powder replenishment stage based on the MBK model, the matching makeup tool held by the robotic arm and dexterous hand can approach the user's face along the planned trajectory, triggering MBK smooth interaction upon detecting physical contact or reaching a predetermined depth. During the variable-parameter smooth interaction process, a lower contact elasticity parameter can be dynamically set based on the physical characteristics of the blush brush and cheek, making the held makeup tool exhibit higher smoothness to conform to facial curves; simultaneously, an appropriate motion retardation parameter is set to absorb high-frequency vibrations, improving the smoothness of the application process. By comparing the force fed back by the sensors with the preset target pressure, the positions of the robotic arm and dexterous hand are automatically adjusted to maintain constant contact. When applying multiple layers using a different strategy, the first layer quickly creates a preliminary base. The robotic arm and dexterous hand then automatically move away from the face and to the powder container, performing a rotating trajectory of "vertical pressure and planar friction" to re-adsorb powder. In the subsequent blending stage, the movement speed is reduced and contact elasticity parameters are fine-tuned to increase contact time and promote natural color integration with the skin. During the visual quality detection and closed-loop intelligent completion stage, after completing the preset number of layers, the robotic arm and dexterous hand are moved to a safe observation position. The makeup quality assessment module, as the main execution unit, performs high-resolution imaging of the blush area and analyzes color uniformity and area coverage completeness, then calculates a weighted makeup quality score.In the intelligent decision-making stage, the score is compared with a preset pass threshold: if the score meets the pass standard, the process proceeds to the task completion node; if the score fails to meet the standard, a makeup defect is identified and the defect location logic is triggered. Subsequently, the local trajectory planner generates a targeted local repair trajectory based on the center coordinates of the defect area, controlling the robotic arm and dexterous hand to perform a second application of the localized material. After repair, the detection and evaluation can be repeated until the quality is satisfactory or the maximum number of safe retries is reached. In the task completion and tool return stage, once the secondary quality inspection confirms that the makeup is satisfactory, the task state machine determines that the current task has been successfully completed. The robotic arm and dexterous hand can carry the blush brush back to its initial position, and the dexterous hand is controlled to perform a release action, smoothly placing the tool back in its original position, thus safely ending the entire automated blush application process. Based on a compliant interactive control strategy with dynamic adjustment of dynamic parameters and a trajectory dynamic compensation mechanism using dual visual and force feedback, the robotic arm and dexterous hand are controlled to operate. This allows for dynamic adjustment of motion inertia, motion stagnation, and contact elasticity parameters according to the surface hardness of the facial area. Simultaneously, visual displacement compensation enables real-time and precise tracking of the user's facial micro-movements and active unloading of abnormal forces. This control method achieves a dynamic balance between soft contact and hard positioning, improving the accuracy of the work trajectory and the safety of the interaction during automated makeup operations. Furthermore, this application incorporates the actual makeup effect on the face, objectively measuring makeup quality through a closed-loop quality inspection and evaluation system. Based on specific defect types, corresponding repair strategies are autonomously generated. These repair strategies then drive the robotic arm and dexterous hand to repair localized makeup imperfections, effectively improving the error tolerance, makeup quality, and user experience of automated makeup operations.

[0163] During the research and development process, the inventors also discovered that related technologies typically employ robotic arms to carry makeup tools to perform makeup interaction actions. However, in actual human-computer interaction scenarios, these technologies struggle to respond promptly to unexpected situations or abnormal states, easily increasing safety risks during human-computer interaction. The lack of flexible and multi-dimensional user intervention and abnormality resolution mechanisms negatively impacts the safety and user experience of the makeup robot performing automated makeup tasks. To address the issue of low makeup quality after automated makeup, the inventors, through in-depth research, designed a control method for a makeup robot. The makeup robot includes an execution component, comprising a robotic arm and a dexterous hand at the end of the robotic arm, used to hold matching makeup tools. The method includes: receiving a first input from the user regarding recommended makeup information displayed by the makeup robot; the first input determining the target makeup information; responding to the first input and outputting an action guidance prompt; the action guidance prompt instructing the user to perform a specified action; generating a current makeup requirement instruction based on the user's facial image information during the specified action and the first input; and controlling the execution component to hold the matching makeup tools and perform the makeup action based on the current makeup requirement instruction. According to the control method of the beauty robot provided in this application embodiment, the target makeup information is determined by receiving the user's first input regarding the recommended makeup look, thus realizing personalized customization of the makeup task. Simultaneously, action guidance prompts are output to guide the user to perform specified actions, and makeup requirement instructions are generated based on facial image information during the dynamic process. This allows the robotic arm and dexterous hand to better match the user's personalized aesthetic and physiological characteristics when performing makeup actions while holding tools, effectively improving the makeup quality and user experience of automated makeup operations.

[0164] like Figure 5 As shown, the control method of the beauty robot includes steps 510, 520, 530 and 540.

[0165] Step 510: Receive the user's first input regarding the recommended makeup information displayed by the beauty robot; in this step, the first input is used to determine the target makeup information. In some embodiments, the source of the recommended makeup information may include: popular makeup styles collected from a public database, personalized schemes matched according to the user's preset preferences or historical data matching, and adaptive generation based on the user's facial data; this application does not limit this.

[0166] In practice, popular makeup looks collected from public databases can be, for example, publicly available makeup templates obtained by combining seasonal trends, specific holiday themes, or celebrity-endorsed looks; personalized solutions matched according to user preset preferences can be, for example, user-selected preference tags such as "daily commute," "evening," or "wild makeup" in the configuration center; makeup solutions determined based on user's historical makeup data can be, for example, directly calling the makeup configuration that the user used most frequently or rated the most in the past month; and personalized recommended makeup looks can be generated adaptively based on user facial data, such as the user's skin tone, facial proportions, and facial contour curvature.

[0167] In practice, the beauty robot can display one or more recommended makeup looks to the user through its built-in interactive interface (such as a display screen panel with augmented reality (AR) makeup try-on function) or a mobile application connected to it. The user can confirm or modify the displayed recommended makeup look information through touch clicks, parameter swiping and dragging, voice commands, or gestures, thus achieving the initial input.

[0168] In some embodiments, the target makeup information may include: makeup style (such as a sweet style, smoky style, natural nude makeup, etc.), makeup intensity (such as a light or high-coverage base makeup, light blending or heavy coloring, etc.), local area morphological features (such as the upward angle and length of the eyeliner, the coverage and shape of the blush, the specific arc of the eyebrow), target color space parameters (such as the standard HSV color value range corresponding to different parts), and cosmetic parameters that match the above makeup effect (such as the specified foundation shade, the physical texture of the cosmetic being powder or cream, etc.) and makeup tool parameters (such as the selection of a fine brush, a sponge beauty blender, or a powder brush).

[0169] In some embodiments, step 510 includes: displaying a virtual makeup try-on screen based on recommended makeup information; and receiving a fifth input from the user regarding the virtual makeup try-on screen. In this embodiment, the fifth input is used to customize makeup parameters.

[0170] In some embodiments, makeup parameters include at least one of makeup intensity, makeup coverage, makeup tool parameters, and makeup product parameters. During actual execution, users can fine-tune the makeup parameters by dragging the interface slider or touching the boundaries of specific makeup areas. Based on the fine-tuned makeup parameters, a recommended makeup image is regenerated and displayed on the user interface for confirmation. Users can manually specify makeup tool and product parameters by clicking the interface menu or toolbar, or provide physical makeup products, which the beauty robot automatically identifies, parses into corresponding parameters based on their identifiers, textures, and attributes.

[0171] Step 520: In response to the first input, output an action guidance prompt; in this step, the action guidance prompt is used to instruct the user to perform a specified action. In some embodiments, the action guidance prompt can be output based on voice broadcast or screen animation through the user interaction device of the beauty robot. In actual execution, the user performs the specified action under the instruction of the action guidance prompt, which may be, for example, guiding the user to adjust their face to the effective acquisition area of ​​the visual sensor, and guiding the user to make specific facial muscle movements or dynamic expressions (such as smiling, raising eyebrows, closing eyes, or slight head movements).

[0172] Step 530: Based on the facial image information of the user during the execution of the specified action and the first input, generate the current makeup requirement instruction. In this step, the current makeup requirement instruction is a comprehensive set of information including at least one of the following: the specific makeup operation that the beauty robot is currently performing, the physical interaction constraints in the makeup operation, the expected presentation characteristics of the target makeup effect, and the user's personalized attributes. The current makeup requirement instruction has been explained in detail above and will not be repeated here.

[0173] In some embodiments, step 530 includes: acquiring facial image information of the user performing a specified action; generating preset trajectory parameters based on facial key points in the facial image information; and generating a current makeup requirement instruction based on the preset trajectory parameters and target makeup information determined by the first input. In actual execution, the process of acquiring facial image information and generating preset trajectory parameters is mainly performed by the visual processing thread within the beauty robot. This thread is continuously activated upon receiving a timer trigger signal or detecting the arrival of a new image frame. It uses the image stream acquired by the RGB-D camera as the processing object and calls the 3DDFA_V2 deep learning model and semantic segmentation network (such as DeepLabV3+) to perform multi-dimensional feature decomposition. Facial image information of the user performing the specified action is acquired through the RGB-D camera, and the three-dimensional spatial coordinates of facial key points are extracted using the 3DDFA_V2 deep learning model. Based on the spatial distribution and dynamic deformation characteristics of these facial key points, preset trajectory parameters that fit the user's facial contours are calculated and generated. The preset trajectory parameters and the target makeup information determined by the first input are analyzed to extract parameters such as makeup tools, cosmetics, makeup intensity, and shape characteristics. These parameters are then transformed into dynamic operational attributes (such as movement speed, number of application layers, target force, and initial target compliance interaction parameters) required for the robotic arm and dexterous hand to perform the task. These dynamic operational attributes are then integrated with the preset trajectory parameters to generate the current makeup requirement instructions, determining the tool selection, makeup path, and force control strategy for subsequent makeup operations.

[0174] It is understandable that the preset trajectory parameters can also be corrected based on the aforementioned visual displacement compensation and force control displacement compensation, and the current makeup requirement instruction can be generated based on the corrected preset trajectory parameters.

[0175] Step 540: Based on the current makeup requirement instruction, control the execution component to hold the matching makeup tool and perform the makeup action. In this step, the current makeup requirement instruction may include the aforementioned: the target smooth interaction parameters for the current makeup stage, the matching makeup tool, the corresponding grasping and dipping parameters of the makeup tool, and the makeup repair strategy matching the defect type, etc., which will not be elaborated further.

[0176] In some embodiments, step 540 includes: configuring target compliant interaction parameters based on the current makeup requirement instruction; and controlling the movement of the execution component based on the target compliant interaction parameters. The target compliant interaction parameters include: target motion inertia parameters, target motion retardation parameters, and target contact elasticity parameters. These parameters are primarily used to construct a compliant control dynamics model (e.g., a second-order dynamics model) for the interaction between the robotic arm's end effector and the user's face. By dynamically adjusting the parameters in the target compliant interaction parameters, the makeup robot can simulate the robotic arm's end effector as a compliant control end with specific physical characteristics, enabling the robotic arm and dexterous hand to adaptively switch between position control and force control. This allows the end effector carrying makeup tools to possess a compliant feel similar to that of a professional makeup artist, improving the safety and comfort of the interaction between the robotic arm and dexterous hand and the face while maintaining the accuracy of the makeup trajectory.

[0177] In some embodiments, step 540 includes: receiving a second input from the user during the makeup application process; and controlling the execution component to trigger an emergency stop or perform a safety retreat action in response to an interrupt command. In this embodiment, the second input is used to input an interrupt command.

[0178] In practice, users can input an interruption command via touch interface, physical emergency stop button, or voice when they realize that the number of makeup touch-ups has reached their expected level, feel discomfort due to excessive facial makeup application, or detect abnormal application pressure. In response to this interruption command, the robotic arm and dexterous hand will stop the current makeup application and trigger an emergency stop, or perform a safe retreat action in the opposite direction of the current facial contact surface (e.g., quickly raising and retreating to a safe observation position). This reduces consumable damage and improves user safety during close-range human-computer interaction.

[0179] According to the control method of the beauty robot provided in the embodiments of this application, when receiving an interruption command input by the user during the makeup operation, the robot arm and dexterous hand are controlled to trigger an emergency stop operation or perform a safe retreat operation in response to the interruption command. This realizes the user's subjective intention to intervene in the automated operation in real time, as well as the beauty robot's timely response to emergency commands. This reduces the potential safety hazards in dealing with emergencies and in close human-machine physical interaction, and improves the makeup quality and user experience of the automated makeup operation.

[0180] In some embodiments, after step 540, the method further includes: receiving a third input from the user; in response to the third input, determining the defect type corresponding to the target sub-facial region of the user's facial area based on newly acquired facial image information; and controlling the execution component to perform a corresponding makeup repair operation on the target sub-facial region based on a makeup repair strategy matching the defect type. In this embodiment, the third input is used to input a makeup repair instruction. In actual execution, after the makeup work is completed, if the user is not satisfied with the local makeup effect, they can input a makeup repair instruction as the third input through interactive methods such as touch, voice, or gesture. In response to the third input, the user's current new facial image information can be collected, and the target sub-facial region with defects can be located through feature analysis, and its corresponding defect type (such as broken lines, uneven color, or blurred boundaries) can be diagnosed. Subsequently, a makeup repair strategy matching the defect type is invoked, and the robotic arm and dexterous hand are controlled to perform a repair operation on the target sub-facial region.

[0181] In some embodiments, users can customize repair strategies. For example, users can manually define the repair area, adjust the color intensity, or specify repair tools and intensity parameters through the interactive interface. The system will prioritize parsing and executing the user's customized parameters to meet their personalized repair needs.

[0182] In some embodiments, based on a makeup repair strategy matching the defect type, the execution component is controlled to perform a corresponding makeup repair operation on the target sub-facial area. This includes: acquiring repair trajectory parameters and force control interaction parameters corresponding to the makeup repair strategy; adjusting the smooth interaction model according to the force control interaction parameters, using the working parameters in regular application as a reference, controlling the execution component to decrease the motion inertia parameter and contact elasticity parameter, increase the motion retardation parameter, and decrease the target force; adjusting the motion planning of the execution component according to the repair trajectory parameters, using the working parameters in regular application as a reference, controlling the execution component to reduce the motion speed, decrease the trajectory point spacing, and perform coverage repair on the target sub-facial area corresponding to the defect type along the target trajectory. In this embodiment, the target trajectory is a locally reciprocating or spiral-shaped trajectory.

[0183] In actual execution, compared to conventional large-area application, the underlying compliant interaction model can be dynamically adjusted based on force control interaction parameters. For example, the motion inertia parameter can be reduced from a larger value (e.g., 2.0-5.0 kg) in conventional application to a lower value (e.g., 0.5-1.0 kg) to improve dynamic response speed, allowing the robotic arm and dexterous hand to more sensitively follow subtle mechanical changes on the face; the larger value in conventional application is to improve motion stability during large-area application. The motion retardation parameter can be increased from a moderate value (e.g., 50-100 N·s / m) in conventional application to a higher value (e.g., 100-200 N·s / m) to effectively suppress high-frequency jitter or force control overshoot in local fine movements by enhancing the motion retardation effect; the moderate value in conventional application is to balance stability and response speed. By reducing the contact elasticity parameter from a standard value (e.g., 30-50 N / m) to a lower level (e.g., 10-20 N / m), the end effector exhibits a smoother interaction characteristic, thus more delicately conforming to the microscopic curves of the face. The standard value (e.g., 30-50 N / m) is used to control the movement of the makeup tool held by the component, ensuring it conforms to facial curves while maintaining trajectory stability. Simultaneously, the target force can be reduced from the moderate value used during regular coloring (e.g., 0.5-1.0 N) to 0.2-0.4 N, reducing the risk of harsh makeup or overly dark colors caused by excessive pressure on areas with existing base color defects. The moderate value used during regular coloring aims to improve the coloring effect during regular makeup application. In the motion trajectory planning dimension, the motion logic can be adjusted accordingly based on the repair trajectory parameters. The movement speed of the robotic arm's end effector is reduced from the higher speed of routine operations (e.g., 0.02-0.05 m / s) to a fine-operation speed (e.g., 0.005-0.01 m / s) to improve the accuracy and safety of local repairs through physical deceleration. The higher speed of routine operations is used to improve the efficiency of automated makeup application during regular tasks. The spacing between path points is reduced from a larger distance (e.g., 2-3 mm) in routine application to a denser spacing (e.g., 0.5-1 mm), thereby generating a high-density local repair path, allowing evenly and fully covered sub-regions of minor defects. The larger spacing between path points in routine application is used to enable coverage of large areas.

[0184] In some embodiments, unlike the large-scale circular or S-shaped trajectories used in the conventional application stage, the target trajectory in the fine repair stage can be mainly limited to local reciprocating motions (e.g., small-scale figure-eight reciprocating motions) or spiral shapes (e.g., small-radius spirals). This allows the execution component to precisely cover and repair the target sub-facial area corresponding to a specific defect type along this target trajectory. According to the control method for the beauty robot provided in this application embodiment, the user can input makeup repair instructions based on their subjective wishes. Responding to these instructions, the beauty robot can accurately locate the target sub-facial area with blemishes and determine its corresponding defect type based on newly acquired facial image information. It then automatically matches the corresponding repair strategy to control the execution component to perform the makeup repair operation, improving the makeup quality, error tolerance, and user experience of automated makeup operations.

[0185] In some embodiments, after controlling the execution unit to perform a corresponding makeup repair operation on the target sub-facial region based on a makeup repair strategy matching the defect type, the process includes: receiving a fourth input from the user; and in response to the fourth input, controlling the execution unit to stop the makeup repair task and output a warning message. In this embodiment, the fourth input is used to instruct the execution unit to stop the makeup repair task and output a warning message.

[0186] In actual implementation, when the robotic arm and dexterous hand perform a closed-loop makeup repair task, users can make subjective final interventions based on the actual progress. For example, if the number of repairs has reached the limit, the current repair effect has met their needs, or if the automatic repair process deviates from expectations, users can actively input a fourth input by clicking the "Stop Repair" button on the touch interface, issuing a stop voice command, or pressing a physical button.

[0187] In response to the fourth input, the robotic arm and dexterous hand are controlled to stop the current local makeup repair task and interrupt the automatic repair loop feedback logic. At the same time, warning information is output through screen pop-ups, voice broadcasts or sound and light indicator lights (such as prompting "automatic repair has been terminated by human intervention" or issuing an intervention status alarm) to clearly inform that the current automated process has been forcibly taken over by human intervention and to wait for the next operation (such as manually completing, resetting the task or ending the makeup).

[0188] The control method for a beauty robot provided in this application can be executed by a control device for the beauty robot. This application uses the example of a control device for the beauty robot executing the control method to illustrate the control device for the beauty robot provided in this application. Figure 9 As shown, the control device of the beauty robot includes: a first processing module 910, a second processing module 920, a third processing module 930, a fourth processing module 940, and a fifth processing module 950.

[0189] The first processing module 910 is used to determine the matching makeup tool based on the current makeup requirement instruction and obtain the corresponding grasping parameters of the matching makeup tool; the second processing module 920 is used to control the execution component to grasp the matching makeup tool based on the grasping parameters; the third processing module 930 is used to determine the corresponding dipping parameters of the matching makeup tool; the dipping parameters include: dipping force and rotation trajectory; the fourth processing module 940 is used to control the execution component to clamp the matching makeup tool and perform interactive actions with other makeup products based on the dipping parameters, so as to dip the makeup material required by the matching makeup tool; the fifth processing module 950 is used to control the execution component to perform interactive actions with the user's face.

[0190] According to the control device for the beauty robot provided in this application embodiment, the robotic arm and dexterous hand are controlled to grasp the matching makeup tools according to the grasping parameters determined by the current makeup demand instruction. This realizes automated tool scheduling and grasping, and improves the stability and safety of the grasping action. At the same time, the robotic arm and dexterous hand are controlled to pick up makeup materials based on the dipping parameters, realizing the automatic dipping of makeup materials in an appropriate amount and evenly. This method establishes a closed-loop operation process from tool matching, automated grasping, precise dipping to facial interaction, which helps to realize intelligent and continuous automatic makeup operation of the beauty robot, and improves the makeup quality and user experience of the automatic makeup operation.

[0191] In some embodiments, the third processing module 930 may also be used to: acquire the texture characteristics of other cosmetic products that perform interactive actions with the matching cosmetic tools held by the robotic arm; and determine the dipping parameters based on the texture characteristics.

[0192] In some embodiments, the third processing module 930 may further be used to: determine the dipping force in the dipping parameters as a first dipping force when the texture is liquid; determine the dipping force in the dipping parameters as a second dipping force when the texture is powdery; and determine the dipping force in the dipping parameters as a third dipping force when the texture is pastey; wherein the first dipping force is less than the third dipping force, and the second dipping force is less than the third dipping force.

[0193] In some embodiments, the third processing module 930 may further be used to: determine the rotation speed as a first rotation speed and the number of rotations as a first number of rotations when the texture is liquid; determine the rotation speed as a second rotation speed and the number of rotations as a second number of rotations when the texture is powdery; and determine the rotation speed as a third rotation speed and the number of rotations as a third number of rotations when the texture is paste-like; wherein the first rotation speed is less than the third rotation speed, and the third rotation speed is less than the second rotation speed. The first number of rotations is less than the second number of rotations, and the second number of rotations is less than the third number of rotations.

[0194] In some embodiments, the second processing module 920 can also be used to: identify the tool image corresponding to the matched makeup tool and obtain the initial position information of the matched makeup tool; convert the initial position information into target position information in the base coordinate system of the robotic arm based on the hand-eye calibration matrix; control the robotic arm to move to the position of the matched makeup tool based on the target position information; control the joints of the dexterous hand to adjust their pose and close based on the target angle sequence to cover and grasp the matched makeup tool, and obtain force feedback value; determine that the grasping action is completed when the force feedback value reaches the grasping force threshold.

[0195] In some embodiments, the fourth processing module 940 can also be used to: obtain the dipping force and rotation trajectory based on the dipping parameters; when the robotic arm moves above other cosmetics, control the execution component to drive the matching makeup tool to move along a first direction toward the surface of the other cosmetics, and perform compliant interactive control in the first direction until the actual force of the matching makeup tool and the surface of the other cosmetics reaches the dipping force; wherein, the first direction is a direction perpendicular to the surface of the other cosmetics; based on the dipping force and rotation trajectory, control the execution component to drive the matching makeup tool to move on a target plane; the target plane is a plane formed by a second direction and a third direction on the surface of the other cosmetics; wherein, the second direction intersects the third direction.

[0196] In some embodiments, the fifth processing module of the control device of the beauty robot is used to: control the execution component to repeatedly perform interactive actions on other cosmetic products when the amount of other cosmetic products applied is insufficient to meet the makeup requirements.

[0197] The control device for the beauty robot in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. Electronic devices can be mobile phones, tablets, laptops, handheld computers, in-vehicle electronic devices, mobile internet devices (MID), augmented reality (AR) / virtual reality (VR) devices, robots, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc. They can also be servers, network attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not specifically limit the specific devices. The control device for the beauty robot in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems, this application embodiment does not specifically limit the specific devices. The control device for the beauty robot provided in this application embodiment can achieve… Figures 1 to 8 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0198] This application also provides a dexterous hand. In some embodiments, the dexterous hand is configured as an end effector of a makeup robot, mounted on the end of the robot's robotic arm, for gripping matching makeup tools. In this embodiment, the dexterous hand operates based on the control method for the makeup robot provided in any of the above embodiments.

[0199] This application also provides a makeup robot. In some embodiments, the makeup robot includes a dexterous hand and a control device for the makeup robot as provided in any of the above embodiments. In this embodiment, the dexterous hand is used to hold matching makeup tools. Matching makeup tools refer to end-effectors with specific physical shapes and material properties that match the current makeup requirements. In actual execution, the specific type of the makeup tool is determined by the type of makeup technique in the current makeup requirement instruction, the characteristics of the target facial area, and the expected physical interaction constraints (e.g., a fine-tipped brush suitable for local line repair, a filling brush or foundation brush suitable for large-area initial coloring or uneven color filling, and a blending stick or beauty sponge suitable for edge blurring). The control device of the makeup robot is electrically connected to the dexterous hand. In some embodiments, the makeup robot further includes a robotic arm. In this embodiment, a dexterous hand is installed at the end of the robotic arm; the robotic arm is electrically connected to the control device of the makeup robot.

[0200] In some embodiments, such as Figure 10 As shown, this application embodiment also provides an electronic device 1000, including a processor 1001, a memory 1002, and a computer program stored in the memory 1002 and executable on the processor 1001. When executed by the processor 1001, this program implements the various processes of the above-described control method embodiment for the beauty robot and achieves the same technical effects. To avoid repetition, it will not be described again here. It should be noted that the electronic device in this application embodiment includes the aforementioned mobile electronic device and non-mobile electronic device.

[0201] This application also provides a non-transitory computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the control method embodiment for the beauty robot described above, achieving the same technical effects. To avoid repetition, it will not be described again here. The processor is the processor in the electronic device described above. The readable storage medium includes computer-readable storage media, such as a computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0202] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned control method for a beauty robot. The processor is the processor in the electronic device described above. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0203] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described control method embodiment for the beauty robot, and can achieve the same technical effect. To avoid repetition, it will not be described again here. It should be understood that the chip mentioned in this application embodiment can also be called a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc. It should be noted that in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of this application is not limited to performing functions in the order shown or discussed. It may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples. Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0204] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art, under the guidance of this application, can make many other forms without departing from the spirit and scope of the claims, all of which are within the protection scope of this application. In the description of this specification, the reference to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., means that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for controlling a robot, characterized in that, The beauty robot includes an execution component, which comprises a robotic arm and a dexterous hand disposed at the end of the robotic arm, the execution component being used to grip matching makeup tools; the method includes: Based on the current makeup requirement command, determine the matching makeup tool and obtain the capture parameters corresponding to the matching makeup tool; Based on the capture parameters, the execution unit is controlled to capture the matching makeup tool; Determine the dipping parameters corresponding to the matched makeup tool; the dipping parameters include: dipping force and rotation trajectory; Based on the dipping parameters, the actuator is controlled to hold the matching makeup tool and other makeup products and perform interactive actions to pick up the makeup materials required by the matching makeup tool; The actuator is controlled to interact with the user's face.

2. The robot control method according to claim 1, characterized in that, Determining the application parameters corresponding to the matched makeup tool includes: Acquire the texture characteristics of other cosmetic products that perform interactive actions with the matching cosmetic tools held by the robotic arm; The dipping parameters are determined based on the texture characteristics.

3. The robot control method according to claim 2, characterized in that, The process of determining the dipping parameters based on the texture characteristics includes: When the texture characteristic is liquid, the dipping force in the dipping parameters is determined as the first dipping force; When the texture is powdery, the dipping force in the dipping parameters is determined to be the second dipping force; When the texture is a paste-like texture, the dipping force in the dipping parameters is determined to be the third dipping force; Wherein, the first dipping force is less than the third dipping force, and the second dipping force is less than the third dipping force.

4. The robot control method according to claim 2, characterized in that, The rotation trajectory includes rotation speed and number of rotations; determining the dipping parameters based on the texture characteristics includes: When the texture characteristic is liquid, the rotation speed is determined as the first rotation speed, and the number of rotations is determined as the first number of rotations; When the texture is powdery, the rotation speed is determined as the second rotation speed, and the number of rotations is determined as the second number of rotations; When the texture is a paste-like texture, the rotation speed is determined as the third rotation speed, and the number of rotations is determined as the third number of rotations; Wherein, the first rotational speed is less than the third rotational speed, and the third rotational speed is less than the second rotational speed; The first number of rotations is less than the second number of rotations, and the second number of rotations is less than the third number of rotations.

5. The robot control method according to any one of claims 1-4, characterized in that, The grasping parameters include at least one of the following: hand-eye calibration matrix, target angle sequence of each joint of the dexterous hand, and grasping force threshold.

6. The robot control method according to claim 5, characterized in that, The step of controlling the execution unit to grasp the matching makeup tool based on the grasping parameters includes: The tool image corresponding to the matched makeup tool is identified to obtain the initial position information of the matched makeup tool; Based on the hand-eye calibration matrix, the initial position information is converted into target position information in the base coordinate system of the robotic arm; Based on the target position information, the robotic arm is controlled to move to the position of the matching makeup tool; Based on the target angle sequence, the joints of the dexterous hand are controlled to adjust their posture and close to cover and grasp the matching makeup tool and obtain force feedback values. When the force feedback value reaches the grasping force threshold, the grasping action is determined to be complete.

7. The robot control method according to any one of claims 1-4, characterized in that, The step of controlling the actuating component to clamp the matching makeup tool and perform interactive actions with other makeup products based on the dipping parameters includes: Based on the dipping parameters, the dipping force and rotation trajectory are obtained; When the robotic arm moves above the other cosmetic products, the actuator is controlled to drive the matching makeup tool to move along a first direction toward the surface of the other cosmetic products, and smooth interactive control is performed in the first direction until the actual force of the matching makeup tool and the surface of the other cosmetic products reaches the scooping force; wherein, the first direction is a direction perpendicular to the surface of the other cosmetic products. Based on the dipping force and the rotation trajectory, the actuator is controlled to move the matching makeup tool on a target plane; the target plane is a plane formed by a second direction and a third direction on the surface of the other makeup product; wherein the second direction intersects with the third direction.

8. The robot control method according to any one of claims 1-4, characterized in that, After controlling the actuating component to clamp the matching makeup tool and perform interactive actions with other makeup products based on the dipping parameters, the method includes: If the amount of other cosmetic products applied is insufficient to meet makeup needs, the actuator is controlled to repeatedly perform the interactive action on the other cosmetic products.

9. A control device for a robot, characterized in that, The beauty robot includes an execution component, which comprises a robotic arm and a dexterous hand disposed at the end of the robotic arm, the execution component being used to grip matching makeup tools; the device includes: The first processing module is used to determine the matching makeup tool based on the current makeup requirement instruction, and to obtain the capture parameters corresponding to the matching makeup tool. The second processing module is used to control the execution unit to capture the matching makeup tool based on the capture parameters; The third processing module is used to determine the dipping parameters corresponding to the matched makeup tool; the dipping parameters include: dipping force and rotation trajectory; The fourth processing module, based on the dipping parameters, controls the execution component to clamp the matching makeup tool and other makeup products to perform interactive actions, so as to dip the matching makeup tool into the makeup materials required by the matching makeup tool; The fifth processing module is used to control the execution component to interact with the user's face.

10. A dexterous hand, characterized in that, The dexterous hand is configured as the end effector of the beauty robot, installed at the end of the robot's robotic arm, and is used to hold matching makeup tools; The dexterous hand operates based on the robot control method as described in any one of claims 1-8.

11. A robot, characterized in that, include: Dexterous hands; The dexterous hand is used to hold the matching makeup tools; The robot control device as described in claim 9; the robot control device is connected to the dexterous flashlight.

12. The robot according to claim 11, characterized in that, include: A robotic arm; the dexterous hand is mounted at the end of the robotic arm; the robotic arm is electrically connected to the robot's control device.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the robot control method as described in any one of claims 1-8.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the robot control method as described in any one of claims 1-8.

Images