A method for human-computer interaction of a feeding auxiliary robot arm based on laser point indication

Abstract

The present application relates to the field of eating auxiliary mechanical arm human-computer interaction, and particularly relates to an eating auxiliary mechanical arm human-computer interaction method based on laser point indication, which comprises deploying a camera in front of a user and a dinner plate to capture an image of the face and lips of the user, setting a computing power node to calculate an interaction instruction and send the instruction to a mechanical arm controller, installing a laser head on a mechanical arm end gripping device to emit a laser point for indicating food, using the laser point to indicate a target food, and using the camera and the computing power node to capture an image of the face of the user, obtain an inner and outer corner point of an eye corner, a nose tip and lips, calculate a triangle and quadrilateral area ratio formed by the corner points, and project the area ratio on a plurality of characteristic value intervals, so as to correspond to a mechanical arm motion instruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of eating auxiliary robot human-computer interaction, in particular to an eating auxiliary robot human-computer interaction method based on laser point indication. BACKGROUND

[0002] The current mainstream eating auxiliary device and system, the user can only passively accept food and cannot enjoy the pleasure of autonomous eating, or needs to use electronic screens, voice, buttons, joysticks and other devices, and long-time staring at the screen, frequent shouting of target food or repeated operation of buttons and joysticks are not friendly to users with weakened physical strength; in addition, if the emerging brain-computer interface interaction technology is used, expensive wearable devices or implantable chips are relied on, and long-time data training is needed to achieve effective human-computer interaction, which is still difficult to be widely applied.

[0003] Therefore, it is urgent to develop a convenient and easy-to-use, energy-saving eating auxiliary human-computer interaction method, so that the user can easily and flexibly guide the food selection process and enjoy the pleasure of autonomous eating by only simple head movements and mouth opening and closing, and it is necessary to provide an eating auxiliary robot human-computer interaction method based on laser point indication. SUMMARY

[0004] In view of the problems in the prior art, the present application provides an eating auxiliary robot human-computer interaction method based on laser point indication.

[0005] The technical scheme adopted by the present application to solve the technical problem is: an eating auxiliary robot human-computer interaction method based on laser point indication, characterized in that: a camera is arranged in front of a user and a food tray to capture images of the user's face and lips, a computing power node is arranged to calculate an interaction instruction and send it to a robot controller, a laser head is installed on a robot end gripping device to emit a laser point for indicating food, and the controller controls the robot to point the laser point to target food and then confirm feeding based on the interaction instruction.

[0006] In use, the following steps are taken:

[0007] Step S1: the user first performs a series of head tilting, twisting and mouth opening and closing actions; the computing power node processes the images captured by the camera to calibrate key parameters of the user's face posture representation value and the mouth opening and closing representation value;

[0008] Step S2: the robot end points downward to emit the laser point to the food tray, flashes several times and then enters a constant light state, prompting the user to select food;

[0009] Step S3: the user twists his head, the computing power node calculates the user's head pitch and twist attitude representation value according to the key parameters calibrated in step S1, projects it on multiple intervals, and generates an interactive instruction to the controller of the mechanical arm; the controller controls the mechanical arm to rotate left and right to change the direction of the laser point indication, or changes the angle of the end gripping device to drive the laser head to retract or stretch the indication distance of the laser point, and controls the mechanical arm to point to the target food and its gripping point;

[0010] Step S4: the user expresses whether to eat through the opening and closing action of the mouth, and the computing power node calculates the mouth opening and closing representation value to make a judgment; if the eating intention is confirmed, the feeding motion instruction is sent to the mechanical arm controller, the controller controls the mechanical arm to grab the target food according to the position indicated by the laser point and feeds it into the user's mouth, which completes a round of eating assistance feeding, and then returns to step S2 for the next round of food selection interaction; if the eating intention is revoked, it directly returns to step S2 for the next round of interaction;

[0011] In step S1, the computing power node identifies the right outer corner of the mouth A, the left outer corner of the mouth B, the left outer corner of the eye C, the right outer corner of the eye D, the nose tip E and the inner and outer corner points of the lips in the user's face image;

[0012] The user first performs a series of head pitch and twist actions and mouth opening and closing actions, and the camera correspondingly captures a series of user head images, and the computing power node calculates the ratio of the pixel area of the upper triangle CED to the pixel area of the lower triangle AEB , the ratio of the pixel area of the left triangle AED to the pixel area of the right triangle BEC , and the ratio of the inner quadrilateral pixel area of the lips to the outer quadrilateral pixel area ; ;

[0013] The computing power node calculates a series of 、 and , and the natural logarithm of 、 and , respectively, and the mean of these natural logarithms is 、 , and the variance 、 and , which are used as key parameters for calculating the user's face posture representation value and calibrating the mouth opening and closing representation value.​​

[0014] The step S3, the computing power node obtains the current user face image in the upper and lower triangular area ratio And the left and right triangular area ratio After, respectively using formula And Respectively calculate the pitch attitude representation value And deflection attitude representation value ; Project in multiple intervals, specifically the following intervals:

[0015] a) "confirmation feeding interval", representation value And Satisfy inequality , Indicates that the user face posture is stable, and the command is stable laser head and its projection laser point of mechanical arm, more than a specific time is considered to be the user basically determine the current laser point of food as the target food, and then enter step S3, prepare to confirm or regret through the mouth opening and closing representation value;

[0016] b) "fine adjustment interval", representation value And Satisfy inequality , Indicates that the user adjusts the face posture to control the mechanical arm: Less than 0, send command to let the mechanical arm with the laser point of the end left turn at low speed, greater than 0, then right turn at low speed; Greater than 0, send command to let the mechanical arm low speed down the end laser head, according to the laser point of the slow contraction, Less than 0, then let the laser head low speed up to the slow expansion of the laser point outside; So that the user through the small face posture adjustment fine control mechanical arm pointing to the target food and its grasp point;

[0017] c) "quick contraction interval", representation value And Satisfy inequality , , , Indicates that the user with a large amplitude down head, and the left and right deflection amplitude is not big, at this time send command to let the mechanical arm faster down the end laser head to the laser point of the fast contraction;

[0018] d) "quick extension interval", representation value And Satisfy inequality , , , Indicates that the user with a large amplitude up head, and the left and right deflection amplitude is not big, at this time send command to let the mechanical arm faster up the end laser head to the laser point of the fast extension;

[0019] e) "fast left turning interval", represented by value and satisfying inequality , , , indicating that the user turns the head to the left with a large amplitude, while the up-down pitch amplitude is not large, at this time, the instruction is issued to make the mechanical arm and the laser point hit by the end fast left turn;

[0020] f) "fast right turning interval", represented by value and satisfying inequality , , , indicating that the user turns the head to the right with a large amplitude, while the up-down pitch amplitude is not large, at this time, the instruction is issued to make the mechanical arm and the laser point hit by the end fast right turn;

[0021] g) "pause interval", represented by value and satisfying inequality constitute an elliptical ring, and the area after excluding the "fast contraction interval", "fast extension interval", "fast left turning interval" and "fast right turning interval" represents that the user simultaneously tilts the head with a large amplitude while deflecting the face with a large amplitude, which is understood as the user hesitates to select the target food, at this time, the instruction is issued to make the mechanical arm pause;

[0022] h) "exit interval", represented by value and satisfying inequality , indicating that the user's head pitch or deflection value is too large, exceeding the effective interval of human-computer interaction, at this time, the instruction is issued to make the mechanical arm pause and time, and after exceeding a certain time, the interaction is exited, and a new round of interaction is started, that is, from the step S1, the key parameters of the user's face posture representation value and the mouth opening and closing representation value are recalibrated;

[0023] In the above inequalities , , , are given thresholds or coefficients.

[0024] Specifically, the step S4 calculates the mouth opening and closing representation value, which uses the key parameters calibrated in the step S1 and , and the current obtained pixel area ratio of the inner quadrilateral and the outer quadrilateral of the lips to calculate the mouth opening and closing representation value by formula ; if If the value exceeds a given threshold, feeding is confirmed. The robotic arm picks up the target food according to the laser dot indication and feeds it into the user's mouth, completing one round of assisted feeding. Then, the process returns to step S2 for the next round of human-machine interaction-based feeding. Before falling below a given threshold, the characterization value and If the user moves away from the "a) confirm feeding range" corresponding to step S3, it is assumed that the user has changed their mind about the selected food target, and the human-computer interaction returns to step S2.

[0025] The beneficial effects of this invention are as follows: The human-computer interaction method of the feeding-assisting robotic arm based on laser point indication described in this invention involves installing a laser head at the end of the feeding-assisting robotic arm to project laser points onto a plate. Using a camera and computing nodes, the user's facial image is captured, and the inner and outer corners of the eyes, nose, and lips are obtained. The area ratios of the triangles and quadrilaterals formed by these corners are calculated, and these area ratios are statistically analyzed and projected onto multiple representation value ranges, corresponding to the robotic arm's movement commands. The user can control the end of the robotic arm to move the laser point to indicate the target food and its grasping point by using small head movements. Then, the user can confirm or cancel the selected food by using the mouth opening and closing representation value, and then let the robotic arm grasp and feed or reselect. The food selection process can be easily and flexibly controlled by simple head movements and lip opening and closing. Attached Figure Description

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] Fig. 1 This is an interactive schematic diagram of a human-machine interaction method for a feeding-assisting robotic arm based on laser dot indication, provided by the present invention.

[0028] Fig. 2 This is a framework diagram of a human-computer interaction method for a feeding-assisting robotic arm based on laser dot indication, provided by the present invention. Detailed Implementation

[0029] To make the technical means, creative features, objectives and effects of this invention easier to understand, the invention will be further described below in conjunction with specific embodiments.

[0030] like Figs. 1-2 As shown, the present invention provides a human-machine interaction method for a feeding-assisted robotic arm based on laser point indication. This method, as described in the present invention, is a human-machine interaction method for a feeding-assisted robotic arm based on laser point indication. Fig. 1As shown, a camera is deployed directly in front of the user and the plate to capture images of their face and lips. A JETSON NANO B01 computing power box is used as a computing power node to process the images and obtain interactive commands, which are then sent to the robotic arm controller. The robotic arm is a five-joint desktop robotic arm, and the controller is made of a GD32F103 microcontroller. The parallel plastic gripper installed at the end of the robotic arm serves as a food grasping device. A 9mm red dot laser head installed on the grasping device emits a light spot to indicate the food in the plate. The controller receives the interactive commands and controls the robotic arm to point the laser point to the target food, thereby confirming the feeding.

[0031] The interaction method reference Fig. 2 The computing nodes run the DLIB face detection algorithm to identify the right outer corner of the mouth (A), left outer corner of the mouth (B), left outer corner of the eye (C), right outer corner of the eye (D), tip of the nose (E), and inner and outer corners of the lips in the user's facial image. These points are used to extract the user's facial pose representation value and calibrate the mouth opening and closing representation value to generate interactive commands.

[0032] The interaction method includes step S1: the user first performs a series of head tilting, twisting, and mouth opening and closing movements, and the camera captures a series of user head images accordingly. The computing node calculates the pixel area of ​​the upper triangle CED in each image. and the pixel area of ​​the lower triangle AEB ratio The pixel area of ​​the left triangle AED and the pixel area of ​​the right triangle BEC ratio Pixel area of ​​the inner quadrilateral of the lips and the area of ​​the outer quadrilateral pixels ratio ; Computing nodes calculate a series of 、 and The natural logarithm is 、 and The mean of these natural logarithms is calculated separately. 、 , and variance 、 and , which serves as a key parameter for calculating user facial posture representation values ​​and calibrating mouth opening and closing representation values.

[0033] Step S2: The end effector of the robotic arm points the laser dot downwards towards the plate, flashes several times, and then enters a constant-on state, prompting the user to enter the food selection state.

[0034] Step S3: The user turns their head. Based on the key parameters marked in step S1, the computing node obtains the ratio of the area of ​​the upper and lower triangles in the current user's facial image. The ratio of the areas of the left and right triangles Then, using the formulas respectively and Calculate the pitch attitude characterization values ​​separately. and deflection attitude characterization value Project it onto multiple intervals, specifically the following intervals:

[0035] a) "Confirm Feeding Range", Characteristic Value and Satisfying inequalities This indicates that the user's facial posture is stable. The command is issued to stabilize the robotic arm's laser head and the laser point it projects. After a certain period of time, it is considered that the user has basically determined that the food where the current laser point is located is the target food. Then, step S3 is entered to prepare to confirm or change the decision by using the mouth opening and closing characterization value.

[0036] b) "Fine-tuning interval", characterization value and Satisfying inequalities This indicates that the user makes slight adjustments to their facial posture to control the robotic arm. When the value is less than 0, a command is issued to make the robotic arm and the laser point emitted from its end effector turn left at a low speed; when the value is greater than 0, it turns right at a low speed. A value greater than 0 sends a command to cause the robotic arm to slowly lower its end-effector laser head, thereby gradually contracting the laser spot inward. When the value is less than 0, the laser head slowly rises and extends the laser point outwards; thus, the user can make small facial adjustments to precisely control the robotic arm to point at the target food and its grasping point.

[0037] c) "Rapid contraction interval", characterization value and Satisfying inequalities , , This indicates that the user tilts their head down significantly, while the left and right tilts are not significant. At this time, a command is issued to make the robotic arm tilt down quickly and the laser head at the end of the arm rapidly retracts the laser point inward.

[0038] d) "Rapid Expansion Interval", characterization value and Satisfying inequalities , , This indicates that the user has raised their head significantly, but the left and right deviation is not significant. At this time, a command is issued to make the robotic arm quickly raise its end laser head and quickly extend the laser point outward.

[0039] e) "Rapid Left Turn Interval", characterization value and Satisfying inequalities , , This indicates that the user turns their head to the left with a relatively large amplitude, but the pitch is not large. At this time, a command is issued to make the robotic arm and the laser point emitted by the end effector turn to the left quickly.

[0040] f) "Rapid Right Turn Interval", characterization value and Satisfying inequalities , , This indicates that the user turns their head to the right with a large degree of rotation, but the pitch is not large. At this time, a command is issued to make the robotic arm and the laser point emitted by its end effector turn to the right quickly.

[0041] g) "Pause Zone", characterization value and Satisfying inequalities The elliptical ring formed by these elements, after excluding the "rapid contraction zone," "rapid extension zone," "rapid left turn zone," and "rapid right turn zone," indicates that the user is simultaneously tilting their head up and down at a large angle while also turning their face at a large angle. This is interpreted as the user being hesitant about selecting the target food. At this point, an instruction is issued to stop the robotic arm from working.

[0042] h) "Exiting the interval", characterization value and Satisfying inequalities If the user's head pitch or tilt value is too large, exceeding the effective range of human-computer interaction, an instruction is issued to make the robotic arm stop working and start timing. After a certain time, the interaction is exited and a new round of interaction is to be started, that is, starting from step S1, the key parameters of the user's facial posture representation value and mouth opening and closing representation value are recalibrated.

[0043] In the above inequalities , , , The values ​​are given as 0.5, 1, 3 and 2 respectively.

[0044] When a user turns their head, the computing node calculates the pitch and twisting posture values ​​of the user's head, projects them into the aforementioned intervals, and generates corresponding interactive commands that are transmitted to the controller of the robotic arm. The controller controls the robotic arm to rotate left and right according to the interactive commands to change the direction of the laser dot, or to change the undulation angle of the end-effector, thereby causing the laser head to retract or extend the distance indicated by the laser dot, and manipulating the robotic arm to point at the target food and its grasping point.

[0045] In step S4, the user expresses whether they are eating by opening and closing their mouth; the computing node obtains the pixel area ratio of the inner quadrilateral and outer quadrilateral of the lips. Using the key parameters specified in step S1 and The process, in formula Calculate the mouth opening and closing characterization value ;like If the value exceeds a given threshold, it is determined that the user has opened their mouth to confirm feeding. The robotic arm then picks up the target food according to the laser dot indication and feeds it into the user's mouth, completing one round of assisted feeding. The process then returns to step S2 for the next round of human-machine interaction and designated feeding. Before falling below a given threshold, the characterization value and If the user leaves the area corresponding to "a) Confirm feeding range" in step S3, it is assumed that the user has changed their mind about the selected food, and the human-computer interaction returns to step S2. This process of iterating through each step completes the meal assistance process.

[0046] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of protection claimed by the present invention. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A human-machine interaction method for a feeding-assisting robotic arm based on laser dot pointing, characterized in that, This includes deploying a camera directly in front of the user and the plate to capture images of the user's face and lips, setting up computing nodes to calculate interactive instructions and sending them to the robotic arm controller, and installing a laser head on the robotic arm's end gripper to emit a laser point to indicate the food. The controller controls the robotic arm to point the laser point at the target food based on the interactive instructions and then confirms the feeding. Follow these steps when using it: Step S1: The user first performs a series of head tilting, twisting and mouth opening and closing movements; the computing node processes the images captured by the camera and calibrates the key parameters of the user's facial posture representation value and mouth opening and closing representation value. Step S2: The end effector of the robotic arm points the laser dot downwards towards the plate, flashes several times, and then enters a constant-on state to prompt the user to select food; Step S3: The user turns their head. The computing node calculates the pitch and twisting posture representation values ​​of the user's head according to the key parameters marked in step S1, projects them into multiple intervals, and generates interactive commands to be transmitted to the controller of the robotic arm. The controller controls the robotic arm to rotate left and right according to the interactive commands to change the direction of the laser point, or to change the undulation angle of the end gripping device, thereby causing the laser head to retract or extend the laser point's indication distance, and manipulating the robotic arm to point at the target food and its gripping point. Step S4: The user expresses whether to eat by opening and closing their mouth. The computing node calculates the mouth opening and closing representation value to make a judgment. If the intention to eat is confirmed, a feeding motion command is sent to the robotic arm controller. The controller controls the robotic arm to grab the target food according to the position indicated by the laser point and feed it into the user's mouth, thus completing one round of feeding assistance. Then, the process returns to step S2 to perform the next round of food selection interaction. If you change your mind about eating, return directly to step S2 for the next round of interaction; In step S1, the computing node identifies the right outer corner of the mouth (A), the left outer corner of the mouth (B), the left outer corner of the eye (C), the right outer corner of the eye (D), the tip of the nose (E), and the inner and outer corners of the lips in the user's facial image. The user first performs a series of head tilting, twisting, and mouth opening and closing movements, and the camera captures a series of images of the user's head accordingly. The computing node calculates the pixel area of ​​the upper triangle CED in each image. and the pixel area of ​​the lower triangle AEB ratio The pixel area of ​​the left triangle AED and the pixel area of ​​the right triangle BEC ratio Pixel area of ​​the inner quadrilateral of the lips and the area of ​​the outer quadrilateral pixels ratio ; The computing node calculates a series of... 、 and The natural logarithm is 、 and The mean of these natural logarithms is calculated separately. 、 , and variance 、 and , serving as a key parameter for calculating user facial posture representation values ​​and calibrating mouth opening and closing representation values; In step S3, the computing node obtains the area ratio of the upper and lower triangles in the current user's facial image. The ratio of the areas of the left and right triangles Then, using the formulas respectively and Calculate the pitch attitude characterization values ​​separately. and deflection attitude characterization value Project it onto multiple intervals, specifically the following intervals: a) "Confirm feeding range", characterization value and Satisfying inequalities This indicates that the user's facial posture is stable. The command is issued to stabilize the robotic arm's laser head and the laser point it projects. After a certain period of time, it is considered that the user has basically determined that the food where the current laser point is located is the target food. Then, step S3 is entered to prepare to confirm or change the decision by using the mouth opening and closing characterization value. b) "Fine-tuning interval", characterization value and Satisfying inequalities This indicates that the user makes slight adjustments to their facial posture to control the robotic arm. When the value is less than 0, a command is issued to make the robotic arm and the laser point emitted from its end effector turn left at a low speed; when the value is greater than 0, it turns right at a low speed. A value greater than 0 sends a command to cause the robotic arm to slowly lower its end-effector laser head, thereby gradually contracting the laser spot inward. When the value is less than 0, the laser head slowly rises and extends the laser point outwards; thus, the user can make small facial adjustments to precisely control the robotic arm to point at the target food and its grasping point. c) "Rapid contraction interval", characterization value and Satisfying inequalities , , This indicates that the user tilts their head down significantly, while the left and right tilts are not significant. At this time, a command is issued to make the robotic arm tilt down quickly and the laser head at the end of the arm rapidly retracts the laser point inward. d) "Rapid Expansion Interval", characterization value and Satisfying inequalities , , This indicates that the user has raised their head significantly, but the left and right deviation is not significant. At this time, a command is issued to make the robotic arm quickly raise its end laser head and quickly extend the laser point outward. e) "Rapid Left Turn Interval", characterization value and Satisfying inequalities , , This indicates that the user turns their head to the left with a relatively large amplitude, but the pitch is not large. At this time, a command is issued to make the robotic arm and the laser point emitted by the end effector turn to the left quickly. f) "Rapid Right Turn Interval", characterization value and Satisfying inequalities , , This indicates that the user turns their head to the right with a large degree of rotation, but the pitch is not large. At this time, a command is issued to make the robotic arm and the laser point emitted by its end effector turn to the right quickly. g) "Pause Zone", characterization value and Satisfying inequalities The elliptical ring formed by these two elements, excluding the "rapid contraction zone," "rapid extension zone," "rapid left turn zone," and "rapid right turn zone," indicates that the user is simultaneously tilting their head up and down at a large angle while also turning their face at a large angle. This is interpreted as the user being hesitant about selecting the target food. At this point, an instruction is issued to pause the robotic arm's work. h) "Exiting the interval", characterization value and Satisfying inequalities If the user's head pitch or tilt value is too large, exceeding the effective range of human-computer interaction, an instruction is issued to make the robotic arm stop working and start timing. After a certain time, the interaction is exited and a new round of interaction is to be started, that is, starting from step S1, the key parameters of the user's facial posture representation value and mouth opening and closing representation value are recalibrated. In the above inequalities , , , For a given threshold or coefficient.

2. The human-machine interaction method for a feeding-assisting robotic arm based on laser dot indication as described in claim 1, characterized in that: In step S4, the computing node calculates the mouth opening and closing characteristic value, which utilizes the key parameters calibrated in step S1. and And the pixel area ratio of the currently acquired inner and outer quadrilaterals of the lips. , in formula Calculate the mouth opening and closing characterization value ;like If the value exceeds a given threshold, feeding is confirmed. The robotic arm picks up the target food according to the laser dot indication and feeds it into the user's mouth, completing one round of assisted feeding. Then, the process returns to step S2 for the next round of human-machine interaction-based feeding. Before falling below a given threshold, the characterization value and If the user leaves the "a) Confirm feeding range" corresponding to step S3, it is assumed that the user has changed their mind about the selected food target, and the human-computer interaction returns to step S2.

Images