A feeding assistance robot

Abstract

This invention discloses an auxiliary feeding robot, in which a rigid robotic arm and a flexible robotic arm are fixedly mounted on an installation module. The rigid robotic arm includes a rigid link and a rotating module, with the rigid link and rotating module fixedly connected at intervals. A gripper is located at the end of the rigid robotic arm, and the rotating module provides driving force to rotate the rigid robotic arm, allowing it to grasp a food tray through the gripper and place the tray in front of the person being fed. The flexible robotic arm includes an elastic arm, a drive module, and a drive rope. The drive module pulls the drive rope to move the elastic arm, allowing the flexible robotic arm to deliver food to the person being fed through utensils attached to its end. This invention utilizes the positioning accuracy advantage of the rigid robotic arm, as well as the passive compliance of the flexible robotic arm and active compliance control using the obtained tension, thereby implementing compliance control and force limiting for the robot, avoiding accidental injury to the person being fed, and improving safety.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to an auxiliary feeding robot. Background Technology

[0002] As the global population ages, the service and consumption market for the elderly is expanding, encompassing not only food and daily necessities but also quality of life, health, and home living. In this context, safe and efficient service robots have a significant market potential. Many elderly people face upper limb mobility issues due to muscle atrophy and joint damage, making independent feeding difficult and requiring external assistance to help them eat.

[0003] Currently, there are two main methods for assisting feeding: manual feeding and feeding robots.

[0004] 1. Artificial feeding of elderly people with weak functions is time-consuming and laborious, especially in nursing homes and other settings where one person needs to care for multiple people. Staff members will become fatigued from long hours of mechanical repetition. Furthermore, the people being cared for often experience psychological burden from being cared for, and may even feel that they have lost their dignity, which is detrimental to their mental health.

[0005] 2. Current feeding robots employ rigid structures, largely developed based on multi-axis rigid robotic arms. While these rigid robots offer advantages in positioning accuracy, they suffer from poor interactivity, instill fear in users, and are prone to injuring users in unexpected situations. They can easily injure the elderly or cause psychological distress during assisted feeding. Some feeding robots utilize flexible mechanisms, but positioning accuracy is difficult to guarantee. Furthermore, existing feeding robots lack intelligent visual or voice interaction algorithms, resulting in poor interactivity and hindering a comfortable dining experience for the elderly. Manual feeding is expensive, time-consuming, and labor-intensive, with varying results and difficulty in monitoring. This invention is a rigid-flexible dual-arm assisted feeding robot, combining the precision of rigid arms with the safety of flexible arms. Incorporating a gaze detection algorithm, it can accurately add and retrieve target food, allowing the elderly to easily and comfortably eat simply by looking at the desired food. This improves the safety of the feeding process, reduces the learning curve for the elderly, and enhances work efficiency and user experience.

[0006] The main body of an existing automatic feeding robot consists of a food tray and a spoon for feeding, which is also the current mainstream form of feeding robots. Automatic feeding robots require button operation, which can be difficult for users with hand and foot disabilities; the tray rotates after each feeding, and users cannot select the food themselves.

[0007] Another existing feeding robot can select dishes through voice recognition. While this gives users room to choose, it requires users to speak clearly and the food on the plate to be easy to describe, which greatly limits its daily use.

[0008] An existing soft feeding robot uses a flexible arm for feeding. Although the arm is flexible, it is pneumatically driven, resulting in poor positioning accuracy, low load capacity, and no sensors, making feedback control impossible.

[0009] The mainstream solutions for existing feeding robots adopt rigid structures and are generally developed based on multi-axis robotic arms. Although they have certain advantages in positioning accuracy, they are very easy to injure users in case of accidents. Some feeding robots use flexible mechanisms, but the positioning accuracy is difficult to guarantee, making it difficult to complete the task of assisting feeding. Summary of the Invention

[0010] The purpose of this invention is to solve the technical problem that existing feeding robots cannot simultaneously achieve both accuracy and safety, and to propose an auxiliary feeding robot.

[0011] The technical problem of this invention is solved by the following technical solution:

[0012] An auxiliary feeding robot includes an installation module, a rigid robotic arm, a flexible robotic arm, and a camera unit. The rigid robotic arm, flexible robotic arm, and camera unit are fixedly installed on the installation module. The end of the rigid robotic arm can grasp a food tray and place it in front of the person being fed. The camera unit is used to capture an image of the person being fed's face and confirm the desired food item on the tray. The end of the flexible robotic arm can deliver the desired food item to the person being fed's mouth. The flexible robotic arm is equipped with a sensing unit inside, enabling compliant end-effector control.

[0013] In some embodiments, the rigid robotic arm includes a rigid link and a rotating module, the rigid link and the rotating module being fixedly connected at intervals, and the rotating module providing driving force to rotate the rigid robotic arm; the flexible robotic arm further includes an elastic arm, a drive module, and a drive rope, the drive module driving the elastic arm to move by pulling the drive rope; the rotating module includes a first motor and a joint, the rotating module driving the joint to rotate by the first motor; the drive module includes a drive unit and a sensing unit, the drive unit and the sensing unit being used to measure the tension signal of the drive rope; the elastic arm includes a spring assembly, one end of the drive rope being fixed to the drive unit and the other end being fixed to the spring assembly.

[0014] In some embodiments, the elastic boom further includes a retaining rope, and the spring assembly includes a spring plate, a concave plate, and a convex plate; both ends of the retaining rope are fixed to the spring assembly; the spring plates are installed alternately in sequence, and the spring plates are positioned by the concave plate and the convex plate; the plane of movement of the elastic boom is always perpendicular to the spring plate and passes through the center point of the spring plate.

[0015] In some embodiments, both the concave and convex disks are provided with opening areas, and at least one opening area on each of the concave and convex disks is through which the drive rope passes, and at least one opening area on each of the concave and convex disks is through which the retaining rope passes.

[0016] In some embodiments, the drive module further includes a driver mounting plate and an arm adapter plate; the drive unit and the sensing unit are respectively fixedly mounted on the driver mounting plate, one side of the arm adapter plate is fixedly connected to the driver mounting plate, and the other side is fixedly connected to the spring assembly of the elastic arm.

[0017] In some embodiments, the drive unit includes a spool, a retainer, and a second motor; the bottom of the spool is connected to the output shaft of the second motor by a key, the spool has a groove, the end of the drive rope is fixed inside the groove, and the drive rope is wound around the groove; the second motor and the retainer are respectively fixed on the driver mounting plate; the second motor can rotate and drive the spool to rotate, so that the drive rope is wound on the spool; the retainer is located outside the groove of the spool and is used to keep the drive rope close to the groove of the spool.

[0018] In some embodiments, the sensing unit includes a tension sensor, a digital transmitter, and a guide pulley; the guide pulley has a groove, and the boom adapter plate has a rope hole; one end of the drive rope is fixed to a spring assembly, and the other end of the drive rope passes through the rope hole of the boom adapter plate and the groove of the guide pulley, and is fixed inside the groove of the cable reel; the tension sensor has a shaft, and the guide pulley is connected to the shaft of the tension sensor; the tension sensor is used to measure the resultant force of the drive rope at the groove of the guide pulley, and the digital transmitter is used to convert the tension signal of the drive rope into a digital signal.

[0019] In some embodiments, the rigid robotic arm is a spatial three-dimensional robotic arm capable of changing the height of the spoon.

[0020] In some embodiments, the camera unit confirms that the food on the plate is what the person being fed wants by using a gaze estimation method.

[0021] In some embodiments, the line-of-sight estimation method includes the following steps:

[0022] S1. Receive facial image;

[0023] S2. Detect the facial position in the face image and obtain facial feature points;

[0024] S3. Perform face alignment based on the facial feature points and estimate head pose.

[0025] S4. Correct the face image based on the head position pose estimation results;

[0026] S5. Extract the features of the corrected face image to obtain the real-time gaze estimation result, and confirm the food on the plate is the food the person being fed wants based on the real-time gaze estimation result.

[0027] The beneficial effects of this invention compared to the prior art include:

[0028] This invention proposes an auxiliary feeding robot that is fixedly mounted on an installation module using a rigid robotic arm and a flexible robotic arm. By combining the characteristics of both rigid and flexible robotic arms, it solves the problem of existing feeding robots being unable to balance accuracy and safety. It utilizes the positioning accuracy advantage of the rigid robotic arm, as well as the inherent passive compliance of the flexible robotic arm and the active compliance control of the tension obtained at the end effector using a sensing unit. This improves safety from both active and passive perspectives, thereby implementing compliance control and force limiting for the robot to avoid accidental injury to the person being fed and improving safety.

[0029] In some embodiments, the present invention also has the following beneficial effects:

[0030] This invention proposes an auxiliary feeding robot that detects facial images, estimates head pose, and extracts features using a user-specific gaze estimation network model for real-time gaze estimation. It considers the influence of head pose on the gaze estimation vector, allowing the gaze estimation algorithm to ignore the head pose effect and primarily focus on the gaze estimation accuracy under standard poses. This enables the gaze estimation network model to adapt well to the user's eye structure and environment, improving human-computer interaction, effectively increasing positioning accuracy, reducing errors, and making it effective for feeding tasks.

[0031] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the feeding robot in an embodiment of the present invention.

[0033] Figure 2 This is a schematic diagram of the structure of the flexible arm of the rope-driven spring sheet in an embodiment of the present invention.

[0034] Figure 3 This is a top view of the flexible arm of the rope-driven spring sheet in an embodiment of the present invention.

[0035] Figure 4 This is an isometric view of the spring plate arm in an embodiment of the present invention.

[0036] Figure 5 This is a schematic diagram of the drive module of the rope-driven spring plate flexible arm in an embodiment of the present invention.

[0037] Figure 6 This is a schematic diagram of the spring sheet unit in an embodiment of the present invention.

[0038] Figure 7 This is a schematic diagram of the driver kit in an embodiment of the present invention.

[0039] Figure 8 This is a schematic diagram of the sensing kit in an embodiment of the present invention.

[0040] Figure 9 This is a flowchart of the line-of-sight estimation method in an embodiment of the present invention.

[0041] Figure 10 This is a schematic diagram of the operation flow of the line-of-sight estimation method in an embodiment of the present invention.

[0042] Figure 11 This is a schematic diagram of the line-of-sight estimation network model generated in an embodiment of the present invention.

[0043] Figure 12 This is a schematic diagram of a rope-driven elastic rod flexible arm in another embodiment of the present invention.

[0044] Figure 13 This is a schematic diagram of a spatial three-dimensional robotic arm in another embodiment of the present invention.

[0045] The attached figures are labeled as follows:

[0046] 1 Mounting plate, 2 Rigid robotic arm, 21 Rigid connecting rod, 22 Rotating module, 3 Rope-driven spring plate flexible arm, 31 Spring plate arm, 311 Spring plate sub-unit, 3111 Spring plate, 3112 Concave disc, 3113 Convex disc, 312 Retaining rope, 32 Drive box, 321 Drive kit, 3211 Winding drum, 3212 Retainer, 3213 Second motor, 3214 Reducer, 3215 Encoder, 322 Sensing kit, 3221 Tension sensor, 3222 Digital transmitter, 3223 Guide pulley, 323 Driver mounting plate, 324 Drive box mounting post, 325 Arm adapter plate, 33 Drive rope, 4 Camera unit, 41 Camera, 42 Camera mounting bracket. Detailed Implementation

[0047] The present invention will be further described below with reference to the accompanying drawings and preferred embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0048] It should be noted that the directional terms such as left, right, up, down, top, and bottom used in this embodiment are only relative concepts or are based on the normal use of the product, and should not be considered as restrictive.

[0049] The current method of fixing plates is to use a fixed base. Some improvements have added a rotating function to the fixed base, which can easily lead to the following two problems:

[0050] 1. The fixed position of the food tray greatly reduces the feeding range due to the limited position of the food tray, making it less flexible.

[0051] 2. The plates cannot be easily replaced. Due to the added degree of rotation, the plates must be fixed to the base and must be made of special materials that cannot be replaced, resulting in low versatility. In contrast, the rigid arm can hold bowls and trays, making it more flexible and convenient.

[0052] The present invention provides a rigid-flexible dual-arm assisted feeding robot, such as... Figure 1 As shown, this design addresses the challenge of balancing accuracy and safety in existing feeding robots, while also enhancing human-robot interaction by introducing gaze detection technology. The feeding robot comprises a mechanical component and an integrated gaze estimation algorithm. The mechanical component uses a combination of rigid and flexible arms, including a mounting module, a rigid robotic arm 2, a flexible robotic arm, and a camera unit 4. In this embodiment, the mounting module is a mounting plate 1, and the flexible robotic arm is a rope-driven spring-loaded flexible arm 3. The camera unit 4 captures facial images. In this embodiment, the camera unit 4 includes a camera 41 and a camera mounting bracket 42. The camera mounting bracket 42 is fixed to the mounting module 1 with screws, and the camera 41 and the camera mounting bracket 42 are also fixed together with screws. The function of the camera mounting bracket 42 is to fix the position of the camera 41 and raise it so that its field of view is not obstructed by the robotic arm, allowing it to see the eyes of the person being fed, thereby implementing the gaze estimation algorithm. The rigid robotic arm 2, the rope-driven spring-loaded flexible arm 3, and the camera unit 4 are fixed to the mounting plate 1 with screws. The algorithm includes a gaze estimation algorithm.

[0053] The rigid robotic arm 2 includes a rigid link 21 and a rotating module 22. The rigid link 21 and the rotating module 22 are fixedly connected at intervals. When performing a feeding task, the rotating module 22 provides driving force to drive the rigid robotic arm 2 to rotate. After the rigid robotic arm 2 grasps the plate with its end gripper, it can place the plate in front of the person being fed.

[0054] Traditional feeding robots lack a gaze estimation module, requiring users to operate them via buttons, resulting in low automation, high operating costs, and a poor human-computer interaction experience. In this embodiment, the feeding robot uses a camera unit 4 to control a rigid robotic arm 2 to grasp a food tray and place it in front of the person being fed. The gaze estimation method proposed in this embodiment confirms that the food tray contains the desired food, and then controls a flexible robotic arm to deliver the food to the person's mouth. Specifically, the camera unit 4 uses a gaze estimation algorithm to confirm the desired food tray. This gaze estimation algorithm relies on the camera unit 4, which in this embodiment is a camera 41. The input to the gaze estimation algorithm is the facial image data captured by the camera 41.

[0055] The rope-driven spring plate flexible arm 3 in this embodiment includes an elastic arm, a drive module, and a drive rope 33. The drive module drives the elastic arm to move by pulling the drive rope 33. When performing a feeding task, the rope-driven spring plate flexible arm 3 uses the utensils installed at its end to deliver the target food to the mouth of the person being fed. The end compliance control is achieved by sensing the rope tension through the rope-driven spring plate flexible arm 3 to ensure feeding safety. At this point, the entire feeding task is completed.

[0056] The rope-driven spring plate flexible arm 3 in this embodiment has the ability to move in the horizontal plane. The specific steps are as follows:

[0057] 1. A rigid robotic arm 2 places the plate in the appropriate place;

[0058] 2. The flexible arm 3 of the rope-driven spring plate is bent into a "C" or "S" shape so that its end reaches near the plate position;

[0059] 3. Camera unit 4 detects the gaze of the person being fed, determines the specific food they want, and converts it into a location signal;

[0060] 4. The rope-driven spring plate flexible arm 3 moves to the position of the specific food item mentioned above, and the end gripper picks up the food item.

[0061] 5. The rope-driven spring plate flexible arm 3 is straightened, and the end is brought close to the mouth of the person being fed, and the food is slowly brought into the mouth.

[0062] It should be noted that: 1. The end gripper is not driven by the motor in the drive box, but is driven by a separate servo motor, just like the end of the rigid arm. 2. Taking one of the motors as an example, the rotation of the motor pulls the rope to shorten (or lengthen). The end of the rope is fixed to the concave-convex disc of the rope-driven spring-plate flexible arm 3 by the rope head lock, which pulls the rope-driven spring-plate flexible arm 3 to bend in the horizontal plane. Other motors are similar. The required rope length for a specific shape can be obtained through kinematic algorithms, which can then be converted into the number of revolutions required by the motor.

[0063] In this embodiment, the installation module is the mounting plate 1, which consists of a metal fixing plate and an aluminum profile bracket, wherein the metal plate has corresponding positioning holes.

[0064] In this embodiment, the rigid robotic arm 2 consists of four rigid connecting rods 21 and a rotating module 22. The rotating module 22 includes three parallel joints and three first motors. The first motors are motors at the joints, and the joints are directly driven by the motors at the joints. That is, the rotation of the rotating motors at the joints directly drives the joints to rotate. Under this joint configuration, the rigid robotic arm 2 has three degrees of freedom, and its end effector can only move in the corresponding plane.

[0065] In this embodiment, the flexible arm 3 of the rope-driven spring plate is as follows: Figure 2 As shown, it includes an elastic arm, a drive module, and a drive rope 33. In this embodiment of the invention, the elastic arm is a spring plate arm 31, and in this example, the drive module is a drive box 32. The drive method of the rope-driven spring plate flexible arm 3 is as follows: Figure 7 The second motor 3213 shown pulls the drive rope 33, thereby causing the spring plate arm 31 to move in a plane, wherein the plane is perpendicular to the rectangular spring plate 3111 and passes through its center point.

[0066] like Figure 2 and Figure 4 As shown, the spring plate arm 31 has a modular structure, consisting of several spring assemblies and a retaining rope 312. In this embodiment, the spring assembly is a spring plate sub-unit 311. The spring plate arm 31 is specifically composed of several small arm segments and a retaining rope 312. In this embodiment, it consists of 3 small arm segments and a retaining rope 312. Each small arm segment contains 4 spring plate units 311, therefore it consists of a total of 12 spring plate units 311. The spring plate unit 311 is as follows: Figure 6 As shown, it consists of a spring plate 3111, a concave disk, and a convex disk. In this embodiment, the concave disk is a concave disc 3112, and the convex disk is a convex disc 3113. The concave and convex discs are engaged with the holes in the spring plate 3111 and the concave disc 3112 via pins on the convex disc 3113. In this embodiment, the convex disc 3113 has two holes that engage with the two holes in the concave disc 3112 for positioning, and is then fixed by screws at the lifting lugs on both sides of the discs. In this embodiment, one side of the arm adapter plate 325 is fixed to the support platform of the drive box 32 by bottom screws, and the other side is fixed to the spring plate arm 31 by screws, serving a connecting function.

[0067] In this embodiment, the retaining rope 312 is a steel wire rope, such as Figure 2 and Figure 4As shown, its two ends are fixed to the spring plate arm 31, specifically to the openings on the concave and convex discs of the first and last spring plate subunits 311 on the spring plate arm 31, which serve to retain the spring plate arm 31 and can greatly reduce the lateral bending of the spring plate arm 31 when it deflects at a large angle.

[0068] like Figure 6 As shown, in this embodiment, the spring sheet 3111 is made of 45 steel with a thickness of 0.7mm, and is positioned by pins on the concave disc 3112 and the convex disc 3113. Specifically, the convex disc has two pin structures (integrated structure), the concave disc has two holes, the pins cooperate for positioning, and then there are two lifting lugs next to it, which are fixed by screws.

[0069] Because the spring plate 3111 has a certain thickness, it needs to be installed alternately from left to right to ensure that the center of the spring plate arm 31 does not deviate to one side. If the spring plate near the end is always on the side near the root, the center of the entire rope-driven flexible arm 3 will shift. Alternating left and right installation will not cause this. Specifically, the spring plate 3111 is positioned by the concave disc 3112 and the convex disc 3113, and the plane of motion of the spring plate arm 31 is always perpendicular to the spring plate 3111 and passes through the center point of the spring plate 3111. Both the concave disc 3112 and the convex disc 3113 have opening areas, and at least one opening area on each of the concave disc 3112 and the convex disc 3113 is through which the drive rope 33 passes, and at least one opening area on each of the concave disc 3112 and the convex disc 3113 is through which the holding rope 312 passes. In this embodiment, specifically, the concave disc 3112 and the convex disc 3113 have four opening areas: top, bottom, left, and right. Figure 2 As shown, the left and right openings are for the movement of the drive rope 33; as Figure 4 As shown, the function of the upper and lower openings is to facilitate the movement of the retaining rope 312.

[0070] like Figure 2 , Figure 3 , Figure 5As shown, in this embodiment, the drive box 32 of the rope-driven spring flexible arm 3 consists of a corresponding number of drive units, sensing units, driver mounting plates 323, drive box mounting posts 324, and arm adapter plates 325. One end of the drive rope 33 is fixed to the drive unit, and the other end is fixed to the spring assembly. The drive units and sensing units are used to measure the tension signal of the drive rope 33. In this example, the drive unit is a drive kit 321, and the sensing unit is a sensing kit 322. Specifically, the drive box 32 consists of 6 drive kits 321, sensing kits 322, driver mounting plates 323, drive box mounting posts 324, and arm adapter plates 325. The drive kits 321 and sensing kits 322 are respectively fixedly mounted on the driver mounting plate 323, and are used to measure the tension signal of the drive rope 33. One side of the arm adapter plate 325 is fixedly connected to the driver mounting plate 323, and the other side is fixedly connected to the spring assembly of the elastic arm.

[0071] like Figure 7 As shown, the drive assembly 321 includes a cable reel 3211, a retainer 3212, a second motor 3213, a reducer 3214, and an encoder 3215. The bottom of the cable reel 3211 is connected to the output shaft of the second motor 3213 via a key. The rotation of the second motor 3213 drives the cable reel 3211 to rotate. At the same time, the second motor 3213 is fixed on the driver mounting plate 323. The cable reel 3211 is provided with a U-shaped groove. The retainer 3212 is fixed to the driver mounting plate 323 by screws and contacts the outside of the groove of the cable reel 3211, without any relative fixing device. The end of the drive rope 33 is fixed inside the groove of the drum 3211. The drive rope 33 winds around the groove. Specifically, the rotation of the second motor 3213 drives the drum 3211 to rotate, thereby winding the drive rope 33 onto the drum 3211. The function of the drum 3211 is to transmit the torque of the second motor 3213, pull the drive rope 33, collect the excess part of the drive rope 33, and fix its trajectory. The retainer 3212 is located outside the groove of the drum 3211. The retainer 3212 is used to limit the wire rope (i.e., retaining rope 312) on the drum 3211 from sticking tightly to the groove of the drum 3211, so that it does not pop out and thus detach from the drum 3211 and cause an accident. Specifically, the key connection is to achieve circumferential fixation between the shaft and the parts on the shaft to transmit motion and torque. Among them, some types can also achieve axial fixation and transmit axial force, and some types can achieve axial dynamic connection.

[0072] The reducer is a standard commercial motor, reducer, and encoder assembly. The reducer's function is to reduce the speed of the output shaft of the second motor 3213, thereby increasing its torque. The encoder is located at the tail of the second motor 3213 and is used to read data from the second motor 3213, such as speed and position information. The reducer, the second motor 3213, and the encoder together constitute a motor drive module.

[0073] like Figure 8 As shown, the sensing kit 322 includes a tension sensor 3221, a digital transmitter 3222, and a guide pulley 3223. The cage 3212 is used to maintain a specific angle of the drive rope 33, aiding in subsequent tension calculation (which requires the geometric relationship of the drive rope 33's path). The guide pulley has a groove; the tension sensor 3221 measures the resultant force of the drive rope 33 at the groove of the guide pulley 3223, and the digital transmitter 3222 converts the tension signal of the drive rope 33 into a digital signal. Viewed from the fixed end of the drum 3211, the path of the drive rope 33 is: drum 3211 - guide pulley 3223 - spring plate unit 311. The second motor 3213, reducer 3214, and encoder 3215 are common motor kits; the encoder 3215 is a sensor that provides feedback on the position, speed, and current of the motor (i.e., the second motor 3213). Tension sensor 3221 and guide pulley 3223 are connected via a shaft on tension sensor 3221. A section of the shaft extends from tension sensor 3221, and a bearing is integrated inside guide pulley 3223. The bearing's inner bore fits onto the shaft of tension sensor 3221, and its position is fixed by a shoulder and a retaining ring. Digital transmitter 3222 is a supporting device for tension sensor 3221. The function of digital transmitter 3222 is to convert the signal from tension sensor 3221 into a digital signal required by the computer. In this embodiment, drive rope 33 only contacts the grooved portion of guide pulley 3223.

[0074] like Figure 1 , Figure 2 , Figure 3 , Figure 7 As shown, in this embodiment, the drive rope 33 is a steel wire rope, one end of which is fixed to the spool 3211 of the drive kit 321, and the other end is fixed to the corresponding spring plate unit 311 of the spring plate arm 31, serving as a transmission device. It is fixed by placing a rope end at the end and tightening it with screws, relying on friction for fixation. Each small segment of the spring plate arm 31 is driven by two drive ropes 33 and can only perform bending motion in a plane. These two drive ropes 33 are respectively fixed to both sides of the spring plate unit 311 at the end of this small segment and to the corresponding second motor 3213. The second motor 3213 changes the bending angle of this small segment by tightening or loosening the drive ropes 33.

[0075] In this embodiment, the sensing method by which the rope-driven flexible robotic arm acquires rope tension data depends on, for example, the sensing method used by the rope-driven flexible robotic arm to acquire rope tension data. Figure 3 and Figure 5 The positional arrangement of the drive assembly 321 and sensing assembly 322 shown is for measuring the tension of the drive rope 33. Specifically, the tension signal of the drive rope 33 is calculated through the positional arrangement of the drive assembly 321 and sensing assembly 322. The force measurement principle is to calculate the rope force using fixed geometric angles. The drive rope 33 passes around the guide pulley 3223 of the sensing assembly 322. The sensing assembly 322 is arranged in layers from the outside to the inside to avoid crossing of the drive rope 33 during movement. Along the direction of movement of the drive rope 33, the tangent of the groove of its guide pulley 3223 coincides with the center of the rope hole of the corresponding rope-driven spring flexible arm 3. Specifically, the rope hole is provided on the arm adapter plate 325. The purpose is to allow the drive rope 33 to pass directly through the rope hole without generating friction for turning, thereby making the tension measurement of the drive rope 33 more accurate. One end of the drive rope 33 is fixed to the spring assembly, and the other end passes through the rope hole of the arm adapter plate 325 and the groove of the guide pulley 3223, and is fixed inside the groove of the reel 3211. In addition, the position of the sensing kit 322 is also related to the position of the drive kit 321, because force measurement requires the geometric position information of these modules. The position of the sensing kit 322 needs to ensure that the angle at which the drive rope 33 passes around the guide pulley 3223 is within a reasonable range (approximately 120 degrees). Furthermore, the center height of the groove of the guide pulley 3223 of the sensing kit 322 must be consistent with the plane of motion of the drive rope 33. Specifically, the resultant force of the drive rope 33 at the guide pulley 3223 is measured by the tension sensor 3221. Then, the tension signal of the drive rope 33 is obtained through geometric conversion. Specifically, the tension is obtained by converting the angle of the ropes on both sides of the guide pulley 3223 to obtain the component force of the drive rope 33. This tension is then converted into a digital signal by the digital transmitter 3222. The geometric relationship is obtained through the common tangent of the cage 3212 and the guide pulley 3223. Compliant control of the flexible robotic arm's end effector can be achieved based on the measured tension data.

[0076] This embodiment provides a real-time gaze estimation algorithm based on facial and binocular features, along with a corresponding system. The real-time gaze estimation method comprises four steps: image acquisition and face detection, head pose correction, eye image extraction, and pupil detection and gaze estimation.

[0077] This system considers the impact of head pose on the gaze estimation vector and eliminates this impact by correcting the image, allowing the gaze estimation algorithm to ignore the influence of head pose and primarily focus on the gaze estimation accuracy under standard poses. This part works in conjunction with a dual-arm feeding robot to detect the feeder's attention in real-time by estimating the gaze, thereby making the correct food selection and feeding actions. The gaze estimation method proposed in this embodiment is as follows: Figure 9 As shown, it includes five steps: receiving face images, face detection and alignment, mesh fitting and head pose estimation, face image correction, and gaze estimation.

[0078] The specific process of the line-of-sight estimation method is as follows: Figure 10 As shown:

[0079] S1, Receives facial images captured by the camera;

[0080] S2, Face Detection and Alignment Step: This step detects the face position in the face image and obtains facial feature points. The input target face image must have sufficient detail to distinguish facial contours and pupils. Specifically, it includes the following steps:

[0081] S21. Detect the face position of the face image using a face detection algorithm; specifically, detect the face position of the person from the input face image, crop the image of the face center and surrounding area and perform facial feature point detection. This process outputs the positions of 6 facial feature points through the BlazeFace (face detection) algorithm, including the centers of the two eyes, the two ears, the nose, and the mouth.

[0082] S22. Based on the angle between the line connecting the current pose feature points and the horizontal line, obtain the transformation relationship between the current pose and the standard pose, and perform the first face alignment. Specifically, perform preliminary face alignment based on the facial feature points. Then, the pose where the horizontal line connecting the left and right feature points is the standard pose, and the pose represented by the facial feature points detected in the current input image is the current pose. Here, face alignment refers to finding the transformation relationship between the current pose and the standard pose, which can be calculated based on the angle between the line connecting the facial feature points of the current pose and the horizontal line.

[0083] Step S3, Mesh Fitting and Head Pose Estimation, involves aligning the face based on facial feature points and estimating the head pose. Step S3 includes the following steps:

[0084] S31. Perform a second face alignment based on the facial feature points using a real-time face alignment method. Specifically, based on the 6 facial feature point positions obtained in S2, perform a second face alignment by further aligning the face using the real-time face alignment method FaceMesh, which outputs 468 face alignment points.

[0085] S32. The head pose matrix is ​​obtained by iteratively finding the most suitable projective 3D transformation from feature points in the standard pose and the current pose using Procrustes analysis. Specifically, the pose with the head upright and facing the camera is called the standard pose, and the head pose in the current input face image is called the current pose. The current pose can be represented by the positions of 468 facial alignment points. Then, the most suitable projective 3D transformation is found iteratively from feature points in the standard pose and the current pose using Procrustes analysis. Here, Procrustes analysis is a method for finding 3D transformations between different poses. Using this method, combined with camera calibration, the current head pose matrix H can be obtained, and the following relationship holds:

[0086]

[0087] In the formula It is the position of the center of the feeder's eye relative to the camera. These are the center positions of the eyes in the standard pose. They can be transformed using the head pose matrix H. The input image is then warped using this pose matrix H to obtain the input image I. H Image I corresponding to the standard pose O The location of the mouth center is given by the feature points of the mouth output by the BlazeFace face detection algorithm, and obtained by rotating the head pose matrix.

[0088] S4, the face image correction step, corrects the face image based on the head pose estimation result. Specifically, the face image is corrected using the head pose matrix H obtained in step S3, ensuring the face image is in a standard pose facing the screen. The corrected image is then used to complete the subsequent gaze estimation step. Specifically, based on the feature points at both eyes, from image I... O The images of the left and right eyes are obtained by cropping the images of the left and right eyes.

[0089] S5. Gaze estimation step: Extract the features of the corrected face image to obtain the real-time gaze estimation result, and confirm the food on the plate is the food the person being fed wants based on the real-time gaze estimation result.

[0090] Specifically, the S4-corrected face image is input into a trained user-specific gaze estimation network model. The user-specific gaze estimation network model is then used to extract features from the corrected face image, thereby estimating the gaze estimation vector in the standard pose. The gaze estimation vector for the current pose is obtained by rotating the gaze estimation vector under the standard pose using the head pose matrix. The gaze estimation vector obtained under the standard pose The gaze estimation vector at the current pose The following relationship exists:

[0091]

[0092] Where H is the head pose matrix.

[0093] Specifically, the left and right eye images extracted from S4 are input into a user-specific gaze estimation network model. The backbone network within this network extracts gaze estimation features from both eyes. Here, the backbone network specifically refers to the portion of the gaze estimation network that extracts features from the image. Various backbone networks exist in computer vision, such as ResNet (Residual Network) and VGG (Visual Geometry Group, a type of convolutional neural network). A backbone network can be any network structure that can be trained to extract gaze estimation feature vectors from images. This embodiment uses a CNN (Convolutional Neural Networks) to build the backbone network for feature extraction. Other embodiments may use other network structures such as ResNet, which could yield different results. The extracted gaze estimation features are then combined with the positions of the left and right eyes in their standard pose and input into a gaze multilayer perceptron used to obtain the final result from the extracted features, resulting in a gaze estimation vector. In contrast, the backbone network described above is the portion used to extract features from the image. Like the backbone network, the gaze multilayer perceptron is a part of the gaze estimation network. Finally, by training on the error between the gaze estimation vector and the true gaze direction, a user-specific gaze estimation network model is obtained.

[0094] The training and optimization of the user-specific gaze estimation network model in this embodiment includes the following steps:

[0095] S51. Optimizations were made to address the estimation errors caused by differences in the eyeball structure among different people. During the training phase, the MAML++ (Model-Agnostic Meta-Learning) framework was used to train the gaze estimation network model. This framework can effectively improve the generalization of the network to gaze estimation tasks for different people, resulting in a general gaze estimation network model.

[0096] S52. Based on this, only the user of the feeding robot needs to be calibrated. By taking pictures of the user gazing at objects in known locations, the general gaze estimation network can be fine-tuned to obtain the user's user-specific gaze estimation network model. This can reduce gaze estimation errors caused by structural differences and environmental differences.

[0097] The main idea behind the meta-learning method, MAML++ framework, is to allow the network to learn how to learn from different tasks. Here, "different tasks" is defined as gaze estimation tasks for different individuals. Figure 11 As shown, the detailed training method is as follows: First,

[0098] More specifically, training the gaze estimation network model in step S51 includes the following steps:

[0099] S511. Divide the gaze estimation dataset used for training according to identity labels. Sample the divided dataset using the method of n-way (n categories) and k-shot (k images per category). Sample k+1 images from the data of different identities in n categories. Use k images for training and the remaining 1 image for testing during the training phase. Here, n and k are constants.

[0100] S512. The n*k images used for training are called a Support set, and the remaining k images are called a Query set. The set consisting of a Support set and a Query set is considered a sample. The gaze estimation network model is trained on the Support set of one sample and tested on the Query set, which is called a task. This gaze estimation network model is a copy of the "original gaze estimation network model," and its structure is consistent with the "original gaze estimation network model." It can also obtain the user's gaze estimation vector from the image. The main function of this gaze estimation network model is to calculate the final Loss during the meta-learning training process for the "original gaze estimation network model" to train.

[0101] S513. Take h samples as a batch for training to obtain the loss of these h tasks. The structure of each gaze estimation network model used for training is the same as the original gaze estimation network model, which is called the same network structure model. Here, h is a constant.

[0102] S514. These h gaze estimation network models (with the same network structure model) are trained on a support set of h samples. Each gaze estimation network model is called a meta-learner, where the first meta-learner is called meta-learner 1, the kth learner is called meta-learner k, and so on, with the last meta-learner being meta-learner h. The h meta-learners are trained on the support set of h samples and validated on the query set using the resulting h losses (loss1 to loss...). h The sum of these losses is used as the final Loss (the sum of the meta-learner's predicted gaze and gaze label losses).

[0103] S515. Finally, the loss is calculated by taking the gradient of the initial parameters φ of the original gaze estimation network model and updating the initial parameters φ of the original gaze estimation network model, for a total of N rounds of training. The final general gaze estimation network model has the following property: it has the smallest sum of losses when trained separately for several rounds on all sampled tasks. That is, this general gaze estimation network model has the fastest convergence of initial parameters among all tasks, which greatly improves the generalization of the gaze estimation network model and enables subsequent fine-tuning to converge quickly, resulting in a general gaze estimation network model. This gaze estimation network model refers to the application of meta-learning to the gaze estimation task, which greatly improves the generalization of the gaze estimation network model, significantly enhancing the generalization of a specific gaze estimation network model.

[0104] The auxiliary feeding robot of this invention has the following effects:

[0105] I. Significantly improves security while maintaining accuracy.

[0106] Traditional feeding robots use a single arm, which cannot control the position of the tray. The lack of coordination between the tray and utensils easily leads to spillage during food transport. Using only a rigid robotic arm compromises safety; using only a flexible robotic arm compromises accuracy. The accuracy of this invention is primarily reflected in the precise food-grabbing process. During food grabbing, the rigid and flexible arms work in conjunction with line-of-sight detection to accurately determine which dish the user wants to eat, such as radish or meat in radish stir-fry, or meat in green pepper stir-fry but not green peppers. A typical flexible arm cannot accomplish this task, but the rigid arm holding the tray can compensate. For example, when the flexible arm is near the target, the rigid arm controls the tray to move the target dish directly below the end of the flexible arm (via the camera unit), allowing the flexible arm to simply grab it. This collaborative approach ensures accuracy.

[0107] This invention proposes a novel rigid-flexible dual-arm system for assisted feeding. By combining a rigid robotic arm with a flexible robotic arm, it addresses the challenge of balancing precision and safety in existing feeding robots. Furthermore, it enhances safety primarily through both active and passive methods.

[0108] 1. Utilize the inherent passive compliance of the rope-driven spring plate flexible arm. Since the flexible arm itself is not a rigid connection, its spring plate has a certain degree of elasticity. Therefore, it will deform accordingly after touching the person being fed, so as to avoid accidentally injuring the person being fed or causing a bad feeding experience.

[0109] 2. A novel tension sensing scheme for a rope-driven flexible arm is proposed, and the obtained tension is used for active compliant control. This scheme involves alternating ropes, using tension sensors to measure the resultant force of the ropes at the guide pulleys, and limiting the position of each rope to maintain geometric relationships. The rope tension is then calculated from the resultant force using these geometric relationships. With the rope tension information, compliant control and force limiting can be implemented on the robot, preventing accidental injury to the feeder and improving safety.

[0110] II. Improve the human-computer interaction experience

[0111] This invention utilizes gaze estimation technology to detect the desired food on the plate based on the feeder's gaze. During this process, the feeder only needs to move their eyes without any other actions, significantly reducing the operational steps and learning costs, improving the dining experience, and enhancing human-computer interaction. The algorithm offers two main benefits:

[0112] 1. Currently, the mainstream method for gaze estimation is to use eye trackers. However, specialized equipment such as eye trackers is expensive and requires an additional screen for gaze estimation. Gaze estimation significantly improves the user-friendliness of feeding tasks, especially for individuals with upper limb motor impairments and those with weakened functional abilities. Deploying eye trackers and screens for feeding tasks would greatly increase the cost and size of feeding robots, contradicting the initial goals of low-cost and user-friendly care.

[0113] 2. Deep learning-based methods for estimating gaze using face and eye images only require a standard RGB camera and sufficient computing power, making them simpler and less costly to deploy compared to eye-tracking methods. However, current methods on the same dataset have an error (the angle between the actual gaze and the estimated gaze) of around 3°. When trained and tested on different datasets without model generalization and personalization, the error can reach as high as 10°-20°. This makes image-based gaze estimation unusable in real-world environments due to excessive error, thus preventing its deployment on feeding robots. The gaze estimation method proposed in this invention comprehensively considers the influence of head pose on the estimated gaze vector, the model's generalization ability, and personalization. By rotating the image after estimating the head pose, the gaze estimation algorithm can ignore the influence of the head pose and mainly consider the gaze estimation accuracy under the standard pose. By adding the meta-learning framework of MAML++, the generalization of the model is improved, resulting in a general gaze estimation model. By using the user's image to fine-tune the general model, the model can be well adapted to the user's eye structure and the environment, thereby improving accuracy and reducing errors. This makes the image-based gaze estimation method effective for feeding tasks.

[0114] III. Save labor costs and improve work efficiency

[0115] The rigid-flexible dual-arm assisted feeding robot of this invention replaces manual labor by automating the feeding process, saving labor costs and improving work efficiency.

[0116] In other embodiments, the elastic arm can be replaced with other flexible robotic arms, such as... Figure 12 The rope-driven elastic rod flexible arm shown is specifically modified by replacing the spring sheet in the above embodiment with an elastic rod, and replacing the concave-convex disk composed of the concave disk and the convex disk with a structural disk.

[0117] In other embodiments, the spring sheet in the spring sheet arm 31 can be changed in size and material.

[0118] The rigid robotic arm 2 in the above embodiments is a planar robotic arm, which can only move on a plane. In a preferred embodiment, its degree of freedom configuration can be changed, and it can be replaced with, for example... Figure 13 The spatial robotic arm shown has spatial movement capabilities and can change the height of the spoon without changing the effect.

[0119] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or purpose, should be considered within the scope of protection of the present invention.

Claims

1. An auxiliary feeding robot, characterized in that, Includes installation modules, rigid robotic arms, flexible robotic arms, and camera units; The rigid robotic arm, the flexible robotic arm, and the camera unit are fixedly mounted on the mounting module. The end of the rigid robotic arm can grasp the plate and place it in front of the person being fed; The camera unit is used to capture images of the face of the person being fed and to confirm the food that the person being fed wants on the plate. The end of the flexible robotic arm can deliver the food that the person being fed wants to their mouth. The flexible robotic arm is equipped with a sensing unit inside, enabling it to achieve compliant end effector control.

2. The auxiliary feeding robot as described in claim 1, characterized in that, The rigid robotic arm includes a rigid link and a rotating module. The rigid link and the rotating module are fixedly connected at intervals. The rotating module provides driving force to drive the rigid robotic arm to rotate. The flexible robotic arm also includes an elastic arm, a drive module, and a drive rope. The drive module drives the elastic arm to move by pulling the drive rope. The rotating module includes a first motor and a joint, and the rotating module drives the joint to rotate through the first motor; The drive module includes a drive unit and a sensing unit, wherein the drive unit and the sensing unit are used to measure the tension signal of the drive rope. The elastic arm includes a spring assembly, and one end of the drive rope is fixed to the drive unit, while the other end is fixed to the spring assembly.

3. The auxiliary feeding robot as described in claim 2, characterized in that, The elastic boom also includes a retaining rope, and the spring assembly includes a spring plate, a concave plate, and a convex plate; Both ends of the retaining rope are fixed to the spring assembly; the spring plates are installed alternately in sequence, and the spring plates are positioned by concave and convex discs. The plane of movement of the elastic arm is always perpendicular to the spring plate and passes through the center point of the spring plate.

4. The auxiliary feeding robot as described in claim 3, characterized in that, Both the concave and convex disks are provided with opening areas, and at least one opening area on each of the concave and convex disks is through which the drive rope passes, and at least one opening area on each of the concave and convex disks is through which the retaining rope passes.

5. The auxiliary feeding robot as described in claim 2, characterized in that, The drive module also includes a driver mounting plate and an arm adapter plate; the drive unit and the sensing unit are respectively fixedly mounted on the driver mounting plate, and one side of the arm adapter plate is fixedly connected to the driver mounting plate, and the other side is fixedly connected to the spring assembly of the elastic arm.

6. The auxiliary feeding robot as described in claim 2, characterized in that, The drive unit includes a winding drum, a retainer, and a second motor; The bottom of the spool is connected to the output shaft of the second motor via a key. The spool has a groove, and the end of the drive rope is fixed inside the groove. The drive rope winds around the groove. The second motor and the retainer are respectively fixed on the driver mounting plate. The second motor can rotate and drive the spool to rotate, so that the drive rope is wound on the spool. The retainer is located outside the groove of the spool and is used to keep the drive rope close to the groove of the spool.

7. The auxiliary feeding robot as described in claim 6, characterized in that, The sensing unit includes a tension sensor, a digital transmitter, and a guide pulley; The guide pulley is provided with a groove, and the boom adapter plate is provided with a rope hole. One end of the drive rope is fixed to the spring assembly, and the other end of the drive rope passes through the rope hole of the boom adapter plate and the groove of the guide pulley and is fixed inside the groove of the winding drum. The tension sensor is provided with a shaft, and the guide pulley is connected to the shaft of the tension sensor. The tension sensor is used to measure the resultant force of the drive rope at the groove of the guide pulley, and the digital transmitter is used to convert the tension signal of the drive rope into a digital signal.

8. The auxiliary feeding robot as described in any one of claims 1-7, characterized in that, The camera unit uses a gaze estimation method to determine the desired food item on the plate for the person being fed.

9. The auxiliary feeding robot as described in claim 8, characterized in that, The line-of-sight estimation method includes the following steps: S1. Receive facial image; S2. Detect the facial position in the face image and obtain facial feature points; S3. Perform face alignment based on the facial feature points and estimate head pose. S4. Correct the face image based on the head position pose estimation results; S5. Extract the features of the corrected face image to obtain the real-time gaze estimation result, and confirm the food on the plate is the food the person being fed wants based on the real-time gaze estimation result.

Images