A fully automatic mouth-following dining assistance method and system based on machine vision

Abstract

The embodiment of the present invention provides a meal-assisting method and system with fully automatic mouth-following based on machine vision, which belongs to the field of human-computer interaction technology of feeding-assisting robotic arms. The meal-assisting method includes: establishing a robotic arm base coordinate system according to the positional relationship between the robotic arm and the depth camera; obtaining a two-dimensional image of the human face; obtaining the mouth pose of the key points of the two-dimensional image of the human face based on an improved direct linear transformation algorithm; judging whether the feeding conditions are met according to the mouth pose; if it is judged that the feeding conditions are met, the robotic arm obtains a feeding trajectory according to teaching learning to perform a feeding operation; judging whether the robotic arm has completed the feeding operation; if it is judged that the robotic arm has completed the feeding operation, the mouth pose is updated and a meal delivery trajectory is obtained according to the mouth pose; the robotic arm performs the meal delivery operation according to the meal delivery trajectory. The present invention improves detection accuracy and robustness by obtaining the mouth pose through an improved direct linear transformation algorithm, and improves feeding accuracy by generating a feeding trajectory according to the real-time mouth pose.

Description

Technical Field

[0001] The present invention relates to the technical field of human-machine interaction of an eating assistance robot arm, and in particular to a fully automatic mouth-following dining assistance method and system based on machine vision. Background Art

[0002] With the aging population and the increasing number of people with upper limb dysfunction due to causes such as stroke, spinal cord injury, and neurodegenerative diseases, helping these people achieve independent eating has become a key social concern. To reduce the burden on caregivers and enhance user autonomy, dining-assistance robots, as an emerging assistive device, are gradually entering the research and application stage.

[0003] Researchers are experimenting with incorporating machine vision technology into meal-serving robots, using cameras to capture facial information and locate the mouth. Currently, some studies use 2D images to detect the user's mouth, but the lack of depth information prevents accurate estimation of spatial position. Some systems combine depth cameras with facial landmark detection algorithms to obtain 3D poses, but this process is slow and accuracy is significantly affected by lighting and occlusion. Furthermore, some commercial meal-serving robots, such as Handy 1 (UK), MYSPOON (Japan), and Bestic (Sweden), generally utilize a pre-set fixed feeding trajectory, which suffers from low accuracy and inflexibility. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a fully automatic mouth-following meal-assisting method and system based on machine vision, which solves the problem of being unable to accurately estimate the mouth posture and dynamically generate a meal delivery trajectory based on the real-time mouth posture.

[0005] To achieve the above objectives, an embodiment of the present invention provides a fully automatic mouth-following dining assistance method based on machine vision, the dining assistance method comprising:

[0006] Establish a robotic arm base coordinate system based on the positional relationship between the robotic arm and the depth camera;

[0007] Obtain a two-dimensional face image;

[0008] Obtaining the mouth pose of key points of the two-dimensional face image based on an improved direct linear transformation algorithm;

[0009] judging whether feeding conditions are met according to the mouth posture;

[0010] When it is determined that the mouth posture meets the feeding conditions, the robotic arm obtains a feeding trajectory according to the teaching learning to perform the feeding operation;

[0011] Determining whether the robotic arm completes the feeding operation;

[0012] When it is determined that the robotic arm completes the feeding operation, updating the mouth posture and obtaining a food delivery trajectory according to the mouth posture;

[0013] The robotic arm performs the food delivery operation according to the food delivery trajectory.

[0014] Optionally, a robotic arm base coordinate system is established according to the positional relationship between the robotic arm and the depth camera, including:

[0015] Install ArUco markers on the end of the robotic arm;

[0016] Get the first spatial position of the ArUco marker through the depth camera;

[0017] Obtain the second spatial position of the end in its own coordinate system through the robotic arm;

[0018] A rigid body transformation matrix is ​​calculated according to the first spatial position, the second spatial position and the ICP algorithm to transform the camera coordinate system into the robotic arm base coordinate system.

[0019] Optionally, obtaining the mouth pose of key points of the two-dimensional face image based on an improved direct linear transformation algorithm includes:

[0020] According to formulas (1) to (2), the key points of the two-dimensional face image and the corresponding three-dimensional key points of the calibrated three-dimensional face model are obtained.

[0021] , (1)

[0022] , (2)

[0023] in, is the key point of the two-dimensional face image, is a three-dimensional key point, is the horizontal coordinate of the key point of the two-dimensional image, is the vertical coordinate of the key point of the two-dimensional image, is the coordinate of the three-dimensional key point in the X-axis direction, is the coordinate of the three-dimensional key point in the Y-axis direction, is the coordinate of the three-dimensional key point in the Z-axis direction, For the A key point, is an integer number;

[0024] Normalizing the key points of the two-dimensional face image;

[0025] According to formula (3), a cross product constraint algebraic model is constructed.

[0026] , (3)

[0027] in, is an antisymmetric cross product matrix and , is the rotation matrix, is the translation vector;

[0028] According to formula (4), the directional projection residual is obtained.

[0029] , (4)

[0030] in, is the directional projection residual, is the horizontal coordinate of the key point of the normalized two-dimensional image, is the vertical coordinate of the key point of the normalized two-dimensional image, is the first component of the translation vector, is the second component of the translation vector, is the x direction of the translation vector, is the y direction of the translation vector, is the first component of the rotation matrix, is the second component of the rotation matrix;

[0031] According to formulas (5) to (6), the objective function is constructed.

[0032] , (5)

[0033] , (6)

[0034] in, is the residual quadratic matrix, is the first residual coefficient matrix, and , is the second residual coefficient matrix, and , is the objective function, is the rotation matrix and , is the number of key points;

[0035] The objective function is minimized to obtain the rotation matrix and translation vector.

[0036] Optionally, minimizing the objective function to obtain the rotation matrix and the translation vector includes:

[0037] Minimize the objective function to obtain the first component and the second component of the rotation matrix and the first component and the second component of the translation vector;

[0038] Get the third component of the rotation matrix based on the geometric relationship;

[0039] Performing singular value decomposition on the rotation matrix to obtain an optimal rotation matrix;

[0040] Obtaining a third component of the translation vector according to the depth coordinate of the three-dimensional key point;

[0041] The pitch angle, roll angle and three-dimensional coordinates of the mouth posture are obtained according to the optimal rotation matrix and translation vector.

[0042] Optionally, judging whether feeding conditions are met according to the mouth posture includes:

[0043] Get the first feeding time;

[0044] determining whether the first feeding time is greater than or equal to a first feeding cycle;

[0045] When it is determined that the first feeding time is greater than or equal to the first feeding cycle, obtaining a pitch angle of the mouth posture;

[0046] Determining whether the pitch angle is greater than a first set threshold;

[0047] If it is determined that the pitch angle is greater than the first set threshold, determining that the feeding condition is met;

[0048] If it is determined that the pitch angle is not greater than the first set threshold, determining that the feeding condition is not met;

[0049] Get the second feeding time;

[0050] determining whether the second feeding time is greater than or equal to a second feeding cycle;

[0051] If it is determined that the second feeding time is greater than or equal to the second feeding cycle, obtaining a swing angle of the mouth posture;

[0052] Determining whether the swing angle is greater than a second set threshold;

[0053] If it is determined that the swing angle is greater than the second set threshold, it is determined that the feeding condition is not met;

[0054] When it is determined that the swing angle is not greater than the second set threshold, it is determined that the feeding condition is met.

[0055] Optionally, determining whether the robotic arm completes the feeding operation includes:

[0056] Obtain image information of the spoon at the end of the robotic arm using the RGB camera;

[0057] Preprocessing the image information;

[0058] Build a lightweight YOLOv5n optimization model;

[0059] Input the image information into a lightweight YOLOv5n optimization model to obtain a binary classification result of whether there is food on the spoon;

[0060] When a binary classification result is obtained indicating that there is food on the spoon, determining that the robotic arm completes the feeding operation;

[0061] When a binary classification result indicating that there is no food on the spoon is obtained, it is determined that the robotic arm has not completed the feeding operation, and the robotic arm is executed to obtain a feeding trajectory according to teaching learning to perform the feeding operation.

[0062] Optionally, build a lightweight YOLOv5n optimization model, including:

[0063] Get the YOLOv5n backbone network as the backbone network of the lightweight YOLOv5n optimization model;

[0064] Delete the Conv module, C3 module and SPPF module of the YOLOv5n backbone network;

[0065] Remove the P5 detection head and the Anchor configuration corresponding to the P5 detection head;

[0066] Configure lightweight parameters.

[0067] Optionally, updating the mouth pose and obtaining a food delivery trajectory according to the mouth pose includes:

[0068] Obtain the three-dimensional coordinates of the mouth pose based on the improved direct linear transformation algorithm;

[0069] Determine the food delivery target point according to the three-dimensional coordinates;

[0070] The delivery trajectory is planned according to the delivery target point and the position of the spoon at the end of the robotic arm.

[0071] Optionally, updating the mouth pose and obtaining a food delivery trajectory according to the mouth pose includes:

[0072] Obtaining an end point of the food delivery trajectory according to the food delivery trajectory;

[0073] Determining whether the distance between the food delivery target point and the end point of the food delivery trajectory is greater than a third set threshold;

[0074] When it is determined that the distance between the delivery target point and the end point of the delivery trajectory is greater than a third set threshold, replanning the delivery trajectory;

[0075] If it is determined that the distance between the delivery target point and the end point of the delivery trajectory is not greater than a third set threshold, executing the delivery operation;

[0076] Get the third feeding time;

[0077] determining whether the third feeding time is greater than a third feeding cycle;

[0078] When it is determined that the third feeding time is greater than the third feeding cycle, the execution robot arm obtains the feeding trajectory according to the teaching learning to perform the feeding operation.

[0079] In another aspect, the present invention provides a fully automatic mouth-following dining assistance system based on machine vision, the system comprising:

[0080] A robotic arm is provided on the table, and a spoon is provided at the end of the robotic arm;

[0081] A visual module, comprising a depth camera and an RGB camera. The depth camera is used to obtain a two-dimensional image of a face, and the RGB camera is used to obtain image information of a spoon.

[0082] The control module is connected to the visual module and the robotic arm, and is used to execute any of the above-mentioned dining assistance methods.

[0083] Through the above technical solution, the present invention provides a fully automatic mouth-following dining assistance method and system based on machine vision. This method uses an improved direct linear transformation algorithm to obtain the mouth pose of key points in a two-dimensional facial image. Building on the classic DLT architecture, it combines orthogonal constraints between rotation vectors with a singular value decomposition strategy to transform the point-n-place problem into a linear algebraic structure solution. This significantly improves computational efficiency while ensuring accuracy, avoiding the error amplification and degradation problems associated with mixed variable solutions in traditional methods. The system is suitable for real-time pose estimation using only sparse structural points, improving detection accuracy and robustness. The learned feeding trajectory enables the robotic arm to perform standardized feeding operations, adapting to different food types and container shapes. By constructing a lightweight YOLOv5n optimization model, the success rate of scooping is increased, empty spoon feeding is avoided, and the user experience is enhanced. The system can also monitor and update the mouth pose in real time, adapting to the user's real-time movements, and generating a smooth, collision-free trajectory based on the mouth pose, improving feeding accuracy and safety.

[0084] Other features and advantages of the embodiments of the present invention will be described in detail in the subsequent detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the following detailed description, they are used to explain the embodiments of the present invention, but do not constitute a limitation of the embodiments of the present invention. In the accompanying drawings:

[0086] Figure 1is a flow chart of a meal assistance method according to one embodiment of the present invention;

[0087] Figure 2 is a flow chart of spatial calibration according to one embodiment of the present invention;

[0088] Figure 3 is a flowchart of obtaining mouth pose according to one embodiment of the present invention;

[0089] Figure 4 is a flow chart of minimizing an objective function according to one embodiment of the present invention;

[0090] Figure 5 is a flow chart for determining whether the mouth posture satisfies feeding conditions according to one embodiment of the present invention;

[0091] Figure 6 is a flow chart for determining whether the mouth posture satisfies feeding conditions according to one embodiment of the present invention;

[0092] Figure 7 is a flow chart for determining whether a robotic arm has completed a feeding operation according to an embodiment of the present invention;

[0093] Figure 8 is a flowchart of constructing a lightweight YOLOv5n optimization model according to one embodiment of the present invention;

[0094] Figure 9 is a flowchart of updating the mouth pose and obtaining the food delivery trajectory based on the mouth pose according to one embodiment of the present invention;

[0095] Figure 10 is a flowchart of updating the mouth pose and obtaining the food delivery trajectory based on the mouth pose according to one embodiment of the present invention;

[0096] Figure 11 is a physical diagram of a dining assistance system according to one embodiment of the present invention;

[0097] Figure 12 is a detection result diagram of spatial calibration according to one embodiment of the present invention;

[0098] Figure 13 is a diagram showing a detection result of obtaining a mouth posture according to an embodiment of the present invention;

[0099] Figure 14 3 is a detection result diagram of a binary classification result according to one embodiment of the present invention. DETAILED DESCRIPTION

[0100] The following describes the specific implementation of the embodiment of the present invention in detail with reference to the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the embodiment of the present invention and is not used to limit the embodiment of the present invention.

[0101] It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of this application are in compliance with the relevant provisions of laws and regulations. In the embodiments of this application, certain software, components, models, and other existing solutions in the industry may be mentioned. These should be considered as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of this application, but it does not mean that the applicant has or will necessarily use such solutions.

[0102] Figure 1 FIG1 is a flow chart of a meal assistance method according to one embodiment of the present invention. In the figure, the meal assistance method includes:

[0103] In step S1, the robot arm base coordinate system is established according to the positional relationship between the robot arm and the depth camera. In the present invention, before the robot arm performs the action of taking and delivering food, the coordinate system is a prerequisite for ensuring the accuracy of the task. Since the depth camera and the robot arm are located in different coordinate systems, in order to accurately convert the mouth posture detected by the camera to the working coordinate system of the robot arm, the spatial calibration between the coordinate systems must be completed before the system is run. Figure 12 As shown, since the present invention adopts an "eyes outside the hands" visual configuration, that is, a fixed depth camera is used to observe the working area and the user's face, and the depth camera and the robotic arm are located in different coordinate systems, the spatial calibration between the coordinate systems must be completed before the system is run.

[0104] In step S2, a two-dimensional face image is obtained.

[0105] In step S3, the mouth poses of the key points of the two-dimensional face image are obtained based on the improved direct linear transformation algorithm.

[0106] In step S4, the mouth position is used to determine whether the feeding conditions are met. If so, step S5 is executed; otherwise, step S2 is executed. The start and stop control of the robotic arm is achieved by the mouth position, eliminating the need for manual operation and providing intuitive, contactless interaction.

[0107] In step S5, the robotic arm learns the feeding trajectory based on the instructional learning process and automatically executes the "scoop-scrape-lift" action based on the feeding trajectory. Specifically, the robotic arm scoops food from the plate; scrapes away excess food along the plate's edge; and finally lifts the spoon to the detection position. Because the feeding trajectory is learned through instructional learning, it can adapt to different food types and container shapes.

[0108] In step S6, it is determined whether the robot arm has completed the feeding operation. If it is determined that the robot arm has completed the feeding operation, step S7 is executed; otherwise, step S5 is executed.

[0109] In step S7, the mouth pose is updated and the food delivery trajectory is obtained according to the mouth pose.

[0110] In step S8, the robotic arm executes the food delivery operation according to the delivery trajectory. Steps S5 to S8 are executed repeatedly to ensure that the feeding is complete. If, during steps S5 to S8, the mouth position is determined not to meet the feeding conditions, the process returns to step S2. That is, during the loop of steps S5 to S8, if it is determined that the feeding conditions are not met, the robotic arm immediately stops.

[0111] In steps S1 to S8, the present invention provides a fully automatic mouth-following dining assistance method based on machine vision. The mouth posture of the key points of the two-dimensional face image is obtained based on the improved direct linear transformation algorithm, which avoids the error amplification and degradation problems caused by solving mixed variables in traditional methods. It is suitable for real-time posture estimation using only sparse structural points, thereby improving the accuracy of mouth positioning. The feeding trajectory learned through teaching enables the robotic arm to perform standardized feeding and adapt to different types of food and container shapes. It can also monitor and update the mouth posture in real time, adapt to the user's real-time movement changes, and generate a smooth and collision-free trajectory based on the mouth posture, thereby improving the accuracy and safety of feeding.

[0112] In step S1, in order to ensure the accuracy of the coordinate mapping from the camera coordinate system to the manipulator workspace, it is necessary to establish a manipulator base coordinate system based on the positional relationship between the manipulator and the depth camera. There are many ways to perform spatial calibration between coordinate systems known to those skilled in the art. In one example of the present invention, specifically, the methods for performing spatial calibration between coordinate systems may include: Figure 2 The method shown in Figure 2 In the method for performing spatial calibration between coordinate systems, the method may include the following steps:

[0113] In step S11 , an ArUco marker is installed at the end of the robotic arm.

[0114] In step S12, a first spatial position of the ArUco marker is acquired by a depth camera.

[0115] In step S13, the second spatial position of the end in its own coordinate system is obtained by the robotic arm.

[0116] In step S14, a rigid body transformation matrix is ​​calculated based on the first spatial position, the second spatial position, and the ICP algorithm to transform the camera coordinate system into the robotic arm base coordinate system. The rigid body transformation matrix consists of a rotation matrix and a translation vector, representing the fixed spatial relationship between the depth camera's camera head and the robotic arm.

[0117] In steps S11 through S14, the rigid body transformation matrix of the robot base coordinate system is used to convert points in the camera coordinate system to the robot base coordinate system. The calibration results are stored, and all subsequent visual inspection results are converted to the robot base coordinate system using this matrix. Calibration ensures the accuracy of the coordinate mapping from the camera coordinate system to the robot workspace and is the basis for subsequent mouth positioning and delivery trajectory planning.

[0118] In step S2, a two-dimensional image of the user's face is obtained by a fixedly installed depth camera. Based on the key points detected in the two-dimensional face image and the corresponding three-dimensional key points of the calibrated three-dimensional face model, the mouth posture is further obtained. The depth camera detects the position and posture of the mouth, which represents the real-time posture of the mouth relative to the camera of the depth camera. There are many ways to obtain the mouth posture known to those skilled in the art. In a preferred example of the present invention, specifically, the way to obtain the mouth posture may include the following: Figure 3 The method shown in Figure 3 In the present invention, the meal assistance method may include the following steps:

[0119] In step S31, the key points of the two-dimensional face image and the corresponding three-dimensional key points of the calibrated three-dimensional face model are obtained according to formulas (1) to (2).

[0120] , (1)

[0121] , (2)

[0122] in, is the key point of the two-dimensional face image, is a three-dimensional key point, is the horizontal coordinate of the key point of the two-dimensional image, is the vertical coordinate of the key point of the two-dimensional image, is the coordinate of the three-dimensional key point in the X-axis direction, is the coordinate of the three-dimensional key point in the Y-axis direction, is the coordinate of the three-dimensional key point in the Z-axis direction, For the A key point, An integer number used to index a set of keypoints.

[0123] In step S32, the key points of the two-dimensional face image are normalized. In order to unify the coordinate scale and eliminate the influence of the camera internal parameters, the key points of the two-dimensional image can be projected into the normalized line of sight direction according to formula (7):

[0124] , (7)

[0125] in, is the key point of the normalized two-dimensional image, in For the A key point, is the camera intrinsic parameter matrix and , is the horizontal focal length of the camera, is the vertical focal length of the camera, is the horizontal coordinate of the optical center in the image, is the vertical coordinate of the optical center in the image, is the horizontal direction of the image (x direction), is the longitudinal direction of the image (y direction), is the horizontal coordinate of the key point of the normalized two-dimensional image, is the vertical coordinate of the key point of the normalized two-dimensional image, and , .

[0126] In theory, the normalized light direction The direction is consistent with the world point after posture transformation,

[0127] , (8)

[0128] According to vector collinearity, we have , (9)

[0129] in, For the The coordinates of the three-dimensional key points in the camera coordinate system, is the rotation matrix, is the translation vector.

[0130] Furthermore, define for The antisymmetric cross product matrix of Substitute into formula (9). In step S33, construct the cross product constraint algebraic model according to formula (3),

[0131] , (3)

[0132] in, is an antisymmetric cross product matrix and , is the rotation matrix, is the translation vector.

[0133] The third row of the antisymmetric cross product matrix is ​​used to construct the error term. In step S34, the directional projection residual is obtained according to formula (4):

[0134] , (4)

[0135] in, is the directional projection residual, is the horizontal coordinate of the key point of the normalized two-dimensional image, is the vertical coordinate of the key point of the normalized two-dimensional image, is the first component of the translation vector, is the second component of the translation vector, is the x direction of the translation vector, is the y direction of the translation vector, is the first component of the rotation matrix, is the second component of the rotation matrix.

[0136] According to formula (10), let the rotation matrix be:

[0137] , (10)

[0138] in, is the rotation matrix, is the first component of the rotation matrix, is the second component of the rotation matrix. Due to the inherent structure of the rotation matrix itself: any two rows are unit vectors, different two rows are orthogonal, and the third row can be obtained by cross-product of the first two rows. Therefore, we only need to find 、 Construct the base plane and then use orthogonality to directly construct , which can ensure that the rotation structure has no deviation. However, the traditional DLT does not consider this structure, and the three "The lines may not form an orthogonal system and must be corrected by post-processing, which often leads to error superposition.

[0139] Determine the residual coefficient row vector of each key point according to formulas (11) to (12),

[0140] , (11)

[0141] , (12)

[0142] in, is the direction of the projected residual with respect to the rotation variable The linear coefficient of The key point direction projection residual is about the rotation variable The linear coefficient of is the direction projection residual with respect to the translation variable The linear coefficient of The key point direction projection residual is about the translation variable The linear coefficients of for , stack all for .

[0143] The translation term in formula (4) 、 After elimination, the residual quadratic matrix is ​​constructed. In step S35, the objective function is constructed according to formulas (5) to (6):

[0144] , (5)

[0145] , (6)

[0146] in, is the residual quadratic matrix, is the first residual coefficient matrix, and , is the first dimension of the normalized two-dimensional image The horizontal coordinates of the key points, is the first dimension of the normalized two-dimensional image The vertical coordinate of the key point, For the 3D key points, is the second residual coefficient matrix, and , is the objective function, is the rotation matrix and , is the first component of the rotation matrix, is the second component of the rotation matrix, is the number of key points.

[0147] In step S36, the objective function is minimized to obtain the rotation matrix and the translation vector.

[0148] In steps S31 to S36, to enable the robot, or robotic arm, to accurately perceive the spatial position and orientation (6-DoF) of the user's mouth in real time, this paper proposes a fast PnP solution method based on orthogonal constraint minimization. This method solves the PnP problem by improving direct linear transformations to estimate mouth pose from facial key points. Building on the classic DLT structure, this method incorporates orthogonal constraints between rotation vectors to transform the PnP problem into a linear algebraic structure for solution. This significantly improves computational efficiency while ensuring accuracy. This method is suitable for low-power, real-time embedded dining assistance robot platforms, achieving an inference speed of approximately 33 FPS on a laptop CPU.

[0149] In this embodiment, the method for minimizing the objective function may be any of a variety of methods known to those skilled in the art. In a preferred embodiment of the present invention, specifically, the method for minimizing the objective function may include: Figure 4 The method shown in Figure 4 In , methods for minimizing the objective function may include:

[0150] In step S361, the objective function is minimized to obtain the first component and the second component of the rotation matrix and the first component and the second component of the translation vector. The first two rows of the rotation matrix and the first two dimensions of the translation vector are solved by the algebraic residual quadratic matrix. The rank of is 4, and the dimension of its homogeneous solution space is 2. Therefore, the general solution of the rotation matrix can be set according to formula (12):

[0151] , (12)

[0152] in, For the general solution, is the first parameter, is the first basis vector, is the second basis vector. , represents the rotation matrix Take the first three elements, corresponding to the first row of the rotation matrix, , represents the rotation matrix The 4th to 6th elements in the rotation matrix correspond to the second row and apply the orthogonal constraint matrix ,have to

[0153] , (13)

[0154] in, is the first coefficient, is the second coefficient, is the third coefficient, and , , , for Corresponding Part is the rotation matrix The first row of subspace components, for Corresponding Part is the rotation matrix The second row of subspace components, for Corresponding Part is the rotation matrix Another orthogonal basis direction component of the first row of for Corresponding Part is the rotation matrix Another orthogonal basis direction component of the second row of . Solve for Then substitute into formula (12) to get the only legal solution rotation matrix .

[0155] Furthermore, the first and second components of the translation vector are restored according to formula (14):

[0156] , (14)

[0157] In step S362, the third component of the rotation matrix is ​​obtained according to the geometric relationship. Specifically, the rotation matrix is ​​obtained according to formula (15):

[0158] , (15)

[0159] in, is the third component of the rotation matrix. Further, we can get the rotation matrix .

[0160] In step S363, singular value decomposition is performed on the rotation matrix to obtain the optimal rotation matrix. According to formulas (16) to (17), the rotation matrix is ​​singular value decomposition is performed to obtain the optimal rotation matrix.

[0161] , (16)

[0162] , (17)

[0163] in, is the left singular vector matrix, is the right singular vector matrix, is a diagonal matrix of singular values, is the rotation matrix, is the optimal rotation matrix.

[0164] In step S364, the third component of the translation vector is obtained according to the depth coordinate of the three-dimensional key point. The third component of the translation vector is obtained according to formula (18):

[0165] , (18)

[0166] in, is the third component of the translation vector, For the The depth coordinates of the key points in the camera coordinate system, that is, the position of the transformed point in the z-axis direction of the camera coordinate system, and finally the translation vector .

[0167] In step S365, the pitch angle, roll angle and three-dimensional coordinates of the mouth pose are obtained according to the optimal rotation matrix and translation vector.

[0168] In step S361 to step S365, the present invention divides the estimation of the mouth pose into two stages. In the first stage, the first two rows of the rotation matrix are solved by the algebraic residual matrix. 、 and the first two components of the translation vector 、 In the second stage, the third row of the rotation matrix is ​​solved analytically based on the geometric structure of the rotation matrix. , and combine the sight direction to inversely solve the third dimension of the depth translation vector , ensuring the orthogonality of the rotation matrix. In the first stage, the algebraic least squares method is used to construct , control the solution structure, decouple rotation and translation, reduce variables and improve robustness. In the second stage, based on The geometric prior restores orthogonality, and then finds , By inversely solving the geometric relationship, the stability is enhanced, the physical constraints are clarified, and deviations caused by post-correction are avoided. Compared with the traditional DLT that requires post-processing to correct the orthogonality of the rotation matrix, this method ensures that the solution process itself meets the geometric consistency conditions through homogeneous solution space analysis and analytical orthogonal constraints, thereby improving numerical stability and physical consistency. Due to the characteristics of the mouth key points such as a small number of points (key points n=6), partial approximate coplanarity, and large confidence changes, this method can still stably estimate the three-dimensional spatial posture of the mouth with only 6 key points, solving the problem that traditional DLT is prone to degradation and instability when the number of points is small and the distribution is dense. By estimating the mouth pose in real time, deviations caused by fixed trajectories are avoided, and dynamic positioning and following are achieved, which is suitable for scenarios such as intelligent feeding and visual interaction. Furthermore, the key point confidence weighted residual row can be introduced to increase the anti-interference ability. Among them, the acquired mouth pose can be as follows Figure 13 shown.

[0169] In step S4, it is determined whether the feeding condition is met according to the mouth posture. There are many ways to determine whether the feeding condition is met, which are known to those skilled in the art. In one example of the present invention, the method of determining whether the feeding condition is met may include: Figure 5 and Figure 6 The method shown in Figure 5 and Figure 6 In the present invention, the meal assistance method may include the following steps:

[0170] In step S41, a first feeding time is acquired.

[0171] In step S42, it is determined whether the first feeding time is greater than or equal to the first feeding cycle. If it is determined that the first feeding time is greater than or equal to the first feeding cycle, step S43 is executed; otherwise, step S41 is executed.

[0172] In step S43, the pitch angle of the mouth pose is obtained.

[0173] In step S44, it is determined whether the pitch angle is greater than a first set threshold. If the pitch angle is greater than the first set threshold, step S45 is executed; otherwise, step S46 is executed. The first set threshold may be 10°.

[0174] In step S45, it is determined that the feeding condition is satisfied.

[0175] In step S46, it is determined that the feeding condition is not satisfied.

[0176] In step S47, the second feeding time is acquired.

[0177] In step S48, it is determined whether the second feeding time is greater than or equal to the second feeding cycle. If it is determined that the second feeding time is greater than or equal to the second feeding cycle, step S49 is executed; otherwise, step S47 is executed.

[0178] In step S49, the swing angle of the mouth pose is acquired.

[0179] In step S410, a determination is made as to whether the swing angle is greater than a second set threshold. If so, step S46 is executed; otherwise, step S45 is executed. The second set threshold can be 15°. Furthermore, during the loop execution of steps S5 through S9, the swing angle of the mouth posture is acquired in real time and a determination is made as to whether the swing angle is greater than the second set threshold. If the swing angle is greater than the second set threshold, the feeding condition is determined to be unsatisfactory, and the robotic arm immediately ceases operation.

[0180] In steps S41 to S410, the user's feeding needs can be obtained by monitoring the pitch angle and the swing angle. When the pitch angle is greater than the first set threshold, it indicates that the user nods, and the feeding conditions are met, and the robotic arm begins to feed; when the swing angle is greater than the second set threshold, it indicates that the user shakes his head, and the feeding conditions are not met, and the robotic arm does not feed or stops feeding. Existing dining assistance robots are mostly controlled by buttons or touch screens, which are complicated and non-intuitive to operate. Some users have difficulty operating them due to mobility problems. The present invention does not require users to operate with their hands, realizing contactless and natural interaction, which significantly improves the convenience of operation.

[0181] In step S6, it is determined whether the robot arm has completed the feeding operation. There are many ways to determine whether the robot arm has completed the feeding operation. In one example of the present invention, the method of determining whether the robot arm has completed the feeding operation may include: Figure 7 The method shown in Figure 7 In the present invention, the meal assistance method may include the following steps:

[0182] In step S61, image information of the spoon at the end of the robotic arm is obtained using the RGB camera.

[0183] In step S62, the image information is pre-processed by scaling the image to 640x640 pixels and performing normalization.

[0184] In step S63, a lightweight YOLOv5n optimization model is constructed.

[0185] In step S64, the image information is input into the lightweight YOLOv5n optimization model to obtain a binary classification result of whether there is food on the spoon. The detection result can be as follows: Figure 14 shown.

[0186] In step S65 , when a binary classification result indicating that there is food on the spoon is obtained, it is determined that the robotic arm completes the feeding operation, and step S7 is executed.

[0187] In step S66 , when a binary classification result indicating that there is no food on the spoon is obtained, it is determined that the robotic arm has not completed the feeding operation, and step S5 is executed.

[0188] If the robot arm is determined to have failed to scoop food in steps S61 to S66, the system returns to step S5. If the robot arm fails to scoop food three times in a row, a buzzer on the robot arm control panel sounds an alarm, and the robot arm stops. This closed-loop feeding system, created by the RGB camera and the robot arm, improves the success rate of scooping food, avoids empty spoons, and enhances the user experience.

[0189] In this embodiment, the method for constructing a lightweight YOLOv5n optimization model can be various methods known to those skilled in the art. In a preferred embodiment of the present invention, specifically, the method for constructing a lightweight YOLOv5n optimization model can include: Figure 8 The method shown in Figure 8 In

[15] , the method for constructing a lightweight YOLOv5n optimization model may include the following steps:

[0190] In step S631, a YOLOv5n backbone network is obtained as the backbone network of the lightweight YOLOv5n optimization model.

[0191] In step S632, the Conv module, C3 module, and SPPF module of the YOLOv5n backbone network are deleted. The number of channels of the feature maps output by the Conv module and the C3 module is 1024, which can also be expressed as Conv (1024), C3 (1024). The deep P5 structure in the original YOLOv5n backbone network is mainly used to detect large targets, which does not match the "food detection in the spoon" in the scenario of the present invention, and its computational complexity is relatively high. Therefore, the Conv module, C3 module, and SPPF module contained in layers 7 to 9 are deleted, and only P4 (P4 / 16) is retained as the deepest feature extraction layer, which significantly reduces the model size and improves the response speed of small targets.

[0192] In step S633, the P5 detection head and the Anchor configuration corresponding to the P5 detection head are removed. In the detection head part, the original YOLOv5n model performs multi-scale detection on the three-layer feature maps of P3, P4, and P5. The present invention removes the P5 detection head and only retains the two-layer detection heads of P3 (suitable for extremely small targets) and P4 (suitable for small and medium-sized targets). The shallow and middle-layer feature maps are fully utilized to express tiny food pieces, improve detection sensitivity and reduce redundant reasoning calculations. As the P5 layer is cropped, its corresponding Anchor configuration is also deleted synchronously, retaining only two sets of Anchor sizes corresponding to P3 / 8 and P4 / 16. This synchronous modification ensures the rationality of the detection frame regression mechanism and avoids the problem of mismatch between the anchor and feature layer.

[0193] In step S634, lightweight parameters are configured. The network structure retains YOLOv5n's original channel width scaling and depth scaling strategies, setting width_multiple = 0.25 and depth_multiple = 0.33. This keeps the model parameters and computational complexity extremely low, making it easy to deploy on resource-constrained platforms such as Jetson Nano, Raspberry Pi, and edge boxes.

[0194] In steps S631 to S634, improvements are made based on the original YOLOv5n v6.0 framework, retaining its structural advantages while optimizing the model size and running speed. The final model size is 1.7MB, and the inference speed on a laptop MX250 graphics card is 23FPS, making it more suitable for deployment on edge computing devices.

[0195] In order to dynamically generate the target point for food delivery and improve the accuracy of food delivery, in this embodiment, it is necessary to update the mouth pose. There are many ways to update the mouth pose known to those skilled in the art. In one example of the present invention, specifically, the way to update the mouth pose may include: Figure 9 The method shown in Figure 9 In the present invention, the meal assistance method may include the following steps:

[0196] In step S71 , the three-dimensional coordinates of the mouth pose are obtained based on the improved direct linear transformation algorithm.

[0197] In step S72, the food delivery target point is determined based on the three-dimensional coordinates.

[0198] In step S73, the delivery trajectory is planned based on the delivery target point and the position of the spoon at the end of the robotic arm. Specifically, it includes:

[0199] Generate the delivery trajectory function according to formula (19):

[0200]

[0201] in, is the delivery trajectory function, is the starting position of the trajectory, is the initial moment, is the first acceleration coefficient, is the second acceleration coefficient, is the third acceleration coefficient, is the fourth acceleration coefficient, is the fifth acceleration coefficient, For the current moment, The robotic arm controls the spoon to stop about 2 cm in front of the mouth, effectively preventing collisions and ensuring smooth and safe food delivery.

[0202] In steps S71 to S73, an improved direct linear transformation algorithm acquires the three-dimensional coordinates of the mouth and accurately determines its position. Next, the delivery target point is determined based on these three-dimensional coordinates to ensure the robot can accurately deliver the food to the user's mouth. Finally, the motion trajectory of the spoon at the end of the robotic arm is planned to ensure a smooth and safe delivery process, avoiding food spills and errors. By capturing the user's mouth position in real time and dynamically adjusting the delivery trajectory based on the current spoon and mouth positions, the accuracy of delivery is ensured and efficiency is improved.

[0203] In order to prevent the spoon from colliding with the mouth and ensure the smoothness and safety of food delivery, in this embodiment, specifically, the method of updating the mouth posture and obtaining the food delivery trajectory according to the mouth posture may include: Figure 10 The method shown in Figure 10 In the present invention, the meal assistance method may include the following steps:

[0204] In step S74, the end point of the food delivery track is obtained according to the food delivery track.

[0205] In step S75, a determination is made as to whether the distance between the delivery target point and the end point of the delivery trajectory is greater than a third threshold. The third threshold is 4 cm. The third threshold may also include a threshold range, which may include ±4 cm. If the distance between the delivery target point and the end point of the delivery trajectory is greater than the third threshold, step S76 is executed; otherwise, step S77 is executed.

[0206] In step S76, the meal delivery route is replanned.

[0207] In step S77, the meal delivery operation is performed.

[0208] In step S78, the third feeding time is acquired.

[0209] In step S79, it is determined whether the third feeding time is greater than the third feeding cycle. The third feeding cycle can be 15 seconds. If the third feeding time is greater than the third feeding cycle, step S5 is executed to proceed to the next meal. Otherwise, step S78 is executed. Wait 15 seconds to ensure that the user has finished eating before continuing to the next meal.

[0210] In steps S74 to S79, the delivery trajectory is dynamically adjusted by monitoring the mouth position in real time to ensure that the trajectory can be replanned in time when the mouth deviates from the target by more than ±4 cm to avoid delivery deviation.

[0211] On the other hand, the present invention provides a fully automatic mouth-following dining assistance system based on machine vision, which includes a robotic arm, a vision module and a control module. Specifically, it can be as follows Figure 11As shown in the figure, the robotic arm is a lightweight 6-DOF robotic arm, with the movement of each joint controlled by an ESP32 main control board. It is mounted on a tabletop, with its base fixedly connected to the tabletop. A spoon is located at the end of the robotic arm. The vision module includes a depth camera and an RGB camera. The RGB-D depth camera is an Intel RealSense D415 model, fixedly mounted in front of the tabletop at a height of 60 cm and approximately 70 cm from the user. It captures information about the user's face and the workbench, providing a two-dimensional facial image and depth data. The RGB camera is mounted at the end of the robotic arm, facing the spoon, and is used to capture images of the inside of the spoon after scooping food. It is connected to the control module via a USB interface. The control module is connected to the vision module and the robotic arm to execute any of the above-mentioned food delivery methods. The control module can be of various types known to those skilled in the art. In one embodiment of the present invention, the control module can include a computer. This computer runs the Ubuntu 16.04 operating system and is equipped with ROS, the YOLOv5n detection module, and an improved direct linear transformation algorithm. It is connected to the depth camera and the end RGB camera via USB and communicates with the ESP32 via the MQTT protocol for commands. In addition, the dining assistance system can also include a dining tray. The tray is placed in the working space of the robotic arm, and the edge of the tray is integrated with a scraping structure to facilitate scraping off excess food after scooping.

[0212] Through the above technical solution, the present invention provides a fully automatic mouth-following dining assistance method and system based on machine vision. This method uses an improved direct linear transformation algorithm to obtain the mouth pose of key points in a two-dimensional facial image. Building on the classic DLT architecture, it combines orthogonal constraints between rotation vectors with a singular value decomposition strategy to transform the point-n-place problem into a linear algebraic structure solution. This significantly improves computational efficiency while ensuring accuracy, avoiding the error amplification and degradation problems associated with mixed variable solutions in traditional methods. The system is suitable for real-time pose estimation using only sparse structural points, improving detection accuracy and robustness. The learned feeding trajectory enables the robotic arm to perform standardized feeding operations, adapting to different food types and container shapes. By constructing a lightweight YOLOv5n optimization model, the success rate of scooping is increased, empty spoon feeding is avoided, and the user experience is enhanced. The system can also monitor and update the mouth pose in real time, adapting to the user's real-time movements, and generating a smooth, collision-free trajectory based on the mouth pose, improving feeding accuracy and safety.

[0213] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0214] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A fully automatic mouth-following dining assistance method based on machine vision, characterized in that: The meal assistance method comprises: Establish a robotic arm base coordinate system based on the positional relationship between the robotic arm and the depth camera; Obtain a two-dimensional face image; Obtaining the mouth pose of key points of the two-dimensional face image based on an improved direct linear transformation algorithm, including: According to formulas (1) to (2), the key points of the two-dimensional face image and the corresponding three-dimensional key points of the calibrated three-dimensional face model are obtained. ，（1） ，（2） in, is the key point of the two-dimensional face image, is a three-dimensional key point, is the horizontal coordinate of the key point of the two-dimensional image, is the vertical coordinate of the key point of the two-dimensional image, is the coordinate of the three-dimensional key point in the X-axis direction, is the coordinate of the three-dimensional key point in the Y-axis direction, is the coordinate of the three-dimensional key point in the Z-axis direction, For the A key point, is an integer number; Normalizing the key points of the two-dimensional face image; According to formula (3), a cross product constraint algebraic model is constructed. ，（3） in, is an antisymmetric cross product matrix and , is the rotation matrix, is the translation vector; According to formula (4), the directional projection residual is obtained. ，（4） in, is the directional projection residual, is the horizontal coordinate of the key point of the normalized two-dimensional image, is the vertical coordinate of the key point of the normalized two-dimensional image, is the first component of the translation vector, is the second component of the translation vector, is the x direction of the translation vector, is the y direction of the translation vector, is the first component of the rotation matrix, is the second component of the rotation matrix; According to formulas (5) to (6), the objective function is constructed. ，（5） ，（6） in, is the residual quadratic matrix, is the first residual coefficient matrix, and , is the second residual coefficient matrix, and , is the objective function, is the rotation matrix and , is the number of key points; Minimizing the objective function to obtain the rotation matrix and translation vector includes: Minimize the objective function to obtain the first component and the second component of the rotation matrix and the first component and the second component of the translation vector; Get the third component of the rotation matrix based on the geometric relationship; Performing singular value decomposition on the rotation matrix to obtain an optimal rotation matrix; Obtaining a third component of the translation vector according to the depth coordinate of the three-dimensional key point; Obtaining the pitch angle, roll angle and three-dimensional coordinates of the mouth posture according to the optimal rotation matrix and translation vector; judging whether feeding conditions are met according to the mouth posture; When it is determined that the mouth posture meets the feeding conditions, the robotic arm obtains a feeding trajectory according to the teaching learning to perform the feeding operation; Determining whether the robotic arm completes the feeding operation; When it is determined that the robotic arm completes the feeding operation, updating the mouth posture and obtaining a food delivery trajectory according to the mouth posture; The robotic arm performs the food delivery operation according to the food delivery trajectory.

2. The meal assistance method according to claim 1, characterized in that: The robot base coordinate system is established based on the positional relationship between the robot arm and the depth camera, including: Install ArUco markers on the end of the robotic arm; Get the first spatial position of the ArUco marker through the depth camera; Obtain the second spatial position of the end in its own coordinate system through the robotic arm; A rigid body transformation matrix is ​​calculated according to the first spatial position, the second spatial position and the ICP algorithm to transform the camera coordinate system into the robotic arm base coordinate system.

3. The meal assistance method according to claim 1, characterized in that: Judging whether feeding conditions are met according to the mouth posture includes: Get the first feeding time; determining whether the first feeding time is greater than or equal to a first feeding cycle; When it is determined that the first feeding time is greater than or equal to the first feeding cycle, obtaining a pitch angle of the mouth posture; Determining whether the pitch angle is greater than a first set threshold; If it is determined that the pitch angle is greater than the first set threshold, determining that the feeding condition is met; If it is determined that the pitch angle is not greater than the first set threshold, determining that the feeding condition is not met; Get the second feeding time; determining whether the second feeding time is greater than or equal to a second feeding cycle; If it is determined that the second feeding time is greater than or equal to the second feeding cycle, obtaining a swing angle of the mouth posture; Determining whether the swing angle is greater than a second set threshold; If it is determined that the swing angle is greater than the second set threshold, it is determined that the feeding condition is not met; When it is determined that the swing angle is not greater than the second set threshold, it is determined that the feeding condition is met.

4. The meal assistance method according to claim 1, characterized in that: Determining whether the robotic arm has completed the feeding operation includes: Obtain image information of the spoon at the end of the robotic arm using the RGB camera; Preprocessing the image information; Build a lightweight YOLOv5n optimization model; Input the image information into a lightweight YOLOv5n optimization model to obtain a binary classification result of whether there is food on the spoon; When a binary classification result is obtained indicating that there is food on the spoon, determining that the robotic arm completes the feeding operation; When a binary classification result indicating that there is no food on the spoon is obtained, it is determined that the robotic arm has not completed the feeding operation, and the robotic arm is executed to obtain a feeding trajectory according to teaching learning to perform the feeding operation.

5. The meal assistance method according to claim 4, characterized in that: Build a lightweight YOLOv5n optimization model, including: Get the YOLOv5n backbone network as the backbone network of the lightweight YOLOv5n optimization model; Delete the Conv module, C3 module and SPPF module of the YOLOv5n backbone network; Remove the P5 detection head and the Anchor configuration corresponding to the P5 detection head; Configure lightweight parameters.

6. The meal assistance method according to claim 1, characterized in that: Updating the mouth pose and obtaining a food delivery trajectory according to the mouth pose, including: Obtain the three-dimensional coordinates of the mouth pose based on the improved direct linear transformation algorithm; Determine the food delivery target point according to the three-dimensional coordinates; The delivery trajectory is planned according to the delivery target point and the position of the spoon at the end of the robotic arm.

7. The meal assistance method according to claim 6, characterized in that: Updating the mouth pose and obtaining a food delivery trajectory according to the mouth pose, including: Obtaining an end point of the food delivery trajectory according to the food delivery trajectory; Determining whether the distance between the food delivery target point and the end point of the food delivery trajectory is greater than a third set threshold; When it is determined that the distance between the delivery target point and the end point of the delivery trajectory is greater than a third set threshold, replanning the delivery trajectory; If it is determined that the distance between the delivery target point and the end point of the delivery trajectory is not greater than a third set threshold, executing the delivery operation; Get the third feeding time; determining whether the third feeding time is greater than a third feeding cycle; When it is determined that the third feeding time is greater than the third feeding cycle, the execution robot arm obtains the feeding trajectory according to the teaching learning to perform the feeding operation.

8. A fully automatic mouth-following dining assistance system based on machine vision, characterized in that: The system comprises: A robotic arm is provided on the table, and a spoon is provided at the end of the robotic arm; A visual module, comprising a depth camera and an RGB camera. The depth camera is used to obtain a two-dimensional image of a face, and the RGB camera is used to obtain image information of a spoon. A control module is connected to the vision module and the robotic arm, and is used to execute the dining assistance method as described in any one of claims 1 to 7.

Images