A control method of a multi-modal large model feeding robot

Abstract

The embodiment of the application provides a control method of a feeding robot based on a multimodal large model, and belongs to the technical field of image recognition. The control method comprises the following steps: acquiring a voice instruction issued by a user; acquiring a food position in a dish according to the voice instruction by using a lightened model of an improved YOLOv8n; picking up the food by a mechanical arm according to the food position; detecting a three-dimensional pose of a mouth based on a weighted point cloud normal estimation method; planning a feeding track according to the three-dimensional pose; and executing a feeding operation by the mechanical arm according to the feeding track. The improved YOLOv8n lightened model and the weighted point cloud normal estimation method are used, so that real-time detection of food blocks in the dish and high-precision estimation of the position of the mouth of the user are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a control method of a feeding robot based on a multi-modal large model. BACKGROUND

[0002] To help the population with limited upper limb function to achieve independent feeding, researchers and enterprises have proposed various feeding robot systems. Such systems are usually composed of a mechanical arm, an end effector, a vision sensor and a human-computer interaction module, and can assist users to complete the tasks of taking food and feeding to a certain extent. However, the existing feeding robots generally have the following shortcomings:

[0003] On the one hand, the detection efficiency is limited during visual detection: the existing target detection algorithm often faces the problems of large computational overhead and insufficient real-time performance when running on embedded or edge hardware, affecting the detection accuracy and user experience; on the other hand, the mouth positioning is not accurate enough: the dynamic perception of the user's mouth position is insufficient, or the positioning method used is not accurate enough, which easily leads to feeding errors. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a control method of a feeding robot based on a multi-modal large model, which solves the problem of low accuracy in detecting food blocks on a plate and estimating the position of a user's mouth.

[0005] In order to achieve the above-mentioned purpose, the embodiments of the present application provide a control method of a feeding robot based on a multi-modal large model, which comprises:

[0006] obtaining a voice instruction issued by a user;

[0007] obtaining the position of food on a plate according to the voice instruction using an improved lightweight model of YOLOv8n, wherein the improved lightweight model of YOLOv8n comprises:

[0008] calculating the loss function of the improved lightweight model of YOLOv8n according to formula (1),

[0009] , (1)

[0010] wherein, is the loss function, is the weighted intersection over union, is the weight factor, is the focal adjustment factor;

[0011] the mechanical arm forks the food according to the position of the food;

[0012] detecting the three-dimensional pose of the mouth based on the weighted point cloud normal estimation method;

[0013] According to the three-dimensional pose planning, a feeding trajectory is obtained;

[0014] The mechanical arm performs a feeding operation according to the feeding trajectory.

[0015] Optionally, a voice instruction issued by a user is acquired, including:

[0016] The voice instruction issued by the user is collected in real time by using a microphone;

[0017] The voice instruction is input into a multi-modal large model to determine whether there is an interrupt intention in the voice instruction;

[0018] In a case where it is determined that there is an interrupt intention in the voice instruction, the mechanical arm stops the feeding operation;

[0019] In a case where it is determined that there is no interrupt intention in the voice instruction, the voice instruction is parsed to generate a structured command;

[0020] The structured command is matched with a local skill instruction according to the structured command;

[0021] The corresponding feeding task function is matched according to the local skill instruction.

[0022] Optionally, the improved YOLOv8n lightweight model is used to obtain the food position in the tray, including:

[0023] The backbone network, the feature fusion network and the detection head of YOLOv8n are used as the backbone network, the feature fusion network and the detection head of the improved YOLOv8n lightweight model;

[0024] The standard convolution module in the backbone network of YOLOv8n is replaced by an ADown module;

[0025] The Bottleneck structure of the C2f module of the backbone network of YOLOv8n is replaced by a lightweight unit UIB constructed based on a MobileNetV4 network;

[0026] The Bottleneck structure of the C2f module of the feature fusion network of YOLOv8n is replaced by a lightweight unit UIB constructed based on a MobileNetV4 network.

[0027] Optionally, the improved YOLOv8n lightweight model is used to obtain the food position in the tray, including:

[0028] The weighted intersection over union is determined according to formula (2),

[0029] , (2)

[0030] wherein, is a weighted intersection over union, is an intersection over union of the predicted box and the ground truth box, is a center point of the predicted box is a center point of the ground truth box is a squared Euclidean distance between the center points of the predicted box and the ground truth box, is a trade-off factor for the aspect ratio constraint, is an aspect ratio consistency measure, is a diagonal length of the minimum enclosing rectangle of the predicted box.

[0031] Optionally, the three-dimensional pose of the mouth is detected based on a weighted point cloud normal estimation method, and the method comprises the following steps:

[0032] A dense three-dimensional point cloud data of a mouth region of a user is acquired by using a depth camera at an end of a mechanical arm;

[0033] A point cloud subset belonging to the mouth is extracted from the point cloud data;

[0034] A normal vector of each point in the point cloud subset is calculated by using a weighted PCA method;

[0035] The normal vectors are fused to estimate an overall orientation of the mouth;

[0036] A mouth center position is determined according to a geometric centroid of the point cloud subset, and the method comprises the following steps:

[0037] The mouth center position is determined according to formula (3),

[0038] , (3)

[0039] wherein, is the mouth center position, is the point cloud subset of the mouth, is an i-th point in the point cloud subset of the mouth.

[0040] Optionally, the normal vector of each point in the point cloud subset is calculated by using a weighted PCA method, and the method comprises the following steps:

[0041] A weight coefficient is determined according to formula (4),

[0042] , (4)

[0043] A covariance matrix is constructed according to formula (5),

[0044] , (5)

[0045] wherein, is the weight coefficient of a point relative to a point , and is ​a neighborhood point set of is a i-th point in a neighborhood point set of , is a distance decay parameter, is a covariance matrix of the neighborhood point set of is a centroid of the neighborhood point set;

[0046] performing eigen decomposition on the covariance matrix to obtain eigenvalues and eigenvectors;

[0047] selecting an eigenvector corresponding to a minimum eigenvalue as a normal vector.

[0048] Optionally, the normal vector is fused to estimate a mouth overall orientation, comprising:

[0049] performing distance-weighted global fusion on the normal vector of each point according to formulas (6) to (8),

[0050] , (6)

[0051] , (7)

[0052] , (8)

[0053] wherein, is a weighted coefficient of the i-th point, is a distance decay factor, is an Euclidean distance between the i-th point and a centroid of a mouth point cloud subset, is a mouth overall normal vector, is a normal vector of the i-th point, is a centroid of the mouth point cloud subset; performing normalization processing on the mouth overall normal vector. Optionally, a feeding trajectory is planned according to the three-dimensional pose, comprising:

[0054] determining a target mouth position according to the three-dimensional pose;

[0055] planning a feeding trajectory according to the target mouth position and a position of an end effector of a robot arm, comprising:

[0056] determining a feeding trajectory according to formula (9),

[0057]

[0058] , (9)

[0059] , (9)

[0060] ​​The constraints of the feeding trajectory are determined according to formula (10).

[0061] (10)

[0062] in, for Feeding trajectory at all times, Let be the coefficient vector of the feeding trajectory. For index number, Schedule time for feeding routes. , The feeding trajectory execution cycle, For the feeding trajectory function, Let be the velocity vector function of the feeding trajectory. Let be the acceleration vector function of the feeding trajectory. Starting position The initial desired speed, Target mouth position, The desired speed at the endpoint;

[0063] Get the current mouth position;

[0064] The feeding trajectory is updated by calculating the mouth offset error vector based on the target mouth position and the current mouth position.

[0065] Optionally, the feeding trajectory is updated by calculating a mouth offset error vector based on the target mouth position and the current mouth position, including:

[0066] The mouth offset error vector is determined according to formula (11).

[0067] (11)

[0068] Update the feeding trajectory according to formula (12).

[0069] (12)

[0070] in, For the first Next sampling time For the first Offset error vector at the next sampling time For the first The current mouth position at the time of the next sample. for Feeding trajectory at all times, Schedule time for the updated feeding route. , The updated feeding trajectory execution cycle, This is the coefficient vector of the updated feeding trajectory. is indexed.

[0071] Optionally, the mechanical arm performs a feeding operation according to the feeding trajectory, comprising:

[0072] obtaining a current mouth position and a position of an end effector of the mechanical arm;

[0073] determining whether a distance between the current mouth position and the position of the end effector of the mechanical arm is greater than a first preset safety threshold;

[0074] in a case where it is determined that the distance between the current mouth position and the position of the end effector of the mechanical arm is greater than the first preset safety threshold, the mechanical arm performs a feeding operation;

[0075] in a case where it is determined that the distance between the current mouth position and the position of the end effector of the mechanical arm is not greater than the first preset safety threshold, the mechanical arm performs a hovering operation.

[0076] Through the above technical solution, the application provides a control method of a feeding robot based on a multi-modal large model. The application realizes intent-driven YOLO model target detection through a coupling mode of voice semantics and visual perception, reduces the detection range, and improves the recognition speed and accuracy. Through the use of an improved lightweight model of YOLOv8n and a weighted point cloud normal estimation method, real-time detection of food blocks on a plate and high-precision estimation of the position of a user's mouth are realized, overcoming the shortcomings of the prior art in terms of detection accuracy and adaptability. Compared with traditional loss functions such as IoU and CIoU, the Focaler-WIoU used in the application pays more attention to the boundaries of targets that are difficult to accurately regress, thereby improving the fitting accuracy and stability of the detection frame, effectively separating adjacent fruit blocks, and reducing the false detection and missed detection rates. In addition, the application generates continuous and smooth trajectories in motion control, making the feeding process more natural and safe, and improving the flexibility and comfort of human-computer interaction.

[0077] Other features and advantages of the embodiments of the application will be described in detail in the following specific implementation part. BRIEF DESCRIPTION OF DRAWINGS

[0078] The accompanying drawings are included to provide a further understanding of the embodiments of the application, and constitute a part of the specification, and are used together with the following specific implementation to explain the embodiments of the application, but do not constitute a limitation on the embodiments of the application. In the drawings:

[0079] Figure 1 is a flowchart of a control method according to an embodiment of the application;

[0080] Figure 2 is a flowchart of obtaining a feeding task function according to an embodiment of the application;

[0081] Figure 3 is a code diagram of a feeding task function according to an embodiment of the present application;

[0082] Figure 4 is a code diagram of a feeding task function according to an embodiment of the present application;

[0083] Figure 5 is a flowchart of a lightweight model of improved YOLOv8n according to an embodiment of the present application;

[0084] Figure 6 is a flowchart of acquiring a three-dimensional pose of a mouth according to an embodiment of the present application;

[0085] Figure 7 is a flowchart of calculating a normal vector according to an embodiment of the present application;

[0086] Figure 8 is a flowchart of fusing a normal vector according to an embodiment of the present application;

[0087] Figure 9 is a flowchart of planning a feeding trajectory according to an embodiment of the present application;

[0088] Figure 10 is a flowchart of determining a feeding trajectory according to an embodiment of the present application;

[0089] Figure 11 is a flowchart of updating a feeding trajectory according to an embodiment of the present application;

[0090] Figure 12 is a flowchart of performing a feeding operation according to an embodiment of the present application;

[0091] Figure 13 is a network structure diagram of a C2f_UIB module and an Adown module according to an embodiment of the present application;

[0092] Figure 14 is a network structure diagram of a lightweight model of improved YOLOv8n according to an embodiment of the present application;

[0093] Figure 15 is a result comparison diagram of experimental results according to an embodiment of the present application;

[0094] Figure 16 is a detection result diagram of model detection according to an embodiment of the present application;

[0095] Figure 17 is a real object diagram of a mechanical arm motion according to an embodiment of the present application;

[0096] Figure 18 is a result map of point cloud detection according to an embodiment of the present application. DETAILED DESCRIPTION

[0097] The specific embodiments of the embodiments of the present application will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely intended to illustrate and explain the embodiments of the present application, and are not intended to limit the embodiments of the present application.

[0098] It should be noted that the acquisition, transmission, storage, use, processing, etc. of data in the technical solutions of the present application comply with relevant provisions of laws and regulations. In the embodiments of the present application, some industry existing solutions, components, models, etc. may be mentioned, which should be considered as exemplary, and the purpose is only to illustrate the feasibility of the implementation of the technical solutions of the present application, but it does not mean that the applicant has or will necessarily use the solutions.

[0099] Figure 1 is a flowchart of a control method according to an embodiment of the present application, in which the control method comprises:

[0100] In step S1, a voice instruction issued by a user is acquired.

[0101] In step S2, a light-weight model of improved YOLOv8n is used to acquire the food position in the tray according to the voice instruction, wherein the light-weight model of improved YOLOv8n comprises:

[0102] The loss function of the light-weight model of improved YOLOv8n is calculated according to formula (1),

[0103] , (1)

[0104] wherein, is a loss function, i.e. Focaler-WIoU loss function, is a weighted intersection-over-union, i.e. the intersection-over-union under the constraints of center distance constraint and aspect ratio consistency constraint, is a weight factor, used to adjust the contribution of different samples in the training process, and the present application preferably sets to 0.9 to increase the optimization proportion of the boundary fuzzy or occluded fruit pieces in the training, is a focal adjustment factor, used to amplify the gradient of low WIoU samples, and the present application preferably sets it to 2.0 to enhance the learning ability of the model to difficult examples such as reflection and adhesion. In the boundary box regression, the Focaler-WIoU loss function is introduced, and the difficult samples are given greater weight through the focal modulation mechanism, effectively improving the fruit piece positioning accuracy and recall rate.

[0105] In step S3, the mechanical arm forks the food according to the food position. The present application adopts an eye-on-hand, that is, a depth camera is fixedly installed near the end effector of the feeding robot mechanical arm, so that the camera view angle can be adjusted synchronously with the end posture, thereby realizing flexible visual perception and interactive control. After detecting the target fruit block, the two-dimensional pixel coordinates of the detection box center of the target (i.e. the category corresponding to food_name in the user voice command) in the image are first extracted, and the corresponding three-dimensional camera coordinates in the coordinate system are obtained in combination with the depth image; then, the point is converted into a three-dimensional workspace coordinate in the base coordinate system of the mechanical arm by using the calibrated hand-eye transformation matrix and the DH model or kinematics parameters of the mechanical arm, as the pose input point of the fork target.

[0106] In step S4, the three-dimensional pose of the mouth is detected based on the weighted point cloud normal estimation method. The three-dimensional pose includes the position and orientation of the mouth.

[0107] In step S5, the feeding trajectory is planned according to the three-dimensional pose.

[0108] In step S6, the mechanical arm performs the feeding operation according to the feeding trajectory.

[0109] In steps S1 to S6, the present application realizes intent-driven YOLO model target detection through the coupling of voice semantics and visual perception, that is, only locates the watermelon target under the instruction of "I want to eat watermelon", narrows the detection range, and improves the recognition speed and accuracy. This mechanism also supports dynamic food category expansion and continuous learning, and has good scalability and generalization ability. By using the improved lightweight model of YOLOv8n and the weighted point cloud normal estimation method, real-time detection of the food blocks on the plate and high-precision estimation of the user's mouth position are realized, overcoming the shortcomings of the prior art in detection accuracy and adaptability. Compared with traditional loss functions such as IoU and CIoU, the Focaler-WIoU used in the present application pays more attention to the target boundary that is difficult to accurately regress, thereby improving the fitting accuracy and stability of the detection box, effectively separating adjacent fruit blocks, and reducing the false detection and missed detection rates. In addition, the present application generates a continuous and smooth trajectory in motion control, making the feeding process more natural and safe, and improving the flexibility and comfort of human-computer interaction.

[0110] In step S1, the present application introduces a multi-modal large language model to realize semantic understanding of user natural language instructions and automatically map them into corresponding operation processes (such as food type identification, path execution, task termination, etc.). The present application discards the traditional interaction mode based on fixed keywords or button triggering, greatly improves the freedom of human-computer interaction and the range of applicable people, and is suitable for special groups such as normal cognitive but action-limited elderly people and paralytics. The step of obtaining the voice instruction issued by the user can be various as known by those skilled in the art. In an example of the present application, the step of obtaining the voice instruction issued by the user can be the method shown in Figure 2 In Figure 2 , the control method further comprises:

[0111] In step S11, the user's voice instruction is collected in real time by a microphone. The voice signal (such as "I want to eat watermelon" / "I am full") is collected in real time by using a local microphone (such as PyAudio), and then the signal is adjusted in sampling rate (such as 16 kHz) and encoded, and uploaded to a voice recognition interface (such as Ali Bai Lian voice recognition interface api / asr / recognize) through a Hypertext Transfer Protocol Secure (HTTPS) request, and finally the text format user intent sentence, i.e. the voice text, is returned.

[0112] In step S12, the voice instruction is input to the multi-modal large model to determine whether there is an interrupt intention in the voice instruction. If there is an interrupt intention in the voice instruction, step S13 is executed, otherwise, step S14 is executed.

[0113] In step S13, the robot arm stops the feeding operation. The voice stream is continuously monitored, and the semantic content thereof is analyzed by the voice large model. When detecting that the typical interrupt instruction (such as "I am full" "I don't want to eat" etc.) expressing "refusal to eat" or "satiety state" appears in the user's voice, the controller immediately sends an emergency stop instruction to interrupt the mechanical arm action of the feeding robot, and at the same time, a text-to-speech module (TTS) is used to broadcast: "the feeding task has been terminated".

[0114] In step S14, the voice instruction is parsed to generate a structured command. The obtained voice text is input to the Ali Bai Lian large model API, and a structured function call format is requested through the tool_call parameter, specifically as shown in Figure 3 and Figure 4 .

[0115] In step S15, the structured command is matched with the local skill instruction. The structured command (such as eat (food_name = "watermelon")) is matched with the local preset skill instruction set, that is, the control function. If the matching is successful, the eat_task ("watermelon") function is called to start the task of feeding watermelon; if the matching is not successful, the TTS prompt "please repeat your requirement" is called.

[0116] In step S16, the corresponding feeding task function is matched according to the local skill instruction.

[0117] In steps S11 to S16, by introducing a multi-modal large model, real-time understanding and semantic analysis of natural language instructions are realized, and the problem of single interaction and poor user experience in the traditional way is solved, with high real-time performance, strong robustness and good user experience.

[0118] In step S2, when the analysis result is a feeding task function (such as eat (food_name = "watermelon")), a lightened model of improved YOLOv8n is called to perform target detection on the meal tray image collected by the end camera, as shown in Figure 16 After the function obtains multiple food detection results of the model output, the detection categories are filtered according to the input parameter food_name, the center coordinates and depth information of the food block detection box of the corresponding category are extracted, and then used to calculate the three-dimensional position of the target in the mechanical arm workspace of the feeding robot. The way of lightening the improved YOLOv8n model can be various as known by those skilled in the art. In an example of the present application, the way of lightening the improved YOLOv8n model can be the method shown in Figure 5 In Figure 5 The control method further comprises:

[0119] In step S21, the backbone network, feature fusion network and detection head of YOLOv8n are used as the backbone network, feature fusion network and detection head of the lightened model of the improved YOLOv8n.

[0120] In step S22, the standard convolution module in the backbone network of YOLOv8n is replaced by an ADown module.

[0121] In step S23, the Bottleneck structure of the C2f module of the backbone network of YOLOv8n is replaced by a lightened unit UIB based on the MobileNetV4 network.

[0122] In step S24, the standard convolution stack of the C2f module of the feature fusion network of YOLOv8n is replaced with a lightweight unit UIB constructed based on the MobileNetV4 network. Tensor specifications (B, C, H, W) represent batch size (Batch size), channel number (Channels), feature map height (Height), and feature map width (Width), respectively. In the inference stage, B = 1 by default, and Conv (k, s, C) represents convolution (kernel size k, stride s, output channel C). Among them, the ADown module and the C2f_UIB module can be as shown in Figure 13 The improved lightweight model of YOLOv8n can be as shown in Figure 14 In Figure 13 and Figure 14 , the backbone network includes:

[0123] S0 | Initial convolution: Conv (3x3, s=2, C=64)

[0124] Input: (B, 3, 640, 640) → Output C1: (B, 64, 320, 320); Function: Extract low-level edges and color texture, and construct subsequent multi-scale representation basis.

[0125] S1 | Efficient downsampling: ADown (C=128) (downsampling information preservation)

[0126] Input: (B, 64, 320, 320) → (B, 128, 160, 160); Structure: DWConv (3x3, s=2) and lightweight convergence branch are parallel, and the output is fused and aligned with channels after 1x1 PWConv; Improvement advantage: Compared with the standard stride=2 convolution, ADown preserves the boundary and fine texture while reducing the resolution, reduces the confusion of the fruit block contour by the dinner plate background (plate edge), and stabilizes the positioning when occluded. At the same time, the main multiplication and addition are reduced from the standard 3x3 convolution of 9C2 to DW of 9C+PW of C2, and the calculation amount is significantly reduced.

[0127] S2 | Lightweight stacking: C2f_UIBx3 (C=128) (lightweight feature extraction)

[0128] Input / Output: (B, 128, 160, 160) → (B, 128, 160, 160); Structure: Replace standard convolution stack with UIB (1x1 expansion → 3x DWConv → 1x1 compression) within the split-multiplex framework of C2f; Improvement advantages: DWConv focuses on spatial fine-grained, PWConv does channel re-labeling, cross-layer multiplexing improves effective feature density; More sensitive to fruit block surface texture, fruit block and plate interface. At the same time, the parameter amount of a single 3x3 standard convolution is about 9C2, and the parameter amount of UIB is about 9C+C2. When C≥64, the single-layer reduction is about ≈88%, and the GFLOPs are reduced synchronously after stacking.

[0129] S3 | Efficient down-sampling: ADown(C=256)

[0130] Input: (B, 128, 160, 160) → P3 (1 / 8): (B, 256, 80, 80); Effect: Provide a complete base for the P3 small / middle scale branch with boundary details.

[0131] S4 | Lightweight stack: C2f_UIBx6 (C=256)

[0132] Input / Output: (B, 256, 80, 80) → (B, 256, 80, 80); Effect: Strengthen boundary / corner geometry at 1 / 8 scale, improve frame alignment robustness when close or lightly occluded.

[0133] S5 | Efficient down-sampling: ADown(C=512)

[0134] Input: (B, 256, 80, 80) → P4 (1 / 16): (B, 512, 40, 40)

[0135] S6 | Lightweight stack: C2f_UIBx6 (C=512)

[0136] Input / Output: (B, 512, 40, 40) → (B, 512, 40, 40); Effect: Integrate the context semantics on the plate, suppress false detection / drifting caused by high contrast and reflection along the plate.

[0137] S7 | Efficient down-sampling: ADown(C=1024)

[0138] Input: (B, 512, 40, 40) → P5 (1 / 32): (B, 1024, 20, 20)

[0139] S8 | Lightweight stack: C2f_UIBx3 (C=1024)

[0140] Input / Output: (B, 1024, 20, 20) → (B, 1024, 20, 20); Effect: High semantic aggregation, conducive to irregular fruit block overall shape discrimination.

[0141] S9 | SPPF (5x5)

[0142] Input / Output: (B, 1024, 20, 20) → (B, 1024, 20, 20); Effect: Expand the effective receptive field with very low overhead, and stabilize global structure perception.

[0143] The detection head of the feature fusion network comprises:

[0144] N1 | Top-down fusion (P5→P4)

[0145] Structure: Upsample(P5, x2)→Concat(P4)→C2f_UIB(C=512); Input / Output: (B, 1024, 40, 40)⊕(B, 512, 40, 40)→(B, 512, 40, 40); Improvement effect: Replace the multi-layer 3x3 convolution stack with C2f_UIB to unify channel alignment and spatial fine-grained selection. Fuse global semantics and medium-scale geometry to preserve fruit block boundaries in the background of plate reflection / textured, reduce plate edge and tableware interference, while also achieving network lightweight.

[0146] N2 | Top-down fusion (P4→P3)

[0147] Structure: Upsample(P4, x2)→Concat(P3)→C2f_UIB(C=256); Input / Output: (B, 512, 80, 80)⊕(B, 256, 80, 80)→(B, 256, 80, 80); Improvement effect: Continue to use C2f_UIB as the only fusion unit to replace "Concat followed by multi-layer standard convolution". Enhance edges / textures at 1 / 8 scale, reduce false detection of adjacent fruit blocks sticking together, and meet the fine positioning before grabbing. At the same time, DW convolution is used at high resolution to significantly save FLOPs and avoid the neck becoming a bottleneck.

[0148] N3 | Bottom-up backflow (P3→P4)

[0149] Structure: Down(3x3, s=2) -> Concat(N1 output) -> C2f_UIB(C=512); Input / Output: (B, 256, 40, 40) (B, 512, 40, 40) (B, 512, 40, 40); Improvement effect: The backflow path still completes channel alignment + demixing with C2f_UIB, avoiding multi-layer stacking redundancy. The fine-grained geometric backflow of P3 is fed into the mesoscale semantics, the frame jitter of the occlusion and disc neighborhood is reduced, and the feeding trajectory is more stable. In terms of parameter quantity, after Concat, 1x1 compression + DW spatial screening is completed once, which is less than the traditional "1x1 + 3x3x n" combination.

[0150] N4 | Bottom-up backflow (P4 -> P5)

[0151] Structure: Down(3x3, s=2) -> Concat(Backbone S8) -> C2f_UIB(C=1024); Input / Output: (B, 512, 20, 20) (B, 1024, 20, 20) (B, 1024, 20, 20); Improvement effect: Channel reorganization and spatial screening are completed with C2f_UIB. In the case of reducing the parameter quantity, the global inhibition ability of P5 is strengthened, the risk of utensils being misjudged as long fruit is reduced, and the key boundary clues are preserved.

[0152] Detect | Three-scale detection head (Anchor-free)

[0153] Structure: Output class and bounding box parameters on P3 / P4 / P5 respectively (necessary 1x1 / 3x3 mapping, available in depth separable form); Input / Output: P3: (B, 256, 80, 80); P4: (B, 512, 40, 40); P5: (B, 1024, 20, 20) -> prediction tensor; Improvement effect: P3 is responsible for boundary complexity / closeness to the target, P4 covers most of the tableware layout, and P5 provides global inhibition; Boundary box regression uses Focaler-WIoU, which gives greater gradient weight to occlusion / boundary ambiguous samples, improving the positioning accuracy and recall rate of tableware and fruit block detection.

[0154] Further, the input of the ADown module is the feature map output by the upper layer network, with a size of HxWxC (heightxwidthxchannel), and the output feature map will be sent to the subsequent module of the backbone or the detection head, carrying more rich and compressed spatial information. Specifically, it includes:

[0155] I. AvgPool downsampling path: use AvgPool2D (average pooling) for preliminary downsampling, window size K=1, step can be set to s=2 to reduce the size of the feature map. The pooled output is sent to the Chunk operation for further processing.

[0156] II. Chunk operation: the feature map after average pooling is equally divided by channel dimension and sent to two different sub-paths to build a multi-branch information extraction structure.

[0157] III. Double-branch convolution processing: left branch (main branch): access a Conv2D convolution with a kernel size of 3x3 and a step size of s=2 to realize spatial dimension down-sampling; right branch (auxiliary branch): first enhance edge and texture response through MaxPool2D (max pooling), then access a 1x1 convolution (step size s=1) for channel conversion or light fusion. The above two sub-paths jointly extract backbone features and texture features to realize information complementation.

[0158] IV. Concat fusion: the feature maps output by the two sub-paths are concatenated in the channel dimension to obtain the final output feature map. The output size is H / 2xW / 2xC out, which has stronger expression ability.

[0159] Further, the C2f_UIB module includes:

[0160] I. Initial feature compression layer: the input features are first compressed by a 1x1 convolution layer to adjust the number of channels and reduce the computational burden of subsequent modules.

[0161] II. Feature branch and stacking stage, including: Split+ multiple UIB sub-blocks: the input features are equally divided into multiple paths (usually two or more), and each path is extracted by multiple series of UIB units. Each UIB unit is essentially an improved Inverted Bottleneck structure, including two types of variants: Shortcut=True branch: with residual connection, composed of Conv1x1-Conv3x3-jump connection, used to preserve original information and enhance gradient propagation; Shortcut=False branch: standard feedforward path, composed of Conv1x1-Conv3x3-direct output, enhances non-linear transformation ability, and each Conv3x3 can use depthwise convolution (Optional Depthwise Conv) to reduce computational complexity.

[0162] III. Feature fusion and projection layer, including Concat+Conv1x1: the features output by multiple UIB branches are concatenated in the channel dimension (Concat), and then a 1x1 convolution is used to reorganize the number of channels, realizing the compressed representation after fusion as the final output of the module.

[0163] In steps S21 to S24, in the existing YOLO series algorithm, although versions such as YOLOv5, YOLOv7 and the like are mature, the feature extraction and fusion structure is relatively backward, and it is difficult to balance the detection accuracy and lightweight; although new versions such as YOLOv9, YOLOv10 and the like further improve the accuracy, the network depth and the calculation overhead are too large, and are not suitable for real-time running on the embedded end. In contrast, YOLOv8n, as a lightweight model of the YOLOv8 series, has the advantages of simple structure, fast inference speed and easy deployment, and is particularly suitable for fruit block detection tasks of the feeding robot in the dish scene. The present application selects YOLOv8n as the basic framework, which can not only ensure the real-time performance on the hardware with limited resources, but also solve the detection difficulties such as fuzzy boundary of fruit blocks, occlusion adhesion, irregular shape and the like by improving the structure and loss function, so as to realize the unification of lightweight and high accuracy. On the basis of YOLOv8n, the present application proposes an improved method combining a lightweight feature enhancement structure and a down-sampling information reservation mechanism to solve the problems of parameter redundancy, insufficient feature utilization, spatial information loss and inaccurate boundary box fitting in the fruit block detection of the feeding robot in the dish scene.

[0164] Compared with the prior art, the present application introduces C2f_UIB structure in the backbone network (Backbone) and feature fusion network (Neck) part, uses lightweight bottleneck unit and cross-layer feature reuse mechanism to reduce the calculation overhead and improve the expression of texture edge features; in the down-sampling link, the ADown structure is adopted, which reserves more spatial detail information while reducing the resolution through multi-branch aggregation, and enhances the detection robustness in complex background and occlusion conditions.

[0165] Further, in the actual application of the feeding robot, the fruit blocks often have fuzzy boundary, adhesion and occlusion, irregular shape and disc edge reflection and the like. The traditional boundary box regression loss function (such as CIoU) only takes the overlapping area as the main optimization basis, and pays insufficient attention to the detection of such difficult samples, which may lead to prediction frame drift and unclear separation of adjacent targets, thereby directly affecting the stability of the grabbing and feeding actions of the mechanical arm of the feeding robot. In order to solve the above problems, the present application introduces an improved Focaler-WIoU loss function in the boundary box regression stage of YOLOv8n. This method first uses WIoU (Weighted IoU) to comprehensively measure the predicted frame and the real frame, and then introduces a focal mechanism to improve the optimization ability of the model for difficult-to-detect fruit block samples. Specifically, it includes:

[0166] In step S25, the weighted intersection-over-union is determined according to formula (2),

[0167] , (2)

[0168] wherein, The weighted intersection-union ratio (IUU) is the IUU under the constraints of center distance and aspect ratio consistency. The intersection-union ratio (IU) of the predicted bounding box and the ground truth bounding box. Center point of the prediction box Center point of the real frame The square of the Euclidean distance, As a trade-off factor for aspect ratio constraints, As a measure of aspect ratio consistency, The length of the diagonal of the circumscribed minimum closure rectangle is given. Based on the introduction of center distance and aspect ratio consistency penalties, the traditional Intersection over Union (IoU) is weighted and modified to obtain the Weighted Intersection over Union (WIoU), which is used to more accurately evaluate the matching degree between the predicted bounding box and the ground truth bounding box. The Focaler-WIoU loss function, as a further improvement after optimization of the ADown and C2f_UIB modules, plays a key role in addressing the practical difficulties in the fruit piece detection task of the feeding robot. Its principle lies in further introducing a focus mechanism based on the comprehensive WIoU metric, giving higher optimization weights to difficult samples with low WIoU (such as fruit pieces with juice reflection, blurred boundaries, adhesion occlusion, or distorted aspect ratios) during training. At the same time, this improvement does not increase the additional network structure or computational complexity, maintaining the model's real-time performance on embedded systems. Through this improvement, this invention achieves a balance between lightweight design and high-precision detection, providing reliable support for the feeding robot to safely and stably grasp and deliver food in complex plate environments.

[0169] To verify the effectiveness of the improved method proposed in this invention, experiments were conducted on a self-built dataset of 1389 images. This dataset contains two types of food blocks: watermelon slices and banana slices, divided into training, validation, and test sets in an 8:1:1 ratio. The experiments were performed on an Ubuntu 22.04 operating system, using the PyTorch 2.1.0 deep learning framework with CUDA 12.1 providing GPU acceleration support. The hardware platform consisted of an Intel Xeon 12-core CPU and two NVIDIA RTX 2080 Ti graphics cards (total 22GB of VRAM). The input image size was set to 640 pixels to balance detection accuracy and computational efficiency. The model was trained for 150 epochs with a batch size of 32. The optimizer used was stochastic gradient descent (SGD), with an initial learning rate of 0.01, a momentum coefficient of 0.937, and a weight decay of 0.0005. Differential regularization strategies were employed for different parameter groups.

[0170] Under the same training strategy and hardware environment, the experimental results are as follows: (The original YOLOv8n model is compared with the improved model that gradually introduces the C2f_UIB module, the ADown module, and the Focaler-WIoU loss function.)Figure 15 As shown. In Figure 15 The improvements clearly demonstrate the following: 1. Significantly reduced model size: The improved model's parameter count decreased from 3.0M to 2.0M, its size from 6.3MB to 4.4MB, and its FLOPs from 8.1G to 5.6G, significantly reducing computational overhead and making it more suitable for deployment in embedded feeding robot systems. 2. Steadily improved detection accuracy: Detection accuracy improved from 0.938 to 0.964 at mAP@0.5 and from 0.694 to 0.726 at mAP@0.5:0.95, effectively enhancing the accuracy of fruit block boundary localization and overall detection. 3. The synergistic effect of module improvements is evident: Introducing either the C2f_UIB or ADown modules individually brings model lightweighting and a certain degree of accuracy improvement. However, when all three are combined and supplemented with the Focaler-WIoU loss function, the overall performance reaches its optimal level, achieving a balance between lightweighting and accuracy.

[0171] On the Jetson Xavier NX embedded platform, the improved YOLOv8n model increased real-time video detection speed from approximately 21 FPS to 25 FPS, effectively reducing computational overhead during model inference. In the actual food tray scenario of the feeding robot, this improved method not only reduced resource consumption during real-time detection but also significantly improved robustness in detecting blurred boundaries, occlusions, and irregularly shaped fruit pieces, thus ensuring more accurate and stable detection boxes. This further improved the accuracy and continuity of the robotic arm in grasping and feeding actions, achieving a balance between lightweight design and high precision.

[0172] In step S4, a trajectory planning algorithm is used to generate a smooth robotic arm motion trajectory, driving the robotic arm at the end of the fork to complete the picking up of food. After picking up the food, the target feeding point is obtained based on a three-dimensional pose detection method of the mouth based on weighted point cloud normal estimation, and the feeding action is executed. Figure 17 As shown. The steps for detecting the three-dimensional pose of the mouth can be various methods known to those skilled in the art. In one example of the present invention, the steps for detecting the three-dimensional pose of the mouth can be as follows: Figure 6 The method shown is illustrated. Figure 6 The control method also includes:

[0173] In step S41, a depth camera at the end effector of the robotic arm acquires dense 3D point cloud data of the user's mouth region. The dense 3D point cloud data of the user's mouth region acquired using the depth camera mounted at the end effector of the robotic arm is denoted as: .in, All are three-dimensional coordinates. Indicates the first The three-dimensional coordinates of the points This represents the total number of points in the point cloud.

[0174] In step S42, a point cloud subset belonging to the mouth in the point cloud data is extracted. In combination with a face detector or a mouth key point detection network (MediaPipe), the subset belonging to the mouth in the point cloud is extracted, defined as: . Wherein, is the index set conforming to the mouth position constraint.

[0175] In step S43, the normal vector of each point in the point cloud subset is calculated by using the weighted PCA method.

[0176] In step S44, the normal vectors are fused to estimate the overall orientation of the mouth.

[0177] In step S45, the mouth center position is determined according to the geometric centroid of the point cloud subset, including:

[0178] The mouth center position is determined according to formula (3),

[0179] , (3)

[0180] Wherein, is the mouth center position, is the point cloud subset of the mouth, is the i-th point in the point cloud subset of the mouth, is the total number of points in the point cloud subset .

[0181] ​In steps S41 to S45, in human-computer interaction systems, especially in assisted feeding robots, acquiring the spatial position and orientation of the user's mouth is a key technology for achieving safe and accurate feeding operations. However, because facial features are easily affected by changes in lighting, expression, and posture, traditional RGB image or key point localization methods suffer from insufficient accuracy in complex environments. The 3D mouth pose detection method based on weighted point cloud normal estimation proposed in this invention has significant technical advantages such as high accuracy, fast computation, strong adaptability, and friendly embedded deployment. First, compared with existing methods that rely solely on 2D image feature points or regular face models for fitting, this invention directly uses the original mouth point cloud acquired by a depth camera. By constructing a weighted normal estimation formula within the local neighborhood, it significantly enhances robustness to sparse boundaries, noise interference, and point cloud occlusion areas, thereby improving the accuracy of normal estimation and the stability of mouth orientation calculation. Secondly, the weighted normal calculation method adopted does not require global point cloud modeling or highly complex iterative optimization. On a Jetson Xavier NX edge computing device, the average computation time per frame is less than 35 milliseconds, meeting the requirements of real-time human-computer interaction. In summary, this invention not only achieves a comprehensive improvement in the accuracy, robustness, and computational efficiency of mouth pose estimation, but also possesses good versatility, deployability, and interoperability. It is of great significance for promoting the performance evolution of intelligent feeding robot systems in areas such as human-machine collaboration, path control, and multimodal perception. The detection results are as follows... Figure 18 As shown.

[0182] Furthermore, for each point Through its neighborhood point set Specifically, to perform normal vector estimation, such as Figure 7 As shown, it includes:

[0183] In step S431, the weighting coefficients are determined according to formula (4).

[0184] (4)

[0185] In step S432, the covariance matrix is ​​constructed according to formula (5).

[0186] (5)

[0187] in, For point Relative to point The weighting coefficients are used to control the weighting of different neighborhood points at the estimation point. The degree of contribution of the normal vector. for The neighborhood point set, for The first in the neighborhood set One point, This is the distance decay parameter, used to control the rate at which the weights of neighboring points far from the center point decay. The smaller the value, the faster the weight decays at points further away. for The covariance matrix of the neighborhood point set, Let be the centroid of the neighborhood point set.

[0188] In step S433, the covariance matrix is ​​decomposed into eigenvalues ​​and eigenvectors.

[0189] In step S434, the eigenvector corresponding to the smallest eigenvalue is selected as the normal vector.

[0190] Furthermore, such as Figure 8 As shown, in step S441, the normal vector of each point is subjected to distance-weighted global fusion according to formulas (6) to (8).

[0191] (6)

[0192] (7)

[0193] (8)

[0194] in, For the first The weighting coefficients for each point, This is the distance attenuation factor, representing the degree of distance attenuation during mouth orientation fusion. It can be understood as the attention attenuation rate during the information fusion stage. The larger it is, the more important the center point becomes. For the first The Euclidean distance between each point and the centroid of the subset of points in the mouth cloud. The normal vector of the mouth is the overall shape. For the first The normal vector of each point Let the centroid of the point cloud subset of the mouth be the centroid.

[0195] In step S442, the overall normal vector of the mouth is normalized.

[0196] In steps S431 to S434 and steps S441 to S442, using Weighted fusion of local neighborhood normals was achieved through weighted fusion estimation, and the following was performed: Perform eigenvalue decomposition and take the eigenvector corresponding to the smallest eigenvalue as the point. normal vector Furthermore, the set of normal vectors for all points. The weighted fusion is performed to calculate the overall normal vector of the mouth, and accurate estimation of the mouth direction is realized.

[0197] In addition, since the present application does not rely on a standard face template or a specific race / gender face prior model, it has strong adaptability to individual differences of users, and significantly improves the generalization ability in complex scenes such as families and nursing homes. In actual operation, the pose estimation result can be seamlessly coupled with the feeding trajectory planning module to realize dynamic real-time compensation. When a slight movement of the user's head is detected, the current trajectory endpoint can be updated through the mouth normal, and the feeding path can be automatically adjusted, effectively reducing the risk of "mouth deviation" and "accidental touch" caused by error accumulation, and improving the feeding accuracy and user comfort. The overall performance is better than the existing iterative registration-based method. Specifically, the steps of planning the feeding trajectory according to the three-dimensional pose can be the method as shown in Figure 9 Figure 9 The control method further comprises:

[0198] In step S51, the target mouth position is determined according to the three-dimensional pose.

[0199] In step S52, the feeding trajectory is planned according to the target mouth position and the position of the end effector of the robot arm. The target mouth position is the final target pose of the end effector, which is represented as the end position constraint of the feeding trajectory in the trajectory planning, and specifically includes:

[0200] In the set feeding task, it is assumed that the initial time is , the initial position of the end effector of the robot arm is , the target mouth position is , and all are three-dimensional coordinates. The trajectory planning duration is . As shown in Figure 10

[0201] In step S521, the feeding trajectory is determined according to formula (9),

[0202] , (9)

[0203] In step S522, the constraint condition of the feeding trajectory is determined according to formula (10),

[0204] , (10)

[0205] Wherein, is the feeding trajectory at time , i.e. the spatial position of the end effector at time , which constitutes the feeding trajectory is the coefficient vector of the feeding trajectory, is the index number, ​​feeding trajectory planning time, , feeding trajectory execution period, feeding trajectory function, feeding trajectory velocity vector function, feeding trajectory acceleration vector function, initial position, initial desired velocity, default is 0, target mouth position, final desired velocity, default is 0. To ensure that the trajectory meets the continuity of position, velocity and acceleration at the start and end points, the above six groups of boundary constraint conditions are required.

[0206] In step S53, the current mouth position is obtained.

[0207] In step S54, the mouth offset error vector is calculated according to the target mouth position and the current mouth position.

[0208] In step S55, the compensation trajectory is planned according to the mouth offset error vector to update the feeding trajectory.

[0209] In steps S51 to S55, for patients with upper limb disorders, in the actual feeding process, the user may cause the mouth position to change due to slight head movement of the user, thereby affecting the ability of the robot arm end (such as a fork) to accurately send food to the edge of the mouth. In order to enhance the fault tolerance and system robustness of the feeding trajectory, the present application realizes the tracking of the real-time dynamic position of the user's mouth and the end trajectory compensation while maintaining the smoothness of the path, ensuring that the feeding end is stably aligned with the target mouth position under the premise of maintaining a safe distance.

[0210] In step S55, the way to update the feeding trajectory can be various as known by those skilled in the art. In one example of the present application, the step of updating the feeding trajectory can be the method shown in Figure 11 In Figure 11 , the method further comprises:

[0211] Let the th sampling time be , then the current mouth estimated position is: .

[0212] In step S551, the mouth offset error vector is determined according to formula (11),

[0213] , (11)

[0214] to compensate for the offset error , a short period of quintic polynomial compensation trajectory is re-planned:

[0215] In step S552, the feeding trajectory is updated according to formula (12),

[0216] (12)

[0217] wherein, is the time of the nth sampling, is the offset error vector at the time of the nth sampling, is the current mouth position at the time of the nth sampling, is the updated feeding trajectory, is the updated feeding trajectory planning time, , is the updated feeding trajectory execution period, after the trajectory is updated, the execution period is dynamically adjusted, denoted as , which is used to describe the new trajectory execution period based on the mouth offset error compensation, and , is the coefficient vector of the updated feeding trajectory, is the index number. The boundary condition of the updated feeding trajectory is set as the initial value of the current end position, speed and acceleration, and the end position is , the speed and acceleration are zero. In steps S551 to S552, the three-dimensional posture information of the user's mouth (including spatial position and orientation) is obtained in real time, and a continuous tracking and dynamic compensation algorithm is used to fine-tune and correct the target endpoint in each trajectory execution process, thereby improving the accuracy of feeding. The present application uses a quintic polynomial interpolation method to generate a smooth trajectory from the initial position to the spatial target point of the mouth, while ensuring the continuity of position, speed and acceleration, and cooperating with the mouth dynamic compensation mechanism to realize closed-loop trajectory reconstruction. Compared with the traditional linear interpolation or cubic interpolation method, this method has better trajectory smoothness, controllability and biocompatibility, reduces the discomfort of the user, and improves the naturalness of feeding.

[0218] In step S6, the robot arm performs a feeding operation according to the feeding trajectory. The specific steps for performing the feeding operation can be various as known by those skilled in the art. In one example of the present application, the specific steps for performing the feeding operation can be the method shown in In

[0219] , the control method further comprises: Figure 12 Figure 12 In step S61, the current mouth position and the position of the end effector of the robot arm are obtained.

[0220] In step S61, the current mouth position and the position of the end effector of the robot arm are obtained.

[0221] ​In step S62, it is judged whether the distance between the current mouth position and the position of the end effector of the mechanical arm is greater than a first preset safety threshold. If the distance between the current mouth position and the position of the end effector of the mechanical arm is greater than the first preset safety threshold, step S63 is executed, otherwise, step S64 is executed. The first preset safety threshold can be 1cm.

[0222] In step S63, the mechanical arm performs the feeding operation. At the same time, the TTS feedback of Ali Bai Lian is called: "I am feeding you with watermelon, please wait a moment."

[0223] In step S64, the mechanical arm performs the hovering operation.

[0224] In steps S61 to S64, a 1cm safety distance is set in front of the mouth to hover the end point, ensuring that the feeding end is precise while avoiding touching. This method effectively adapts to slight head deviation or posture changes of the user, improving robustness and feeding safety.

[0225] Through the above technical solution, the application provides a control method of a feeding robot based on a multi-modal large model. The application realizes intent-driven YOLO model target detection through the coupling of voice semantics and visual perception, narrows the detection range, and improves the recognition speed and accuracy. By using the improved lightweight model of YOLOv8n and the weighted point cloud normal estimation method, real-time detection of the food blocks on the plate and high-precision estimation of the user's mouth position are realized, overcoming the shortcomings of the prior art in detection accuracy and adaptability. Compared with traditional loss functions such as IoU and CIoU, the Focaler-WIoU used in the application pays more attention to the boundaries of targets that are difficult to accurately regress, thereby improving the fitting accuracy and stability of the detection frame, effectively separating adjacent fruit blocks, and reducing the false detection and missed detection rates. In addition, the application generates a continuous and smooth trajectory in motion control, making the feeding process more natural and safe, and improving the flexibility and comfort of human-computer interaction.

Claims

1. A method for controlling a multi-modal large model feeding robot, the method comprising: receiving a user input; determining a feeding plan based on the user input; and controlling the multi-modal large model feeding robot based on the feeding plan. The control method comprises: acquiring a voice instruction issued by a user; acquiring a food position in a tray according to the voice instruction using a lightweight model of an improved YOLOv8n, wherein the lightweight model of the improved YOLOv8n comprises: using a backbone network, a feature fusion network and a detection head of YOLOv8n as a backbone network, a feature fusion network and a detection head of the lightweight model of the improved YOLOv8n; replacing a standard convolution module in the backbone network of YOLOv8n with an ADown module; replacing a Bottleneck structure of a C2f module of the backbone network of YOLOv8n with a lightweight unit UIB constructed based on a MobileNetV4 network; replacing a Bottleneck structure of a C2f module of the feature fusion network of YOLOv8n with a lightweight unit UIB constructed based on a MobileNetV4 network; calculating a loss function of the lightweight model of the improved YOLOv8n according to formula (1), ，（1） wherein, is a loss function, is a weighted intersection over union, is a weight factor, is a focal adjustment factor; a mechanical arm forks food according to the food position; detecting a three-dimensional pose of a mouth based on a weighted point cloud normal estimation method; planning a feeding trajectory according to the three-dimensional pose; the mechanical arm performs a feeding operation according to the feeding trajectory.

2. The control method according to claim 1, characterized by, Acquiring a voice instruction issued by a user comprises: using a microphone to collect a user voice instruction in real time; inputting the voice instruction into a multi-modal large model to determine whether there is an interrupt intention in the voice instruction; in a case where it is determined that there is an interrupt intention in the voice instruction, the mechanical arm stops the feeding operation; in a case where it is determined that there is no interrupt intention in the voice instruction, the voice instruction is parsed to generate a structured command; matching a local skill instruction according to the structured command; matching a corresponding feeding task function according to the local skill instruction.

3. The control method according to claim 1, characterized by, Acquiring a food position in a tray using a lightweight model of an improved YOLOv8n comprises: determining a weighted intersection-over-union according to formula (2), ，（2） wherein, is a weighted intersection over union, is an intersection over union of the predicted box and the ground truth box, is the Euclidean distance squared of the predicted box center point and the ground truth box center point is a width-to-height ratio constraint weighing factor, is a width-to-height ratio consistency measure, is a diagonal length of the minimum enclosing bounding box.​ 4. The control method according to claim 1, characterized by, Detecting a three-dimensional pose of a mouth based on a weighted point cloud normal estimation method comprises: using a depth camera at the end of a mechanical arm to acquire dense three-dimensional point cloud data of a mouth region of a user; extracting a point cloud subset belonging to the mouth from the point cloud data; calculating a normal vector of each point in the point cloud subset using a weighted PCA method; fusing the normal vectors to estimate the overall orientation of the mouth; determining a mouth center position according to the geometric centroid of the point cloud subset comprises: determining the mouth center position according to formula (3), ，（3） wherein, is the mouth center position, is the point cloud subset of the mouth, is the i-th point in the point cloud subset of the mouth, is the i-th point in the point cloud subset of the mouth.

5. The control method according to claim 4, characterized by calculating a normal vector of each point in the point cloud subset using a weighted PCA method comprises: determining a weight coefficient according to formula (4), ，（4） constructing a covariance matrix according to formula (5), ，（5） in, For point Relative to point The weighting coefficients, for The neighborhood point set, for The first in the neighborhood set One point, For distance attenuation parameters, for The covariance matrix of the neighborhood point set, Let be the centroid of the neighborhood point set; performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues and eigenvectors; selecting an eigenvector corresponding to the smallest eigenvalue as the normal vector.

6. The control method according to claim 4, characterized by Fusing the normal vectors to estimate the overall orientation of the mouth comprises: performing global fusion of the normal vector of each point based on distance weighting according to formulas (6) to (8), ，（6） ，（7） ，（8） wherein, is a weighting factor for the th point, is a distance decay factor, is a Euclidean distance between the th point and the centroid of the mouth point cloud subset, is a mouth overall normal vector, is a normal vector of the th point, is a centroid of the mouth point cloud subset; performing normalization processing on the overall normal vector of the mouth.

7. The control method according to claim 1, characterized by, Planning a feeding trajectory according to the three-dimensional pose comprises: determining a target mouth position according to the three-dimensional pose; Planning a feeding trajectory according to the target mouth position and a position of an end effector of the robotic arm, comprising: determining the feeding trajectory according to formula (9), ，（9） determining a constraint condition of the feeding trajectory according to formula (10), ，（10） wherein, is a feeding trajectory at a time instant, is a coefficient vector of the feeding trajectory, is an index number, is a feeding trajectory planning time, , is a feeding trajectory execution period, is a feeding trajectory function, is a feeding trajectory velocity vector function, is a feeding trajectory acceleration vector function, is a start position, is a start desired velocity, is a target mouth position, is an end desired velocity; obtaining a current mouth position; calculating a mouth offset error vector according to the target mouth position and the current mouth position to update the feeding trajectory.

8. The control method according to claim 7, characterized by calculating a mouth offset error vector according to the target mouth position and the current mouth position to update the feeding trajectory, comprising: determining the mouth offset error vector according to formula (11), ，（11） updating the feeding trajectory according to formula (12), ，（12） wherein, is the is the is the is the is the is the is the is the is the is the is the is the is the 9. The control method according to claim 8, characterized by, the robotic arm performing a feeding operation according to the feeding trajectory, comprising: obtaining a current mouth position and a position of an end effector of the robotic arm; judging whether a distance between the current mouth position and the position of the end effector of the robotic arm is greater than a first preset safety threshold; in a case where it is judged that the distance between the current mouth position and the position of the end effector of the robotic arm is greater than the first preset safety threshold, the robotic arm performing a feeding operation; in a case where it is judged that the distance between the current mouth position and the position of the end effector of the robotic arm is not greater than the first preset safety threshold, the robotic arm performing a hovering operation.

Images