A meal picking system and method

Abstract

The present invention belongs to the technical field of intelligent meal picking, and specifically relates to a meal picking method, comprising the following steps: Step 1: voice recognition, in which a customer inputs meal ordering information by voice through a microphone; Step 2: natural language processing, in which the information is converted into machine strategy instructions that can be recognized by a machine; Step 3: detection and positioning, in which the meal is determined by an image recognition module; Step 4: path planning, in which an optimal path is found; Step 5: path execution, in which a robot arm automatically completes the meal picking operation. The present invention can control the operation of the robot arm through voice recognition and recognition of the meal, and at the same time utilizes path planning to select and execute a closest trajectory without collision in the same direction of the selected grabbing process, thereby improving the operating efficiency of the robot arm, reducing costs, and bringing huge benefits to commercial catering.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent meal collection, and in particular relates to a meal collection system and method. Background Art

[0002] With the development of automation and intelligent technology, complex systems with multiple inputs and outputs, such as robotic arms, have been widely used in various industries to carry out operations such as handling, placing, and transferring objects. However, in the catering industry, robotic arms need to perform gripping, transporting, and identifying tasks in an unordered and unstructured environment. Faced with the changing environment and the complexity of robotic arm operations, the robotic arms need to plan their working methods in advance to cope with different working environments and working conditions. This requires improving the control accuracy and requirements of the robotic arms, while also making the robot's robotic arms have stronger cognitive and executive capabilities. Therefore, it is necessary to provide a control method that can improve the intelligence and adaptability of the robot to solve the above problems. Summary of the invention

[0003] The purpose of the present invention is to provide a meal collection system and method to solve the problems existing in the background technology.

[0004] In order to achieve the above technical objectives, the technical solution adopted by the present invention is as follows:

[0005] A method for taking a meal, characterized in that it comprises the following steps:

[0006] Step 1: Voice recognition: The customer uses a microphone to input the order information, and the voice recognition module recognizes the voice and converts it into natural language;

[0007] Step 2: Natural language processing: the natural language input by the customer is processed and the language processing module converts it into machine policy instructions that can be recognized by the machine;

[0008] Step 3: Detection and positioning: determine the food through the image recognition module, locate the food, and return the system positioning information;

[0009] Step 4: Path planning: Based on the detection and positioning information, the central processing module plans the motion trajectory of the robot arm and finds the optimal path;

[0010] Step 5: Path execution: Based on the instructions given by online planning, the robot arm automatically completes the food pickup operation.

[0011] The step 2 uses the independently pre-trained LLM framework ChatGPT3.0 and the VLM framework CLIP to perform natural language processing and computer visual language processing respectively. The specific steps are:

[0012] Step 2.1: Integration of LLM, the natural language input in step 1 is processed by the pre-trained LLM, and then the output is mapped to the robot's actionable commands or queries. The LLMNode function is described as a mapping from natural language input to robot actions, that is, LLMNode: Where L represents the space of natural language input, A represents the set of possible actions of the robot, and the formula is as follows:

[0013]

[0014] Where LM(l) is the language model for input l i ∈L, Sen(Data) represents the sensor data that informs the context command, and then REM(RobotExecutionMechanism) converts it into an executable command for the robot;

[0015] Step 2.2: Integration of VLM, using OpenAI Contrastive Language Image Pre-trained Model (CLIP) to provide RGB image stream of observation source (Intel Realsense D435i) and text description of objects in the image,

[0016]

[0017] The step 3 acquires the image of the food through a monocular vision system, and then detects and locates the food. The specific steps are as follows:

[0018] Step 3.1: Tracking, extract features from the image using FAST and BRIEF algorithms, with the goal of using the ORB algorithm;

[0019] Step 3.2: Mapping, the location of the feature is estimated, a local map is derived using BA, and then new equivalent points of the unmatched ORB points are searched from the remaining keyframes and the new points are triangulated;

[0020] Step 3.3: The loop ends, and the last step of ORB-SLAM is to compare the obtained local map with the last keyframe to optimize the duplicate points to be merged, and then ORB-SLAM continues to search for the next keyframe;

[0021] Step 3.4: Point cloud clustering, aggregate neighbor points in the cloud, which may be obtained from the following video frames, in order to detect all points describing the same object;

[0022] Step 3.5: Bounding box evaluation, assuming that the bounding box is a rectangle and then consider all the points assigned to the cluster as the data source of PCA, calculate the covariance matrix after normalization, and perform eigendecomposition on it to obtain two eigenvector value pairs in the two-dimensional case

[0023] The step 4 uses a real-time iterative kinematics solver (IK solver) of joint angle differential evolution to plan, select and execute a collision-free trajectory leading to the selected grasping trajectory for the robot arm, and the specific steps are as follows:

[0024] Step 4.1: First, grab the path points of the trajectory (sv w ,c h ) ij , and then converted to a trajectory in the robot's full joint space:

[0025] where i = 1…m j

[0026] Hand joint configuration Joint configuration of the arm Where N a is the number of arm joints;

[0027] Step 4.2: Measure the wrist posture sv by task space error w and K mapped by forward kinematics for : From the approximate solution Calculated pose:

[0028]

[0029] Among them, lin(.) extracts the R in the parameter 3 translation vector, ang(.) extracts the SO quaternion in the parameters, ||·|| is the Euclidean norm, · is the quaternion dot product, and a∈<0…1> is the coefficient that weighs the linear and angular distance.

[0030] Step 5 will load the previously planned path, perform an initial check to ensure that the path is collision-free and kinematically feasible, and then execute the path. The specific steps are:

[0031] Step 5.1: Find the current robot configuration c c The approach path point c 1r Two tracking trajectories of Then according to c c and c 1r The distance between them determines the execution trajectory:

[0032]

[0033] in It is the threshold for controlling the switching between global and local planning;

[0034] Step 5.2: The robot performs the tracking trajectory at the end of each control cycle Try to stay at the approach waypoint of the current grab track until tracking is interrupted.

[0035] The present invention can control the operation of the robotic arm through voice recognition and identification of food, and at the same time use path planning to select and execute a collision-free shortest trajectory for the selected grasping process in the same direction, thereby improving the operating efficiency of the robotic arm, reducing costs, and bringing huge benefits to commercial catering. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention is further illustrated by means of the non-limiting examples given in the accompanying drawings.

[0037] Figure 1 It is a schematic diagram of the natural language processing process of the present invention;

[0038] Figure 2 It is a schematic diagram of the detection and positioning process of the present invention. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the present invention, the technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] like Figure 1-2 As shown, a meal taking system and method of the present invention comprises the following steps:

[0041] 1. Voice recognition: customers use a microphone to input their order information, and the system recognizes the voice and converts it into natural language.

[0042] 2. Natural language processing: Process the natural language input by customers and convert it into machine policy instructions that can be recognized by machines.

[0043] 3. Detection and positioning: determine the food after image recognition, locate the food, and return the system positioning information.

[0044] 4. Path planning: Based on the detection and positioning information, plan the motion trajectory of the robot arm and find the optimal path.

[0045] 5. Path execution: Based on the instructions given by online planning, the robot arm automatically completes the food collection operation.

[0046] Step 1: Speech recognition. In the present invention, speech recognition is used as an input to input natural language text for computer processing. In the present invention, OpenAI's Whisper speech-to-text model will be used for speech-to-text conversion. Whisper is an automatic speech recognition (ASR) system that has been trained with 680,000 hours of multi-language and multi-task supervised data. Through a large and diverse data set, the robustness to accents, background noise, and technical language will be effectively improved.

[0047] The Whisper architecture is a simple end-to-end approach implemented as an encoder-decoder converter. The input audio is segmented into 30-second segments, converted into log-Mel spectrograms, and then passed to the encoder. The decoder is trained to predict the corresponding text description, which is interspersed with special tags, guiding a single model to perform tasks such as language identification, phrase-level timestamping, multi-language speech transcription, and English speech translation. The use of the Whisper model can effectively meet the speech recognition function in the present invention and provide natural language text to the computer.

[0048] Step 2: Natural language processing. In this step, the natural language input by the customer needs to be processed and then converted into computer assembly language for the computer to understand and proceed to the next step. In the present invention, the independently pre-trained LLM framework ChatGPT3.0 and VLM framework CLIP are introduced to perform natural language processing and computer visual language processing respectively, where LLMNode decodes natural language dialogue and CLIPNode provides visual and semantic understanding of the robot task environment.

[0049] Specific steps:

[0050] 1. Integration of LLM. In order to decode natural language dialogue and abstract it into robot actions, a ROS (Robot Operation) node LLMNode is used in the present invention to establish communication between the pre-trained LLM and the interface in the ROS ecosystem.

[0051] The present invention processes the natural language input in step 1 through a pre-trained LLM, and then maps the output to an actionable command or query for the robot. During the mapping process, the present invention uses pattern matching to align the generated text with predefined actions or information requests (for example, navigation commands are converted into goals that the robot pursues in its environment. In order to enable real-time feedback, LLMNode also supervises the execution and feedback process, which will facilitate the system to record interaction data for subsequent analysis.

[0052] In summary, the LLMNode function can be described as a mapping from natural language input to robot action, namely LLMNode: Where L represents the space of natural language input, and A represents the set of possible actions of the robot. This mapping is a combination of multiple functions, and the formula is as follows:

[0053]

[0054] In formula (1), LM(l) is the language model for input l i ∈L, Sen(Data) represents the sensor data that informs the contextual command. Then REM(RobotExecutionMechanism) converts it into an executable command for the robot.

[0055] 2. Integration of VLM. In addition, to provide visual and semantic understanding of the task environment, the present invention uses the OpenAI Contrastive Language Image Pre-training Model (CLIP). The CLIP model consists of language and image encoders that have been trained on an astonishing 400 million image-text pairs. Therefore, the present invention will use CLIP to provide a stream of RGB images from the observation source (Intel Realsense D435i) and text descriptions of objects in the image.

[0056] Given an image I at time t t , we consider a set of predefined text descriptions D = {d1,d2,d3,…,d n Each description d i ∈D is mapped to a tokenized representation, forming a set T = {t1, t2, t3, …, t n This set contains human-readable labels of common food names, such as "Kung Pao Chicken", "Fish-flavored Eggplant", "Tomato Scrambled Eggs", etc. Using the CLIP model, extract the feature vector of the image, i.e., f I = CLIP_encode_image(I t ). For each tokenized description t i ∈T, its image features and text features, using Therefore, the present invention determines the object to be identified by selecting the text description with the highest similarity score. The formula is as follows:

[0057]

[0058] Step 3: Detection and positioning. In this step, the food image will be acquired through the monocular vision system, and then the food will be detected and positioned to facilitate the path planning of the robot arm in the next step. In this invention, SLAM-OR

[0059] (Simultaneous localization and mapping-Object Recognition) algorithm, in the absence of prior information, uses the global reference position estimation for long-term synchronous map construction and positioning, and combines the object recognition algorithm based on YOLO for object recognition. Then used in the present invention, the image processing pipeline will adopt the monocular ORB-SLAM algorithm, which is divided into three steps: tracking, local mapping and loop closing.

[0060] Specific steps:

[0061] 1. Tracking. The first step of the algorithm is to extract features from the image, and the purpose is to use the ORB algorithm. Feature extraction uses the FAST and BRIEF algorithms. The image is converted to grayscale, and then the FAST algorithm is used to search for corners. FAST involves detecting a set of pixels N in the image that meet a threshold within a certain radius. The ORB algorithm uses the intensity centroid method, which is able to determine the direction of the corner. Assuming that the intensity is offset from its center, the resulting vector will help determine the direction. The main idea of ​​the BRIEF algorithm is to randomly select points around the feature point (corner), and then combine the values ​​of these points in a binary vector, using the vector as a descriptor. Formula (3) represents a binary vector T, where g assumes that the image is smooth:

[0062]

[0063] where g(x)id is defined as (4)

[0064]

[0065] ORB extracts the BRIEF descriptor according to the direction obtained by formula (5)

[0066]

[0067] We can use the appropriate rotation matrix R θ and S to construct

[0068] S θ S θ =R θ S(6)

[0069] The returned descriptors are sensitive to interference and noise, which can result in large variability in each feature. ORB dynamically adjusts the thresholds of the descriptors and selects the threshold with the largest value. In cases where there are no features in some parts of the image (for example, due to a uniform background or low contrast differences), ORB uses other descriptors. If there are not enough features to perform tracking, the search for points is repeated based on the current position. After obtaining appropriate features from the image and using the DBoW2 mechanism, the predicted position of the camera is determined.

[0070] 2. Mapping. Once the location of the features is estimated, a local map is derived using bundle adjustment. New equivalent points of the unmatched ORB points are then searched from the remaining keyframes and the new points are triangulated. The algorithm monitors the points generated and, based on the information collected during tracking, eliminates them so that only the highest quality points remain.

[0071] 3. The loop ends. The last step of ORB-SLAM is to compare the obtained local map with the last keyframe to optimize the duplicate points to be merged. After that, ORB-SLAM continues to search for the next keyframe.

[0072] 4. Point cloud clustering. The goal is to aggregate neighboring points in the cloud that may be obtained from the following several video frames in order to detect all points that describe the same object. After aggregation, we can estimate the spatial coordinates of the bounding box of the detected object. Since we cannot make any assumptions about the number of clusters present in the recordings, as cluster membership is determined by spatial proximity and point density, a straightforward choice is to apply density-based clustering such as DBSCAN or OPTICS.

[0073] 5. Bounding Box Evaluation. This paper proposes a heuristic method based on principal component analysis (PCA). Let us assume that the bounding box is rectangular. Let us consider all the points assigned to the clusters as the data source for PCA. After normalization, we use these to calculate the covariance matrix and perform eigendecomposition on it, obtaining (in the two-dimensional case) two pairs of eigenvector values: In the 2D case (which can be easily extended to 3D), we can assume that the first principal component is parallel to the two bounding box boundaries (b1, b3), while the second principal component is parallel to the remaining two bounding box boundaries (b2, b4):

[0074]

[0075] The center of the bounding box is located at a point determined by the mean of all points in a given cluster M, and the length of the bounding box is proportional to the variance explained by a given principal component. and var ％1 ,right and var ％2 The bounding box can be calculated using the following formula:

[0076]

[0077] α is a scaling parameter, which we set to 2.

[0078] Step 4: Path Planning. In this step, our method will provide online path planning for the robotic arm and provide an effective grasping path for the robotic arm to grasp the food. We will convert the grasping trajectory into a new object frame based on a real-time iterative kinematic solver (IK solver) based on differential evolution of joint angles. At any given moment, a single grasping trajectory is selected online that has the smallest task space error due to kinematic constraints and collisions. Finally, the real-time arm trajectory planning selects and executes a collision-free trajectory leading to the selected grasping trajectory. In the present invention, some different approaches will be taken.

[0079] 1. Rely on a generative grasping model to generate a set of diverse grasping trajectories.

[0080] 2. Grasp trajectories are selected using a soft metric of trajectory quality, which is a task space error that can be incorporated into the robot’s motion planning by solving the inverse kinematics using a nonclassical approach based on iterative numerical optimization of the robot’s joint angles to minimize the soft task space error. The robot continuously “tracks” the current grasp trajectory as it approaches the waypoint by executing the trajectory that produces the minimum feasible task space error at any moment in time. This technique can handle situations where no error-free trajectory exists, but a feasible grasp may be available again in the near future.

[0081] 3. Our approach relies on a dynamic switch between two planners, depending on how far the robot is from the current destination pre-grasp position. This technique produces smooth “local” robot motion (in terms of joint space) as long as the current task space error is not larger than a predefined threshold. However, when the current target pre-grasp position becomes highly infeasible (the task space error becomes large), the robot switches to a “global” planner, which triggers a global discontinuity when the robot selects and moves to a new pre-grasp pose relative to the moving object (necessary because the object’s motion has caused the previously desired pre-grasp pose to become unattainable). It is worth mentioning that for local trajectory planning, we use a fast, anywhere-anytime local planner that plans collision-free trajectories that guarantee that waypoints remain close to each other in joint space. In this step, since the generation of the grasping path will be completed by the LLM integrated in step 2, this step will specifically describe the planning and selection of the path.

[0082] Specific steps:

[0083] 1. Arbitrary time iterative inverse kinematics solver. First, grab the path points of the trajectory (sv w ,c h ) ij (Wrist gesturesv w ∈SE) is converted into the trajectory in the robot's complete joint space through the inverse transformation (inverse()):

[0084]

[0085] Hand joint configuration Joint configuration of the arm Where N a is the number of arm joints. These trajectories are computed using an arbitrary-time iterative inverse kinematics solver (IKsolver) based on differentially evolving joint angles, rather than by inverting or pseudo-inverting the Jacobian matrix as in the classical approach. The IK solver is used for any input target Provides collision-free solutions taking into account various task-space constraints (such as collision boundaries or joint limits). This is because the solution allows for inaccuracies and accuracy is measured by the task-space error

[0086] 2. Error-based selection. The quality of the inverse kinematics solution is measured by the task space error, which compares the desired wrist pose sv w and K mapped by forward kinematics for : From the approximate solution Calculated pose:

[0087] Among them, lin(.) extracts the R in the parameter 3 translation vector, ang(.) extracts the SO quaternion in the parameters, ||·|| is the Euclidean norm, · is the quaternion dot product, and a∈<0…1> is the coefficient that weighs the linear and angular distance.

[0088] The overall j-th trajectory error ε j is the sum of the task space errors of all waypoints:

[0089]

[0090] Calculate all grasping trajectories and their corresponding errors Then, find the trajectory with the minimum error ε k The kth trajectory of:

[0091]

[0092] If its error ε k Less than the previous minimum error And an additional offset δ w ∈R, then the kth trajectory becomes the current grasping trajectory r.

[0093] Step 5: Path execution. In this step, ROS will load the previously planned path and perform an initial check to ensure that the path is collision-free and kinematically feasible, and then execute the path. In the actual meal collection process, many unexpected situations may occur, such as the location of the meal being transferred. Therefore, the present invention will also use real-time path tracking to achieve the effect of tracking the meal and avoiding obstacles.

[0094] Specific steps:

[0095] 1. Tracking trajectory planning. At this stage, we need to find the trajectory from the current robot arm configuration c c The approach path point c 1r Tracking trajectory The track is executed immediately. Use one of the following two trajectory planners to find: 1. A local planner based on local iterative optimization, denoted as local(); 2. A global planner based on a probabilistic roadmap (PRM), denoted as global(). Both planners will find a path from the current robot configuration c c To the target approach path point c 1r The local planner ensures that all trajectory path points {c i} are all in c c and c 1r The global planner does not guarantee this property, but tries to find c to c 1r The shortest path of c and c 1r The distance between them determines whether to use a global planner or a local planner:

[0096]

[0097] in is the threshold for switching between global and local planning. c Values ​​are between 0 and 1, with smaller values ​​allowing faster switching.

[0098] 2. Path tracking execution. The robot executes the tracking trajectory at the end of each control cycle In this way, the robot arm attempts to stay at the approach waypoint of the current grasping trajectory until tracking is interrupted.

[0099] The specific implementation method of step 2 is as follows:

[0100] The speech processing module consists of three main modules:

[0101] LLMNode: Processes natural language input and generates instructions that the robot can execute.

[0102] CLIPNode: Providing visual and semantic understanding of the task environment.

[0103] REMNode: Abstracting high-level language understanding and environment perception into actual robot actions.

[0104] 1.LLMNode (Large Language Model Node)

[0105] Function: Decode natural language conversations and abstract them into robot action instructions.

[0106] Processing process:

[0107] 1. Input processing: accepts natural language input from speech recognition and passes it to the pre-trained large language model (GPT-4)

[0108] 2. Language decoding: Use pattern matching to align the generated text to predefined actions or information requests.

[0109] 3. Action generation: Map the decoded language into executable instructions for the robot.

[0110] 4. Real-time feedback: Provide real-time feedback by publishing messages and record interaction data for subsequent analysis.

[0111] 2. CLIPNode (Comparative Language Image Pre-training Node)

[0112] Function: Provide visual and semantic understanding of the task environment.

[0113] Processing process:

[0114] 1. Image encoding: Use the CLIP model to encode the RGB image and extract the feature vector of the image.

[0115] 2. Text encoding: Map predefined text descriptions into feature vectors.

[0116] 3. Similarity calculation: Calculate the similarity score between the image features and the text features to determine the objects recognized in the image.

[0117] 4. Spatial localization: Use the bounding box of the YOLO model to determine the location of the object in the image, and publish the identity and location of the object as a ROS topic.

[0118] 3.REMNode (Robot Execution Mechanism Node)

[0119] Function: Abstract high-level language understanding and environmental perception into actual robot actions.

[0120] Processing process:

[0121] 1. Navigation target processing: Convert the target position described in text into precise coordinates within the robot's operating environment.

[0122] 2. Motion command processing: Process motion commands for unspecified target positions and convert them into Twist messages.

[0123] 3. Safety mechanism: When an unrecognized command is received, all movements of the robot are stopped to ensure safe operation.

[0124] The specific implementation method of step 3 is as follows:

[0125] 1. Sensor data acquisition: Use sensors such as cameras and lidar to collect environmental data, including images and point cloud data.

[0126] 2. Preprocessing: Preprocess the sensor data, such as filtering, denoising and feature processing.

[0127] 3. SLAM initialization: Initialize the SLAM algorithm and determine the initial position and posture of the robot arm.

[0128] 4. Feature extraction: Extract environmental feature points from the preprocessed images and point clouds.

[0129] 5. Feature matching: Match the features of the current frame with the features in the map for positioning and map update.

[0130] 6. Posture estimation: Use matching feature points to match the current robot arm posture for positioning and map update.

[0131] 7. Map update: Update the environment map based on the pose estimation results.

[0132] 8. Object Detection: Objects are identified in the image using the object detection algorithm YOLO. YOLO (You Only Look Once) is a neural network based object recognition method. The image that is initially subject to the detection process is divided into a grid of dimensions S*S. After the image is divided, each sector is identified at the same time and if an object is detected within the frame, a bounding box B is defined. The bounding box stores the following information: x , c y --The center position of the object, w, h--the dimension of the object, and the conditional probability that the object exists in the frame. The probability is defined as:

[0133]

[0134] Among them, IOU is expressed as a value in the range of 0 to 1, which is the sum of the intersections of the predicted frame and the ground truth. If the object is not in the grid, then it gets a value of 0. Then determine the class probability of each grid:

[0135]

[0136] Only one class probability is predicted in each grid, regardless of the number of bounding boxes. Therefore, we get the output tensor S*S*(B*5+Con). The center and frame position are then corrected by the loss function (17).

[0137]

[0138] Where B is the bounding box in each grid, λcoord and λnoobj are used to enhance the emphasis on the grid. i (Con) Reference classification prediction, value In the range 0 to 1, it depends on the probability that the r object is in the jth bounding box and the i-th cell. If there is an object in the i-th cell, then 1 if yes, 0 if no.

[0139] 9. Object positioning: Based on the object detection results and point cloud data, estimate the 3D position and posture of the object.

[0140] 10. Object marking: Mark the detected object location on the map and update the object information.

[0141] 11. Feedback adjustment: Use object positioning information to adjust the map and pose estimation in the SLAM algorithm to increase the robustness of the system.

[0142] 12. Data storage: Store the updated map, location and object information in the database.

[0143] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical ideas disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A method for taking a meal, characterized in that: The following steps are involved: Step 1: Voice recognition: The customer uses a microphone to input the order information, and the voice recognition module recognizes the voice and converts it into natural language; Step 2: Natural language processing: the natural language input by the customer is processed and the language processing module converts it into machine policy instructions that can be recognized by the machine; Step 3: Detection and positioning: determine the food through the image recognition module, locate the food, and return the system positioning information; Step 4: Path planning: Based on the detection and positioning information, the central processing module plans the motion trajectory of the robot arm and finds the optimal path; Step 5: Path execution: Based on the instructions given by online planning, the robot arm automatically completes the meal picking operation; The step 2 uses the independently pre-trained LLM framework ChatGPT3.0 and the VLM framework CLIP to perform natural language processing and computer visual language processing respectively. The specific steps are: Step 2.1: Integration of LLM, the natural language input in step 1 is processed by the pre-trained LLM, and then the output is mapped to the robot's actionable commands or queries. The LLMNode function is described as a mapping from natural language input to robot actions, that is, LLMNode: Where L represents the space of natural language input, A represents the set of possible actions of the robot, and the formula is as follows: Where LM(l) is the language model for input l i ∈L, Sen(Data) represents the sensor data that informs the context command, and then REM(RobotExecutionMechanism) converts it into an executable command for the robot; Step 2.2: Integration of VLM, using OpenAI Contrastive Language Image pre-trained model to provide RGB image stream of observation source and text description of objects in the image, 2. A meal taking method according to claim 1, characterized in that: The step 3 acquires the image of the food through a monocular vision system, and then detects and locates the food. The specific steps are as follows: Step 3.1: Tracking, extract features from the image using FAST and BRIEF algorithms, with the goal of using the ORB algorithm; Step 3.2: Mapping, the location of the feature is estimated, a local map is derived using BA, and then new equivalent points of the unmatched ORB points are searched from the remaining keyframes and the new points are triangulated; Step 3.3: The loop ends, and the last step of ORB-SLAM is to compare the obtained local map with the last keyframe to optimize the duplicate points to be merged, and then ORB-SLAM continues to search for the next keyframe; Step 3.4: Point cloud clustering, aggregate neighbor points in the cloud; Step 3.5: Bounding box evaluation, assuming that the bounding box is a rectangle and then consider all the points assigned to the cluster as the data source of PCA, calculate the covariance matrix after normalization, and perform eigendecomposition on it to obtain two eigenvector value pairs in the two-dimensional case 3. A meal taking method according to claim 1, characterized in that: The step 4 uses a real-time iterative inverse kinematics solver of joint angle differential evolution to plan, select and execute a collision-free trajectory leading to the selected grasping trajectory for the robot arm, and the specific steps are as follows: Step 4.1: First, grab the path points of the trajectory (sv w ,c h ) ij , and then converted to a trajectory in the robot's full joint space: where i = 1…m j Hand joint configuration Joint configuration of the arm Where N a is the number of arm joints; Step 4.2: Measure the wrist posture sv by task space error w and K mapped by forward kinematics for : From the approximate solution Calculated pose: Among them, lin(.) extracts the R in the parameter 3 translation vector, ang(.) extracts the SO quaternion in the parameters, ||·|| is the Euclidean norm, · is the quaternion dot product, and a∈<0…1> is the coefficient that weighs the linear and angular distance.

4. A meal taking method according to claim 1, characterized in that: Step 5 will load the previously planned path, perform an initial check to ensure that the path is collision-free and kinematically feasible, and then execute the path. The specific steps are: Step 5.1: Find the current robot configuration c c The approach path point c 1r Two tracking trajectories of Then according to c c and c 1r The distance between them determines the execution trajectory: in It is the threshold for controlling the switching between global and local planning; Step 5.2: The robot performs the tracking trajectory at the end of each control cycle Try to stay at the approach waypoint of the current grab track until tracking is interrupted.

5. A meal taking system, characterized in that: A meal picking method as claimed in claim 1 is used for processing, comprising a speech recognition module and a language processing module, wherein the output end of the speech recognition module is connected to the input end of the language processing module, the output end of the language processing module and the image recognition module are connected to the input end of the central processing module, and the central processing module is connected to the robotic arm.

6. A meal taking system according to claim 5, characterized in that: The language processing module includes an LLMNode module, a CLIPNode module and a REMNode module; LLMNode: processes natural language input and generates instructions that the robot can execute; CLIPNode: provides visual and semantic understanding of the task environment; REMNode: Abstracting high-level language understanding and environment perception into actual robot actions.

Images