Kitchen service robot operation method based on VLN large model

Abstract

The application provides a kitchen service robot operation method based on a VLN large model, and relates to the technical field of intelligent robots, and comprises the following steps: acquiring environment information in real time based on a binocular camera and a depth sensor to construct a three-dimensional environment model; decomposing a user natural language instruction obtained into a subtask sequence, evaluating the influence of the subtask sequence on the environment information to obtain a decomposition evaluation result; planning a mechanical arm motion path based on the three-dimensional environment model and the decomposition evaluation result; determining a grabbing parameter based on the characteristics of an article to adjust the contact stiffness and grabbing strength of the mechanical arm; and adjusting the execution state of the mechanical arm based on feedback information obtained in real time when the mechanical arm grabs the article and in combination with the environment information.

Description

Technical Field

[0001] This application belongs to the field of intelligent robot technology, specifically relating to a method for operating a kitchen service robot based on a VLN large model. Background Technology

[0002] With the rapid development of artificial intelligence and robotics, service robots are increasingly being used in home settings, especially in complex and dynamic environments such as kitchens, where the demand for tasks such as assisting with cooking and moving items is constantly growing. However, existing kitchen service robots still face many technical bottlenecks in practical applications.

[0003] On the one hand, the kitchen environment is highly unstructured: there are many types of items, their placement is random, lighting conditions are variable, and there are frequent human interferences and dynamic obstacles. Traditional robots mostly rely on a single visual modality or a pre-set map for environmental perception, making it difficult to accurately identify items and their spatial positions under complex lighting and occlusion conditions, resulting in insufficient accuracy in environmental modeling.

[0004] On the other hand, users typically issue commands via natural language (such as "put the tomatoes on the cutting board into the microwave"), but existing systems only understand natural language at the keyword matching level, lacking the ability to deeply analyze the semantics of commands and structurally decompose tasks. Furthermore, even when tasks are decomposed, the system often ignores the impact of the current environmental state on the feasibility of subtasks, resulting in path planning that is detached from reality.

[0005] Furthermore, at the execution level, robotic arms often employ fixed-parameter strategies for grasping operations, failing to dynamically adjust contact stiffness and grasping force based on the material of the object (such as a fragile glass or a soft tomato) or its deformation characteristics. This can easily lead to the object slipping or being damaged. Even when some systems introduce force feedback, they lack a closed-loop mechanism to integrate real-time grasping feedback with environmental information to dynamically adjust the execution state, resulting in poor operational robustness and low success rate.

[0006] In summary, existing technologies are fragmented in the four key aspects of environmental perception, task understanding, path planning, and adaptive execution, making it difficult to achieve high-precision and high-reliability end-to-end operations in kitchen scenarios. Summary of the Invention

[0007] This application provides a method for operating a kitchen service robot based on a VLN large model to solve one of the aforementioned technical problems.

[0008] The technical solution adopted in this application is as follows:

[0009] This application provides a method for operating a kitchen service robot based on a VLN large model, including:

[0010] Real-time environmental information is acquired using binocular cameras and depth sensors to construct a 3D environment model;

[0011] The acquired user natural language instructions are decomposed into a sequence of subtasks, and the impact on the subtask sequences is evaluated based on environmental information to obtain the decomposition and evaluation results.

[0012] Based on the 3D environment model and decomposition evaluation results, the motion path of the robotic arm is planned;

[0013] The gripping parameters are determined based on the characteristics of the object in order to adjust the contact stiffness and gripping force of the robotic arm.

[0014] Based on real-time feedback information obtained from the robotic arm's grasping of objects, combined with environmental information, the robotic arm's execution state is adjusted.

[0015] According to one embodiment of this application, the step of acquiring environmental information in real time based on a binocular camera and a depth sensor to construct a three-dimensional environment model specifically includes:

[0016] Based on a binocular camera and a depth sensor, the kitchen environment is scanned to identify the initial position and basic state of items in the kitchen in order to obtain environmental information, including visual images and three-dimensional spatial data of the current environment. A three-dimensional environment model is then constructed based on the environmental information.

[0017] According to one embodiment of this application, the process of decomposing the acquired user natural language instructions into a sequence of sub-tasks and evaluating the impact on the sub-task sequences based on environmental information to obtain a decomposition evaluation result is as follows:

[0018] Receive and parse user natural language commands, and identify key elements in the user natural language commands, including: the object of operation, the location, and the target location;

[0019] Based on key elements, understand the semantics and operational intent of the user's natural language commands, decompose the user's natural language commands into a sequence of sub-tasks, and determine the specific operation steps and parameters of each sub-task sequence;

[0020] Based on environmental information, the impact of environmental changes on the execution of the subtask sequence is assessed to determine the feasibility of executing each subtask in the current environment.

[0021] According to one embodiment of this application, the step of planning the robotic arm motion path based on the three-dimensional environment model and decomposition evaluation results specifically includes:

[0022] Based on the environmental model and task decomposition results, the global path of the robotic arm end effector from the current position to the target position is determined;

[0023] An improved A* search algorithm or a conflict-based multi-agent path planning algorithm is used to ensure that the path is conflict-free and meets the basic constraints.

[0024] Convert global path points into joint angle values, and use inverse kinematics to calculate the joint angle corresponding to each path point;

[0025] A 3-5-3 piecewise polynomial interpolation method is used to smooth the path points in the joint space to ensure the continuity of position, velocity, and acceleration.

[0026] Optimize the interpolation time of each segment to ensure the optimal running time of the robotic arm while satisfying the speed and acceleration constraints.

[0027] Based on obstacle information in the 3D environment model, the planned path is adjusted for obstacle avoidance.

[0028] The Cartesian repulsion velocity of the robotic arm is calculated using a differential kinematics method based on the Jacobian matrix.

[0029] Based on the global path, obtain the initial local path for each joint on the robotic arm;

[0030] Using an improved particle swarm optimization algorithm, the time required between adjacent path nodes in the initial local path corresponding to each joint is optimized to obtain the final local path.

[0031] According to one embodiment of this application, the step of determining the gripping parameters based on the characteristics of the object to adjust the contact stiffness and gripping force of the robotic arm specifically includes:

[0032] By combining the object classification information in the 3D environment model, the material type of the object is identified, and the structural characteristics of the object are judged through visual images;

[0033] For items made of different materials, a preset database of gripping parameters is established, including standard contact stiffness values ​​and recommended gripping force ranges.

[0034] During the grasping process, the deformation and surface condition of the object are monitored in real time, and the contact stiffness and grasping force are dynamically adjusted based on the real-time feedback from the object.

[0035] According to one embodiment of this application, the step of adjusting the execution state of the robotic arm based on real-time feedback information obtained from the robotic arm grasping the object, combined with environmental information, specifically involves:

[0036] According to the planned path, the robotic arm is moved to the gripping point of the object, and the attitude of the end effector is adjusted to maintain the best contact angle with the surface of the object.

[0037] Set the initial gripping parameters, including contact stiffness and gripping force;

[0038] Approaching the object along a planned grasping path, gradually increasing the grasping force, while monitoring the deformation and surface condition of the object, and sensing the force feedback during the grasping process in real time through force sensors;

[0039] Continuously monitor the joint torque of the robotic arm, the contact force of the end effector, and the deformation of the object;

[0040] Observe the changes in the state of the item in real time during the grasping process;

[0041] Compare the actual capture parameters with the preset parameters to evaluate the capture effect;

[0042] When abnormal deformation of the object is detected or excessive gripping force is detected, reduce the gripping force;

[0043] When a smooth surface of an object is detected, causing unstable gripping, the contact stiffness is increased;

[0044] The robot arm's movement speed and trajectory are dynamically adjusted based on real-time feedback from the object.

[0045] According to one embodiment of this application, the step of scanning the kitchen environment and identifying the initial positions and basic states of items in the kitchen based on a binocular camera and a depth sensor to obtain environmental information further includes:

[0046] Based on ambient lighting conditions, the processing parameters of visual images are automatically adjusted using an adaptive lighting compensation algorithm.

[0047] According to one embodiment of this application, it also includes:

[0048] Once the item is grasped, the grasping status is confirmed to verify that the item has been correctly grasped. After verification, the robotic arm performs subsequent operations according to the planned movement path.

[0049] During the grasping process, data on the deformation of the object, contact force, and the status of the robotic arm joints are collected.

[0050] Record key indicators such as the success rate and item damage for each grabbing operation;

[0051] Record the parameters of each successful capture into the item characteristic parameter library;

[0052] Optimize and update preset parameters based on the actual grasping effect of objects of different materials and shapes;

[0053] Based on historical crawling data, the crawling parameter library is continuously optimized;

[0054] When a crawl fails, analyze the reason for the failure, adjust the subsequent crawling parameters based on the reason, and re-execute the crawling operation.

[0055] A second aspect of this application provides a computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the steps described in the method.

[0056] A third aspect of this application provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method as described.

[0057] Due to the adoption of the above technical solution, the beneficial effects achieved by this application are as follows:

[0058] This application uses a combination of binocular cameras and depth sensors to obtain environmental information in real time and build a 3D environmental model. This effectively overcomes problems such as uneven lighting and severe occlusion in the kitchen environment, providing an accurate spatial semantic basis for subsequent tasks and significantly improving the accuracy of object recognition and positioning.

[0059] This application decomposes user natural language commands into a sequence of subtasks, and then uses a 3D environment model to evaluate the impact of the environment on each subtask (such as whether the target item is occluded or the path is blocked), generating a "decomposition evaluation result" to avoid executing infeasible tasks and improve the rationality and security of system decisions.

[0060] This application plans the robotic arm's motion path based on a three-dimensional environment model and decomposition evaluation results, ensuring that the path not only meets the task logic requirements but also effectively avoids static and dynamic obstacles, significantly improving the practicality of path planning and the success rate of execution.

[0061] This application dynamically determines the gripping parameters based on the characteristics of the item (such as material and deformability), and adjusts the contact stiffness and gripping force of the robotic arm accordingly to avoid damage to fragile or soft items, while ensuring gripping stability and significantly reducing the item damage rate and slippage rate.

[0062] During the grasping process, this application dynamically adjusts the execution state of the robotic arm (such as speed, force, and posture) based on real-time feedback information (such as contact force and object deformation) and environmental information, forming a closed-loop control of "perception-decision-execution-feedback-readjustment", which significantly improves the operational robustness and task completion rate in complex kitchen environments. Attached Figure Description

[0063] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0064] Figure 1A flowchart illustrating a method for operating a kitchen service robot based on a VLN large model, provided in an embodiment of this application;

[0065] Figure 2 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0066] Figure label:

[0067] 810, Processor; 820, Communication interface; 830, Memory; 840, Communication bus. Detailed Implementation

[0068] To more clearly illustrate the overall concept of this application, a detailed explanation is provided below with reference to the accompanying drawings.

[0069] Many specific details are set forth in the following description to provide a thorough understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below. It should be noted that, unless otherwise specified, the embodiments of this application and the features thereof can be combined with each other.

[0070] In this application, unless otherwise expressly specified and limited, the "above" or "below" of the second feature can mean that the first and second features are in direct contact, or that the first and second features are in indirect contact through an intermediate medium. In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples.

[0071] Example 1

[0072] like Figure 1 As shown, a method for operating a kitchen service robot based on a VLN large model includes:

[0073] The S100 uses a binocular camera and a depth sensor to acquire environmental information in real time in order to build a three-dimensional environment model.

[0074] As mentioned above, binocular cameras calculate the depth information of objects in a scene by measuring the image differences (parallax) between the left and right perspectives, and have good texture recognition capabilities and long-distance perception advantages;

[0075] Depth sensors (such as structured light, time-of-flight ToF, or active infrared sensors) directly output distance data for each point in the scene, providing high-precision short-range ranging capabilities, especially performing well in low-light or textureless areas.

[0076] The two sets of data are aligned in time and space and fused with features to complement each other’s limitations in terms of illumination sensitivity, ranging accuracy, and effective distance, generating a dense 3D point cloud that includes geometric structure, object boundaries, surface normals and spatial occupancy status.

[0077] Based on this point cloud data, the system further performs semantic segmentation, object clustering and spatial relationship modeling, and finally constructs a structured three-dimensional environment model that includes object categories, positions, poses and operable areas.

[0078] This model not only reflects static environments (such as cabinets and stoves), but also dynamically updates the positional changes of moving objects (such as user movement and temporarily placed food), providing a real-time and accurate spatial semantic foundation for subsequent task understanding and path planning.

[0079] For example, in a kitchen scenario, the user instruction is "Put the apples on the table into the refrigerator." In this case:

[0080] The dual-camera system detected multiple objects (apple, cup, tissue) on the tabletop, but due to the dim lighting, the textures in some areas were blurry.

[0081] The depth sensor clearly detected a raised, circular object on the table, whose height and diameter matched the typical size of an apple.

[0082] The system fuses the color and texture features in the binocular images with the geometric shape in the depth data to confirm that the object is an "apple" and accurately calculates its three-dimensional position in the world coordinate system (e.g., x=0.85m, y=0.32m, z=0.78m).

[0083] At the same time, the system recognizes that the refrigerator door is closed, and the position of its handle and the operating space required to open the door are also modeled into the environment model;

[0084] The final 3D environment model not only includes the spatial coordinates of the apple, but also labels it as a "grabable object" and associates it with its material properties (such as "deformable fruit"), providing a basis for setting subsequent grasping parameters.

[0085] It should be noted that, in specific implementation scenarios, the binocular camera can be replaced with a multi-view vision system (such as tri-view or surround view) based on the above solution, and the depth sensor can be a device with different principles (such as structured light, ToF, or lidar). As long as the combination can achieve real-time acquisition and fusion of three-dimensional environmental information, it is an equivalent implementation of this solution.

[0086] In specific implementation scenarios, based on the above solutions, the 3D environment model can be expressed in the form of point cloud, voxel grid, triangular mesh, or semantic map. As long as it can carry the spatial location, category, and operability information of the object, it falls within the protection scope of this solution.

[0087] In specific implementation scenarios, based on the above solutions, the environment model can also be updated incrementally at a fixed frequency (e.g., 5 frames per second) or by event triggering (e.g., detecting object movement) to ensure that the model always reflects the current real environment state.

[0088] In specific implementation scenarios, in addition to binocular and depth data, the above solutions can also be combined with inertial measurement unit (IMU) data to compensate for the perception bias caused by the robot's own movement, or historical task data can be introduced to assist in understanding the current environment (such as "this location is usually where a chopping board is placed").

[0089] In specific implementation scenarios, based on the above solutions, during the construction of the 3D model, a pre-trained object recognition model or kitchen knowledge graph can be combined to assign high-level semantic labels (such as "ingredients", "tableware", "dangerous areas") to the objects in the model, thereby supporting subsequent task decomposition and safety assessment.

[0090] S200. Decompose the acquired user natural language instructions into a sequence of subtasks, and evaluate the impact on the subtask sequences based on environmental information to obtain the decomposition evaluation results.

[0091] As mentioned above, instruction parsing first identifies key semantic elements in the instruction through the natural language understanding module, including operation verbs (such as "take", "put", "cut"), operation objects (such as "tomato"), source location (such as "on the cutting board"), and target location (such as "inside the microwave oven").

[0092] Subtask decomposition: Based on preset task templates or knowledge rules, high-level instructions are broken down into several atomic-level subtasks. Typical subtasks include "locating the target item", "navigating to the vicinity of the item", "grabbing the item", "moving to the target location", and "placing the item".

[0093] Environmental Impact Assessment: This involves matching each subtask with the currently constructed 3D environment model to assess the impact of environmental factors on the execution of the subtask. For example, it determines whether the target object is visible, whether it is obscured, whether the target area is accessible, whether there are dynamic obstacles along the path, and whether the target container is in an operable state (e.g., whether the microwave oven door is closed).

[0094] Generate decomposition evaluation results: Based on the above evaluation, mark the execution status (such as "executable", "wait", "infeasible"), priority, dependencies and potential risk warnings for each subtask to form a structured "decomposition evaluation result", which serves as the basis for subsequent path planning and execution strategy formulation.

[0095] This process bridges the gap between fuzzy semantic instructions and environment-aware task planning, significantly improving the robot's ability to understand complex instructions and its adaptability to task execution.

[0096] For example, a user might give the instruction: "Take the milk out of the refrigerator and put it on the table."

[0097] Instruction Analysis: The system identifies the object of the operation as "milk", the source location as "inside the refrigerator", the target location as "table", and the action as "take out and place".

[0098] Subtask decomposition: The system decomposes it into the following sequence of subtasks:

[0099] Navigate to the refrigerator;

[0100] Open the refrigerator door;

[0101] Position the milk inside the refrigerator;

[0102] Grab the milk;

[0103] Close the refrigerator door;

[0104] Move to the dining table;

[0105] Place the milk on the table;

[0106] Environmental Impact Assessment:

[0107] The 3D environment model shows that the refrigerator door is currently closed, but there are no obstacles in front of it, so subtasks 1 and 2 can be executed.

[0108] Point cloud data inside the refrigerator shows that the milk is located on the second shelf and is not obstructed by other items, making subtasks 3 and 4 feasible.

[0109] There are currently users dining in the dining area, creating a temporary obstacle. Subtasks 6 and 7 need to wait or have their placement adjusted.

[0110] Decompose the evaluation results: The system generates evaluation results, marking subtasks 1-5 as "can be executed immediately" and subtasks 6-7 as "conditions are limited, and we need to wait for the table area to be cleared or negotiate the placement point", and suggests prioritizing the completion of the first five steps, while continuously monitoring the status of the table.

[0111] The assessment results avoided collisions or interference caused by the robot blindly performing all tasks, demonstrating a deep synergy between the task and the environment.

[0112] It should be noted that, in specific implementation scenarios, in addition to the above solutions, natural language commands are not limited to voice input, but can also come from text input (such as mobile app commands, touch screen input) or cross-modal fusion commands (such as "point to an item and say 'take this'"), as long as they are ultimately converted into semantically clear operational intentions, they fall within the scope of this solution.

[0113] In specific implementation scenarios, based on the above solutions, subtask decomposition can be achieved based on a predefined rule base, task knowledge graph, or learning-based task parsing model. Different kitchen operations (such as "chopping vegetables", "pouring water", and "heating") correspond to different subtask combination templates.

[0114] In specific implementation scenarios, in addition to spatial accessibility and obstruction judgment, the environmental impact assessment can also include multiple factors such as time constraints (e.g., "The microwave oven is running and needs to be stopped"), safety constraints (e.g., "A child is detected approaching, so operation is suspended"), and item status (e.g., "The milk carton has been opened and needs to be tilted carefully"). As long as the feasibility of the sub-task is judged based on environmental information, it falls within the protection scope of this scheme.

[0115] In specific implementation scenarios, the evaluation results can be further decomposed based on the above scheme and presented as a task dependency graph, a weighted execution queue, a feasibility score with confidence level, or a task list with risk labels. Different data structure forms do not affect their essential function as input for path planning.

[0116] In specific implementation scenarios, based on the above solutions, when there is uncertainty in the environmental assessment results (such as not being able to confirm whether an item is the target object), the system can proactively initiate a clarification request (such as "Do you mean red tomatoes or yellow tomatoes?") and incorporate user feedback into the assessment result update.

[0117] S300, based on the three-dimensional environment model and decomposition evaluation results, plans the motion path of the robotic arm.

[0118] As mentioned above, the input fusion: the path planning module receives two core inputs simultaneously: one is a 3D environment model constructed by a binocular camera and a depth sensor, which includes geometric and semantic information such as obstacle distribution, passable areas, and precise poses of objects; the other is the decomposition and evaluation results generated by natural language instruction decomposition and environmental assessment, which includes sub-task target points, execution order, feasibility status, and potential risk warnings.

[0119] Path generation logic: The planning process first determines the starting pose and target pose of the robotic arm end effector based on the sub-task objective (such as "grabbing tomatoes"); then, it identifies all static and dynamic obstacles (such as the edge of the stove, temporarily placed dishes, and moving human bodies) in the 3D environment model; next, it combines the information on task priority, execution conditions, and safety constraints in the decomposition evaluation results to select candidate paths that meet the task logic and are collision-free.

[0120] Path optimization mechanism: Under the premise of meeting the task objectives and obstacle avoidance, the smoothness, movement time, energy consumption, and operational stability of the path are further optimized. For example, for sub-tasks requiring delicate operation (such as placing fragile items), the path will prioritize low-speed, stable trajectories; for movement in obstacle-free areas, more efficient straight or curved paths can be used.

[0121] Output format: The final output of the robotic arm motion path is a continuous trajectory defined in joint space or Cartesian space, which includes the pose, velocity, acceleration and time parameters of each key point, and can be directly used for subsequent motion control module execution.

[0122] This step achieves the crucial transformation from "task intent" to "physical action" and is the core link connecting high-level semantic understanding and low-level execution control.

[0123] For example, if the user command is "put the spatula on the stove into the drawer," the system has completed command decomposition and environmental assessment, obtaining the following information:

[0124] The 3D environment model shows that the spatula is located on the right side of the stove with its handle facing outwards and there are no obstructions around it; the target drawer is located in the cabinet below and is currently closed, but there is ample space in front of it; there are hanging kitchen utensils above the stove, which constitute an obstacle above.

[0125] The breakdown evaluation results include a sequence of subtasks: "1. Grab the spatula; 2. Open the drawer; 3. Place the spatula in; 4. Close the drawer", where subtask 2 is marked as "Confirm that the drawer handle is accessible first".

[0126] Based on the above information, the path planning process is as follows:

[0127] For subtask 1 (grabbing the spatula): plan a path from the current position of the robotic arm around the edge of the stove, avoid the overhead hanging objects, and accurately approach the spatula handle, and adjust the end effector posture to a suitable angle for gripping the handle;

[0128] For subtask 2 (opening the drawer): After the grabbing is completed, plan the robotic arm to move to the front of the drawer with the spatula, and then release the spatula (or temporarily store it). Plan a separate path to the drawer handle to ensure that the fingers do not collide with the cabinet.

[0129] For subtask 3 (putting in the spatula): After the drawer is opened, grab the spatula again and plan a path to drop it vertically from above, avoiding the edge of the drawer, and place it inside smoothly;

[0130] All paths avoid known obstacles and employ trajectory parameters with moderate speed and appropriate stiffness based on assessment information such as "the spatula is a hard metal object" and "the internal space of the drawer is limited".

[0131] The final generated path is not only reachable and collision-free, but also takes into account both operational efficiency and safety.

[0132] It should be noted that, in specific implementation scenarios, the path can be planned in Cartesian space (end-effector pose space) or joint space (angle space of each joint) based on the above scheme, or a hybrid space strategy can be adopted. As long as it generates a feasible trajectory based on the three-dimensional environment model and decomposition evaluation results, it belongs to the equivalent implementation of this scheme.

[0133] In specific implementation scenarios, based on the above solutions, path planning can also support real-time re-planning. When a new dynamic obstacle is detected (such as a user suddenly reaching into the work area), the system can adjust the path in real time based on the updated environment model.

[0134] In specific implementation scenarios, based on the above solutions, when there are multiple parallel or sequential subtasks (such as needing to grab two items at the same time), a task scheduling and path collaborative optimization strategy can be adopted to generate a composite path that takes into account both temporal and spatial efficiency.

[0135] In specific implementation scenarios, based on the above solutions, path planning can also integrate safety strategies such as safe distance constraints, speed-limited areas (such as automatic speed reduction when approaching a human body), and emergency stop triggering areas. These safe path generation methods based on environmental semantics (such as "human body area" and "high temperature area") are part of this solution.

[0136] In specific implementation scenarios, based on the above solutions, the path generation process can be adapted and optimized by combining the kinematic model of the robotic arm (such as joint constraints, singularity avoidance, and maximum speed / acceleration limits) to ensure that the output path is physically executable.

[0137] In specific implementation scenarios, in addition to the above solutions, collision detection, reachability verification, and time rationality verification can be performed before outputting the final path. If it is found to be infeasible, the order of subtasks can be adjusted or environmental intervention can be requested (such as "Please move the bowl blocking the drawer").

[0138] S400: Determine gripping parameters based on the characteristics of the object to adjust the contact stiffness and gripping force of the robotic arm.

[0139] As mentioned above, the acquisition of item characteristics includes, but is not limited to, material (such as glass, plastic, metal, fruits and vegetables), shape (regular / irregular), surface condition (smooth / rough), structural strength (rigid / deformable), weight, and center of gravity distribution. These characteristics can be derived from semantic recognition results in 3D environment models (such as "tomato" and "glass"), point cloud geometric analysis (such as curvature and volume estimation), historical task databases, or user-preset tags.

[0140] Grasping parameters are defined as follows: These parameters mainly include contact stiffness (i.e., the compliance of the robotic arm's end effector when in contact with the object, reflecting its sensitivity to external disturbances) and gripping force (i.e., the magnitude of the clamping force applied by the actuator). Together, they determine the "hardness" and "grip" of the gripping process.

[0141] Parameter mapping logic: The system has a built-in or dynamically constructed "item characteristics - grasping parameters" mapping rule library. For example, for fragile or easily deformable items (such as eggs or cooked tomatoes), a lower contact stiffness (high flexibility) and a smaller grasping force are set; for hard, rough-surfaced items (such as metal spatulas or ceramic bowls), a higher stiffness and a larger grasping force can be used to ensure stability.

[0142] Parameter output and execution: The determined grasping parameters are transmitted to the force control or impedance control module of the robotic arm, which adjusts the force output and compliance behavior of the actuator in real time during the grasping action to achieve adaptive grasping based on the object.

[0143] This step enables an intelligent leap from "seeing the item" to "knowing how to take it," and is a key technology for improving the operational safety and versatility of kitchen service robots.

[0144] For example, the user instruction might be "Put the cooked tomatoes from the plate into the food storage container".

[0145] Item characteristic recognition: The system identifies the target item as "ripe tomato" through a 3D environment model. Combined with its reddish color, weak surface reflection, and slight indentation in the point cloud, it determines that it is a food item that is "easily deformable, has low structural strength, and has a moist surface".

[0146] Fetching parameters are determined: The system automatically sets them based on built-in rules.

[0147] Contact stiffness: Set to a low level to give the robotic arm high compliance when contacting the tomato, so that it can adapt to the small deformations of its surface without applying abrupt impact.

[0148] Grasping force: Set to a gentle level (such as 30% of the normal force), which only needs to overcome gravity to avoid squeezing and causing juice to spill or break.

[0149] Results: The robotic arm slowly approaches the tomato, then gently follows its contours upon contact, applying minimal gripping force to complete the grasping process without slipping or causing damage.

[0150] For example, if the target item is a "stainless steel soup spoon":

[0151] The system identifies it as metallic, rigid, with a smooth but textured surface;

[0152] Set a high contact stiffness (to ensure stable gripping posture) and a medium-high gripping force (to prevent slippage due to smooth surface).

[0153] The grasping process is fast and stable, with no risk of slipping.

[0154] It should be noted that, in specific implementation scenarios, the characteristics of items along the path can not only come from visual and depth perception, but also integrate tactile feedback (such as deformation response after the first light touch), weight estimation (derived from torque), user historical operation preferences (such as "user prefers to hold tomatoes lightly") and other multi-source information.

[0155] In specific implementation scenarios, in addition to contact stiffness and gripping force, gripping parameters can also include gripping speed, gripper closing rate, gripping posture angle, pre-gripping gap, etc., based on the above scheme. As long as they are used to adjust the interaction behavior between the robotic arm and the object and are set based on the characteristics of the object, they fall within the protection scope of this invention.

[0156] In specific implementation scenarios, the mapping rules can be based on expert experience rule bases, machine learning models (such as classifier output parameter ranges), or online optimization strategies (such as feedback adjustment after trial capture) on the basis of the above scheme. Different implementation methods are all equivalent technical means of this scheme.

[0157] In specific implementation scenarios, this method, based on the above scheme, is applicable to various end effectors, including two-finger grippers, multi-finger dexterous hands, vacuum suction cups, magnetic clamps, etc. The specific physical meaning of the grasping parameters may differ for different actuators (e.g., negative pressure intensity for suction cups), but the core logic of "setting interaction parameters based on object characteristics" remains unchanged and falls within the scope of this invention.

[0158] In specific implementation scenarios, based on the above scheme, the system can set safe upper and lower limits for the grasping parameters for each type of item, and monitor whether the threshold is exceeded during execution (such as automatically releasing the grip if abnormal deformation is detected).

[0159] In specific implementation scenarios, based on the above solutions, if the same item requires different grasping methods in different tasks (such as "picking up the tomato" vs. "fixing the tomato before cutting it"), the system can fine-tune the parameters according to the sub-task type.

[0160] The S500 adjusts the robotic arm's execution state based on real-time feedback information obtained from the robotic arm's grasping of objects, combined with environmental information.

[0161] As mentioned above, the sources of feedback information include internal state data such as the joint torque sensor of the robotic arm itself, the contact force / pressure sensor of the end effector, the opening and closing state of the gripper, and changes in motor current, as well as visual and spatial information continuously updated by external sensing systems (such as binocular cameras and depth sensors) during operation (such as whether the object has shifted, deformed, or slipped).

[0162] Environmental information fusion: The above feedback information is compared and correlated with the constructed 3D environment model. For example, if the vision detects a slight shift in the object after grasping it, the environmental model is used to determine whether the shift has caused the object to be too close to surrounding obstacles; if the force sensor shows a sudden drop in gripping force, the material of the object is used to determine whether it is a sign of impending slippage.

[0163] Execution state adjustment: Based on the fusion analysis results, the system can adjust the execution state of the robotic arm in multiple dimensions, including but not limited to:

[0164] Adjust the speed or acceleration of movement (e.g., decelerate when passing through a narrow area);

[0165] Correct the end-effector pose (e.g., fine-tune the gripping angle to compensate for item offset);

[0166] Dynamically reset the gripping force or contact stiffness (e.g., immediately reduce the force if deformation of the object is detected).

[0167] Trigger task rollback or replanning (if it is confirmed that the item has been dropped, the current subtask will be aborted and the pickup recovery process will be started).

[0168] This step represents a leap from "open-loop execution" to "perception-decision-adjustment" closed-loop control, significantly improving the robot's operational robustness and fault tolerance in uncertain kitchen environments.

[0169] For example, a user instruction might be "Take the glass from the sink to the sterilizer".

[0170] Grasping in progress: The robotic arm has successfully grasped the glass and begun moving it toward the sterilizer;

[0171] Real-time feedback: The force sensor detected a slight fluctuation in the clamping force, while the depth camera detected a slight tilt of the cup.

[0172] Environmental information integration: The system calls the 3D environment model to confirm that there is a towel hanging above the current path. If the cup continues to tilt, it may collide.

[0173] Execution status adjustment:

[0174] Immediately reduce the robotic arm's movement speed;

[0175] Fine-tune the wrist joint angle to restore the cup to a vertical position;

[0176] Slightly increase grip strength (still within safe limits) to enhance grip stability;

[0177] At the same time, slightly raise the overall path to avoid areas with hanging objects.

[0178] For example, when grabbing a piece of ripe tomato:

[0179] The initial gripping force is set to the preset gentle parameter;

[0180] However, force feedback showed that the deformation increased rapidly after contact, and visual detection also showed that surface sap seeped out.

[0181] The system determined that the current force was still too strong, so it immediately reduced the gripping force and paused the movement, waiting for the deformation to stabilize before slowly resuming the task.

[0182] All of the above adjustments were completed within milliseconds, preventing item damage or mission failure.

[0183] It should be noted that, in specific implementation scenarios, in addition to force, posture, and visual feedback, the above solutions can also include multimodal sensing information such as temperature sensors (e.g., detecting a hot pot), vibration signals (e.g., determining whether an object is loose), and sound feedback (e.g., the sound of glass colliding).

[0184] In specific implementation scenarios, in addition to the above solutions, the execution status adjustment is not limited to intensity and speed, but can also include high-level strategy adjustments such as task priority reordering (e.g., pausing the current task to handle sudden obstacles), subtask skipping (e.g., the target item is no longer in its original position), and operation mode switching (e.g., switching from "grab" to "push").

[0185] In specific implementation scenarios, the above scheme can be adjusted to be triggered by threshold (e.g., force exceeding a certain value), trend prediction (e.g., deformation acceleration), or fuzzy judgment based on confidence (e.g., "may slip"). Different triggering logics are all equivalent implementations of this scheme.

[0186] In specific implementation scenarios, based on the above solutions, when the system cannot automatically adjust to restore a stable state (such as when an item has slipped significantly), it can proactively pause execution and request user confirmation or assistance (such as a voice prompt saying "The cup is a bit slippery, please hold it steady").

[0187] In specific implementation scenarios, based on the above solutions, the system can also record the state and results before and after each adjustment, which can be used to optimize future parameter settings or adjustment strategies (such as "a certain type of tomato is more prone to deformation when the humidity is high, and it needs to be picked up more gently next time").

[0188] In specific implementation scenarios, based on the above solutions, if the robot is equipped with multiple execution units (such as dual arms, mobile chassis + robotic arm), adjustments can be made to involve the coordinated actions of multiple actuators (such as slight chassis movement to improve the grasping angle).

[0189] According to one embodiment of this application, the step of acquiring environmental information in real time based on a binocular camera and a depth sensor to construct a three-dimensional environment model specifically includes:

[0190] Based on a binocular camera and a depth sensor, the kitchen environment is scanned to identify the initial position and basic state of items in the kitchen in order to obtain environmental information, including visual images and three-dimensional spatial data of the current environment. A three-dimensional environment model is then constructed based on the environmental information.

[0191] As described above, the kitchen environment is scanned synchronously using a binocular camera and a depth sensor to acquire visual images and 3D spatial data of the current scene. The binocular camera captures color images from both left and right perspectives to extract textures, colors, and surface features of objects in the environment, while the depth sensor directly measures the distance from each point in the scene to the sensor, generating corresponding depth maps or point cloud data. The two types of data are aligned and fused in time and space to form dense environmental perception information containing geometric structure and visual semantics. Based on this, the initial position, orientation, visibility, and basic state (such as whether containers are open, whether items are stacked, and whether surfaces are wet) of various items in the kitchen are identified. Finally, based on the fused environmental information, a 3D environmental model is constructed that can represent object categories, spatial coordinates, size contours, and interrelationships. This model can be updated in real time to support subsequent task understanding, path planning, and adaptive execution.

[0192] According to one embodiment of this application, the process of decomposing the acquired user natural language instructions into a sequence of sub-tasks and evaluating the impact on the sub-task sequences based on environmental information to obtain a decomposition evaluation result is as follows:

[0193] Receive and parse user natural language commands, and identify key elements in the user natural language commands, including: the object of operation, the location, and the target location;

[0194] Based on key elements, understand the semantics and operational intent of the user's natural language commands, decompose the user's natural language commands into a sequence of sub-tasks, and determine the specific operation steps and parameters of each sub-task sequence;

[0195] Based on environmental information, the impact of environmental changes on the execution of the subtask sequence is assessed to determine the feasibility of executing each subtask in the current environment.

[0196] As described above, the system first receives natural language instructions input by the user and performs semantic parsing to identify key elements, including the object to be manipulated (e.g., "tomato" or "cup"), the object's current location (e.g., "on the cutting board" or "in the refrigerator"), and the target location (e.g., "in the microwave oven" or "on the dining table"). Based on these key elements, and combined with pre-defined task understanding rules or a knowledge base, the system accurately understands the user's operational intent, such as "move something from point A to point B" or "perform a specific operation on something." Subsequently, the high-level instruction is decomposed into a series of ordered, robot-executable sub-task sequences according to operational logic. Each sub-task corresponds to a specific action unit, such as "navigate to the vicinity of the item." The process involves tasks such as "opening the container," "grabbing the item," "moving to the target location," and "placing the item," with each subtask having its required operational parameters defined, including target coordinates, execution order, dependencies, and expected state. Next, using the environmental information provided by the constructed 3D environment model, the execution conditions of each subtask in the current environment are evaluated, determining factors such as whether there are obstacles obstructing the view, whether the target is visible, whether the operating space is sufficient, and whether relevant equipment is operational. Finally, based on the evaluation results, the feasibility of each subtask in the current environment is determined, generating a decomposed evaluation result including feasibility status, risk warnings, and execution constraints, providing a basis for subsequent path planning and execution control.

[0197] According to one embodiment of this application, the step of planning the robotic arm motion path based on the three-dimensional environment model and decomposition evaluation results specifically includes:

[0198] Based on the environmental model and task decomposition results, the global path of the robotic arm end effector from the current position to the target position is determined;

[0199] An improved A* search algorithm or a conflict-based multi-agent path planning algorithm is used to ensure that the path is conflict-free and meets the basic constraints.

[0200] Convert global path points into joint angle values, and use inverse kinematics to calculate the joint angle corresponding to each path point;

[0201] A 3-5-3 piecewise polynomial interpolation method is used to smooth the path points in the joint space to ensure the continuity of position, velocity, and acceleration.

[0202] Optimize the interpolation time of each segment to ensure the optimal running time of the robotic arm while satisfying the speed and acceleration constraints.

[0203] Based on obstacle information in the 3D environment model, the planned path is adjusted for obstacle avoidance.

[0204] The Cartesian repulsion velocity of the robotic arm is calculated using a differential kinematics method based on the Jacobian matrix.

[0205] Based on the global path, obtain the initial local path for each joint on the robotic arm;

[0206] Using an improved particle swarm optimization algorithm, the time required between adjacent path nodes in the initial local path corresponding to each joint is optimized to obtain the final local path.

[0207] As described above, firstly, based on the constructed 3D environment model and the aforementioned decomposition and evaluation results, the sub-task objectives that the robotic arm's end effector needs to complete are clarified, including the initial pose and the target pose, thereby determining the global motion path from the current position to the target position; during the path search process, an improved A* search algorithm is adopted, or in multi-robotic arm collaborative scenarios, a conflict-based search algorithm is used. A multi-agent path planning algorithm (CBS) is used to generate a feasible path that satisfies the task logic, is collision-free, and meets basic motion constraints (such as workspace limitations and obstacle avoidance requirements). Subsequently, the Cartesian space path points in this global path are converted into angle values ​​corresponding to each joint of the robotic arm using inverse kinematics, resulting in a discrete path point sequence in joint space. To ensure the smoothness of the motion process, a 3-5-3 piecewise polynomial interpolation method is used for trajectory planning of the joint space path points, ensuring the continuity of position, velocity, and acceleration of each joint during motion and avoiding impacts or vibrations. Based on this, the time parameters of each interpolation segment are optimized to minimize the overall running time and improve performance while meeting the physical constraints of the robotic arm's maximum velocity and maximum acceleration. To improve efficiency, the system dynamically adjusts the initially planned path based on real-time obstacle information updated in the 3D environment model, ensuring safety in complex kitchen environments. Specific obstacle avoidance strategies include: using a Jacobian matrix-based differential kinematics method to calculate the Cartesian repulsion velocity required for the robotic arm's end effector or link to approach an obstacle, and mapping it back to joint space to correct the motion direction; furthermore, for each joint, the system extracts its corresponding initial local path from the global path and employs an improved particle swarm optimization (PSO) algorithm to finely adjust the running time between adjacent path nodes, further optimizing the smoothness and time efficiency of the local trajectory, ultimately generating a complete robotic arm motion path that meets task requirements, environmental constraints, and dynamic limitations.

[0208] According to one embodiment of this application, the step of determining the gripping parameters based on the characteristics of the object to adjust the contact stiffness and gripping force of the robotic arm specifically includes:

[0209] By combining the object classification information in the 3D environment model, the material type of the object is identified, and the structural characteristics of the object are judged through visual images;

[0210] For items made of different materials, a preset database of gripping parameters is established, including standard contact stiffness values ​​and recommended gripping force ranges.

[0211] During the grasping process, the deformation and surface condition of the object are monitored in real time, and the contact stiffness and grasping force are dynamically adjusted based on the real-time feedback from the object.

[0212] As described above, firstly, the category of the target item is determined by combining the identified item classification information in the 3D environment model. Then, its surface texture, color, reflectivity, and geometry are analyzed through visual imagery to determine the item's material type (e.g., glass, plastic, metal, fruits and vegetables) and structural characteristics (e.g., whether it is easily deformable, whether the surface is smooth, and its overall rigidity or softness). Based on the above identification results, gripping parameters matching the item type are retrieved from a pre-set gripping parameter library. This parameter library sets standard contact stiffness values ​​and recommended gripping force ranges for items with different materials and structural characteristics. For example, for fragile items... For easily deformable items, a lower contact stiffness and a smaller gripping force are set, while for hard or rough-surfaced items, a higher contact stiffness and a moderate gripping force are set. During the actual gripping action of the robotic arm, the real-time deformation, displacement state, and surface contact response of the item are continuously monitored through force sensors, visual feedback, or depth changes. Once an abnormality is detected (such as excessive deformation, sliding tendency, or sudden change in contact force), the current contact stiffness and gripping force are dynamically adjusted immediately based on the feedback information. This ensures gripping stability and avoids damage to the item or operation failure, thereby achieving adaptive and safe gripping of different items.

[0213] According to one embodiment of this application, the step of adjusting the execution state of the robotic arm based on real-time feedback information obtained from the robotic arm grasping the object, combined with environmental information, specifically involves:

[0214] According to the planned path, the robotic arm is moved to the gripping point of the object, and the attitude of the end effector is adjusted to maintain the best contact angle with the surface of the object.

[0215] Set the initial gripping parameters, including contact stiffness and gripping force;

[0216] Approaching the object along a planned grasping path, gradually increasing the grasping force, while monitoring the deformation and surface condition of the object, and sensing the force feedback during the grasping process in real time through force sensors;

[0217] Continuously monitor the joint torque of the robotic arm, the contact force of the end effector, and the deformation of the object;

[0218] Observe the changes in the state of the item in real time during the grasping process;

[0219] Compare the actual capture parameters with the preset parameters to evaluate the capture effect;

[0220] When abnormal deformation of the object is detected or excessive gripping force is detected, reduce the gripping force;

[0221] When a smooth surface of an object is detected, causing unstable gripping, the contact stiffness is increased;

[0222] The robot arm's movement speed and trajectory are dynamically adjusted based on real-time feedback from the object.

[0223] As described above, when the robotic arm performs a grasping operation, it first moves to the preset grasping point of the target object according to the planned motion path, and adjusts the posture of the end effector to form the optimal contact angle with the object surface to improve grasping stability. Then, initial grasping parameters, including contact stiffness and grasping force, are set according to the object's characteristics. The robotic arm slowly approaches the object along the planned grasping path, gradually increasing the grasping force during contact. Simultaneously, force sensors detect the contact force feedback in real time, and visual or depth information is used to monitor the object's deformation and surface condition changes. Throughout the grasping process, the system continuously collects and analyzes the torque data of each joint of the robotic arm, the contact force values ​​of the end effector, and the deformation or displacement of the object. Simultaneously, the system observes the changes in the object's state in real time through a vision system; it compares the parameters obtained during the actual grasping process with the preset grasping parameters to assess whether the current grasping effect meets the requirements of stability and safety; if abnormal deformation of the object is detected, the contact force exceeds the safety threshold, or there is a risk of slippage, the grasping force is immediately reduced to prevent damage; if it is determined that the grasping is unstable due to the smooth surface of the object, the contact stiffness is appropriately increased to enhance the gripping compliance and friction; in addition, based on the above real-time feedback information, the system also dynamically adjusts the movement speed, acceleration, and local trajectory of the robotic arm, such as decelerating when approaching fragile items and fine-tuning the path when deviation is detected, thereby achieving closed-loop optimization and adaptive control of the execution state to ensure the reliability and safety of the grasping operation.

[0224] According to one embodiment of this application, the step of scanning the kitchen environment and identifying the initial positions and basic states of items in the kitchen based on a binocular camera and a depth sensor to obtain environmental information further includes:

[0225] Based on ambient lighting conditions, the processing parameters of visual images are automatically adjusted using an adaptive lighting compensation algorithm.

[0226] As described above, during the scanning of the kitchen environment, the current lighting conditions are detected in real time, including light intensity, color temperature, and the presence of strong light reflection or shadow occlusion. Based on the detection results, an adaptive lighting compensation algorithm is used to automatically adjust the processing parameters of the visual images acquired by the binocular cameras, such as exposure time, gain, contrast, and white balance. This adjustment effectively suppresses image distortion caused by uneven lighting, overexposure, or underexposure, and improves image clarity and color reproduction in low-light, backlight, or high-contrast scenes. This enhances the accuracy of subsequent object recognition, edge detection, and 3D reconstruction, ensuring that the acquired environmental information has higher reliability and consistency, and providing a high-quality visual input foundation for constructing an accurate 3D environment model.

[0227] According to one embodiment of this application, it also includes:

[0228] Once the item is grasped, the grasping status is confirmed to verify that the item has been correctly grasped. After verification, the robotic arm performs subsequent operations according to the planned movement path.

[0229] During the grasping process, data on the deformation of the object, contact force, and the status of the robotic arm joints are collected.

[0230] Record key indicators such as the success rate and item damage for each grabbing operation;

[0231] Record the parameters of each successful capture into the item characteristic parameter library;

[0232] Optimize and update preset parameters based on the actual grasping effect of objects of different materials and shapes;

[0233] Based on historical crawling data, the crawling parameter library is continuously optimized;

[0234] When a crawl fails, analyze the reason for the failure, adjust the subsequent crawling parameters based on the reason, and re-execute the crawling operation.

[0235] As described above, after an item is grasped, the system confirms the grasping status by using visual detection, force feedback, and pose consistency judgment to verify whether the item has been grasped correctly and stably. Only after confirming successful grasping does the robotic arm execute subsequent operations, such as handling or placement, according to the pre-planned movement path. During the grasping process, the system simultaneously collects and records multi-dimensional data, including the deformation data of the item, the contact force data of the end effector, and the torque, angle, and motion state information of each joint of the robotic arm. Simultaneously, the system statistically analyzes key performance indicators for each grasping operation, including grasping success rate, whether the item is damaged, and the number of slippages. For successfully completed grasping operations, the grasping parameters used are recorded. (Such as contact stiffness, gripping force, and clamping posture) are associated with the material, shape, and state information of the corresponding items and stored in the item characteristic parameter library; based on the accumulated historical gripping data, the system continuously analyzes the actual gripping effect of items with different materials, shapes, or surface states, optimizes and updates the preset gripping parameters, and improves the adaptability and accuracy of the parameter library; when a gripping failure occurs, the system automatically analyzes the reasons for the failure, including insufficient gripping force, mismatched contact stiffness, item slippage, or environmental interference, and dynamically adjusts the gripping parameters for the next attempt accordingly, replans the gripping action and executes it, forming a closed-loop learning and optimization mechanism, and continuously improving the robotic arm's adaptive gripping ability in complex kitchen environments.

[0236] A second aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any of the embodiments of the first aspect above.

[0237] Figure 2 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 2 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 may call logical instructions in the memory 830 to execute the method in any of the embodiments of the first aspect described above, the method including:

[0238] Real-time environmental information is acquired using binocular cameras and depth sensors to construct a 3D environment model;

[0239] The acquired user natural language instructions are decomposed into a sequence of subtasks, and the impact on the subtask sequences is evaluated based on environmental information to obtain the decomposition and evaluation results.

[0240] Based on the 3D environment model and decomposition evaluation results, the motion path of the robotic arm is planned;

[0241] The gripping parameters are determined based on the characteristics of the object in order to adjust the contact stiffness and gripping force of the robotic arm.

[0242] Based on real-time feedback information obtained from the robotic arm's grasping of objects, combined with environmental information, the robotic arm's execution state is adjusted.

[0243] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0244] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer being able to perform the methods provided by the above methods, the method comprising:

[0245] Real-time environmental information is acquired using binocular cameras and depth sensors to construct a 3D environment model;

[0246] The acquired user natural language instructions are decomposed into a sequence of subtasks, and the impact on the subtask sequences is evaluated based on environmental information to obtain the decomposition and evaluation results.

[0247] Based on the 3D environment model and decomposition evaluation results, the motion path of the robotic arm is planned;

[0248] The gripping parameters are determined based on the characteristics of the object in order to adjust the contact stiffness and gripping force of the robotic arm.

[0249] Based on real-time feedback information obtained from the robotic arm's grasping of objects, combined with environmental information, the robotic arm's execution state is adjusted.

[0250] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the methods provided by the above methods, the method comprising:

[0251] Real-time environmental information is acquired using binocular cameras and depth sensors to construct a 3D environment model;

[0252] The acquired user natural language instructions are decomposed into a sequence of subtasks, and the impact on the subtask sequences is evaluated based on environmental information to obtain the decomposition and evaluation results.

[0253] Based on the 3D environment model and decomposition evaluation results, the motion path of the robotic arm is planned;

[0254] The gripping parameters are determined based on the characteristics of the object in order to adjust the contact stiffness and gripping force of the robotic arm.

[0255] Based on real-time feedback information obtained from the robotic arm's grasping of objects, combined with environmental information, the robotic arm's execution state is adjusted.

[0256] Example 2

[0257] I. Environmental Perception Phase: Multimodal Data Fusion and Adaptive Processing

[0258] Data acquisition and fusion from binocular cameras and depth sensors:

[0259] The system acquires left and right view images using binocular cameras and calculates the disparity map:

[0260]

[0261] Where f=320 pixels is the camera focal length, B=0.12 meters is the binocular baseline, and Z(x,y) is the depth value.

[0262] Depth maps are acquired by a depth sensor (ToF type), and the raw depth data is as follows:

[0263]

[0264] Where c = 3 × 10 8 m / s is the speed of light, and Δt is the time it takes for light to travel.

[0265] Dense point clouds are obtained by fusing depth data with stereo images through spatiotemporal alignment:

[0266]

[0267] Where P(x,y) represents the coordinates of a point in three-dimensional space.

[0268] Adaptive illumination compensation algorithm:

[0269] The system monitors ambient lighting conditions in real time and calculates the average brightness value.

[0270]

[0271] Where W=640 and H=480 are the image resolutions. These are the pixel values ​​of the original image.

[0272] Automatically adjust parameters based on lighting conditions:

[0273]

[0274] in, This represents the desired brightness value.

[0275] Application of the compensation formula:

[0276]

[0277] In this embodiment, the brightness of the left area was detected to be 85 and that of the right area was 42. The system automatically calculated k=1.5 and c=12. After compensation, the brightness of the right area was increased to 128, which effectively solved the problem of uneven lighting.

[0278] Item identification and status analysis:

[0279] Calculate the geometric features of tomatoes using point cloud data:

[0280]

[0281] Where N=1500 is the number of tomato point clouds, and r is the average radius.

[0282] Identifying tomatoes as "easily deformable" materials (based on surface curvature and point cloud density):

[0283]

[0284] Among them, κ=0.35 (threshold >0.25 is judged as easily deformable).

[0285] The final three-dimensional coordinates of the tomato were determined to be (x=0.75m, y=0.42m, z=0.81m), and its surface condition was "moist and easily deformable".

[0286] II. Semantic Understanding and Task Decomposition Stage: Application of VLN Multimodal Large Model

[0287] Instruction parsing and semantic feature extraction:

[0288] The user instruction "put the tomatoes on the cutting board into the microwave" was analyzed as the key element:

[0289] Target of operation: Tomato

[0290] Source location: on the cutting board

[0291] Target location: Inside the microwave oven

[0292] Cross-modal feature alignment using a VLN multimodal large model:

[0293]

[0294] Where V is the visual feature vector (dimension 512), L is the language feature vector (dimension 512), and S is the spatial location feature (dimension 128). , , This is a learnable weight matrix.

[0295] Task breakdown and feasibility assessment:

[0296] Break down instructions into a sequence of subtasks:

[0297] (1) Navigate to the cutting board

[0298] (2) Grab the tomatoes

[0299] (3) Move to microwave oven

[0300] (4) Open the microwave oven door

[0301] (5) Place the tomatoes

[0302] (6) Close the microwave oven door

[0303] Feasibility assessment of each sub-task based on environmental model:

[0304]

[0305] Where ΔE represents the change in environmental state (such as the change in the state of the microwave oven door). It is an adaptation factor to environmental changes.

[0306] Evaluation results:

[0307] Subtask (2): Feasibility = 0.85 (Tomatoes are easily deformed, and the grabbing parameters need to be dynamically adjusted);

[0308] Subtask (4): Feasibility = 0.78 (Microwave oven door is currently closed, operation is required).

[0309] III. Path Planning and Fetch Parameter Determination Stage

[0310] Robotic arm path planning:

[0311] Path planning using the improved A* algorithm:

[0312]

[0313] Where g(n) is the actual cost from the starting point to the current point, and h(n) is the heuristic estimate to the target point.

[0314] Optimize heuristic functions:

[0315]

[0316] Where α = 1.2, β = 0.8, The distance to the target point. The distance to the obstacle.

[0317] Path point conversion to joint angle (inverse kinematics calculation):

[0318]

[0319] in, is the pseudo-inverse of the Jacobian matrix, and v is the end effector speed.

[0320] Fetching parameters determined:

[0321] Based on the item characteristic parameter library, the corresponding parameters for tomatoes are:

[0322] Contact stiffness: (Standard values ​​for easily deformable items)

[0323] Scraping strength: (Recommended range: 0.5-1.2N)

[0324] After the system presets the initial parameters, the robotic arm begins to perform the grasping action.

[0325] IV. Implementation and Closed-Loop Adjustment Phase

[0326] Crawling process control:

[0327] The robotic arm approaches the tomato along a planned path, gradually increasing its gripping force.

[0328]

[0329] Where F(t) is the gripping force at time t.

[0330] Real-time deformation monitoring:

[0331]

[0332] Where δ(t) is the deformation, and N=1500 is the number of point clouds.

[0333] When δ(t) > 0.4 mm (deformability threshold) is detected, the grasping parameters are dynamically adjusted:

[0334]

[0335] Where L is the loss function and θ is the meta-network parameter.

[0336] Execution status adjustment:

[0337] During the data crawling process, the system continuously monitors:

[0338] Joint torque: (Shoulder, elbow, wrist, base)

[0339] Contact force: (Stable value)

[0340] Deformation variables: (Below the threshold)

[0341] Based on the monitoring data, the system confirmed that the capture status was stable and proceeded with subsequent operations.

[0342] V. Effect Verification and Parameter Optimization Phase

[0343] Operation result recording and analysis:

[0344] This operation was successfully completed. The system has recorded the following:

[0345] Success rate of data capture: 100%

[0346] Tomato Deformation: Maximum Indentation ;

[0347] Contact force: stable at ;

[0348] Joint condition: No abnormal fluctuations;

[0349] The system associates this parameter with tomato characteristics:

[0350] Material: Easily deformable;

[0351] Surface condition: Moist;

[0352] Optimal parameters: K = 0.3 N / m, F = 0.6 N;

[0353] Parameter library update and optimization:

[0354] The system updates the item attribute parameter database:

[0355]

[0356] Where n=10 represents historical Tomato data scraped.

[0357] Optimize parameters for similar items:

[0358]

[0359] Where H represents the percentage of ambient humidity (the intensity decreases by 10% when humidity > 50%).

[0360] In this embodiment, the ambient humidity is 65%, and the system automatically updates the tomato gripping force range to 0.45-1.08N.

[0361] Failure handling mechanism:

[0362] Suppose a fetching attempt fails (the tomato slips), the system analyzes the reason for the failure:

[0363]

[0364] Where c represents the category of failure reason (insufficient force, stiffness mismatch, etc.).

[0365] Based on the analysis results, the system adjusted the parameters and re-executed:

[0366] New parameters: , ;

[0367] The fetch was re-executed and completed successfully.

[0368] For any parts not mentioned in this application, existing technologies may be used or referenced.

[0369] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0370] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for operating a kitchen service robot based on a VLN large model, characterized in that, include: Real-time environmental information is acquired using binocular cameras and depth sensors to construct a 3D environment model; The acquired user natural language instructions are decomposed into a sequence of subtasks, and the impact on the subtask sequences is evaluated based on environmental information to obtain the decomposition and evaluation results. Based on the 3D environment model and decomposition evaluation results, the motion path of the robotic arm is planned; The gripping parameters are determined based on the characteristics of the object to adjust the contact stiffness and gripping force of the robotic arm, specifically: By combining the object classification information in the 3D environment model, the material type of the object is identified, and the structural characteristics of the object are judged through visual images; For items made of different materials, a preset database of gripping parameters is established, including standard contact stiffness values ​​and recommended gripping force ranges. During the grasping process, the deformation and surface condition of the object are monitored in real time, and the contact stiffness and grasping force are dynamically adjusted based on the real-time feedback from the object. Based on the real-time feedback information obtained by the robotic arm when grasping objects, combined with environmental information, the execution state of the robotic arm is adjusted. When the robotic arm performs a grasping operation, it first moves the robotic arm to the preset grasping point of the target object according to the planned motion path, and adjusts the posture of the end effector to form the best contact angle with the surface of the object in order to improve the grasping stability. Subsequently, initial gripping parameters, including contact stiffness and gripping force, are set according to the characteristics of the object. The robotic arm slowly approaches the object along the planned gripping path, gradually increasing the gripping force during contact. Simultaneously, force sensors detect the contact force feedback in real time, and visual or depth information is used to monitor the deformation and surface condition changes of the object. Throughout the gripping process, the system continuously collects and analyzes the torque data of each joint of the robotic arm, the contact force values ​​of the end effector, and the deformation or displacement of the object. The vision system also observes the changes in the object's state in real time. The parameters obtained during the actual gripping process are compared with the preset gripping parameters. The system compares the data to assess whether the current grasping effect meets the requirements of stability and safety. If abnormal deformation of the item is detected, the contact force exceeds the safety threshold, or there is a risk of slippage, the grasping force is immediately reduced to prevent damage. If it is determined that the grasping is unstable due to the smooth surface of the item, the contact stiffness is appropriately increased to enhance the gripping compliance and friction. In addition, based on real-time feedback information, the system also dynamically adjusts the movement speed, acceleration, and local trajectory of the robotic arm, decelerates when approaching fragile items, and fine-tunes the path when deviation is detected, thereby achieving closed-loop optimization and adaptive control of the execution state to ensure the reliability and safety of the grasping operation. After an item is grasped, the system confirms the grasping status through visual inspection, force feedback, and pose consistency judgment to verify whether the item has been grasped. Only after successful grasping is confirmed does the robotic arm execute subsequent operations, including handling or placement, according to the pre-planned movement path. During the grasping process, the system simultaneously collects and records multi-dimensional data, including the deformation data of the item, the contact force data of the end effector, and the torque, angle, and motion state information of each joint of the robotic arm. Simultaneously, the system statistically analyzes key performance indicators for each grasping operation, including grasping success rate, whether the item is damaged, and the number of slippages. For successfully completed grasping operations, the system records the contact stiffness, grasping... The gripping parameters, including force and gripping posture, are associated with the material, shape, and state information of the corresponding items and stored in the item characteristic parameter library. Based on accumulated historical gripping data, the system continuously analyzes the actual gripping effect of items with different materials, shapes, or surface states, optimizes and updates the preset gripping parameters, and improves the adaptability and accuracy of the parameter library. When a gripping failure occurs, the system automatically analyzes the cause of the failure, including insufficient gripping force, mismatched contact stiffness, item slippage, or environmental interference, and dynamically adjusts the gripping parameters for the next attempt accordingly, replans the gripping action, and executes it, forming a closed-loop learning and optimization mechanism to continuously improve the robotic arm's adaptive gripping capability in complex kitchen environments.

2. The method according to claim 1, characterized in that, The method of acquiring environmental information in real time based on binocular cameras and depth sensors to construct a three-dimensional environment model specifically includes: Based on a binocular camera and a depth sensor, the kitchen environment is scanned to identify the initial position and basic state of items in the kitchen in order to obtain environmental information, including visual images and three-dimensional spatial data of the current environment. A three-dimensional environment model is then constructed based on the environmental information.

3. The method according to claim 1, characterized in that, The process of decomposing the acquired user natural language instructions into a sequence of subtasks and evaluating the impact on the subtask sequences based on environmental information to obtain the decomposition and evaluation results is as follows: Receive and parse user natural language commands, and identify key elements in the user natural language commands, including: the object of operation, the location, and the target location; Based on key elements, understand the semantics and operational intent of the user's natural language commands, decompose the user's natural language commands into a sequence of sub-tasks, and determine the specific operation steps and parameters of each sub-task sequence; Based on environmental information, the impact of environmental changes on the execution of the subtask sequence is assessed to determine the feasibility of executing each subtask in the current environment.

4. The method according to claim 1, characterized in that, The robotic arm motion path is planned based on the 3D environment model and decomposition evaluation results, specifically as follows: Based on the environmental model and task decomposition results, the global path of the robotic arm end effector from the current position to the target position is determined; An improved A* search algorithm or a conflict-based multi-agent path planning algorithm is used to ensure that the path is conflict-free and meets the basic constraints. Convert global path points into joint angle values, and use inverse kinematics to calculate the joint angle corresponding to each path point; A 3-5-3 piecewise polynomial interpolation method is used to smooth the path points in the joint space to ensure the continuity of position, velocity, and acceleration. Optimize the interpolation time of each segment to ensure the optimal running time of the robotic arm while satisfying the speed and acceleration constraints. Based on obstacle information in the 3D environment model, the planned path is adjusted for obstacle avoidance. The Cartesian repulsion velocity of the robotic arm is calculated using a differential kinematics method based on the Jacobian matrix. Based on the global path, obtain the initial local path for each joint on the robotic arm; Using an improved particle swarm optimization algorithm, the time required between adjacent path nodes in the initial local path corresponding to each joint is optimized to obtain the final local path.

5. The method according to claim 2, characterized in that, The method of scanning the kitchen environment based on a binocular camera and a depth sensor to identify the initial position and basic state of items in the kitchen in order to obtain environmental information also includes: Based on ambient lighting conditions, the processing parameters of visual images are automatically adjusted using an adaptive lighting compensation algorithm.

6. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-5.

7. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method as described in any one of claims 1-5.

Images