Kitchen service robot operation method based on VLN large model

Abstract

The invention provides a kitchen service robot operation method based on a VLN large model, and relates to the technical field of intelligent robots, and the method comprises the steps: obtaining environment information in real time based on a binocular camera and a depth sensor, so as to construct a three-dimensional environment model; decomposing the obtained natural language instruction of the user into a sub-task sequence, and evaluating the influence on the sub-task sequence according to the environmental information to obtain a decomposition evaluation result; planning a motion path of the mechanical arm based on the three-dimensional environment model and the decomposition evaluation result; grabbing parameters are determined based on the object characteristics so as to adjust the contact rigidity and grabbing force of the mechanical arm; and based on feedback information obtained in real time when the mechanical arm grabs the objects, the execution state of the mechanical arm is adjusted in combination with the environment information.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent robots, and particularly relates to a kitchen service robot operation method based on a VLN large model. BACKGROUND

[0002] With the rapid development of artificial intelligence and robot technology, service robots are increasingly widely used in home scenarios, especially in complex dynamic environments such as kitchens, and the demand for assisting cooking, carrying objects and other tasks is growing. However, existing kitchen service robots still face many technical bottlenecks in actual application.

[0003] On the one hand, the kitchen environment has highly unstructured characteristics: a wide variety of items, random placement, variable lighting conditions, and frequent human interference and dynamic obstacles. Traditional robots rely on a single visual modality or a pre-set map for environment perception, making it difficult to accurately identify items and their spatial positions under complex lighting and occlusion conditions, resulting in insufficient environment modeling accuracy.

[0004] On the other hand, users usually issue operation instructions in natural language (such as "put the tomatoes on the cutting board into the microwave oven"), but existing systems only match keywords at the keyword matching level, lacking the ability to deeply analyze instruction semantics and structure task decomposition. At the same time, even if the task is decomposed, the system often ignores the influence of the current environment state on the feasibility of sub-tasks, causing the path planning to be detached from reality.

[0005] In addition, at the execution level, the fixed parameter strategy is often used for robotic arm grasping operations, which cannot dynamically adjust the contact stiffness and grasping force according to the material of the object (such as fragile glass cups and soft tomatoes) or deformation characteristics, easily leading to object slipping or damage. Even if some systems introduce force feedback, they lack a closed-loop mechanism to integrate real-time grasping feedback with environmental information to dynamically adjust the execution state, resulting in poor operation robustness and low success rate.

[0006] In summary, existing technologies are fragmented in the four key links of environment perception, task understanding, path planning and adaptive execution, making it difficult to achieve high-precision and high-reliability end-to-end operations in kitchen scenarios. SUMMARY

[0007] The application provides a kitchen service robot operation method based on a VLN large model to solve one of the above technical problems.

[0008] The technical solution adopted by the application is: The application embodiment provides a kitchen service robot operation method based on a VLN large model, comprising: real-time acquisition of environment information based on binocular cameras and depth sensors to construct a three-dimensional environment model; The obtained user natural language instruction is decomposed into a subtask sequence, and the influence of the subtask sequence on the environment information is evaluated to obtain a decomposition evaluation result; A motion path of the robot arm is planned based on the three-dimensional environment model and the decomposition evaluation result; A grasping parameter is determined based on the characteristics of the object to adjust the contact stiffness and grasping strength of the robot arm; Based on the feedback information obtained in real time by the robot arm grasping the object, the execution state of the robot arm is adjusted in combination with the environment information.

[0009] According to an embodiment of the present application, the environment information is obtained in real time based on the binocular camera and the depth sensor to construct a three-dimensional environment model, specifically: Based on the binocular camera and the depth sensor, the kitchen environment is scanned, and the initial position and basic state of the object in the kitchen are identified to obtain environment information, which includes visual images and three-dimensional spatial data of the current environment, and a three-dimensional environment model is constructed based on the environment information. According to an embodiment of the present application, the obtained user natural language instruction is decomposed into a subtask sequence, and the influence of the subtask sequence on the environment information is evaluated to obtain a decomposition evaluation result, specifically: The user natural language instruction is received and parsed, and key elements in the user natural language instruction are identified, including: operation object, position and target position; The semantic and operation intention of the user natural language instruction are understood based on the key elements, the user natural language instruction is decomposed into a subtask sequence, and specific operation steps and parameters of each subtask sequence are determined; Based on the environment information, the influence of environmental changes on the execution of the subtask sequence is evaluated to obtain the execution feasibility of each subtask in the current environment.

[0010] According to an embodiment of the present application, a motion path of the robot arm is planned based on the three-dimensional environment model and the decomposition evaluation result, specifically: Based on the environment model and the task decomposition result, a global path of the robot arm end effector from the current position to the target position is determined; An improved A* search algorithm or a multi-agent path planning algorithm based on conflict search is used to ensure that the path is conflict-free and meets the basic constraint conditions; The global path points are converted into joint angle values, and the inverse kinematics method is used to calculate the joint angle corresponding to each path point; A 3-5-3 segmented polynomial interpolation method is used to smooth the path points in the joint space to ensure continuous position, velocity and acceleration; The interpolation time of each segment is optimized to ensure optimal robot arm running time while meeting the speed and acceleration constraint conditions; The planned path is adjusted based on obstacle information in the three-dimensional environment model. A differential kinematics method based on a Jacobian matrix is used to calculate the Cartesian repulsion velocity of the robot arm. According to the global path, the initial local path of each joint of the robot arm is obtained. An improved particle swarm algorithm is used to optimize the time required between adjacent path nodes in the initial local path corresponding to each joint, respectively, to obtain the final local path.

[0011] According to an embodiment of the present application, the gripping parameters are determined based on the characteristics of the object to adjust the contact stiffness and gripping force of the robot arm, specifically: In combination with the classification information of the object in the three-dimensional environment model, the material type of the object is identified, and the structural characteristics of the object are determined through visual images; A preset gripping parameter library is established for objects of different materials, including standard contact stiffness values and recommended gripping force ranges; During the gripping process, the deformation and surface state of the object are monitored in real time, and the contact stiffness and gripping force are dynamically adjusted according to the real-time feedback of the object.

[0012] According to an embodiment of the present application, the feedback information obtained in real time by the robot arm gripping the object is combined with the environmental information to adjust the execution state of the robot arm, specifically: According to the planned path, the robot arm is moved to the gripping point of the object, and the posture of the end effector is adjusted to maintain the optimal contact angle with the surface of the object; The initial gripping parameters, including contact stiffness and gripping force, are set; The object is approached along the planned gripping path, and the gripping force is gradually increased, while the deformation and surface state of the object are monitored, and the force feedback during the gripping process is sensed in real time through the force sensor; The joint torque of the robot arm, the contact force of the end effector, and the deformation of the object are continuously monitored; The state change of the object during the gripping process is observed in real time; The difference between the actual gripping parameters and the preset parameters is compared to evaluate the gripping effect; When abnormal deformation of the object or excessive gripping force is detected, the gripping force is reduced; When it is detected that the smooth surface of the object causes unstable gripping, the contact stiffness is increased; According to the real-time feedback of the object, the motion speed and trajectory of the robot arm are dynamically adjusted.

[0013] According to an embodiment of the present application, the binocular camera and the depth sensor are used to scan the kitchen environment and identify the initial position and basic state of the object in the kitchen to obtain environmental information, which further includes: Based on the ambient light condition, the processing parameters of the visual image are automatically adjusted through an adaptive light compensation algorithm.

[0014] According to one embodiment of the application, further comprising: When the object is grabbed, the grabbing state is confirmed, it is verified that the object has been correctly grabbed, after verification confirmation, the mechanical arm performs subsequent operation according to the planned moving path; In the grabbing process, the object deformation data, contact force data and mechanical arm joint state information are collected; The success rate of each grabbing operation and the key indicators of object damage are recorded; The successful grabbing parameters of each time are recorded into the object characteristic parameter library; According to the actual grabbing effect of objects with different materials and shapes, the preset parameters are optimized and updated; Based on the historical grabbing data, the grabbing parameter library is continuously optimized; When the grabbing fails, the failure reason is analyzed, the subsequent grabbing parameters are adjusted according to the failure reason, and the grabbing operation is re-executed.

[0015] The second aspect embodiment of the application provides a computer readable storage medium, which stores a program, and the program is executed by a processor to realize the steps in the method.

[0016] The third aspect embodiment of the application provides an electronic device, which includes a memory, a processor and a program stored in the memory and executable on the processor, and the processor executes the program to realize the steps in the method.

[0017] Due to the adoption of the above technical solutions, the application has the following beneficial effects: The application realizes fusion perception of binocular cameras and depth sensors, acquires environmental information in real time and constructs a three-dimensional environmental model, effectively overcomes the problems of uneven light and serious occlusion in the kitchen environment, provides accurate spatial semantic basis for subsequent tasks, and significantly improves the accuracy of object recognition and positioning.

[0018] After the natural language instruction of the user is decomposed into a subtask sequence, the influence of the environment on each subtask (such as whether the target object is occluded or the path is blocked) is evaluated in combination with the three-dimensional environmental model to generate a “decomposition evaluation result”, avoid the execution of unfeasible tasks, and improve the rationality and safety of system decision-making.

[0019] The application plans the moving path of the mechanical arm based on the three-dimensional environmental model and the decomposition evaluation result, ensures that the path meets the requirements of task logic and effectively avoids static and dynamic obstacles, and significantly improves the practicability and execution success rate of path planning.

[0020] The application dynamically determines the grabbing parameters according to the characteristics of the objects (such as material, deformability), and adjusts the contact stiffness and grabbing force of the mechanical arm accordingly, to avoid damage to fragile or soft objects, while ensuring the stability of grabbing, greatly reducing the damage rate and slip rate of the objects.

[0021] During the grabbing process, the application dynamically adjusts the execution state (such as speed, force, posture) of the mechanical arm based on the feedback information (such as contact force, object deformation) obtained in real time by the mechanical arm and combined with environmental information, forming a closed-loop control of "perception-decision-execution-feedback-readjustment", significantly improving the operation robustness and task completion rate in complex kitchen environment. BRIEF DESCRIPTION OF DRAWINGS

[0022] The drawings described herein are used to provide further understanding of the application, and form a part of the application. The illustrative embodiments of the application and their descriptions are used to explain the application, and do not constitute an improper limitation on the application. In the drawings: Figure 1 A flowchart of a kitchen service robot operation method based on a VLN large model provided by an embodiment of the application; Figure 2 A structural schematic diagram of an electronic device provided by an embodiment of the application.

[0023] Reference signs: 810, processor; 820, communication interface; 830, memory; 840, communication bus. DETAILED DESCRIPTION

[0024] In order to more clearly explain the overall concept of the application, the following will be described in detail in an exemplary manner with reference to the drawings.

[0025] In the following description, many specific details are set forth in order to provide a thorough understanding of the application, but the application can also be implemented in other ways different from those described herein, therefore, the scope of protection of the application is not limited by the specific embodiments disclosed below. It should be noted that the embodiments of the application and the features in each embodiment can be combined with each other without conflict.

[0026] In the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact through an intermediate medium. In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0027] Embodiment 1 As Figure 1 indicated, a kitchen service robot operation method based on a VLN large model includes: S100, real-time environment information is acquired based on a binocular camera and a depth sensor to construct a three-dimensional environment model.

[0028] As described above, the binocular camera calculates the depth information of the objects in the scene through the image difference (parallax) of the left and right two views, has good texture recognition ability and long-distance perception advantage; The depth sensor (such as structured light, time of flight ToF or active infrared sensor) directly outputs the distance data of each point in the scene, has high-precision close-range ranging capability, and especially performs excellently in weak light or textureless area; The data of the two are aligned in space and time and fused with features, complementing each other's limitations in light sensitivity, ranging accuracy, effective distance, etc., to generate a dense three-dimensional point cloud containing geometric structure, object boundary, surface normal vector and space occupation state; Based on the point cloud data, the system further performs semantic segmentation, object clustering and space relationship modeling, and finally constructs a structured three-dimensional environment model containing object categories, positions, attitudes and operable regions; The model not only reflects the static environment (such as cabinets, stoves), but also dynamically updates the position changes of moving objects (such as users walking, temporarily placed food), providing real-time and accurate spatial semantic basis for subsequent task understanding and path planning.

[0029] For example, in a kitchen scene, the user instruction is "put the apple on the table into the refrigerator". At this time: The binocular camera captures that there are multiple objects (apple, cup, paper towel) on the table, but due to the dark light, the texture of some areas is blurred; The depth sensor clearly detects a convex circular object on the table, whose height, diameter and typical size of the apple match; The system fuses the color and texture features in the binocular image with the geometric shapes in the depth data, confirms that the object is an "apple", and accurately calculates its three-dimensional position in the world coordinate system (such as x=0.85m, y=0.32m, z=0.78m); At the same time, the system identifies that the refrigerator door is in a closed state, and the handle position and the operation space required to open the door are also modeled into the environment model; The finally constructed three-dimensional environment model not only contains the spatial coordinates of the apple, but also labels it as a "graspable object" and associates its material attribute (such as "easy-to-deform fruit"), providing a basis for subsequent grasp parameter setting.

[0030] It should be noted that in specific implementation scenarios, the binocular camera can be replaced by a multi-view vision system (such as three views, surround view) based on the above scheme, and the depth sensor can use devices of different principles (such as structured light, ToF, laser radar), as long as their combination can realize real-time acquisition and fusion of three-dimensional environment information, which belongs to the equivalent implementation mode of the present scheme.

[0031] In specific implementation scenarios, the three-dimensional environment model can be expressed in the form of point cloud, voxel grid (Voxel Grid), triangular mesh (Mesh), or semantic map (Semantic Map) based on the above scheme, as long as it can carry the spatial position, category and operability information of the object, which falls within the protection scope of the present scheme.

[0032] In specific implementation scenarios, the environment model can support incremental updating at a fixed frequency (such as 5 frames per second) or event triggering (such as detecting object movement) based on the above scheme, to ensure that the model always reflects the current real environment state.

[0033] In specific implementation scenarios, in addition to binocular and depth data, inertial measurement unit (IMU) data can also be fused to compensate for the perception deviation caused by the robot's own motion, or historical task data can be introduced to assist current environment understanding (such as "the location usually places a cutting board").

[0034] In specific implementation scenarios, in the process of constructing the three-dimensional model, a pre-trained object recognition model or kitchen knowledge graph can be combined to assign high-level semantic labels (such as "food", "utensil", "dangerous area") to the objects in the model, thereby supporting subsequent task decomposition and safety evaluation.

[0035] S200, decompose the obtained user natural language instruction into a sub-task sequence, and evaluate the influence of the sub-task sequence on the environment information to obtain a decomposition evaluation result.

[0036] As mentioned above, instruction analysis: first identify the key semantic elements in the instruction through natural language understanding module, including operation verbs (such as "take" "put" "cut"), operation objects (such as "tomato"), source location (such as "on the cutting board") and target location (such as "in the microwave"); Subtask decomposition: based on the preset task template or knowledge rules, the high-level instruction is decomposed into several atomic subtasks, typical subtasks include "locate target object", "navigate to object nearby", "grab object", "move to target location", "place object" and so on; Environmental impact assessment: match each subtask with the three-dimensional environment model currently constructed, assess the impact of environmental factors on the execution of subtasks. For example, judge whether the target object is visible, whether it is blocked, whether the target area is accessible, whether the path exists dynamic obstacles, whether the target container is in an operable state (such as whether the microwave door is closed) and so on; Generate decomposition evaluation results: integrate the above evaluation, label the execution status (such as "executable" "need to wait" "unfeasible"), priority, dependency relationship and potential risk prompt for each subtask, form a structured "decomposition evaluation result", as the basis for subsequent path planning and execution strategy.

[0037] This process realizes the bridge from fuzzy semantic instruction to environment perception type task planning, significantly improves the understanding ability of robot to complex instruction and the adaptability of task execution.

[0038] For example, the user issues the instruction: "take the milk in the refrigerator and put it on the dining table." Instruction analysis: the system identifies the operation object as "milk", the source location as "in the refrigerator", the target location as "dining table", and the action as "take out and place"; Subtask decomposition: the system decomposes it into the following subtask sequence: Navigate to the front of the refrigerator; Open the refrigerator door; Locate the milk inside the refrigerator; Grab the milk; Close the refrigerator door; Move to the dining table; Place the milk on the dining table; Environmental impact assessment: Through the three-dimensional environment model, it is found that the refrigerator door is currently closed, but there is no obstacle in front, subtasks 1 and 2 are executable; The point cloud data inside the refrigerator shows that the milk is located on the second layer of partition and is not blocked by other objects, subtasks 3 and 4 are feasible; There is a user currently dining in the dining table area, there is a temporary obstacle, subtasks 6 and 7 need to be waited or adjusted the placement position; Decomposition evaluation result: the system generates an evaluation result, marking subtasks 1-5 as "immediately executable", and subtasks 6-7 as "conditionally limited, waiting for the table area to be cleared or negotiating a placement point", and recommends prioritizing the first five steps while continuously monitoring the table status.

[0039] This evaluation result avoids collisions or interference caused by the robot blindly executing all tasks, reflecting the deep collaboration between tasks and the environment.

[0040] It should be noted that in specific implementation scenarios, natural language instructions can not only be limited to voice input, but also come from text input (such as mobile app instructions, touch screen input), or cross-modal fusion instructions (such as "pointing to an object and saying 'take this'"), as long as they are ultimately converted into semantic explicit operation intentions, which fall within the scope of this solution.

[0041] In specific implementation scenarios, subtask decomposition can be based on a pre-defined rule base, task knowledge graph, or learning-based task parsing model, with different kitchen operations (such as "cutting vegetables", "pouring water", "heating") corresponding to different subtask combination templates.

[0042] In specific implementation scenarios, in addition to space accessibility and occlusion judgment, environmental impact assessment can also include time constraints (such as "the microwave is running, please wait until it's finished"), safety constraints (such as "a child is detected near, suspend operation"), and item state (such as "the milk carton has been opened, be careful not to tilt it"), as long as it is based on environmental information to judge the feasibility of subtasks, it falls within the scope of this solution.

[0043] In specific implementation scenarios, decomposition evaluation results can be represented as task dependency graphs, weighted execution queues, confidence-based feasibility scores, or risk-labeled task lists, and different data structure forms do not affect their essential function as path planning inputs.

[0044] In specific implementation scenarios, when there is uncertainty in the environmental evaluation result (such as whether the object is the target object), the system can actively initiate a clarification request (such as "do you mean the red tomato or the yellow tomato?"), and incorporate user feedback into the evaluation result update.

[0045] S300, based on the three-dimensional environment model and the decomposition evaluation result, plan the motion path of the mechanical arm.

[0046] As mentioned above, the input fusion: path planning module receives two core inputs simultaneously: one is a three-dimensional environment model constructed by binocular cameras and depth sensors, containing geometric and semantic information such as obstacle distribution, passable area, and precise pose of objects; the other is the decomposition and evaluation results generated by natural language instruction decomposition and environment evaluation, including subtask target points, execution order, feasibility status, and potential risk prompts.

[0047] Path generation logic: the planning process first determines the starting pose and target pose of the end effector of the robot arm according to the target of the subtask (such as "grab the tomato"); then identifies all static and dynamic obstacles (such as the edge of the stove, temporarily placed dishes, and moving human bodies) in the three-dimensional environment model; then combines the information about task priority, execution conditions, and safety constraints in the decomposition and evaluation results to filter out candidate paths that meet the task logic and have no collision.

[0048] Path optimization mechanism: under the premise of meeting task targets and obstacle avoidance, the smoothness, motion time, energy consumption, and operation stability of the path are further optimized. For example, for subtasks that require fine operation (such as placing fragile objects), the path will prefer low-speed and smooth trajectories; for movement in obstacle-free areas, more efficient straight or curved paths can be used.

[0049] Output form: the final output of the robot arm motion path is a continuous trajectory defined in joint space or Cartesian space, containing the pose, velocity, acceleration, and time parameters of each key point, which can be directly used for subsequent motion control module execution.

[0050] This step realizes the key conversion from "task intent" to "physical action", which is the core link connecting high-level semantic understanding and low-level execution control.

[0051] For example, the user instruction is "put the spatula on the stove into the drawer", and the system has completed instruction decomposition and environment evaluation, obtaining the following information: The three-dimensional environment model shows that the spatula is located on the right side of the stove, with its handle facing outward, and there is no obstruction around it; the target drawer is located in the lower cabinet, which is currently in a closed state, but the space in front is sufficient; there are hanging kitchen utensils above the stove, forming an upper obstacle.

[0052] The decomposition and evaluation results contain the subtask sequence: "1. grab the spatula; 2. open the drawer; 3. put the spatula in; 4. close the drawer", among which subtask 2 is marked as "need to confirm that the drawer handle is accessible".

[0053] Based on the above information, the path planning process is as follows: For subtask 1 (grab the spatula): plan a path from the current position of the robot arm around the edge of the stove, avoiding the overhead hanging object, and accurately approaching the spatula handle. Adjust the end pose to an angle suitable for grasping the handle; For subtask 2 (open the drawer): after the grab is complete, plan a path for the robot arm to move the spatula to the front of the drawer, then release the spatula (or temporarily store it). Plan a separate path to the drawer handle to ensure that the fingers do not collide with the cabinet; For subtask 3 (put in the spatula): after the drawer is opened, re-grab the spatula and plan a path that descends vertically from above, avoids the edge of the drawer, and smoothly places it inside. All paths avoid known obstacles and, based on evaluation information such as "spatula is a metal hard object" and "drawer interior space is limited", use moderate speed and appropriate rigidity trajectory parameters.

[0054] The final generated path is not only reachable and collision-free, but also takes into account operation efficiency and safety.

[0055] It should be noted that in specific implementation scenarios, the path can be planned in Cartesian space (end effector pose space) or joint space (joint angle space) based on the above scheme, or a hybrid space strategy can be used, as long as it generates a feasible trajectory based on a three-dimensional environment model and decomposition evaluation results, which is an equivalent implementation of the present scheme.

[0056] In specific implementation scenarios, the path planning can support real-time re-planning based on the above scheme. When a new dynamic obstacle (such as a user suddenly reaching into the work area) is detected, the system can adjust the path based on the updated environment model in real time.

[0057] In specific implementation scenarios, when there are multiple parallel or serial subtasks (such as grabbing two objects at the same time), task scheduling and path coordination optimization strategies can be used to generate composite paths that take into account timing and spatial efficiency.

[0058] In specific implementation scenarios, the path planning can integrate safety distance constraints, speed limit areas (such as automatically reducing speed when approaching a human body), emergency stop trigger zones, and other safety strategies based on environmental semantics (such as "human body area" and "high temperature area"). These safety path generation methods based on environmental semantics are part of the present scheme.

[0059] In specific implementation scenarios, the path generation process can be combined with the kinematic model of the robot arm (such as joint limits, singularity avoidance, maximum speed / acceleration limits) for adaptive optimization to ensure that the output path is physically executable.

[0060] In a specific implementation scenario, on the basis of the above scheme, collision detection, reachability verification and time rationality checking can be performed before outputting the final path. If it is found that it is not feasible, the order of the sub-tasks is adjusted or the environment is intervened (such as "please move the bowl in front of the drawer").

[0061] S400, determine the grabbing parameter based on the object characteristics to adjust the contact stiffness and the grabbing strength of the mechanical arm.

[0062] As described above, the object characteristics are obtained: the object characteristics include but are not limited to material (such as glass, plastic, metal, fruits and vegetables), shape (regular / irregular), surface state (smooth / rough), structural strength (rigid / soft), weight and gravity distribution, etc. These characteristics can be derived from semantic recognition results in the three-dimensional environment model (such as "tomato" and "glass"), point cloud geometry analysis (such as curvature and volume estimation), historical task database, or user preset labels.

[0063] Grabbing parameter definition: the grabbing parameter mainly includes contact stiffness (i.e. the compliance of the mechanical arm end effector when contacting the object, reflecting the response sensitivity to external force disturbance) and grabbing strength (i.e. the clamping force applied by the effector). Both of them determine the "softness" and "tightness" of the grabbing process.

[0064] Parameter mapping logic: the system internally or dynamically constructs a "object characteristics-grabbing parameter" mapping rule library. For example, for fragile or soft objects (such as eggs and ripe tomatoes), a lower contact stiffness (high compliance) and a smaller grabbing strength are set; for hard and rough surface objects (such as metal spatula and ceramic bowl), a higher stiffness and a larger grabbing strength can be used to ensure stability.

[0065] Parameter output and execution: the determined grabbing parameter is transmitted to the force control or impedance control module of the mechanical arm, which adjusts the force output and compliance behavior of the effector in real time during the execution of the grabbing action, realizing the adaptive grabbing of "different objects with different strategies".

[0066] This step realizes the intelligent transition from "seeing the object" to "knowing how to take it", which is a key technology to improve the operation safety and universality of the kitchen service robot.

[0067] For example, the user instruction is "put the ripe tomatoes in the dish into the preservation box".

[0068] Object characteristic identification: the system identifies the target object as "ripe tomato" through the three-dimensional environment model, and judges it as "soft, low structural strength, and surface wet" based on its red color, weak surface reflection, and slight concave features displayed in the point cloud.

[0069] Grasping parameter determination: according to the built-in rules, the system automatically sets: Contact stiffness: adjust to a lower level, so that the mechanical arm has higher flexibility when contacting the tomato, can conform to the surface of the tomato and not apply a sudden impact; Grasping force: set to a gentle level (such as 30% of the regular force), only need to overcome gravity, avoid extrusion leading to juice overflow or damage.

[0070] Execution effect: the mechanical arm approaches the tomato at a slow speed, slightly conforms to its contour after contact, applies a small clamping force to complete the grasping, and there is no slipping or damage throughout the process.

[0071] For example, if the target object is a "stainless steel spoon": The system identifies it as a metal, rigid, smooth surface but with texture; Set a higher contact stiffness (to ensure stable grasping posture) and a medium to high grasping force (to prevent slipping due to smooth surface); The grasping process is fast and stable, with no risk of slipping.

[0072] It should be noted that in specific implementation scenarios, in addition to the above scheme, the object characteristics can come from vision and depth perception, and can also integrate multi-source information such as tactile feedback (such as deformation response after first light touch), weight estimation (through torque backpropagation), user historical operation preferences (such as "user prefers to hold tomatoes gently").

[0073] In specific implementation scenarios, in addition to the above scheme, in addition to contact stiffness and grasping force, grasping parameters can also include grasping speed, jaw closing rate, grasping posture angle, pre-clamping gap, etc., as long as they are used to adjust the interaction behavior of the mechanical arm and the object, and are set based on the characteristics of the object, which falls within the scope of the present application.

[0074] In specific implementation scenarios, in addition to the above scheme, the mapping rules can be based on expert experience rule base, machine learning model (such as classifier output parameter interval), or online optimization strategy (such as feedback adjustment after trial grasping), and different implementation ways all belong to equivalent technical means of the present scheme.

[0075] In specific implementation scenarios, in addition to the above scheme, the present method is applicable to various end effectors, including two-fingered gripper, multi-fingered dexterous hand, vacuum suction cup, magnetic clamp, etc. For different effectors, the specific physical meaning of the grasping parameters may be different (such as negative pressure intensity for suction cup), but the core logic of "setting interaction parameters based on object characteristics" remains unchanged, which belongs to the coverage of the present application.

[0076] In specific implementation scenarios, the system can also set safety upper and lower limits for each type of item based on the above scheme, and monitor whether the threshold is exceeded during execution (such as detecting abnormal deformation and automatically releasing).

[0077] In specific implementation scenarios, the system can fine-tune the parameters according to the sub-task type if the same item requires different grabbing methods in different tasks (such as "pick up the tomato" vs "fix before cutting the tomato").

[0078] S500, based on the feedback information obtained in real time by the mechanical arm grabbing the item, combined with environmental information, to adjust the execution state of the mechanical arm.

[0079] As mentioned above, the feedback information comes from: the joint torque sensor of the mechanical arm itself, the contact force / pressure sensor of the end effector, the opening and closing state of the gripper, the motor current change and other internal state data, as well as the visual and spatial information (such as whether the item has shifted, deformed, slipped, etc.) continuously updated by the external perception system (such as binocular cameras, depth sensors) during operation.

[0080] Environmental information fusion: compare and correlate the above feedback information with the three-dimensional environment model constructed. For example, if visual detection shows that the item has shifted slightly after being grabbed, it is combined with the environmental model to determine whether the shift has caused the distance to the surrounding obstacles to be too close; if the force sensor shows that the clamping force has dropped sharply, it is combined with the material of the item to determine whether it is a precursor to slipping.

[0081] Execution state adjustment: based on the fusion analysis results, the system can adjust the execution state of the mechanical arm in multiple dimensions, including but not limited to: Adjust the motion speed or acceleration (such as reducing speed through narrow areas); Correct the end pose (such as fine-tuning the grabbing angle to compensate for item shift); Dynamically reset the grabbing force or contact stiffness (such as detecting that the item has started to deform and immediately reducing the force); Trigger task rollback or re-planning (such as confirming that the item has fallen, then aborting the current sub-task and starting the pick-up recovery process).

[0082] This step realizes the leap from "open-loop execution" to "perception-decision-adjustment" closed-loop control, significantly improving the operation robustness and fault tolerance of robots in uncertain kitchen environments.

[0083] For example, the user instruction is "pick up the glass from the sink and put it in the disinfection cabinet".

[0084] During the grabbing execution: the mechanical arm successfully grabs the glass and starts moving towards the disinfection cabinet; Real-time feedback: Force sensor detects a small fluctuation in clamping force, while depth camera finds a slight tilt of the cup; Environmental information integration: The system calls the three-dimensional environment model and confirms that there is a hanging towel above the current path. If the cup continues to tilt, it may collide; Adjust the execution state: Immediately reduce the moving speed of the mechanical arm; Fine-tune the wrist joint angle to restore the vertical posture of the cup; Slightly increase the gripping force (still within the safe range) to enhance the grip stability; At the same time, slightly raise the overall path to avoid the hanging area.

[0085] For example, when grabbing a ripe tomato: The initial gripping force is executed according to the preset gentle parameter; But the force feedback shows that the deformation increases rapidly after contact, and the vision also detects that the surface juice is oozing out; The system determines that the current force is still too large, immediately reduces the gripping force further, and pauses the movement, waiting for the deformation to stabilize before continuing the task slowly.

[0086] The above adjustments are completed within milliseconds, avoiding damage to the item or failure of the task.

[0087] It should be noted that in specific implementation scenarios, in addition to force, pose, and visual feedback, temperature sensors (such as detecting hot pots), vibration signals (such as determining whether the item is loose), sound feedback (such as glass collision sound), and other multi-modal sensing information can also be included based on the above-mentioned solutions.

[0088] In specific implementation scenarios, in addition to force and speed, the execution state adjustment can also include task priority rearrangement (such as pausing the current task processing to deal with unexpected obstacles), sub-task skipping (such as the target item is no longer in its original position), and operation mode switching (such as switching from "grabbing" to "pushing and pulling"), and other high-level strategy adjustments based on the above-mentioned solutions.

[0089] In specific implementation scenarios, in addition to the above-mentioned solutions, adjustments can be triggered by threshold values (such as force exceeding a certain value), trend prediction (such as accelerated deformation), or fuzzy judgment based on confidence (such as "possible to slip"), and different triggering logic belongs to the equivalent implementation manner of this solution.

[0090] In specific implementation scenarios, in addition to the above-mentioned solutions, when the system cannot restore the stable state through automatic adjustment (such as the item slipping seriously), it can actively pause the execution and request user confirmation or assistance (such as voice prompt "The cup is a bit slippery, please hold it").

[0091] In a specific implementation scenario, on the basis of the above scheme, the system can record the state and result before and after each adjustment, which can be used to optimize future parameter settings or adjustment strategies (e.g., “tomatoes of a certain type are more prone to deformation in high humidity, so next time the grabbing needs to be lighter”).

[0092] In a specific implementation scenario, on the basis of the above scheme, if the robot is equipped with multiple execution units (such as dual arms, mobile chassis + mechanical arm), the adjustment can involve the coordinated action of multiple actuators (such as micro-movement of the chassis to improve the grabbing angle).

[0093] According to an embodiment of the present application, the environment information is obtained in real time based on the binocular camera and the depth sensor to construct a three-dimensional environment model, specifically: Based on the binocular camera and the depth sensor, the kitchen environment is scanned to identify the initial position and basic state of the objects in the kitchen to obtain environment information, which includes visual images and three-dimensional spatial data of the current environment, and a three-dimensional environment model is constructed based on the environment information. As described above, the binocular camera and the depth sensor are used to scan the kitchen environment synchronously to obtain visual images and three-dimensional spatial data of the current scene; the binocular camera collects color images from left and right perspectives to extract texture, color and surface features of objects in the environment, and the depth sensor directly measures the distance from each point in the scene to the sensor to generate corresponding depth maps or point cloud data; the two types of data are aligned and fused in time and space to form dense environment perception information containing geometric structure and visual semantics; on this basis, the initial position, orientation, visibility and basic state (such as whether the container is open, whether the objects are stacked, whether the surface is wet, etc.) of various objects in the kitchen are identified; finally, based on the above fused environment information, a three-dimensional environment model is constructed, which can represent object categories, spatial coordinates, size profiles and mutual relationships, and the model can be updated in real time to support subsequent task understanding, path planning and adaptive execution.

[0094] According to an embodiment of the present application, the obtained user natural language instruction is decomposed into a sub-task sequence, and the influence of the environment information on the sub-task sequence is evaluated to obtain a decomposition evaluation result, specifically: The user natural language instruction is received and parsed, and key elements in the user natural language instruction are identified, including: operation object, position and target position; The semantic and operation intent of the user natural language instruction are understood according to the key elements, the user natural language instruction is decomposed into a sub-task sequence, and the specific operation steps and parameters of each sub-task sequence are determined; Based on the environment information, the influence of the environment change on the execution of the sub-task sequence is evaluated to obtain the execution feasibility of each sub-task in the current environment.

[0095] As described above, the system first receives natural language instructions input by the user and performs semantic parsing to identify key elements, including the object to be manipulated (e.g., "tomato" or "cup"), the object's current location (e.g., "on the cutting board" or "in the refrigerator"), and the target location (e.g., "in the microwave oven" or "on the dining table"). Based on these key elements, and combined with pre-defined task understanding rules or a knowledge base, the system accurately understands the user's operational intent, such as "move something from point A to point B" or "perform a specific operation on something." Subsequently, the high-level instruction is decomposed into a series of ordered, robot-executable sub-task sequences according to operational logic. Each sub-task corresponds to a specific action unit, such as "navigate to the vicinity of the item." The process involves tasks such as "opening the container," "grabbing the item," "moving to the target location," and "placing the item," with each subtask having its required operational parameters defined, including target coordinates, execution order, dependencies, and expected state. Next, using the environmental information provided by the constructed 3D environment model, the execution conditions of each subtask in the current environment are evaluated, determining factors such as whether there are obstacles obstructing the view, whether the target is visible, whether the operating space is sufficient, and whether relevant equipment is operational. Finally, based on the evaluation results, the feasibility of each subtask in the current environment is determined, generating a decomposed evaluation result including feasibility status, risk warnings, and execution constraints, providing a basis for subsequent path planning and execution control.

[0096] According to one embodiment of this application, the step of planning the robotic arm motion path based on the three-dimensional environment model and decomposition evaluation results specifically includes: Based on the environmental model and task decomposition results, the global path of the robotic arm end effector from the current position to the target position is determined; An improved A* search algorithm or a conflict-based multi-agent path planning algorithm is used to ensure that the path is conflict-free and meets the basic constraints. Convert global path points into joint angle values, and use inverse kinematics to calculate the joint angle corresponding to each path point; A 3-5-3 piecewise polynomial interpolation method is used to smooth the path points in the joint space to ensure the continuity of position, velocity, and acceleration. Optimize the interpolation time of each segment to ensure the optimal running time of the robotic arm while satisfying the speed and acceleration constraints. Based on obstacle information in the 3D environment model, the planned path is adjusted for obstacle avoidance. The Cartesian repulsion velocity of the robotic arm is calculated using a differential kinematics method based on the Jacobian matrix. Based on the global path, obtain the initial local path for each joint on the robotic arm; Using the improved particle swarm algorithm, the required time between adjacent path nodes in the initial local path corresponding to each joint is optimized respectively to obtain the final local path.

[0097] As described above, first, based on the constructed three-dimensional environment model and the aforementioned decomposition evaluation results, the sub-task goals that the robot end effector needs to complete are determined, including the initial pose and the target pose, and then the global motion path from the current position to the target position is determined; in the path search process, an improved A* search algorithm is used, or in the multi-robot collaborative scenario, a multi-agent path planning algorithm based on conflict search (CBS) is used to generate a feasible path that meets the task logic, has no collision and conforms to the basic motion constraints (such as workspace restrictions and obstacle avoidance requirements); subsequently, the Cartesian space path points in the global path are converted into angle values corresponding to the joints of the robot through inverse kinematics method to obtain a sequence of discrete path points in the joint space; in order to ensure the smoothness of the motion process, a 3-5-3 segmented polynomial interpolation method is used for trajectory planning of the joint space path points, so that the continuity of position, velocity and acceleration is realized in the motion process of each joint, and impact or jitter is avoided; on this basis, the time parameters of each segment interpolation are optimized, the overall running time is shortened as much as possible under the premise of meeting the physical constraints of the maximum speed and the maximum acceleration of the robot, and the execution efficiency is improved; at the same time, combined with the real-time updated obstacle information in the three-dimensional environment model, the initially planned path is dynamically adjusted for obstacle avoidance to ensure that the path remains safe in the complex kitchen environment; the specific obstacle avoidance strategy includes: using the differential kinematics method based on the Jacobian matrix to calculate the required Cartesian repulsion speed of the robot end effector or the connecting rod when approaching the obstacle, and mapping it back to the joint space to correct the motion direction; in addition, for each joint, the initial local path corresponding to it is extracted from the global path, and an improved particle swarm optimization algorithm (PSO) is used to finely adjust the running time between adjacent path nodes to further optimize the smoothness and time efficiency of the local trajectory, and finally generate a complete robot motion path that meets the task requirements, environmental constraints and dynamic limitations.

[0098] According to one embodiment of the present application, the grasping parameters are determined based on the characteristics of the object to adjust the contact stiffness and the grasping force of the robot arm, specifically: Combined with the classification information of the object in the three-dimensional environment model, the material type of the object is identified, and the structural characteristics of the object are judged through the visual image; For objects of different materials, a preset grasping parameter library is established, including the standard contact stiffness value and the recommended grasping force range; During the grasping process, the deformation and surface state of the object are monitored in real time, and the contact stiffness and the grasping force are dynamically adjusted according to the real-time feedback of the object.

[0099] As described above, first, the category of the target object is determined in combination with the identified object classification information in the three-dimensional environment model, and further the surface texture, color, reflective properties and geometric shape of the object are analyzed through visual image to judge the material type (such as glass, plastic, metal, fruits and vegetables, etc.) and structural characteristics (such as whether it is easy to deform, whether the surface is smooth, the overall rigidity or softness) of the object; based on the above identification results, the grasping parameters matched with the object are retrieved from the pre-set grasping parameter library, and the parameter library sets the standard contact stiffness value and the recommended grasping force range for objects with different material and structural characteristics, for example, a lower contact stiffness and a smaller grasping force are set for fragile or easily deformed objects, and a higher contact stiffness and a moderate grasping force are set for hard or rough surface objects; during the actual execution of the grasping action of the robot arm, the real-time deformation, displacement state and surface contact response of the object are continuously monitored through force sensors, visual feedback or depth changes, and once an abnormality (such as excessive deformation, sliding trend or contact force mutation) is detected, the current contact stiffness and grasping force are dynamically adjusted according to the feedback information, which avoids damage to the object or operation failure under the premise of ensuring the stability of grasping, so as to realize adaptive and safe grasping of different objects.

[0100] According to one embodiment of the present application, the feedback information obtained in real time based on the grasping of the object by the robot arm is combined with the environmental information to adjust the execution state of the robot arm, specifically: According to the planned path, the robot arm is moved to the grasping point of the object, and the pose of the end effector is adjusted to maintain the best contact angle with the surface of the object; The initial grasping parameters, including the contact stiffness and the grasping force, are set; The object is approached along the planned grasping path, and the grasping force is gradually increased while the deformation and surface state of the object are monitored, and the force feedback in the grasping process is sensed in real time through the force sensor; The joint torque of the robot arm, the contact force of the end effector and the deformation of the object are continuously monitored; The state change of the object during the grasping process is observed in real time; The difference between the actual grasping parameters and the pre-set parameters is compared to evaluate the grasping effect; When abnormal deformation of the object or excessive grasping force is detected, the grasping force is reduced; When it is detected that the smooth surface of the object leads to unstable grasping, the contact stiffness is increased; The motion speed and trajectory of the robot arm are dynamically adjusted according to the real-time feedback of the object.

[0101] As mentioned above, when the robot arm performs the grabbing operation, first, according to the planned motion path, the robot arm is moved to the preset grabbing point of the target object, and the posture of the end effector is adjusted to form an optimal contact angle with the surface of the object to improve the stability of grabbing; then, the initial grabbing parameters are set according to the characteristics of the object, including the contact stiffness and the grabbing force; the robot arm slowly approaches the object along the planned grabbing path, gradually increases the grabbing force during the contact process, and simultaneously senses the contact force feedback in the grabbing process in real time through the force sensor, and monitors the deformation degree and surface state change of the object in combination with the visual or depth information; in the whole grabbing process, the system continuously collects and analyzes the torque data of each joint of the robot arm, the contact force value of the end effector, and the deformation or displacement of the object, and simultaneously observes the state change of the object in real time through the visual system; the parameters obtained in the actual grabbing process are compared with the preset grabbing parameters to evaluate whether the current grabbing effect meets the requirements of stability and safety; if it is detected that the object abnormally deforms, the contact force exceeds the safety threshold, or there is a risk of falling, the grabbing force is immediately reduced to prevent damage; if it is judged that the object is unstable due to a smooth surface, the contact stiffness is appropriately increased to enhance the clamping compliance and friction; in addition, according to the above real-time feedback information, the system also dynamically adjusts the motion speed, acceleration and local trajectory of the robot arm, such as slowing down when approaching a fragile object and fine-tuning the path when detecting a deviation, so as to realize closed-loop optimization and adaptive control of the execution state, and ensure the reliability and safety of the grabbing operation.

[0102] According to an embodiment of the present application, the binocular camera and the depth sensor are used to scan the kitchen environment and identify the initial position and basic state of the object in the kitchen to obtain environmental information, which further comprises: Based on the light conditions, the processing parameters of the visual image are automatically adjusted by an adaptive light compensation algorithm.

[0103] As mentioned above, during the scanning of the kitchen environment, the light conditions of the current environment are detected in real time, including the light intensity, color temperature, and whether there is strong light reflection or shadow obstruction; based on the detection results, an adaptive light compensation algorithm is used to automatically adjust the processing parameters of the visual image collected by the binocular camera, such as exposure time, gain, contrast, and white balance; through this adjustment, the image distortion caused by uneven light, overexposure or underexposure is effectively suppressed, and the image clarity and color restoration in low light, backlight or high contrast scenes are improved; further, the accuracy of subsequent object recognition, edge detection and three-dimensional reconstruction is enhanced, and the environmental information obtained has higher reliability and consistency, providing high-quality visual input basis for constructing an accurate three-dimensional environmental model.

[0104] According to an embodiment of the present application, it further comprises: When the object is grabbed, the grabbing state is confirmed, it is verified that the object has been correctly grabbed, after verification, the mechanical arm performs subsequent operations according to the planned movement path; During the grabbing process, the object deformation data, contact force data, and mechanical arm joint state information are collected; The success rate of each grabbing operation, and the key indicators of object damage are recorded; The successful grabbing parameters of each time are recorded into the object characteristic parameter library; According to the actual grabbing effect of objects of different materials and shapes, the preset parameters are optimized and updated; Based on historical grabbing data, the grabbing parameter library is continuously optimized; When the grabbing fails, the failure reason is analyzed, the subsequent grabbing parameters are adjusted according to the failure reason, and the grabbing operation is re-executed.

[0105] As described above, after the object is grabbed, the system confirms the grabbing state, verifies whether the object has been correctly and stably grabbed through visual detection, force feedback and pose consistency judgment; after confirming that the grabbing is successful, the mechanical arm executes subsequent operations such as carrying or placing according to the pre-planned movement path; during the grabbing process, the system synchronously collects and records multi-dimensional data, including the deformation data of the object, the contact force data of the end effector, and the torque, angle and motion state information of each joint of the mechanical arm; at the same time, the system statistically analyzes the key performance indicators of each grabbing operation, including the grabbing success rate, whether the object is damaged, the number of slipping times, etc.; for the successfully completed grabbing operation, the grabbing parameters used (such as contact stiffness, grabbing force, clamping posture, etc.) are associated with the material, shape and state information of the corresponding object, and are stored in the object characteristic parameter library; based on the accumulated historical grabbing data, the system continuously analyzes the actual grabbing effect of objects of different materials, shapes or surface states, optimizes and updates the preset grabbing parameters, and improves the adaptability and precision of the parameter library; when the grabbing fails, the system automatically analyzes the failure reason, including insufficient grabbing force, mismatched contact stiffness, object slipping or environmental interference, etc., and dynamically adjusts the grabbing parameters for the next attempt, re-plans the grabbing action and executes it, forming a closed-loop learning and optimization mechanism, and continuously improving the self-adaptive grabbing capability of the mechanical arm in the complex kitchen environment.

[0106] The second aspect embodiment of the application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to realize the method in any of the embodiments of the first aspect.

[0107] Figure 2 An example of the physical structure diagram of an electronic device is shown in Figure 2As shown, the electronic device can include a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 complete mutual communication through the communication bus 840. The processor 810 can invoke the logical instructions in the memory 830 to execute the method in any of the embodiments of the first aspect described above, and the method includes: acquiring environment information in real time based on a binocular camera and a depth sensor to construct a three-dimensional environment model; decomposing the obtained user natural language instruction into a subtask sequence, and evaluating the influence of the subtask sequence on the environment information to obtain a decomposition evaluation result; planning a motion path of a mechanical arm based on the three-dimensional environment model and the decomposition evaluation result; determining a grasping parameter based on the characteristics of the object to adjust the contact stiffness and grasping strength of the mechanical arm; adjusting the execution state of the mechanical arm based on feedback information obtained in real time by the mechanical arm grasping the object, in combination with the environment information.

[0108] In addition, the logical instructions in the memory 830 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product stored in a storage medium includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0109] On the other hand, the present application also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, when the computer program is executed by a processor, the computer can execute the method provided by the above-mentioned method, and the method includes: acquiring environment information in real time based on a binocular camera and a depth sensor to construct a three-dimensional environment model; decomposing the obtained user natural language instruction into a subtask sequence, and evaluating the influence of the subtask sequence on the environment information to obtain a decomposition evaluation result; planning a motion path of a mechanical arm based on the three-dimensional environment model and the decomposition evaluation result; determining a grasping parameter based on the characteristics of the object to adjust the contact stiffness and grasping strength of the mechanical arm; Based on the feedback information obtained in real time by the mechanical arm grabbing the object, combined with the environmental information, the execution state of the mechanical arm is adjusted.

[0110] In another aspect, the application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the method provided by the above method, and the method comprises: Based on the real-time acquisition of environmental information by the binocular camera and the depth sensor, a three-dimensional environmental model is constructed; The obtained user natural language instruction is decomposed into a sub-task sequence, and the influence of the sub-task sequence on the environment information is evaluated to obtain a decomposition evaluation result; Based on the three-dimensional environmental model and the decomposition evaluation result, a motion path of the mechanical arm is planned; Based on the characteristics of the object, the grabbing parameters are determined to adjust the contact stiffness and the grabbing force of the mechanical arm; Based on the feedback information obtained in real time by the mechanical arm grabbing the object, combined with the environmental information, the execution state of the mechanical arm is adjusted.

[0111] Embodiment 2 I. Environment perception stage: multi-modal data fusion and adaptive processing Binocular camera and depth sensor data acquisition and fusion: The system obtains left and right view images through the binocular camera and calculates the parallax map:

[0112] Wherein, f=320 pixels is the camera focal length, B=0.12 meters is the binocular baseline, and Z(x,y) is the depth value.

[0113] The depth sensor (ToF type) obtains the depth map, and the original depth data is:

[0114] Wherein, c=3×10 8 m / s is the speed of light, and Δt is the light flight time.

[0115] The depth data and binocular image are fused through space-time alignment to obtain a dense point cloud:

[0116] Wherein, P(x,y) is the three-dimensional space point coordinate.

[0117] Adaptive light compensation algorithm: The system detects the environmental light condition in real time, calculates the average brightness value:

[0118] Where W=640, H=480 are the image resolutions, is the original image pixel value.

[0119] Automatically adjust parameters according to lighting conditions:

[0120] Where, is the expected brightness value.

[0121] Apply the compensation formula:

[0122] In this embodiment, the left region brightness is detected as 85, and the right region is 42. The system automatically calculates k=1.5 and c=12. After compensation, the right region brightness is raised to 128, effectively solving the uneven lighting problem.

[0123] Item recognition and state analysis: Calculate the geometric features of the tomato through point cloud data:

[0124] Where N=1500 is the number of tomato point clouds, and r is the average radius.

[0125] Recognize the tomato as "easy to deform" material (judged according to surface curvature and point cloud density):

[0126] Where κ=0.35 (threshold >0.25 is determined as easy to deform).

[0127] Finally, the three-dimensional coordinates of the tomato are determined as (x=0.75m, y=0.42m, z=0.81m), and the surface state is "wet and easy to deform".

[0128] II. Semantic understanding and task decomposition phase: VLN multi-modal large model application Instruction analysis and semantic feature extraction: The user instruction "put the tomato on the cutting board into the microwave" is parsed into key elements: Operation object: Tomato Source location: Cutting board Target location: Inside the microwave Cross-modal feature alignment through VLN multi-modal large model:

[0129] Where V is the visual feature vector (dimension 512), L is the language feature vector (dimension 512), and S is the spatial position feature (dimension 128), , 、 is the learnable weight matrix.

[0130] Task decomposition and feasibility assessment: Decompose the instruction into a sequence of subtasks: (1) Navigate to in front of the cutting board (2) Grasp the tomato (3) Move to the microwave (4) Open the microwave door (5) Place the tomato (6) Close the microwave door Assess the feasibility of each subtask based on the environmental model:

[0131] where ΔE is the environmental state change (such as the microwave door state change), is the environmental change adaptation factor.

[0132] Evaluation results: Subtask (2): Feasibility = 0.85 (tomato is easy to deform, need to dynamically adjust the grasping parameters); Subtask (4): Feasibility = 0.78 (microwave door is currently closed, need to operate).

[0133] Three, path planning and grasping parameter determination stage Mechanical arm path planning: Use improved A* algorithm to plan the path:

[0134] where g(n) is the actual cost from the starting point to the current point, h(n) is the heuristic estimate to the target point.

[0135] Optimize the heuristic function:

[0136] where α = 1.2, β = 0.8, is the distance to the target point, is the distance to the obstacle.

[0137] Convert path points to joint angles (inverse kinematics calculation):

[0138] where is the pseudo-inverse of the Jacobian matrix, v is the end effector speed.

[0139] Grasping parameter determination: Based on the item characteristic parameter library, the corresponding parameters of the tomato are: Contact stiffness: (Deformable object standard value) Grabbing force: (Recommended range 0.5-1.2N) After the system presets the initial parameters, the robotic arm starts executing the grabbing.

[0140] Four, Execution and Closed-loop Adjustment Phase Grabbing process control: The robotic arm approaches the tomato with a planned path, gradually increasing the grabbing force:

[0141] Where F(t) is the grabbing force at time t.

[0142] Real-time deformation monitoring:

[0143] Where δ(t) is the deformation, N=1500 is the number of point clouds.

[0144] When δ(t)>0.4mm (Deformable threshold) is detected, dynamically adjust the grabbing parameters:

[0145] Where L is the loss function, θ is the meta-network parameter.

[0146] Execution state adjustment: During the grabbing process, the system continuously monitors: Joint torque: (Shoulder, elbow, wrist, base) Contact force: (Stable value) Deformation: (Lower threshold) According to the monitoring data, the system confirms that the grabbing state is stable and executes the subsequent operation.

[0147] Five, Effect Verification and Parameter Optimization Phase Operation result recording and analysis: This operation is successfully completed, and the system records: Grabbing success rate: 100% Tomato deformation: maximum concave ; Contact force: stable at ; Joint state: no abnormal fluctuation; The system associates this parameter with the characteristics of the tomato: Material: easy to deform; Surface state: wet; Optimal parameters: K = 0.3 N / m, F = 0.6 N; Parameter library update and optimization: System updates the item characteristic parameter library:

[0148] Wherein, n = 10 is the historical tomato grabbing data.

[0149] Optimize similar item parameters:

[0150] Wherein, H is the percentage of environmental humidity (when humidity > 50%, the force is reduced by 10%).

[0151] In this embodiment, the environmental humidity is 65%, and the system automatically updates the tomato grabbing force range to 0.45-1.08N.

[0152] Failure processing mechanism: Suppose that the grabbing fails (tomato slips) this time, the system analyzes the failure reason:

[0153] Wherein, c is the failure reason category (insufficient force, stiffness mismatch, etc.).

[0154] According to the analysis result, the system adjusts the parameters and executes again: New parameters: , ; Re-execute the grabbing and complete successfully.

[0155] The places not mentioned in this application can be realized by using or referring to the existing technology.

[0156] Each embodiment in the specification is described in a progressive manner, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the different places from other embodiments.

[0157] The above only describes the embodiments of the present application and is not intended to limit the present application. Those skilled in the art can make various modifications and changes to the present application. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the scope of the claims of the present application.

Claims

1. A kitchen service robot operation method based on a VLN large model, characterized by, The method comprises the following steps: Real-time acquisition of environment information based on binocular camera and depth sensor to construct a three-dimensional environment model; Decompose the obtained user natural language instruction into a sub-task sequence, and evaluate the influence of the sub-task sequence on the environment information to obtain a decomposition evaluation result; Planning the motion path of the robot arm based on the three-dimensional environment model and the decomposition evaluation result; Determine the grasping parameters based on the characteristics of the object to adjust the contact stiffness and grasping force of the robot arm; Adjust the execution state of the robot arm based on the feedback information obtained in real time by the robot arm grasping the object, combined with the environment information.

2. The method of claim 1, wherein, The real-time acquisition of environment information based on binocular camera and depth sensor to construct a three-dimensional environment model is specifically: Based on binocular camera and depth sensor, scan the kitchen environment, identify the initial position and basic state of the objects in the kitchen, to obtain environment information, including obtaining the visual image and three-dimensional space data of the current environment, based on the environment information to construct a three-dimensional environment model.

3. The method of claim 1, wherein, The decomposition of the obtained user natural language instruction into a sub-task sequence, and the evaluation of the influence of the sub-task sequence on the environment information to obtain a decomposition evaluation result is specifically: Receive and analyze the user natural language instruction, identify the key elements in the user natural language instruction, including: operation object, position and target position; Understand the semantics and operation intention of the user natural language instruction based on the key elements, decompose the user natural language instruction into a sub-task sequence, and determine the specific operation steps and parameters of each sub-task sequence; Based on the environment information, evaluate the influence of environmental changes on the execution of the sub-task sequence to obtain the execution feasibility of each sub-task in the current environment.

4. The method of claim 1, wherein, The planning of the motion path of the robot arm based on the three-dimensional environment model and the decomposition evaluation result is specifically: Based on the environment model and the task decomposition result, determine the global path of the robot arm end effector from the current position to the target position; Use improved A* search algorithm or multi-agent path planning algorithm based on conflict search to ensure that the path is conflict-free and meets the basic constraint conditions; Convert the global path points to joint angle values, and use inverse kinematics method to calculate the joint angle corresponding to each path point; Use 3-5-3 segmented polynomial interpolation method to smooth the path points in joint space, ensuring continuous position, velocity and acceleration; Optimize the interpolation time of each segment to ensure the optimal running time of the robot arm while meeting the speed and acceleration constraint conditions; Based on the obstacle information in the three-dimensional environment model, adjust the planned path to avoid obstacles; Use the differential kinematics method based on Jacobian matrix to calculate the Cartesian repulsion velocity of the robot arm; According to the global path, obtain the initial local path of each joint of the robot arm; Use the improved particle swarm optimization algorithm to optimize the time required between adjacent path nodes in the initial local path of each joint to obtain the final local path.

5. The method of claim 1, wherein, The determination of grasping parameters based on the characteristics of the object to adjust the contact stiffness and grasping force of the robot arm is specifically: Combine the object classification information in the three-dimensional environment model to identify the material type of the object, and judge the structural characteristics of the object through the visual image; A pre-set grabbing parameter library is established for different materials, including standard contact stiffness value and recommended grabbing force range; During the grabbing process, the deformation and surface state of the object are monitored in real time, and the contact stiffness and grabbing force are dynamically adjusted according to the real-time feedback of the object.

6. The method of claim 1, wherein, The feedback information obtained in real time by the mechanical arm grabbing the object is combined with the environmental information to adjust the execution state of the mechanical arm, specifically: According to the planned path, the mechanical arm is moved to the grabbing point of the object, the posture of the end effector is adjusted, and the optimal contact angle with the object surface is maintained; The initial grabbing parameters, including contact stiffness and grabbing force, are set; The object is approached along the planned grabbing path, and the grabbing force is gradually increased while monitoring the deformation and surface state of the object, and the force feedback during the grabbing process is sensed in real time through the force sensor; The joint torque of the mechanical arm, the contact force of the end effector, and the deformation of the object are continuously monitored; The state change of the object during the grabbing process is observed in real time; The difference between the actual grabbing parameters and the pre-set parameters is compared to evaluate the grabbing effect; When abnormal deformation of the object or excessive grabbing force is detected, the grabbing force is reduced; When the smooth surface of the object is detected, which leads to unstable grabbing, the contact stiffness is increased; According to the real-time feedback of the object, the motion speed and trajectory of the mechanical arm are dynamically adjusted.

7. The method of claim 2, wherein, The binocular camera and depth sensor are used to scan the kitchen environment, identify the initial position and basic state of the object in the kitchen, and obtain environmental information, which also includes: Based on the lighting conditions, the processing parameters of the visual image are automatically adjusted through an adaptive lighting compensation algorithm.

8. The method of claim 6, wherein, It also includes: After the object is grabbed, the grabbing state is confirmed, and it is verified that the object has been correctly grabbed. After verification, the mechanical arm performs subsequent operations according to the planned movement path; During the grabbing process, the deformation data of the object, the contact force data, and the joint state information of the mechanical arm are collected; The success rate of each grabbing operation and the key indicators of object damage are recorded; The successful grabbing parameters of each time are recorded in the object characteristic parameter library; According to the actual grabbing effect of different materials and shapes of objects, the pre-set parameters are optimized and updated; Based on historical grabbing data, the grabbing parameter library is continuously optimized; When the grabbing fails, the failure reason is analyzed, and the subsequent grabbing parameters are adjusted according to the failure reason to re-execute the grabbing operation.

9. A computer-readable storage medium having stored thereon a program, characterized in that, The program is executed by the processor to implement the steps in the method of any one of claims 1-8.

10. An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the steps in the method of any one of claims 1-8.

Images