A coffee making robot control system and method based on voice instructions and VLA model

Abstract

The application provides a coffee making robot control system and method based on a voice instruction and a VLA model, converts a user natural language instruction into a structured task description through a voice interaction unit, decomposes the structured task description into an executable subtask sequence through a task arrangement unit, obtains three-dimensional pose information of key components in a coffee making scene through an environment perception unit, combines action suggestions output by a VLA strategy unit, plans and executes actions such as mobile navigation and mechanical arm operation through a hybrid control mode of model suggestions and rule constraints, and simultaneously performs constraint checking and processing of execution exceptions on the actions through a safety constraint and exception recovery unit, so as to realize closed-loop control of the whole coffee making process. The scheme of the application can improve the environment generalization capability, interaction friendliness and execution reliability of the coffee making robot, reduce the scene landing and maintenance cost, and realize the large-scale deployment of the coffee making robot.

Description

Technical Field

[0001] This invention relates to the field of service robot technology, specifically to a coffee-making robot control system and method based on voice commands and a VLA model. Background Technology

[0002] Coffee-making robots, as an important application scenario for service robots, are gradually moving from the laboratory to commercial use. Current coffee-making robots mostly adopt a technical solution of "fixed script + preset coordinates / jigs." This involves offline calibration of the coffee machine, cup, and button positions, allowing them to perform grasping, placement, and pressing operations along a fixed trajectory when executing coffee-making tasks. While this solution can achieve basic coffee-making functions in highly structured, fixed scenarios, its technical limitations are significant, making it difficult to meet the diverse needs of real-world commercial applications.

[0003] The core problems of existing technologies are mainly reflected in four aspects: First, poor generalization ability. Even slight changes in coffee machine models, button layouts, coffee spout positions, or table placement require engineers to recalibrate the robot or rewrite the execution script. It cannot adapt to different brands and models of coffee machines, nor can it cope with minor environmental changes in the scenario. Second, unfriendly human-computer interaction. Users cannot flexibly issue coffee-making instructions through natural language and can only select from the robot's preset fixed menu. The interaction is monotonous, resulting in a poor user experience. Third, lack of closed-loop control and safety constraints. The robot only relies on preset trajectories to execute actions. If there are problems such as cup deviation, grasping failure, robotic arm collision, or button mispressing, it cannot autonomously identify and correct them, which can easily lead to task failure or even safety risks such as equipment collisions and coffee spills. Fourth, high deployment and maintenance costs. Each deployment point requires professional engineers for on-site debugging, calibration, and subsequent maintenance, resulting in high labor costs and making it impossible to achieve large-scale replication and rapid deployment.

[0004] To address the aforementioned issues, there is an urgent need for a coffee-making robot control system and method that can adapt to unstructured coffee-making environments, support natural language interaction, and possess closed-loop control capabilities for perception, decision-making, and execution. By integrating voice interaction, environmental perception, VLA models, and safety constraint technologies, the robot's environmental adaptability, interactivity, and execution reliability can be improved, while reducing deployment and maintenance costs and promoting the commercial-scale development of coffee-making robots. Summary of the Invention

[0005] This invention provides a coffee-making robot control system and method based on voice commands and VLA models, aiming to solve the technical problems of poor generalization ability, unnatural interaction, low execution reliability, and high deployment cost of existing coffee-making robots. It improves the robot's adaptability to weakly structured environments, realizes closed-loop control of the entire coffee-making process driven by natural language, and reduces the deployment and maintenance costs of the scene.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0007] The first aspect of this invention provides a coffee-making robot control system based on voice commands and a VLA model, comprising:

[0008] The voice interaction unit is used to complete voice wake-up, automatic speech recognition and intent parsing, and convert the user's natural language commands into structured task descriptions;

[0009] The task orchestration unit is used to decompose the structured task description into a sequence of executable subtasks, and to implement the precondition check, step completion determination and abnormal branch handling of the subtasks through the task state machine.

[0010] The environmental perception unit, based on visual sensors and force / tactile sensors, detects and estimates the three-dimensional pose and operable area of ​​the cup and key components of the coffee machine in the coffee making scene.

[0011] The mobile navigation unit, based on SLAM localization and path planning, moves the robot to the working pose near the coffee machine and completes precise alignment.

[0012] The robotic arm and end effector control unit is used for trajectory planning and execution of actions such as grasping, placing, and pressing during the coffee making process;

[0013] VLA strategy unit takes language commands, real-time visual observations and robot body state as input, and outputs high-level actions or continuous control increments.

[0014] The safety constraint and anomaly recovery unit performs constraint verification on the action suggestions output by the VLA strategy unit and the trajectory planned by the mobile navigation and robotic arm control unit, and triggers anomaly recovery operation when the action execution fails or a risk occurs; the VLA strategy unit and the task orchestration unit adopt a hybrid control of model suggestion + rule constraint, and the safety constraint and anomaly recovery unit filters the action suggestions output by the VLA strategy unit before they are executed by the corresponding execution unit.

[0015] As a further improvement to the technical solution of the present invention, the safety constraint and anomaly recovery unit filters candidate actions through a cost function, wherein the cost function is:

[0016] ,

[0017] Where d_pose represents the pose error from the end point to the center of the target operable area, r_collision represents the collision risk, r_force represents the expected contact force exceeding the limit risk, t_exec represents the expected execution time; w1~w4 are weight coefficients, and the system selects the action with the smallest J that satisfies the constraints to execute.

[0018] As a further improvement to the technical solution of the present invention, the visual sensor is an RGB-D camera or a multi-view camera, and the operable area of ​​the environmental perception unit includes a cup grasping area, a coffee machine button area, and a cup holding area. The operable area is represented by a planar area / voxel set or parameterized area in three-dimensional space, and includes normal, allowable contact force range, and preferred contact point attributes.

[0019] As a further improvement to the technical solution of the present invention, the states of the task state machine include at least WAIT_CMD, NAV_TO_MACHINE, PICK_CUP, PLACE_CUP_UNDER_SPOUT, PRESS_BUTTON, WAIT_BREW_FINISH, PICK_UP_CUP, PLACE_CUP_TO_TARGET, DONE, and ERROR. Each state includes an entry condition, an execution action, a completion condition, and an exception transfer rule.

[0020] As a further improvement to the technical solution of the present invention, the input of the VLA strategy unit includes the current image I, the language command L, and the robot state s; the output is action a, which includes one or more of the following: end-effector pose increment Δx, gripper command g, chassis micro-motion command v, and pressing duration τ.

[0021] As a further improvement to the technical solution of the present invention, the anomaly recovery operation includes re-observation, adjusting pose, limiting the number of retries, and reverting to a safe position. When the anomaly exceeds a preset threshold, a user prompt or a user confirmation request is triggered.

[0022] A second aspect of the present invention provides a coffee-making robot control method based on voice commands and a VLA model, applied to the aforementioned coffee-making robot control system based on voice commands and a VLA model, comprising the following steps:

[0023] S1: Receives user voice commands through the voice interaction unit, completes automatic speech recognition and intent parsing, and obtains the target task and optional parameters;

[0024] S2: The target task is decomposed into a sub-task sequence by the task orchestration unit. The sub-task sequence includes at least arriving at the coffee machine work position, taking a cup, placing the cup in front of the coffee spout, triggering coffee dispensing, retrieving the cup, and sending the cup to the designated position.

[0025] S3: The robot is moved to the vicinity of the coffee machine by the mobile navigation unit based on SLAM localization and path planning, and completes precise alignment at the work station;

[0026] S4: Detect the three-dimensional pose of the cup and key components of the coffee machine through the environmental perception unit, and generate candidate actions and operable areas for grasping, placing, and pressing;

[0027] S5: During the execution of each subtask, the VLA strategy unit outputs action suggestions, which are then checked and corrected by the safety constraint and anomaly recovery unit before being executed by the corresponding execution unit. Visual / force feedback is used to determine whether the subtask has been completed.

[0028] S6: If the action fails or a security risk occurs, trigger the exception branch through the security constraint and exception recovery unit to perform the exception recovery operation;

[0029] S7: After all subtasks are completed, place the coffee cup in the designated area and announce it via voice. At the same time, record the execution data for the robot's continuous learning and strategy optimization.

[0030] As a further improvement to the technical solution of the present invention, step S5 includes determining the completion of the subtask through visual / force feedback: the grasping action confirms successful grasping through force feedback and visual observation; the placement action aligns the cup to the placement center through visual servo closed loop and confirms the stability of the cup; the pressing action determines whether the coffee machine button is triggered through force / displacement threshold; and the coffee dispensing action determines the completion of the dispensing action by visually detecting liquid flow / changes in the liquid level in the cup or recognizing the coffee machine indicator light / sound.

[0031] A third aspect of the present invention provides a computer device comprising a memory and a processor, the memory storing computer program code, and the processor being configured to acquire the computer program code and execute the coffee-making robot control method based on voice commands and a VLA model as described above.

[0032] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the coffee-making robot control method based on voice commands and a VLA model as described above.

[0033] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0034] This invention relates to a coffee-making robot control system based on voice commands and a VLA model. By integrating seven functional units—voice interaction, task orchestration, environmental perception, mobile navigation, robotic arm control, VLA strategy, and safety constraints and anomaly recovery—it constructs a fully closed-loop control system encompassing perception, decision-making, and execution. The VLA strategy unit and task orchestration unit employ a hybrid control approach of "model suggestion + rule constraint." First, the VLA strategy unit combines voice commands, real-time visual observation, and the robot's own state to output flexible action suggestions. Then, the safety constraint and anomaly recovery unit performs multi-dimensional constraint verification before execution. This effectively solves the technical problems of traditional coffee-making robots, such as poor generalization ability, unfriendly human-machine interaction, and lack of closed-loop execution and safety constraints. This system eliminates the need for offline calibration of coffee machines, adapting to different models, button layouts, and scene placement errors. This significantly improves its adaptability to unstructured coffee-making environments. It also supports flexible user commands via natural language, enabling autonomous execution of the entire coffee-making process. Furthermore, by verifying the accessibility, collision, and force / torque thresholds of action suggestions and planned trajectories, and by promptly handling execution anomalies, the system reduces the failure rate of robot operations and effectively avoids safety risks such as collisions and force over-limits. The collaborative working mode of each unit also significantly reduces on-site debugging and maintenance workload, lowering the cost of scenario deployment and providing reliable system support for the large-scale deployment and commercial application of coffee-making robots. Attached Figure Description

[0035] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0036] Figure 1 This is a general structural block diagram of a coffee-making robot control system according to some embodiments of the present invention;

[0037] Figure 2 This is an exemplary flowchart of a coffee-making robot control method according to some embodiments of the present invention;

[0038] Figure 3 This is a schematic diagram of the structure of a computer device for implementing a coffee-making robot control method according to some embodiments of the present invention. Detailed Implementation

[0039] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0040] The present invention will be further described in detail below with reference to the accompanying drawings.

[0041] Reference Figure 1 In a first aspect, this embodiment provides a coffee-making robot control system based on voice commands and a VLA model, including:

[0042] The voice interaction unit is used to complete voice wake-up, automatic speech recognition and intent parsing, and convert the user's natural language commands into structured task descriptions;

[0043] The task orchestration unit is used to decompose the structured task description into a sequence of executable subtasks, and to implement the precondition check, step completion determination and abnormal branch handling of the subtasks through the task state machine.

[0044] The environmental perception unit, based on visual sensors and force / tactile sensors, detects and estimates the three-dimensional pose and operable area of ​​the cup and key components of the coffee machine in the coffee making scene.

[0045] The mobile navigation unit, based on SLAM localization and path planning, moves the robot to the working pose near the coffee machine and completes precise alignment.

[0046] The robotic arm and end effector control unit is used for trajectory planning and execution of actions such as grasping, placing, and pressing during the coffee making process;

[0047] VLA strategy unit takes language commands, real-time visual observations and robot body state as input, and outputs high-level actions or continuous control increments.

[0048] The safety constraint and anomaly recovery unit performs constraint verification on the action suggestions output by the VLA strategy unit and the trajectory planned by the mobile navigation and robotic arm control unit, and triggers anomaly recovery operation when the action execution fails or a risk occurs; the VLA strategy unit and the task orchestration unit adopt a hybrid control of model suggestion + rule constraint, and the safety constraint and anomaly recovery unit filters the action suggestions output by the VLA strategy unit before they are executed by the corresponding execution unit.

[0049] This invention constructs a fully closed-loop control system through the collaborative work of seven functional units, fundamentally solving the technical problems of traditional coffee-making robots, such as poor generalization ability, unfriendly human-machine interaction, lack of closed-loop and safety constraints in execution, and high deployment costs. The hybrid control mode balances the flexibility of action decision-making with the standardization of execution, eliminating the need for offline calibration of coffee machines. It is adaptable to different coffee machine models, button layouts, and placement errors in various scenarios, significantly improving adaptability to unstructured coffee-making environments. It supports flexible issuance of user natural language commands, enhancing the friendliness of human-machine interaction and user experience. Multi-dimensional constraint verification enables safe control of the robot's operation process, and combined with anomaly recovery operations, it effectively reduces the task failure rate and avoids safety risks such as collisions and force over-limits. The standardized collaborative working mode of each unit reduces the workload of on-site debugging and maintenance, lowers deployment costs, and provides reliable system support for the large-scale deployment and commercialization of coffee-making robots.

[0050] In some embodiments, the safety constraint and anomaly recovery unit filters candidate actions using a cost function, wherein the cost function is:

[0051] ,

[0052] Where d_pose represents the pose error from the end point to the center of the target operable area, r_collision represents the collision risk, r_force represents the expected contact force exceeding the limit risk, t_exec represents the expected execution time; w1~w4 are weight coefficients, and the system selects the action with the smallest J that satisfies the constraints to execute.

[0053] It should be noted that the safety constraint and anomaly recovery unit introduces a cost function when screening candidate actions. Where d_pose is the pose error from the robot's end effector to the center of the target operable area, r_collision is the collision risk during action execution, r_force is the expected contact force exceeding the limit risk during action execution, t_exec is the expected execution time of the action, and w1 to w4 are weight coefficients set according to the actual coffee making scenario requirements. The system calculates the cost function value for all candidate actions and selects the action with the smallest cost function value that meets all constraints such as reachability, collision, force / torque thresholds as the execution action.

[0054] This invention uses a cost function to quantitatively screen candidate actions, achieving multi-dimensional quantitative evaluation and optimal selection of robot actions. It incorporates key performance indicators such as pose accuracy, collision risk, force over-limit risk, and execution efficiency into a unified evaluation system, avoiding action execution defects caused by single-indicator selection. The weighting coefficients can be flexibly adjusted according to scenario requirements, enabling the action selection strategy to adapt to different coffee-making scenarios and operational requirements. This further improves the accuracy and rationality of action execution, effectively reducing the probability of task failure due to improper action selection, while simultaneously considering both operational efficiency and safety.

[0055] In some embodiments, the visual sensor is an RGB-D camera or a multi-view camera, and the operable area of ​​the environmental perception unit includes a cup gripping area, a coffee machine button area, and a cup holding area. The operable area is represented by a planar region / voxel set or a parameterized region in three-dimensional space, and includes normal, allowable contact force range, and preferred contact point attributes.

[0056] It should be noted that the environmental perception unit uses an RGB-D camera or multi-view camera as a visual sensor, combined with a force / tactile sensor to complete scene perception. The identified operable areas cover key operational areas in coffee making, such as the cup gripping area, coffee machine button area, and cup holder area. This operable area is digitally represented using a planar region, voxel set, or parametric region in three-dimensional space. Simultaneously, each operable area is configured with attribute information such as normal, allowable contact force range, and preferred contact point, providing precise spatial and operational parameters for the robotic arm's motion planning and execution.

[0057] Employing RGB-D cameras or multi-view cameras ensures the depth and accuracy of visual perception. Combining force / tactile sensors enables multimodal perception fusion, improving the accuracy of key component pose detection and operable area identification. Digital representation of the operable area and configuration of multi-dimensional attribute information provide precise and detailed parameter support for robot motion planning and execution. This allows the robotic arm to accurately match the spatial characteristics and operational requirements of the work area, reducing alignment errors and improving the execution accuracy of actions such as grasping, placing, and pressing. Simultaneously, it provides a reference for controlling contact force, preventing equipment damage or task failure due to missing operational parameters.

[0058] In some embodiments, the states of the task state machine include at least WAIT_CMD, NAV_TO_MACHINE, PICK_CUP, PLACE_CUP_UNDER_SPOUT, PRESS_BUTTON, WAIT_BREW_FINISH, PICK_UP_CUP, PLACE_CUP_TO_TARGET, DONE, and ERROR. Each state includes an entry condition, an execution action, a completion condition, and an exception transfer rule.

[0059] It should be noted that the task state machine of the task orchestration unit sets at least ten states: WAIT_CMD, NAV_TO_MACHINE, PICK_CUP, PLACE_CUP_UNDER_SPOUT, PRESS_BUTTON, WAIT_BREW_FINISH, PICK_UP_CUP, PLACE_CUP_TO_TARGET, DONE, and ERROR. Each state is configured with explicit entry conditions, execution actions, completion conditions, and exception transition rules. During the execution of the coffee-making task, the task state machine, based on the execution feedback from each unit, achieves orderly transitions between different states according to preset rules. When the completion condition of a certain state is not met and an exception rule is triggered, the state machine transitions to the ERROR state and initiates the exception handling process.

[0060] By breaking down the entire coffee-making process into standardized state nodes and configuring comprehensive flow rules for each node, refined and standardized control over coffee-making tasks is achieved, making the task execution logic clearer and the process more standardized. Clear exception transfer rules enable rapid identification and response to task execution anomalies, ensuring that the system can promptly trigger processing procedures when anomalies occur. This avoids task delays or disordered execution due to a lack of process control, improving the continuity and reliability of task execution. It also provides clear node references for system troubleshooting and maintenance.

[0061] In some embodiments, the input of the VLA strategy unit includes the current image I, the language command L, and the robot state s; the output is an action a, which includes one or more of the following: end-effector pose increment Δx, gripper command g, chassis micro-motion command v, and pressing duration τ.

[0062] It should be noted that the VLA strategy unit constructs an end-to-end decision model encompassing vision, language, and action. The model input includes the current image I provided by the environment perception unit, the language command L provided by the voice interaction unit, and the real-time state s of the robot body. The robot body state covers key parameters such as the joint state of the robotic arm, the end effector pose, and the position of the mobile chassis. The model output is the robot's action a, which includes one or more of the following: end effector pose increment Δx, gripper command g, chassis micro-motion command v, and pressing duration τ. This provides direct control increments or high-level action suggestions for the robot's action execution.

[0063] The end-to-end VLA decision model achieves the fusion processing of multimodal information from vision, language, and robot state. It can directly output action commands based on multimodal inputs, simplifying the intermediate links between information processing and action decision-making, and improving decision-making efficiency and real-time performance. The diverse output action parameters can accurately match different action requirements in the coffee-making process, such as grasping, placing, pressing, and chassis fine-tuning, providing precise control basis for the robot's refined operation. This enables the robot to output adaptive action suggestions based on real-time changes in the scene, improving its adaptability to unstructured environments and the flexibility of action decision-making.

[0064] In some embodiments, the anomaly recovery operation includes re-observation, adjusting pose, limiting the number of retries, and reverting to a safe position. When the anomaly exceeds a preset threshold, a user prompt or user confirmation request is triggered.

[0065] It should be noted that the safety constraint and anomaly recovery unit has preset multiple anomaly recovery operations, including re-observing the poses of key components in the scene, adjusting the poses of the robot's end effector or chassis, setting a preset retry threshold for the abnormal operation, and withdrawing the robot to a safe position. When the robot's action fails or a safety risk occurs, the system triggers the corresponding recovery operation according to the anomaly type. If the problem is not resolved after the preset retry count for the abnormal operation reaches the threshold, the system triggers a user prompt or requests user confirmation for user intervention.

[0066] Corresponding recovery operations are configured for common anomalies in the coffee-making process, enabling targeted and automated handling of anomalies without real-time human intervention, thus enhancing the system's autonomous fault tolerance. Setting a retry threshold avoids unnecessary repetitive operations, reducing waste of system resources. Simultaneously, timely user prompts are triggered when automated processing fails, effectively connecting automated processing with human intervention. This ensures the system's autonomous operation while preventing equipment damage or safety issues caused by unresolved anomalies, further improving the system's safety and reliability.

[0067] Reference Figure 2The second aspect of the present invention provides a coffee-making robot control method based on voice commands and a VLA model, applied to the aforementioned coffee-making robot control system based on voice commands and a VLA model, comprising the following steps:

[0068] S1: Receives user voice commands through the voice interaction unit, completes automatic speech recognition and intent parsing, and obtains the target task and optional parameters;

[0069] S2: The target task is decomposed into a sub-task sequence by the task orchestration unit. The sub-task sequence includes at least arriving at the coffee machine work position, taking a cup, placing the cup in front of the coffee spout, triggering coffee dispensing, retrieving the cup, and sending the cup to the designated position.

[0070] S3: The robot is moved to the vicinity of the coffee machine by the mobile navigation unit based on SLAM localization and path planning, and completes precise alignment at the work station;

[0071] S4: Detect the three-dimensional pose of the cup and key components of the coffee machine through the environmental perception unit, and generate candidate actions and operable areas for grasping, placing, and pressing;

[0072] S5: During the execution of each subtask, the VLA strategy unit outputs action suggestions, which are then checked and corrected by the safety constraint and anomaly recovery unit before being executed by the corresponding execution unit. Visual / force feedback is used to determine whether the subtask has been completed.

[0073] S6: If the action fails or a security risk occurs, trigger the exception branch through the security constraint and exception recovery unit to perform the exception recovery operation;

[0074] S7: After all subtasks are completed, place the coffee cup in the designated area and announce it via voice. At the same time, record the execution data for the robot's continuous learning and strategy optimization.

[0075] In practice, the execution flow is as follows: First, the voice interaction unit receives user voice commands, performs automatic speech recognition and intent parsing, and obtains the target task and optional parameters. Second, the task orchestration unit decomposes the target task into a sequence of sub-tasks, including at least reaching the coffee machine workstation, retrieving the cup, placing the cup in front of the coffee spout, dispensing coffee, retrieving the cup, and delivering the cup to the designated location. Next, the mobile navigation unit moves the robot to the vicinity of the coffee machine based on SLAM localization and path planning, and performs precise alignment at the workstation. Finally, the environmental perception unit detects the 3D poses of the cup and key components of the coffee machine, generating... Candidate actions and operable areas for grasping, placing, and pressing; during the execution of each subtask, the VLA strategy unit outputs action suggestions, which are then verified and corrected by the safety constraint and anomaly recovery unit before being executed by the corresponding execution unit. Visual / force feedback is used to determine whether the subtask is completed; if the action fails or a safety risk occurs, the safety constraint and anomaly recovery unit triggers an anomaly branch and executes the corresponding anomaly recovery operation; after all subtasks are completed, the coffee cup is placed in the designated area and a voice announcement is made, while the execution data of this task is recorded for the robot's continuous learning and strategy optimization.

[0076] The control method of this invention achieves standardized and automated execution of the entire coffee-making process based on the seven functional units of the control system. It breaks down the complex coffee-making task into ordered sub-tasks, and combines closed-loop perception, decision-making, execution, and verification processes to achieve precise control over each step. Visual / force feedback completion judgment improves the accuracy of sub-task execution, and the setting of abnormal branches enables timely handling of execution anomalies, ensuring the smooth progress of the entire process. The recording and learning of task execution data provides data support for robot strategy optimization, enabling continuous improvement of the robot's operational capabilities, further reducing the task failure rate, and increasing operational efficiency. At the same time, the standardized execution process facilitates the promotion and reuse of the system, promoting the large-scale application of coffee-making robots.

[0077] In some embodiments, the completion of the subtask in step S5 by visual / force feedback includes: the grasping action confirms successful grasping through force feedback and visual observation; the placement action aligns the cup to the placement center and confirms the cup is stable through visual servo closed loop; the pressing action determines whether the coffee machine button is triggered through force / displacement threshold; and the coffee dispensing action determines completion by visually detecting liquid flow / changes in the liquid level in the cup or recognizing the coffee machine indicator light / sound.

[0078] It should be noted that during the execution of subtasks, the completion determination of subtasks is completed through a multimodal fusion method of visual / force feedback: when the grasping action is executed, the gripping force of the gripper is detected by force / tactile sensors, and at the same time, the real-time observation of the visual sensor is combined to double confirm whether the cup is stably grasped; when the placing action is executed, the visual servo algorithm achieves closed-loop alignment, and after the cup is placed in the designated position, the visual observation is used to confirm that the cup is placed stably and without deviation; when the pressing action is executed, the threshold change of force / displacement is detected to determine whether the coffee machine button is successfully triggered; when the coffee dispensing action is executed, the liquid flow at the coffee spout or the liquid level change in the cup is detected in real time by the visual sensor, or the indicator lights, sounds and other signals of the coffee machine are identified to determine whether the coffee making is complete.

[0079] This invention employs a fusion of visual and force feedback for subtask completion determination, overcoming the limitations of single feedback methods and improving the accuracy and reliability of the determination results. It avoids misjudgments caused by errors in single feedback information. Targeted determination methods are designed for the operational characteristics of different subtasks, ensuring a high degree of match between the determination criteria and the execution requirements of the subtasks. This allows for precise identification of the subtask's execution status, timely detection of execution deviations, and triggering corrective actions. This effectively reduces the probability of subsequent task failures due to determination errors, ensuring the execution accuracy and consistency of the entire coffee-making process.

[0080] Reference Figure 3 A third aspect of this invention provides a computer device, comprising a memory and a processor. The memory stores computer program code, and the processor is configured to acquire the computer program code and execute the aforementioned control method for a coffee-making robot based on voice commands and a VLA model. This invention achieves programmatic execution of the control method through the memory and processor of the computer device, enabling the control logic of the coffee-making robot to be stored and run in standardized program code form. This improves the stability and portability of the control logic, facilitating system upgrades and maintenance. The processor's rapid execution capability of the program code ensures the real-time issuance and execution of control commands, enabling rapid response to scene changes and abnormal situations during the coffee-making process, thus improving the robot's real-time performance and response efficiency. Simultaneously, the hardware support of the computer device provides computational power for multimodal information processing, VLA model inference, and complex trajectory planning, ensuring the smooth implementation of various functions of the control system.

[0081] In some embodiments, the coffee-making robot control method based on voice commands and VLA models described above can be implemented by a computer device, which includes at least one processor, a communication bus, a memory, and at least one communication interface.

[0082] A processor can be a general-purpose central processing unit (CPU) or an application-specific integrated circuit (ASIC).

[0083] A communication bus can be used to transmit information between the aforementioned components.

[0084] The memory can be read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), magnetic disks or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited to these. The memory can exist independently and be connected to the processor via a communication bus. The memory can also be integrated with the processor.

[0085] The memory stores the program code for executing the solution of this application, and its execution is controlled by the processor. The processor executes the program code stored in the memory. The program code may include one or more software modules. In the above embodiments, the coffee-making robot control method based on voice commands and a VLA model can be implemented by the processor and one or more software modules in the program code in the memory.

[0086] A communication interface is a device that uses any transceiver or similar device to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0087] In a specific implementation, as one example, a computer device may include multiple processors, each of which may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. Here, a processor may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0088] The aforementioned computer device can be a general-purpose computer device or a special-purpose computer device. In specific implementations, the computer device can be a desktop computer, a portable computer, a network server, a handheld digital assistant (PDA), a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. This application does not limit the type of computer device.

[0089] Fourthly, this invention provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the aforementioned control method for a coffee-making robot based on voice commands and a VLA model. Storing the control method for the coffee-making robot as a computer program on the computer-readable storage medium enables digital storage and convenient transmission of the control method, facilitating the installation, updating, and copying of the control program, and reducing system deployment and maintenance costs. The adaptability to various storage media formats enhances the system's flexibility, allowing selection of a suitable storage medium based on the coffee-making robot's hardware configuration. Simultaneously, the standardized execution of the computer program ensures the accuracy and consistency of the control method's execution, avoiding errors from manual operation and ensuring that the coffee-making robot can complete its tasks according to a unified standard in different deployment scenarios, thus improving the system's standardization and scalability.

[0090] To provide a clearer understanding of the invention, the invention is further described below:

[0091] The coffee-making robot control system and method based on voice commands and VLA models of the present invention are applicable to various coffee-making service robots and can be deployed in weakly structured coffee-making scenarios such as offices, convenience stores, coffee shops, and office building lobbies. It is compatible with different brands and models of commercial coffee machines and realizes autonomous execution of the entire coffee-making process driven by natural language.

[0092] refer to Figure 1The figure is a general structural block diagram of a coffee-making robot control system according to some embodiments of the present invention. The control system mainly includes a voice interaction unit, a task scheduling unit, an environmental perception unit, a mobile navigation unit, a robotic arm and end effector control unit, a VLA strategy unit, and a safety constraint and anomaly recovery unit. The units work together to achieve closed-loop control of the entire coffee-making task.

[0093] In some embodiments, the hardware carrier of the control system is a coffee-making robot platform, which includes a mobile chassis, a six-degree-of-freedom robotic arm, an end effector, a vision sensor, a force / torque sensor, a microphone array, and a computing unit. The end effector can be a two-finger gripper, a suction-grip combination gripper, or the like. The vision sensor is an RGB-D camera (which can be deployed on the robot's head or the wrist of the robotic arm). The computing unit is equipped with a GPU to provide edge computing power to meet the computing power requirements for VLA model inference, environmental perception, and trajectory planning.

[0094] I. Specific Implementation of Each Unit of the Control System

[0095] Voice interaction unit

[0096] This unit integrates a microphone array, a voice wake-up module, an automatic speech recognition (ASR) module, and a natural language understanding (NLU) module. The microphone array performs speech noise reduction and sound source localization. The voice wake-up module triggers the robot's interactive functions through a preset wake-up word. The ASR module converts the user's natural language commands into text information. The NLU module performs intent parsing and parameter extraction on the text information, ultimately converting the user's commands into a structured task description. For example, if the user says "Make me a cup of coffee," the unit parses it and obtains the target task as "MAKE_COFFEE," without any additional parameters. If the user says "Make me a cup of hot coffee," then the parameter "temperature: hot" is added to the structured task description.

[0097] Task orchestration unit

[0098] The core of this unit is the task state machine, whose states include at least WAIT_CMD (waiting for instructions), NAV_TO_MACHINE (navigating to the coffee machine), PICK_CUP (taking the cup), PLACE_CUP_UNDER_SPOUT (placing the cup under the coffee spout), PRESS_BUTTON (pressing the button), WAIT_BREW_FINISH (waiting for the coffee to finish making), PICK_UP_CUP (retrieving the coffee cup), PLACE_CUP_TO_TARGET (placing the cup in the designated position), DONE (task completed), and ERROR (error).

[0099] Each state includes clear entry conditions, execution actions, completion conditions, and exception transfer rules. For example, the entry condition for the PICK_CUP state is that the robot has reached the coffee machine work position and completed precise alignment. The execution action is that the environmental perception unit detects the cup pose, the VLA model outputs the grasping action, and the robotic arm performs the grasping. The completion condition is that the grasping is confirmed to be successful through force feedback and visual observation. The exception transfer rule is that if the grasping fails more than 3 times, it will transfer to the ERROR state and trigger a user prompt.

[0100] This unit decomposes the structured task description into a sequence of subtasks to be executed sequentially, and uses a task state machine to realize the flow and control of subtasks. At the same time, it sends task execution instructions to other units and receives execution status feedback from each unit.

[0101] Environmental sensing unit

[0102] This unit achieves scene perception based on an RGB-D camera (visual sensor) and wrist / gripper force / torque sensors (force / tactile sensors). First, the RGB-D camera acquires color and depth images of the coffee-making scene. Through target detection and pose estimation algorithms, it detects and estimates the 3D poses of key components such as the cup, coffee machine spout, buttons / knobs, etc. Second, based on the detected 3D poses, it generates operable areas such as the cup gripping area, coffee machine button area, and cup holding area. These operable areas are represented as planar regions in 3D space and include attributes such as normal, allowable contact force range, and preferred contact point. Finally, the 3D pose information of key components and operable area information are sent in real time to the VLA strategy unit, robotic arm, and end effector control unit to provide data support for motion decision-making and trajectory planning.

[0103] Mobile navigation unit

[0104] This unit uses SLAM (Simultaneous Localization and Mapping) algorithm to realize robot localization and map building. According to the instructions of the task orchestration unit and combined with the constructed scene map, the optimal path from the robot's current position to the work position near the coffee machine is planned through the path planning algorithm, and the mobile chassis is controlled to move along the planned path. When the robot arrives near the work position, it performs centimeter-level precision alignment based on the visual detection results of the environmental perception unit to ensure that the working space of the robotic arm covers the operable area of ​​the coffee machine, providing a positional basis for subsequent robotic arm operations.

[0105] Robotic arm and end effector control unit

[0106] This unit receives action execution instructions from the task orchestration unit and action suggestions from the VLA strategy unit. Combined with the operable area information provided by the environmental perception unit, it performs inverse kinematics solution and trajectory planning for the robotic arm, generating motion trajectories for each joint of the robotic arm and action instructions for the end effector (such as the opening and closing of the gripper, and the force and duration of pressing). At the same time, this unit collects the joint status, end-effector pose, and feedback information from the force / tactile sensors of the robotic arm in real time, achieving precise control of the robotic arm's movements and completing key coffee-making actions such as grasping, placing, and pressing.

[0107] VLA Strategy Unit

[0108] This unit is the decision-making core of the control system. It uses a vision-language-action (VLA) model to achieve end-to-end action decision-making. The model input includes three parts: the current image I (provided by the RGB-D camera of the environment perception unit), the language command L (a structured task description provided by the voice interaction unit), and the robot body state s (including the joint state of the robotic arm, the end effector pose, the position of the mobile chassis, etc.). The model output is the action a, including the end effector pose increment Δx, the gripper command g, the chassis micro-motion command v, the pressing duration τ, and other continuous control increments or high-level action suggestions.

[0109] The VLA model of this unit can be implemented through offline training and online inference. In the offline training stage, a multimodal dataset (images, language, and actions) of the coffee making scenario is used to train the model. In the online inference stage, action suggestions are output based on real-time input and output, and the action suggestions are sent to the safety constraint and anomaly recovery unit for filtering.

[0110] Safety constraints and anomaly recovery unit

[0111] This unit is the core of the control system's safety assurance, and it is divided into two parts: constraint verification and anomaly recovery.

[0112] Constraint Verification: The motion suggestions output by the VLA strategy unit and the trajectory planned by the mobile navigation and robotic arm control unit are verified using multi-dimensional constraints including reachability, collision, force / torque thresholds, and stability; specifically through a cost function. Candidate actions are screened, where w1, w2, w3, and w4 are weight coefficients that can be adjusted according to the actual scenario requirements. The system selects the action with the smallest cost function J value that satisfies all constraints for execution. If a candidate action does not meet the constraints, it is corrected, the action is projected to the feasible set, and then output to the execution unit.

[0113] Anomaly Recovery: When a task failure or safety risk feedback is received from the execution unit (such as grasping failure, placement deviation, button not triggered, collision / force exceeding limit), the corresponding anomaly recovery operation is triggered. If grasping fails, the system reverts to a safe position, re-observes the cup's pose, and changes the candidate grasping pose. If placement deviation occurs, end-effector fine-tuning or re-grabbing and placing is performed. If button is not triggered, the contact point is repositioned, and the pressing direction and duration are adjusted. If collision / force exceeds limit, a soft emergency stop is triggered, and the system reverts, recording the collision information. All anomaly operations have a limited number of retries. If the number of retries exceeds a preset threshold, a user prompt or a request for user confirmation is triggered.

[0114] Furthermore, the VLA strategy unit and task orchestration unit adopt a hybrid control of "model suggestion + rule constraint". The VLA model is used to provide flexible action suggestions in weakly structured scenarios such as positional deviation, button layout differences, and slight cup displacement. The rule constraints of the task orchestration unit ensure the execution logic and order of sub-tasks, taking into account both the flexibility of action decision-making and the logic of task execution.

[0115] II. Specific Implementation of the Control Method for Coffee Making Robots

[0116] refer to Figure 2 The figure is an exemplary flowchart of a coffee-making robot control method according to some embodiments of the present invention. The method is applied to the above-mentioned control system and mainly includes the following steps:

[0117] Step S1: Voice command reception and parsing

[0118] The voice interaction unit receives the user's natural language voice commands via its microphone array. After voice noise reduction and wake-up detection, the ASR module converts the voice commands into text information. The NLU module then performs intent parsing and parameter extraction on the text information to obtain the target task and optional parameters, which are then converted into a structured task description and sent to the task orchestration unit. For example, if the user's voice command is "Make me a cup of coffee," the parsed result would be a structured task description of "Task: Make coffee, Parameter: None."

[0119] Step S2: Subtask sequence generation

[0120] After receiving the structured task description, the task orchestration unit decomposes it into an executable sequence of subtasks according to the preset coffee-making task logic. The subtask sequence is: arrive at the coffee machine workstation → retrieve the cup → place the cup in front of the coffee spout → trigger coffee dispensing → retrieve the cup → deliver the cup to the designated location. Simultaneously, the task orchestration unit initializes the task state machine, setting the initial state to NAV_TO_MACHINE (navigation to the coffee machine).

[0121] Step S3: Mobile Navigation and Precise Alignment

[0122] The mobile navigation unit receives navigation instructions from the task orchestration unit, realizes real-time localization of the robot based on the SLAM algorithm, plans the optimal path in combination with the scene map, and controls the mobile chassis to move to the work position near the coffee machine. After arriving at the work position, it combines the detection results of the RGB-D camera of the environmental perception unit to perform centimeter-level precision alignment of the robot's position to ensure that the robotic arm can cover the operable area of ​​the coffee machine. After completion, it sends a "navigation completed" status feedback to the task orchestration unit, and the task state machine transitions to the PICK_CUP (cup take) state.

[0123] Step S4: Scene Awareness and Operational Area Generation

[0124] After the robot completes precise alignment, the environmental perception unit activates the RGB-D camera to acquire color and depth images of the coffee machine and cup area. It then uses a target detection algorithm to identify key components such as the cup, coffee spout, and buttons, and uses a pose estimation algorithm to obtain the 3D pose of each key component. Based on the 3D pose information, it generates operable areas such as the cup gripping area, the coffee machine cup-holding area, and the button pressing area, along with attributes such as normal and allowable contact force range. The 3D pose of the key components and the operable area information are then sent in real-time to the VLA strategy unit, the robotic arm, and the end effector control unit.

[0125] Step S5: Subtask Execution and Loop Closure Determination

[0126] During the execution of each subtask, the VLA strategy unit outputs action suggestions based on the current image I, language command L, and robot body state s. After the action suggestions are sent to the safety constraint and anomaly recovery unit, they are filtered and corrected through a cost function to select the optimal action that meets the constraints, and then sent to the robotic arm and end effector control unit. This unit performs trajectory planning based on the filtered action suggestions and controls the robotic arm and end effector to perform the corresponding actions (grasping, placing, pressing, etc.).

[0127] Simultaneously, visual / force feedback is used to perform closed-loop determination of the completion status of each sub-task:

[0128] Cup retrieval task: The clamping force of the gripper is detected by the force / torque sensor, and combined with the visual observation of the RGB-D camera, the cup is confirmed to be stably held, and the cup retrieval task is determined to be completed.

[0129] Cup placement task: The closed-loop alignment of the robotic arm end effector is achieved through a visual servoing algorithm. The cup is placed in the cup-holding area directly below the coffee outlet of the coffee machine. The cup placement task is considered complete when the visual observation confirms that the cup is placed stably and without deviation.

[0130] Button pressing task: Control the end effector of the robotic arm to press the coffee machine's button. Determine whether the button has been successfully triggered by force / displacement threshold detection, and the pressing task is considered complete.

[0131] Coffee making task: The system uses an RGB-D camera to detect the liquid flow at the coffee spout or changes in the liquid level in the cup in real time, or to identify the coffee machine's completion indicator light / beep sound to determine that the coffee is ready.

[0132] After each subtask is completed, a status feedback is sent to the task orchestration unit, and the task state machine transitions to the next subtask state.

[0133] Step S6: Exception Handling and Recovery

[0134] If, during the execution of a subtask, issues such as capture failure, placement deviation, button failure, or collision / force exceeding limits occur, the execution unit sends an exception feedback to the safety constraint and exception recovery unit. This unit then triggers the corresponding exception recovery operation based on the exception type, specifically:

[0135] Grasping failure: Control the robotic arm to retract to the safe position, the environmental perception unit re-detects the cup pose and generates a new candidate grasping pose, the VLA strategy unit outputs a new grasping action suggestion, and the grasping operation is re-executed;

[0136] Placement deviation: The offset of the cup is identified by visual detection, and the end of the robotic arm is controlled to make fine adjustments, or the cup is picked up and placed again to ensure that the cup is centered in the cup-holding area;

[0137] If the button is not triggered: the environmental perception unit repositions the button's three-dimensional pose, adjusts the pressing contact point and pressing direction, and the VLA strategy unit outputs a new pressing action suggestion to re-execute the pressing operation;

[0138] Collision / Force Exceedance: Immediately trigger the robotic arm's soft emergency stop, control the robotic arm to retract to a safe position, record information such as the collision location and force / torque values, and request user confirmation before continuing execution if necessary.

[0139] All abnormal recovery operations are configured with a retry threshold (e.g., 3 times). If the number of retries exceeds the threshold, the task state machine transitions to the ERROR state, triggering a voice prompt to inform the user.

[0140] Step S7: Task Completion and Data Recording

[0141] Once all subtasks are completed, the robotic arm will grab the prepared coffee cup and move it to the user-designated placement area. After placement, the voice interaction unit will announce (e.g., "The coffee is ready, please take your cup"), and the task state machine will transition to the DONE (task completed) state.

[0142] Meanwhile, the control system automatically records the execution data of this task, including scene observation images, poses of key components, VLA model inputs and outputs, robot motion trajectories, force / tactile feedback data, task execution results and abnormal information, and labels the data. This data will be used for offline retraining and strategy optimization of the VLA model to continuously improve the robot's motion decision-making ability and task execution reliability.

[0143] Alternative implementation methods

[0144] Without departing from the concept of the present invention, the technical solution of the present invention can be implemented in various alternative ways, as follows:

[0145] Input method replacement: The voice input of the voice interaction unit can be replaced by touch screen / APP / gesture input, or a hybrid input method combining voice and touch screen can be adopted. The touch screen / APP / gesture recognition module converts the user's operation commands into structured task descriptions to adapt to different user interaction needs.

[0146] VLA Model Replacement: The end-to-end VLA model of the VLA policy unit can be replaced by a two-stage architecture of "visual-language model + motion planner". First, the visual-language model completes semantic understanding and target localization, and then the motion planner generates specific action trajectories. Alternatively, diffusion strategies, behavior cloning + reinforcement learning strategies can be used to replace the existing VLA model to achieve action decision-making.

[0147] Coffee machine triggering method replacement: The triggering method of the robotic arm pressing the button can be replaced by the coffee machine communication interface control, which establishes communication with the coffee machine via Bluetooth / Wi-Fi / serial port and directly sends the brewing instructions to the coffee machine. Alternatively, a mechanical linkage triggering method can be used to replace the direct pressing of the robotic arm, which is compatible with coffee machines of different structures.

[0148] Replacement of sensing devices: The RGB-D camera of the environmental sensing unit can be replaced with a 2D camera + marker positioning, or a fusion sensing method of LiDAR + depth camera can be adopted to realize the pose detection of key components and adapt to different scene sensing needs.

[0149] End effector replacement: The two-finger gripper of the robotic arm can be replaced with a suction cup, a three-finger gripper, a soft hand, or a special fixture. The appropriate end effector can be selected according to the material and shape of the cup to improve the stability of the grip.

[0150] The technical solutions provided by the embodiments disclosed in this invention have the following beneficial effects:

[0151] With strong generalization ability, this invention uses an environmental perception unit to detect the three-dimensional pose of key components in a coffee-making scenario in real time. Combined with the end-to-end decision-making capability of the VLA model, it eliminates the need for offline calibration of the coffee machine. It can adapt to different models of coffee machines, different button layouts, and placement errors in the scenario, greatly reducing the workload of on-site debugging and calibration, and improving the robot's adaptability to weakly structured environments.

[0152] The human-computer interaction is natural. This invention realizes the recognition and intent parsing of users' natural language commands through the voice interaction unit. Users can give coffee making instructions through flexible natural language without being limited to a fixed menu, which significantly improves the friendliness of human-computer interaction and user experience.

[0153] The execution process is closed-loop and reliable. This invention integrates feedback information from visual sensors and force / tactile sensors to determine the completion status of each step, such as grasping, placing, pressing, and dispensing coffee, forming a closed-loop control of perception-decision-execution, which effectively reduces the task failure rate and improves the reliability of coffee making.

[0154] With high safety and controllability, this invention performs multi-dimensional constraint verification on the robot's action suggestions and planned trajectory through a safety constraint and anomaly recovery unit. At the same time, it designs corresponding recovery strategies for abnormal situations such as grasping failure, placement deviation, button failure, collision / force exceeding limits, etc., effectively reducing safety risks such as equipment collision and coffee spill, and ensuring the safety of the robot and the scene.

[0155] This invention enables sustainable optimization and large-scale deployment. After the coffee-making task is completed, it automatically records execution data such as observations, actions, and result labels. This data can be used for offline retraining of the VLA model and policy optimization to continuously improve the robot's performance. At the same time, the robot's high generalization ability greatly reduces on-site debugging and post-maintenance costs, enabling the large-scale replication and rapid deployment of the coffee-making robot.

[0156] The technical solutions provided by the embodiments of the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the embodiments of the present invention. The descriptions of the embodiments above are only for helping to understand the principles of the embodiments of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the embodiments of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A coffee-making robot control system based on voice commands and a VLA model, characterized in that, include: The voice interaction unit is used to complete voice wake-up, automatic speech recognition and intent parsing, and convert the user's natural language commands into structured task descriptions; The task orchestration unit is used to decompose the structured task description into a sequence of executable subtasks, and to implement the precondition check, step completion determination and abnormal branch handling of the subtasks through the task state machine. The environmental perception unit, based on visual sensors and force / tactile sensors, detects and estimates the three-dimensional pose and operable area of ​​the cup and key components of the coffee machine in the coffee making scene. The mobile navigation unit, based on SLAM localization and path planning, moves the robot to the working pose near the coffee machine and completes precise alignment. The robotic arm and end effector control unit is used for trajectory planning and execution of grasping, placing, and pressing actions during the coffee making process; VLA strategy unit takes language commands, real-time visual observations and robot body state as input, and outputs high-level actions or continuous control increments. The safety constraint and anomaly recovery unit performs constraint verification on the action suggestions output by the VLA strategy unit and the trajectory planned by the mobile navigation and robotic arm control unit, and triggers anomaly recovery operation when the action execution fails or a risk occurs; the VLA strategy unit and the task orchestration unit adopt a hybrid control of model suggestion + rule constraint, and the safety constraint and anomaly recovery unit filters the action suggestions output by the VLA strategy unit before they are executed by the corresponding execution unit.

2. The coffee-making robot control system based on voice commands and VLA model as described in claim 1, characterized in that, The safety constraint and anomaly recovery unit filters candidate actions using a cost function, which is: ， Where d_pose represents the pose error from the end point to the center of the target operable area, r_collision represents the collision risk, r_force represents the expected contact force exceeding the limit risk, t_exec represents the expected execution time; w1~w4 are weight coefficients, and the system selects the action with the smallest J that satisfies the constraints to execute.

3. The coffee-making robot control system based on voice commands and VLA model as described in claim 1, characterized in that, The visual sensor is an RGB-D camera or a multi-view camera. The operable area of ​​the environmental perception unit includes a cup gripping area, a coffee machine button area, and a cup holding area. The operable area is represented by a planar region / voxel set or parameterized region in three-dimensional space, and includes normal, allowable contact force range, and preferred contact point attributes.

4. The coffee-making robot control system based on voice commands and VLA model as described in claim 1, characterized in that, The states of the task state machine include at least WAIT_CMD, NAV_TO_MACHINE, PICK_CUP, PLACE_CUP_UNDER_SPOUT, PRESS_BUTTON, WAIT_BREW_FINISH, PICK_UP_CUP, PLACE_CUP_TO_TARGET, DONE, and ERROR. Each state includes an entry condition, an execution action, a completion condition, and an exception transfer rule.

5. The coffee-making robot control system based on voice commands and VLA model as described in claim 1, characterized in that, The input of the VLA strategy unit includes the current image I, the language command L, and the robot state s; the output is action a, which includes one or more of the following: end-effector pose increment Δx, gripper command g, chassis micro-motion command v, and pressing duration τ.

6. The coffee-making robot control system based on voice commands and VLA model as described in claim 1, characterized in that, The anomaly recovery operation includes re-observation, pose adjustment, limiting the number of retries, and reverting to a safe position. When the anomaly exceeds a preset threshold, a user prompt or user confirmation request is triggered.

7. A control method for a coffee-making robot based on voice commands and a VLA model, characterized in that, The coffee-making robot control system based on voice commands and VLA models as described in any one of claims 1-6 includes the following steps: S1: Receives user voice commands through the voice interaction unit, completes automatic speech recognition and intent parsing, and obtains the target task and optional parameters; S2: The target task is decomposed into a sub-task sequence by the task orchestration unit. The sub-task sequence includes at least arriving at the coffee machine work position, taking a cup, placing the cup in front of the coffee spout, triggering coffee dispensing, retrieving the cup, and sending the cup to the designated position. S3: The robot is moved to the vicinity of the coffee machine by the mobile navigation unit based on SLAM localization and path planning, and completes precise alignment at the work station; S4: Detect the three-dimensional pose of the cup and key components of the coffee machine through the environmental perception unit, and generate candidate actions and operable areas for grasping, placing, and pressing; S5: During the execution of each subtask, the VLA strategy unit outputs action suggestions, which are then checked and corrected by the safety constraint and anomaly recovery unit before being executed by the corresponding execution unit. Visual / force feedback is used to determine whether the subtask has been completed. S6: If the action fails or a security risk occurs, trigger the exception branch through the security constraint and exception recovery unit to perform the exception recovery operation; S7: After all subtasks are completed, place the coffee cup in the designated area and announce it via voice. At the same time, record the execution data for the robot's continuous learning and strategy optimization.

8. The coffee-making robot control method based on voice commands and VLA model as described in claim 7, characterized in that, Step S5 involves determining the completion of the subtask through visual / force feedback, including: the grasping action confirms successful grasping through force feedback and visual observation; the placement action aligns the cup to the placement center through visual servo closed loop and confirms the cup is stable; the pressing action determines whether the coffee machine button is triggered through force / displacement threshold; and the coffee dispensing action determines completion by visually detecting liquid flow / changes in the liquid level inside the cup or recognizing the coffee machine indicator light / sound.

9. A computer device, the computer device comprising a memory and a processor, the memory storing computer program code, characterized in that, The processor is configured to acquire the computer program code and execute the coffee-making robot control method based on voice commands and VLA model as described in any one of claims 7-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the coffee-making robot control method based on voice commands and VLA model as described in any one of claims 7-8.

Images