Robot ordering method and system

Abstract

The invention relates to the field of robots, and discloses a robot ordering method and system, and the method comprises the steps: obtaining a depth map and a color map of a current scene; obtaining a detection frame of a target object, and cutting out three-dimensional point cloud data of the corresponding object from the depth map according to a detection result; reasoning by taking the three-dimensional point cloud data as input of a reasoning model, and predicting candidate grabbing poses; and according to the predicted candidate grabbing poses, trajectory planning and motion execution are carried out. Reasonable action information can be generated according to information provided by the visual language model in combination with specific task requirements, and therefore control over the mechanical arm is achieved.

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular to a robot ordering method and system. Background Art The robot system uses its multimodal interaction model to allow guests to interact with the robot using simple everyday language, just like talking to a real person, to order food and have the food delivered to them.

[0002] Existing ordering robots have problems in hardware driving, message transmission, and node management, which makes the existing robot-based ordering methods complex and the control efficiency low. Summary of the Invention

[0003] The main purpose of the present invention is to solve the technical problems of the existing technology of complex ordering methods and low ordering control efficiency. A robot ordering method includes the following steps: Obtain a depth map and color map of the current scene; obtain a detection frame for the target object, and based on the detection results, crop the corresponding 3D point cloud data from the depth map; use the 3D point cloud data as input to the inference model to perform inference and predict candidate grasping poses; perform trajectory planning and motion execution based on the predicted candidate grasping poses.

[0004] As a preferred technical solution, the predicted candidate grasping poses include: The three-dimensional point cloud data is mapped to a unified coordinate system through the camera pose and reference alignment matrix; based on the object pose of the current frame, candidate grasps of the corresponding object are retrieved from a pre-generated object-level grasp set, and the object-level grasp set includes the discretized viewing direction, rotation angle around the normal, proximity depth parameter, and friction coefficient at the sampling points on the object surface; the retrieved candidate grasps are transformed to the current coordinate system at one time, and Boolean screening is performed using the friction coefficient threshold and collision-free mask to obtain multiple filtered candidate grasps; for each filtered candidate grasp, local neighborhood features centered on the grasp center are extracted from the three-dimensional point cloud, and input into the pre-trained GraspNet general object grasping network for inference, and the network outputs a grasp quality prediction value; all candidate grasps are sorted based on the grasp quality prediction value, and duplicates are removed by non-maximum suppression of dual thresholds of position and orientation, and finally the top K candidate grasp postures are output as the execution plan input.

[0005] As a preferred technical solution, the ordering method further includes a desktop robotic arm control step: The instruction acquisition step obtains the voice instructions issued by the user and converts them into original text through the speech recognition module; In the semantic parsing step, the LLM model is activated to perform semantic parsing on the original text, converting it into standardized structured task instructions and generating responses to customers. The scene perception and task planning step starts the VLM model, converts the standardized structured task instructions into a task sequence including multiple subtasks, and passes the multiple subtasks to the VLA model one by one. Action generation step, scene perception and task planning step, the VLA model converts subtasks into continuous action instructions that can be executed by the robot in real time; The robot executes the control steps and sends the generated action instructions to the robotic arm controller, driving the robotic arm to complete the ordering action.

[0006] As a preferred technical solution, the instruction acquisition step includes: The user's voice commands are captured through an audio acquisition device; the original audio signal is preprocessed; and a pre-trained automatic speech recognition model is used to convert the processed voice signal into a text string, wherein the text string serves as an initial load of the user's intention.

[0007] As a preferred technical solution, the instruction acquisition step includes: The user's voice commands are captured through an audio acquisition device; the original audio signal is preprocessed; and the processed voice signal is converted into a text string using a pre-trained automatic speech recognition model; the text string is used as the initial load of the user's intention.

[0008] As a preferred technical solution, the semantic parsing step includes: A pre-configured LLM model is called to understand the intent of the acquired text string; the LLM model outputs a standardized structured task instruction, which is a pre-defined JSON object that includes the task type and records the user's original instruction; the LLM model also generates a personified and concise reply; The scene perception and task planning steps include: Receive standardized structured task instructions generated by the semantic parsing step; acquire real-time multi-view RGB images from multiple cameras deployed in different locations; the VLM model understands the relationship between instructions and visual scenes, identifies all objects on the desktop and their spatial locations, and generates a task sequence including multiple subtasks based on the task type.

[0009] As a preferred technical solution, the action generation step includes: The VLA model acquires a single subtask in a task sequence; the VLA model acquires the real-time status and visual feedback of the robot body; and the VLA model outputs a series of control instructions, including the target joint position, velocity or end effector posture and the opening and closing of the gripper output at a specific frequency.

[0010] As a preferred technical solution, when the shelf robot performs a task with a collision risk, it sends an immediate pause and waiting communication signal through an HTTP request; upon receiving the signal, the desktop robot will immediately pause the current action and remain stationary until the shelf robot completes the high-risk operation and sends a "release" signal, and then resume the action. The stopping and resumption of the desktop robot is achieved through the ROS topic / action control mechanism.

[0011] A second aspect of the present invention provides a robot ordering system, the system including a shelf robotic arm main control program configured as follows: Get the depth map and color map of the current scene; Get the detection frame of the target object and, based on the detection results, crop the 3D point cloud data of the corresponding object from the depth map; Using the three-dimensional point cloud data as input to the inference model to perform inference and predict candidate grasping poses; Based on the predicted candidate grasping poses, trajectory planning and motion execution are performed.

[0012] The third aspect of the present invention provides an electronic device, comprising: a memory and at least one processor, wherein instructions are stored in the memory, and the memory and the at least one processor are interconnected through lines; the at least one processor calls the instructions in the memory so that the electronic device executes the above-mentioned robot ordering method.

[0013] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, enable the computer to execute the aforementioned robot ordering method.

[0014] The present invention has the following beneficial effects: The present invention can generate reasonable action information based on the information provided by the visual language model and combined with specific task requirements, thereby realizing the control of the robotic arm.

[0015] By combining the intelligent prediction of the VLA model with the mandatory intervention of communication control, the present invention enables the dual-manipulator system to achieve multi-level safety protection throughout the entire process from prediction to execution, effectively ensuring the continuity and stability of collaborative tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1This is a flowchart of the robot ordering method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The terms "first," "second," "third," "fourth," and so on (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar items and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances, so that the embodiments described herein can be implemented in an order other than that shown or described herein. In addition, the terms "including," "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product, or apparatus.

[0018] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 The first embodiment of the robot ordering method in the embodiment of the present invention includes: Obtain a depth map and color map of the current scene; obtain a detection frame for the target object, and based on the detection results, crop the corresponding 3D point cloud data from the depth map; use the 3D point cloud data as input to the inference model to perform inference and predict candidate grasping poses; perform trajectory planning and motion execution based on the predicted candidate grasping poses.

[0019] The predicted candidate grasping poses include: The three-dimensional point cloud data is mapped to a unified coordinate system through the camera pose and reference alignment matrix; based on the object pose of the current frame, candidate grasps of the corresponding object are retrieved from a pre-generated object-level grasp set, and the object-level grasp set includes the discretized viewing direction, rotation angle around the normal, proximity depth parameter, and friction coefficient at the sampling points on the object surface; the retrieved candidate grasps are transformed to the current coordinate system at one time, and Boolean screening is performed using the friction coefficient threshold and collision-free mask to obtain multiple filtered candidate grasps; for each filtered candidate grasp, local neighborhood features centered on the grasp center are extracted from the three-dimensional point cloud, and input into the pre-trained GraspNet general object grasping network for inference, and the network outputs a grasp quality prediction value; all candidate grasps are sorted based on the grasp quality prediction value, and duplicates are removed by non-maximum suppression of dual thresholds of position and orientation, and finally the top K candidate grasp postures are output as the execution plan input.

[0020] The method for ordering food further includes a desktop robotic arm control step: The instruction acquisition step obtains the voice instructions issued by the user and converts them into original text through the speech recognition module; In the semantic parsing step, the LLM model is activated to perform semantic parsing on the original text, converting it into standardized structured task instructions and generating responses to customers. The scene perception and task planning step starts the VLM model, converts the standardized structured task instructions into a task sequence including multiple subtasks, and passes the multiple subtasks to the VLA model one by one. Action generation step, scene perception and task planning step, the VLA model converts subtasks into continuous action instructions that can be executed by the robot in real time; The robot executes the control steps and sends the generated action instructions to the robotic arm controller, driving the robotic arm to complete the ordering action.

[0021] The instruction acquisition step includes: The user's voice commands are captured through an audio acquisition device; the original audio signal is preprocessed; and a pre-trained automatic speech recognition model is used to convert the processed voice signal into a text string, wherein the text string serves as an initial load of the user's intention.

[0022] The instruction acquisition step includes: The user's voice commands are captured through an audio acquisition device; the original audio signal is preprocessed; and a pre-trained automatic speech recognition model is used to convert the processed voice signal into a text string, wherein the text string serves as an initial load of the user's intention.

[0023] The semantic parsing step includes: A pre-configured LLM model is called to understand the intent of the acquired text string; the LLM model outputs a standardized structured task instruction, which is a pre-defined JSON object that includes the task type and records the user's original instruction; the LLM model also generates a personalized and concise reply; The scene perception and task planning steps include: Receive standardized structured task instructions generated by the semantic parsing step; acquire real-time multi-view RGB images from multiple cameras deployed in different locations; the VLM model understands the relationship between instructions and visual scenes, identifies all objects on the desktop and their spatial locations, and generates a task sequence including multiple subtasks based on the task type.

[0024] The action generation step includes: The VLA model acquires a single subtask in a task sequence; the VLA model acquires the real-time status and visual feedback of the robot body; and the VLA model outputs a series of control instructions, including the target joint position, velocity or end effector posture and the opening and closing of the gripper output at a specific frequency.

[0025] The specific plan is as follows: The user's voice content is captured from the microphone hardware device. The input audio data undergoes audio processing, including normalization and trimming of silence. FastAPI is then used for API routing and cross-origin processing (CORS). A WebUI is built to enable remote recording button control, and recording status is published via ROS communication. The audio content is processed using Whisper, a speech-to-text model (ASR), to accurately convert the speech content into text.

[0026] By setting a dedicated prompt, the large language model (LLM) is guided to perform structured parsing of the content of user interaction dialogues. Based on the text instructions of the user interaction dialogue, the system uses the large language model to identify the user's intent and output the parsing results in a standardized JSON format. The JSON result contains key information such as the task type, the original user instruction, the system's natural language response, and the name and quantity of the product required by the user. The system only identifies and processes the preset range of products to ensure the accuracy and standardization of the parsing results. For instructions that cannot be clearly identified as ordering tasks or requests that exceed the range of products, the system classifies them as "other tasks" and uses natural language text responses to guide users to supplement their specific needs.

[0027] The system parses the JSON text returned by the LLM, analyzes the task type, and processes information such as product quantity, shelf inventory, and unusual items. Based on the task content, the system then issues a natural language response using a large language model. This process is implemented using a text-to-speech (TTS) model. The system also updates shelf stack information and issues task signals to the shelf and desktop robotic arms.

[0028] The system receives tasks (e.g., an order for multiple items) parsed by the language model and stores the requested items in a task queue. The system then sends these requests to the shelving robot via HTTP. Upon receiving the signal, the shelving robot grabs the corresponding items. The system also sets a state lock to prevent new instructions from being sent before the current action is complete, ensuring serial execution of actions.

[0029] After the shelf robotic arm completes picking up an item, it sends a completion signal back via the Robot Operating System (ROS) topic. Upon receiving confirmation of the robot's action, the system updates its internal state and marks the robotic arm as idle, ready to receive the next action instruction. Simultaneously, the system constructs a formatted instruction for the Visual Language Model (VLM) input (e.g., "[order food] water") and sends it to the desktop robotic arm as input. The desktop robotic arm then takes charge of the next action on the item. Through the cascade of reasoning between the brain's Visual Language Model (VLM) and the cerebellum's Visual Language Action Model (VLA), the desktop robotic arm performs a series of autonomous actions to pick up and place the relevant item in front of the customer.

[0030] The following is a detailed description of the technology used by the two robotic arms.

[0031] This ordering system consists of two robotic arms: a shelf robotic arm and a desktop robotic arm.

[0032] The shelving robot, used to pick up items from shelves and place them on the table, is equipped with a wrist camera capable of capturing depth information. The master control program receives product labels transmitted via HTTP and calls the camera process to acquire real-time image data. The image is first fed into the YOLO object detection neural network to obtain a bounding box for the target object. Based on the detection results, the point cloud data of the corresponding object is cropped from the depth map and fed into the GraspNet general object grasping network for inference, predicting a series of candidate grasping poses (including position, posture, and gripper width). The master control program then filters these candidate grasping poses for feasibility, selects the optimal grasping method, and sends it to the slave robot for trajectory planning and motion execution. After completing the grasping and placing operation, the robot returns to the preset pose and publishes a signal to provide feedback on the action execution result via ROS, achieving closed-loop control of the entire grasping process.

[0033] The desktop robotic arm, used to place products in front of customers, is equipped with two cameras: a wrist camera and a global desktop camera. Two multimodal models, the Visual Language Model (VLM) and the Visual Language Action Model (VLA), form the brain and brain structure to control the robotic arm. The Visual Language Model (VLM) accurately recognizes and deeply understands visual information such as images and videos, and can also correlate and fuse this visual information with natural language. This capability enables the desktop robotic arm to accurately understand the characteristics, location, and relationships of objects in the environment and convert this information into processable language descriptions. The Visual Language Action Model (VLA) empowers the robotic arm to translate this understanding into action. Based on the information provided by the Visual Language Model and the specific task requirements, it generates appropriate action information to control the robotic arm.

[0034] As an implementation method, the overall process of the GraspNet network of the shelf robotic arm is as follows: 1. Point Cloud Data Alignment Camera pose matrix: 4×4 homogeneous transformation matrix, containing rotation component R∈SO(3) and translation component t∈R³; Reference alignment matrix: 4×4 rigid body transformation matrix that calibrates the relationship between the tool coordinate system and the robot base coordinate system.

[0035] The 3D point clouds with a resolution of ≤2mm collected from multiple perspectives are uniformly converted to the robot base coordinate system to eliminate the differences in sensor poses.

[0036] 2. Object-level crawling collection retrieval Pre-generated crawl collection parameters: Surface sampling points: Poisson disk sampling, density 1 point / cm², covering the key grasping area on the object surface; View direction, with an angle interval of ≤15°, a 642-hedron discrete sphere, used to define the grasping approach direction; The rotation angle around the normal direction is divided into 12 equal parts at 30° intervals to optimize the contact angle between the fixture and the object. The approach depth adopts three levels {5mm, 10mm, 15mm} to adapt to the different thickness of the jaws.

[0037] The friction coefficient ranges from 0.2 to 0.8 with a step size of 0.1. This parameter ensures physical stability.

[0038] Retrieval mechanism: Based on the object pose in the current frame, the candidate set of the corresponding object is indexed in the pre-stored grasp database.

[0039] 3. Candidate Capture, Transformation, and Screening Coordinate transformation: Formula: G_current = T_object_current × G_object_canonical Where T_object_current is the current pose matrix of the object Boolean filter conditions: Friction coefficient threshold, friction coefficient μ ≥ 0.5, excludes low-friction gripping that is prone to slippage The collision-free mask is obtained based on the OBB detection of the gripper's bounding box to ensure that there is no robot self-collision in the grasping path.

[0040] Output: Screening retains about 200 candidate crawls.

[0041] 4. Local Neighborhood Feature Extraction and Network Inference Using GraspNet network architecture, feature extraction parameters: Neighborhood range: 20cm×20cm×20cm cube with the grasping center as the origin; Point cloud features: a total of 7-dimensional features, including normal vector, curvature, and RGB color; Downsampling resolution: 2mm voxel grid; Output: Grasp quality score Q∈[0,1], set Q>0.8 to be stable grasp.

[0042] 5. Sorting and deduplication optimization Sorting rules: sort by quality score Q in descending order Non-maximum suppression parameters: Position distance, the threshold selected is 15 mm, which is used to achieve merged spatial proximity grasping; The threshold of orientation difference is selected as 20° to eliminate redundant grasps with similar orientations.

[0043] Output: Keep the first K grasping postures. The value of K is generally 10. The grasping posture that ranks first is selected as the grasping posture.

[0044] Of course, there may be other options for the parameters given in the above solution in practice, and the above is just an example.

[0045] The specific working principle of the desktop robotic arm is: The Visual Language Model (VLM) uses real-time image information from the wrist camera and the desktop camera, along with user textual commands, as input to infer natural language skill labels for the VLA in the current state. The skill labels generated by the VLM serve as textual command inputs for the Visual Language Action Model (VLA). Combined with the real-time camera footage, the VLM infers the basic actions that the robotic arm should currently perform. The VLA inference specifically addresses the EEF position and posture of the robotic arm's end effector, the incremental 6-DOF pose of the end effector, and the degree of gripper closure.

[0046] Action description: The user issues vague or high-level commands (e.g., "Get me a carton of milk") via voice or text. The wrist camera captures the close-up view, the desktop camera provides an overall third-person perspective, and the speech recognition model converts the speech into text input. The VLM model aligns language with images to output more structured and specific language commands. The camera's image observation information and user instructions are mapped to verbal responses and natural language skill labels for the VLA (e.g., "pick up the milk, put the milk on the table, return to the initial position, etc."). The VLA module's processing logic integrates the skill labels output by the VLM with the observed image to generate the robot's action control decisions (e.g., movement direction, movement distance, gripper opening and closing).

[0047] Based on the real-time, high-frequency information from the camera, the system continuously updates its perception, providing feedback and corrections. The VLM and VLA continuously reason based on the current scene, predicting and outputting the robotic arm's skill tags and control decisions in real time. For example, if an item is misplaced and knocked over, the tabletop robotic arm can re-grip it and straighten it. If an object deviates from its movement, the system replans its movements and corrects. If there are other items in the dining area, the current item can be placed in the vacant space on the table.

[0048] The input data for VLM training is video images (wrist, desktop) and the user's verbal task description, and the output is a predicted natural language skill label. The training goal is to map fuzzy language instructions into clear action skill label descriptions. The input data for VLA training is video images from two cameras, skill labels, and the current position of the robotic arm. The output is a predicted inference action sequence (end position and posture, joint information, action type, etc.).

[0049] To train the collaborative model for the two robotic arms, customized data collection processes were designed for the shelf robotic arm and the desktop robotic arm, respectively. For the shelf robotic arm, data collection primarily consists of two parts: first, capturing image data of goods on the shelf through multi-angle and multi-scene shooting, selecting and labeling each item for training the YOLO model to improve the accuracy of product detection and recognition; second, collecting manual grasping demonstration data through drag-and-drop teaching, which is used to optimize the robotic arm's grasping action execution strategy (such as trajectory planning, speed adjustment, and anomaly compensation). The grasping pose is determined using the GraspNet pre-trained model, which performs real-time inference on point cloud data captured by the depth camera to obtain candidate grasping poses for execution by the robotic arm, thereby achieving accurate and stable grasping operations in complex shelf environments.

[0050] The training data for the desktop robotic arm primarily serves the learning needs of its "brain" (VLM) and "cerebellum" (VLA). This data utilizes a master-slave control model to collect extensive real-world operational data. This data not only covers standard correct gripping and placement operations but also includes corrections for various anomalies, such as a bottle accidentally knocked over, an item slipping during placement, and a failed grasp. This rich data on corrections for these anomalies significantly improves the robustness of the model, enabling the trained desktop robotic arm to not only stably complete tasks under ideal conditions but also rapidly perceive, judge, and adjust its actions in the face of unexpected situations in real-world operations, achieving greater adaptability and robustness. The data used to train the desktop robotic arm is in the form of video image sequences, annotated with the natural language labels for the currently executed skills. This data, along with information on the robotic arm's position, state, and motion, serves as the model's training data.

[0051] To ensure the safety of dual-arm collaboration and avoid potential collisions between robotic arms, this system has built a two-layer obstacle avoidance system that integrates predictive planning and imperative control. The core of integrated foresight lies in the "big brain and small brain" model architecture of the desktop robotic arm. In this architecture, the high-level visual language model (VLM) perceives the environment in real time. When the VLM determines that there is a high risk of collision between the desktop robotic arm and the shelf robotic arm based on the current information of the camera, it sends a language instruction skill label for obstacle avoidance to the visual language action model (VLA). This design enables the system to autonomously predict and resolve potential collision risks during the task execution phase through the large model, dynamically optimize actions, and thus eliminate interference. During the reasoning process, the model actively adjusts its actions based on environmental information to complete obstacle avoidance and achieve predictive planning and prediction.

[0052] As a supplement and safeguard, a communication-based, command-controlled safety mechanism operates synchronously at the execution layer. When the shelf robotic arm performs a task involving a collision risk, its main control program sends an immediate pause and wait communication signal to the desktop robotic arm's main control program via HTTP request. Upon receiving the signal, the desktop robotic arm immediately pauses its current action and remains stationary until the shelf robotic arm completes the high-risk operation and issues a "release" signal, at which point it resumes action. This stopping and resuming of the desktop robotic arm is achieved through the ROS topic / action control mechanism. This emergency stop and resume mechanism, implemented through HTTP and ROS communication, forms a solid foundational safety defense.

[0053] By combining the intelligent prediction of the VLA model with the mandatory intervention of communication control, the dual-arm system achieves multi-level safety protection throughout the entire process from prediction to execution, effectively ensuring the continuity and stability of collaborative tasks.

[0054] The specific contents of the prompt for the large language model are as follows: You will play the role of a bartender, receiving instructions from users and providing order services: - The items that can be ordered are limited to the following **5** items: `Coffee, cookies, milk, water, chocolate` - The current quantity of each product is: ##rack_info##. Please respond to user needs based on the quantity.

[0055] - Your task is to accurately parse the user's order, extracting the task type, user instructions, your language response, and the product labels and quantities the user wants.

[0056] - The returned result must be in strict **JSON format**.

[0057] ## Example - **User command:** "I'd like a bottle of coffee and a box of chocolates, please." - **Response:** ```json { "task": "Ordering Food", "user": "I'd like a bottle of coffee and a box of chocolates, please.", "utterance": "Okay, I'll give you a bottle of coffee and a box of chocolates right away.", "objs": [{ "obj1": "coffee", "num": "1"}, { "obj2": "Chocolate", "num": "1"}] }``` - **Standard response format:** ```json { "task": "Ordering Food", "user": "User command", "utterance": "The corresponding response to the user's command", "objs": [{ "obj1": "Item Name", "num": "Number of items"}, { "obj2": "Item Name", "num": "Number of items"}] } ``` Example usage: 1. The user turns on the microphone and speaks a command, such as "Hello, please get me a bottle of coffee and a box of chocolates." 2. The system's large language model analyzes the instructions, outputs a voice response result, and determines the corresponding items and quantities. The shelf robotic arm receives the item information instruction.

[0058] 3. The shelf robotic arm clamps the coffee from the shelf to the preparation area on the table.

[0059] 4. After the shelf robot places the coffee on the table preparation area, the table robot picks up the coffee and places it in front of the customer. Simultaneously, the shelf robot picks up the chocolate.

[0060] 5. The shelf robot arm takes the chocolate from the shelf to the preparation area on the table.

[0061] 6. After the shelf robotic arm places the chocolate on the table preparation area, the desktop robotic arm will grab the chocolate and place it in front of the customer, staggered with the coffee.

[0062] 7. The task is completed and the desktop robotic arm returns to its initial state.

[0063] The system also includes a desktop robotic arm master control program configured as follows: The instruction acquisition step obtains the voice instructions issued by the user and converts them into original text through the speech recognition module; In the semantic parsing step, the LLM model is activated to perform semantic parsing on the original text, converting it into standardized structured task instructions and generating responses to customers. The scene perception and task planning step starts the VLM model, converts the standardized structured task instructions into a task sequence including multiple subtasks, and passes the multiple subtasks to the VLA model one by one. Action generation step, scene perception and task planning step, the VLA model converts subtasks into continuous action instructions that can be executed by the robot in real time; The robot executes the control steps and sends the generated action instructions to the robotic arm controller, driving the robotic arm to complete the ordering action.

[0064] An embodiment of the present invention further provides an electronic device that may vary significantly due to configuration or performance differences and may include one or more central processing units (CPUs) (e.g., one or more processors) and memory, and one or more storage media (e.g., one or more mass storage devices) for storing applications or data. The memory and storage media may be either transient or persistent storage. The program stored on the storage medium may include one or more modules, each of which may include a series of instruction operations on the electronic device. Furthermore, the processor may be configured to communicate with the storage medium and execute the series of instruction operations on the storage medium on the electronic device.

[0065] The electronic device may further include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the electronic device structure in this embodiment does not constitute a limitation to electronic devices, and may include more or fewer components, or combine certain components, or arrange the components differently.

[0066] An embodiment of the present invention provides a structure of an electronic device, which may vary significantly due to different configurations or performance, and may include one or more processors (central processing units, CPUs) (for example, one or more processors) and memories, and one or more storage media for storing applications or data (for example, one or more mass storage devices). The memories and storage media may be either transient or persistent storage. The program stored in the storage medium may include one or more modules, each of which may include a series of instruction operations on the electronic device. Furthermore, the processor may be configured to communicate with the storage medium to execute the series of instruction operations in the storage medium on the electronic device.

[0067] The electronic device may further include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the electronic device structure does not constitute a limitation on the electronic device, and may include more or fewer components than those described above, or combine certain components, or arrange the components differently.

[0068] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. The computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to execute the steps of the aforementioned method.

[0069] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0070] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0071] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot ordering method, characterized in that: The ordering method includes the following steps: Get the depth map and color map of the current scene; Get the detection frame of the target object and, based on the detection results, crop the 3D point cloud data of the corresponding object from the depth map; Using the three-dimensional point cloud data as input to the inference model to perform inference and predict candidate grasping poses; Based on the predicted candidate grasping poses, trajectory planning and motion execution are performed.

2. A robot ordering method according to claim 1, characterized in that: The predicted candidate grasping poses include: Mapping the three-dimensional point cloud data to a unified coordinate system through camera pose and reference alignment matrix; Based on the object pose in the current frame, a candidate grasp of the corresponding object is retrieved from a pre-generated object-level grasp set, where the object-level grasp set includes the view direction, rotation angle around the normal, proximity depth parameter, and friction coefficient discretized at sampling points on the object surface. The retrieved candidate grasps are transformed into the current coordinate system at one time, and Boolean screening is performed using the friction coefficient threshold and the collision-free mask to obtain multiple filtered candidate grasps; For each candidate grasp after screening, local neighborhood features centered on the grasp center are extracted from the 3D point cloud and input into the pre-trained GraspNet general object grasping network for inference. The network outputs a grasp quality prediction value. All candidate grasps are sorted based on the grasp quality prediction value, and duplicates are removed by non-maximum suppression of position and orientation double thresholds. Finally, the top K candidate grasp poses are output as the execution plan input.

3. A robot ordering method according to claim 1, characterized in that: The method for ordering food further includes a desktop robotic arm control step: The instruction acquisition step obtains the voice instructions issued by the user and converts them into original text through the speech recognition module; In the semantic parsing step, the LLM model is activated to perform semantic parsing on the original text, converting it into standardized structured task instructions and generating responses to customers. The scene perception and task planning step starts the VLM model, converts the standardized structured task instructions into a task sequence including multiple subtasks, and passes the multiple subtasks to the VLA model one by one. Action generation step, scene perception and task planning step, the VLA model converts subtasks into continuous action instructions that can be executed by the robot in real time; The robot executes the control steps and sends the generated action instructions to the robotic arm controller, driving the robotic arm to complete the ordering action.

4. A robot ordering method according to claim 1, characterized in that: The instruction acquisition step includes: Capturing the user's voice commands through an audio acquisition device; Preprocess the original audio signal; A pre-trained automatic speech recognition model is used to convert the processed speech signal into a text string, which serves as an initial load of the user's intention.

5. A robot ordering method according to claim 1, characterized in that: The semantic parsing step includes: Call a pre-configured LLM model to understand the intent of the acquired text string; The LLM model outputs a standardized structured task instruction, which is a predefined JSON object. The JSON object includes a task type and records the user's original instruction. The LLM model also generates an anthropomorphic and concise reply; The scene perception and task planning steps include: receiving standardized structured task instructions generated by the semantic parsing step; Acquire real-time multi-view RGB images from multiple cameras deployed at different locations; The VLM model understands the relationship between instructions and visual scenes, identifies all objects on the desktop and their spatial locations, and generates a task sequence consisting of multiple subtasks based on the task type.

6. A robot ordering method according to claim 1, characterized in that: The action generation step includes: The VLA model obtains a single subtask in a task sequence; The VLA model obtains the real-time status and visual feedback of the robot body; The VLA model outputs a series of control commands, including target joint positions, velocities, or end-effector postures, as well as gripper opening and closing, output at a specific frequency.

7. A robot ordering method according to claim 1, characterized in that: When the shelf robot performs a task with a collision risk, it sends an immediate pause and waiting communication signal via HTTP request. Upon receiving the signal, the desktop robot immediately pauses its current action and remains stationary until the shelf robot completes the high-risk operation and sends a "release" signal, at which point it resumes action. The stopping and resuming of the desktop robot is achieved through the ROS topic / action control mechanism.

8. A robot ordering system, characterized in that: The system includes a rack robotic arm master control program configured as follows: Get the depth map and color map of the current scene; Get the detection frame of the target object and, based on the detection results, crop the 3D point cloud data of the corresponding object from the depth map; Using the three-dimensional point cloud data as input to the inference model to perform inference and predict candidate grasping poses; Based on the predicted candidate grasping poses, trajectory planning and motion execution are performed.

9. An electronic device comprising a memory and at least one processor, wherein the memory stores instructions; The at least one processor calls the instructions in the memory to enable the electronic device to execute the various steps of the robot ordering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the robot ordering method as described in any one of claims 1 to 7 are implemented.

Images