Man-machine collaborative robot teleoperation method and system based on visual interaction

Abstract

The invention discloses a man-machine collaborative robot teleoperation method and system based on visual interaction, and belongs to the technical field of robot control and interaction. In order to solve the problems that a traditional teleoperation instruction is not clear, execution is not verified and dependence on a network is high, the method comprises the steps that a robot video stream is displayed on a client; determining a target two-dimensional coordinate in response to a click operation of a user on the video picture; and the coordinates and the three-dimensional environment data space are mapped, and the execution pose of the robot is calculated and verified. The system allows a user to visually specify a target through frame selection / click selection, and a background automatically converts two-dimensional interaction information into an accurate three-dimensional robot instruction subjected to feasibility verification. According to the method, the coordinate precision is guaranteed by introducing the static reference image, batch planning and submission are supported through the task queue to reduce network dependence, and safe, accurate and high-robustness man-machine collaborative operation is realized.

Description

Technical Field

[0001] This invention relates to the field of robot control and human-computer interaction technology, specifically to a method and system for teleoperating a human-computer collaborative robot based on visual interaction. Background Technology

[0002] Robot teleoperation technology has important applications in remote operations and hazardous environment handling. Currently, the main interaction and control methods have the following limitations:

[0003] 1. Traditional manual remote control method: The operator remotely drives the robot's movement using a joystick or controller. In this method, the operator can only send rough directional and speed commands such as "up" or "left," and cannot directly specify the final operational goal such as "grab that cup." Where the robot "goes" and "does" entirely depends on the operator's continuous micro-control, much like performing precise operations remotely with a game controller—a laborious and error-prone task. More importantly, the system executes commands directly after they are issued, lacking automatic verification of critical steps regarding the reachability and safety of the action; safety relies entirely on the operator's personal judgment.

[0004] 2. Voice Control: Users can control the robot via voice commands. While this method is more natural, its reliability is poor in noisy industrial environments or outdoor settings, and voice recognition is easily interfered with. A more fundamental drawback is that voice commands struggle to describe precise spatial positions and postures (e.g., "grab the red bolt on the left front, near the box"). The robot is prone to misjudgments or erroneous operations due to semantic misunderstandings, and there is also a lack of automatic spatial feasibility verification before execution.

[0005] 3. Pre-programmed fully autonomous mode: The robot operates entirely according to a pre-set program. While this mode eliminates the need for real-time human intervention, it is extremely inflexible, unable to adapt to dynamic changes in task content or environmental layout, and cannot handle any unprogrammed situations.

[0006] In summary, existing technologies lack a human-machine collaborative method that enables users to intuitively, accurately, and reliably specify spatial operation targets in complex real-world environments, while the system automatically and in a closed loop ensures safe and successful execution. Summary of the Invention

[0007] 1. The technical problem that the invention aims to solve

[0008] This invention provides a robot teleoperation method and system based on visual interaction. By automatically converting the user's intuitive selections into rigorously verified robot execution commands, it enables safe, accurate, and highly robust human-robot collaborative operations in complex environments.

[0009] 2. Technical Solution

[0010] To solve the above problems, the technical solution provided by the present invention is as follows:

[0011] A method for teleoperating a human-machine collaborative robot based on visual interaction, including

[0012] The user interface displays the real-time video feed transmitted from the robot.

[0013] In response to the user's tapping action on the video screen, the two-dimensional coordinates of the target in the image are determined;

[0014] The two-dimensional coordinates are spatially mapped to the three-dimensional environmental data collected by the robot to calculate the target's execution pose in the robot's base coordinate system;

[0015] Perform robot executability verification on the executed pose.

[0016] As a preferred embodiment of the present invention, the point selection operation includes a box selection operation, and the closed area information defined by the box selection operation is used as the input basis for triggering the background to perform target recognition and three-dimensional pose calculation.

[0017] As a preferred embodiment of the present invention, after the user completes the selection operation, operable instruction options are immediately provided in the user interface.

[0018] As a preferred embodiment of the present invention, the spatial mapping of two-dimensional coordinates to three-dimensional environmental data includes:

[0019] Record the timestamp of the video frame corresponding to when the selection operation occurs;

[0020] The synchronously acquired depth image data is obtained based on the timestamp;

[0021] Based on the depth image data obtained from the camera, the two-dimensional coordinates are back-projected onto three-dimensional spatial points in the robot's base coordinate system.

[0022] As a preferred embodiment of the present invention, determining the two-dimensional coordinates of the target in the image includes: converting the display coordinates generated based on the user's click operation into the original image pixel coordinates and performing normalization processing; the spatial mapping of the two-dimensional coordinates with the three-dimensional environment data collected by the robot includes: restoring the normalized coordinates to pixel coordinates and then using them to perform the spatial mapping.

[0023] As a preferred embodiment of the present invention, the executability verification includes verification of at least one of the following: robot kinematic reachability; collision interference on the motion path; and mechanical stability of the operation process.

[0024] As a preferred embodiment of the present invention, the verified tasks are added to an ordered task queue and the queue is displayed in the user interface; in response to the execution command, the robot is controlled to automatically and sequentially execute the tasks in the queue.

[0025] As a preferred embodiment of the present invention, the task queue is constructed and managed locally on the user interface; the response to the execution instruction includes: serializing the entire task queue and submitting it to the robot as a one-time transaction, which is then executed autonomously by the robot in sequence.

[0026] As a preferred embodiment of the present invention, if multiple candidate targets are detected within the selected area before adding a task to the queue, the user interface will display them separately and receive secondary confirmation from the user regarding the target entities.

[0027] A human-machine collaborative robot teleoperation system based on visual interaction, including

[0028] The display and interaction unit is used to display the video feed transmitted in real time from the robot and to receive user input.

[0029] A task processing unit, used to determine target coordinates and perform spatial mapping and executability verification in response to a selection operation; a memory storing a computer program; and

[0030] A processor configured to execute the program to implement the method steps described above.

[0031] 3. Beneficial effects

[0032] Compared with the prior art, the technical solution provided by this invention has the following advantages:

[0033] This invention effectively addresses the core pain points of ambiguous instructions and unreliable execution in traditional teleoperation through an innovative "what you see is what you get" visual interaction paradigm. The system allows users to specify task objectives through intuitive box selection or point selection, and then automatically completes high-precision mapping and calculation from 2D image coordinates to 3D robot executable poses. It also introduces a multi-layered, real-time verification mechanism (including kinematics, collision, and stability verification) throughout the entire task lifecycle, fundamentally ensuring the safety and success rate of operations. Simultaneously, the unique offline planning and batch task queue design significantly reduces the demanding dependence on network conditions and enhances availability under weak network conditions. This system deeply integrates human intuitive judgment with the precise execution capabilities of machines, achieving efficient, reliable, and easy-to-use human-machine collaborative operation. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the following specific embodiments are provided to further illustrate the invention in detail. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of the invention.

[0035] It should be noted that in this document, terms such as "comprising," "including," and "having" should be understood in an open-ended sense, meaning including but not limited to. When a step is referred to as "based on," "according to," or "in response to" a condition, signal, or data, it may be performed directly based on that condition, signal, or data, or it may undergo intermediate processing or transformation. Furthermore, the order of the steps in the method is for illustrative purposes only. Unless their logical dependencies are explicitly stated, the order of certain steps may be appropriately adjusted or they may be executed in parallel without departing from the spirit of the invention. Terms such as "teleoperation," "control command," and "feedback information" used herein refer to their meanings as commonly understood by those skilled in the art of robot control and interaction.

[0036] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0037] The terms "first" and "second" used in this invention do not represent a specific quantity or order, but are merely used to distinguish names.

[0038] This embodiment of a human-machine collaborative robot teleoperation system based on visual interaction mainly includes three parts: a teleoperation client, a backend server, and a robot body system.

[0039] The remote operation client runs on a computer, tablet, smartphone or other smart terminal. The client provides a graphical user interface to display the video stream transmitted back in real time from the robot and to receive interactive operations such as clicking and selecting by the user. The client is also responsible for managing the local task queue and communicating with the backend server via the MQTT protocol.

[0040] The backend server is deployed in the cloud or on an edge computing node on the same local area network as the robot. The server receives verification requests from the client, executes core spatial mapping, pose calculation and feasibility verification algorithms. The server integrates computer vision and robotics algorithm libraries and can interact with the robot through ROS2 service or MQTT.

[0041] The robot body system includes a robot mobile chassis (or robotic legs), robotic arm, end effector, RGB-D depth camera, industrial computer and underlying controller. The robot end uses ROS2 for internal high real-time communication and exchanges data with the external network through a lightweight MQTT-ROS2 bridge node.

[0042] The communication architecture system adopts a hybrid architecture of MQTT+ROS2 to balance the robustness of cross-network transmission with the real-time performance of local control.

[0043] The remote operation client and the backend server communicate via the MQTT protocol. The communication data includes verification requests, target pose results, and task queue information, which are encapsulated in a lightweight structured form in JSON format. The system adopts a hierarchical QoS communication mechanism based on the security and real-time requirements of different messages. Non-critical status and control information related to real-time interaction adopts the QoS0 "at most once" transmission mode to reduce communication latency, while critical information involving task security, queue transactions, and execution control adopts a reliable QoS transmission method with an acknowledgment mechanism to ensure the integrity and consistency of task execution.

[0044] The robot's internal sensors and controllers communicate with the industrial computer using ROS2 for millisecond-level real-time communication. The backend server triggers the robot's perception and actions by calling ROS2 services or publishing messages to specific topics.

[0045] The detailed workflow of this vision-interaction-based human-computer collaborative robot teleoperation system is as follows:

[0046] Step 1: The RGB-D camera on the robot continuously acquires color video streams and depth image streams, which are then encoded using H.264 and transmitted over the network to the client. The client decodes the video and plays it in real time in the display area of ​​the graphical user interface.

[0047] Step 2: Users can directly select (click) or drag (click and then drag) in the video screen.

[0048] There are two options for this step:

[0049] Option 1: Users watch live video and directly perform point (click) or rectangle (click and drag) operations on the video screen. All interactive visual feedback (such as mouse trajectory, selection rectangle) is superimposed on the dynamic video stream in real time. When the user releases the mouse (completes the operation), the client immediately captures the currently displayed video frame. This frame image reflects the screen state at the moment the user's operation is completed, and the system uses this captured frame as the calculation benchmark.

[0050] Option 2: When the system detects that the user begins a point-and-select operation (such as the instant the left mouse button is pressed), the client immediately freezes the video stream display. From this moment on, the screen displayed is fixed as the static reference image captured at the time of freezing. Subsequent user operations (such as dragging to complete the selection or observing the target location) are all performed on this static image, and the screen content no longer changes. After the user completes the operation, the system directly extracts the precise coordinate information from this static reference image.

[0051] Compared to Solution 1, Solution 2 maintains an absolutely fixed visual reference throughout the entire process from interaction to coordinate extraction. This fundamentally eliminates the coordinate reference drift problem caused by dynamic updates of the video stream, minor network transmission delays, or post-processing. This optimized solution provides spatiotemporally consistent and unambiguous input data for subsequent spatial mapping and pose calculation, and is one of the key technical measures to ensure the high accuracy and reliability of the entire system.

[0052] Step 3: The target is selected after the user releases the mouse.

[0053] If it is a point selection: The system records the two-dimensional pixel coordinates (u,v) of the mouse release point in the image. These coordinates indicate the approximate location of the user's focus. Subsequent background processing will use this point as the center to search for and identify potential operation targets within a preset neighborhood.

[0054] If the selection is a bounding box (preferred method): The system records the closed area defined by the mouse drag trajectory. This area is usually represented by the upper left corner (x_min, y_min) and the lower right corner (x_max, y_max), with the unit being pixels. This area information does not directly correspond to the operation target, but rather limits the search range for the target recognition and pose calculation algorithm in the background. The background will analyze the scene content within this area, identify specific target entities (such as a single object, a specific component, or an operation point), and calculate the corresponding operation pose.

[0055] To ensure the stability and consistency of target coordinate extraction and to avoid spatial mapping errors caused by changes in client display status (such as window size adjustment, display scaling, and resolution differences), the system clearly distinguishes between the display coordinate system and the image content coordinate system during coordinate processing.

[0056] User clicks or selections first occur on the client-side display interface. The system then precisely maps the display coordinates back to the corresponding original video frame pixel coordinate system based on the current video rendering parameters (including scaling ratio, cropping area, and display offset), obtaining pixel coordinates (u_img, v_img) based on the original image resolution.

[0057] Subsequently, the system normalizes the pixel coordinates based on the actual width W_img and height H_img of the original image:

[0058] u_norm=u_img / W_img;

[0059] v_norm=v_img / H_img;

[0060] In this way, the normalized coordinates are strictly anchored to the image content itself, independent of the client display layout, window size, and terminal device resolution, thus ensuring the consistency of spatial mapping and pose calculation results under different display environments. The normalized coordinates or bounding box information are encapsulated in the task request and transmitted to the background processing module. Step 4: After the user completes the point / box selection operation, a context menu immediately pops up on the client interface, providing optional robot operation instructions, such as "grasp" and "release". The menu options can be generated based on simple image analysis of the selected area (such as color and texture) or predefined scene rules, with the aim of reducing the steps for the user to search for and select instructions.

[0061] Step 5: After the user selects a specific command (such as "fetch") from the pop-up context menu, the client will generate a structured task verification request, which will be sent via the publish / subscribe mode of the MQTT protocol.

[0062] Specifically, the client publishes the request to a specific topic corresponding to the target robot and the verification function. In a preferred embodiment, the topic uses a clear hierarchical naming structure, such as robot / {robot_id} / check, where {robot_id} is the unique identifier of the specific robot. This design allows the backend service to easily distinguish the message source and type based on the topic, achieving efficient and accurate message routing and supporting the parallel management of multiple robots.

[0063] The message body of the verification request is encapsulated in a lightweight structured data format such as JSON to clearly convey the task identifier, operation type, and target area information. An example of its specific structure is as follows:

[0064] {

[0065] "id":"task_20250101_001",

[0066] "action_type":"grasp",

[0067] "mask":[[x_min,y_min],[x_max,y_min],[x_max,y_max],[x_min,y_max]]}

[0068] Here, id is a unique task identifier, action_type is the operation type, and mask is the coordinates (in pixels) of the quadrilateral vertices of the selected area. For release tasks, the bbox field is used to pass the rectangle [x_min, y_min, x_max, y_max].

[0069] Step Six: The backend server subscribes to the verification request topic. Upon receiving a request, it executes the following core calculation process:

[0070] The server records the arrival time of the request and, through a ROS2 service call, requests from the robot a synchronized frame of color and depth image corresponding to that timestamp, or the most recent frame or the closest frame in the cache. The robot's camera driver ensures that the RGB image and depth map are hardware-aligned.

[0071] Formula link reference:

[0072] (u_norm,v_norm)→u_img=u_norm*W,v_img=v_norm*H

[0073] Z = depth[u_img, v_img]

[0074] Camera coordinate system:

[0075] X_c=(u_img-cx)*Z / fx

[0076] Y_c=(v_img-cy)*Z / fy

[0077] Z_c=Z

[0078] Homogeneous coordinates:

[0079] P_camera_hom=[X_c,Y_c,Z_c,1]^T

[0080] Robot base coordinate system:

[0081] P_base_hom=T_base_cam*P_camera_hom

[0082] X_base,Y_base,Z_base=P_base_hom[:3]

[0083] The server extracts the valid depth values ​​for the corresponding region from the depth map based on the mask or bounding box information in the request. All depth values ​​are in meters and have undergone distortion correction and denoising. For the user-provided normalized coordinates (u_norm, v_norm), the system first converts them into raw pixel coordinates based on the original image width W_img and height H_img.

[0084] u_img=u_norm*W_img

[0085] v_img=v_norm*H_img

[0086] Subsequently, using the camera intrinsic parameters (obtained through calibration, including focal lengths fx and fy, and optical centers cx and cy), the coordinates of each pixel are back-projected to obtain a 3D point in the camera coordinate system:

[0087] X_c=(u_img-cx)*Z / fx

[0088] Y_c=(v_img-cy)*Z / fy

[0089] Z_c=Z

[0090] Where Z represents the effective depth value of the corresponding pixel in the depth map. The point cloud is expanded to homogeneous coordinates:

[0091] P_camera_hom=[X_c,Y_c,Z_c,1]^T

[0092] Using a pre-calibrated hand-eye transformation matrix T_base_cam (4x4 homogeneous matrix), points in the camera coordinate system are transformed to the robot base coordinate system:

[0093] P_base_hom=T_base_cam*P_camera_hom

[0094] P_base=[X_base,Y_base,Z_base] / / Take the first 3 components as the 3D position

[0095] The target point cloud, transformed to the robot's base coordinate system, is then input into a pre-trained 6D grasping pose estimation model (e.g., based on the PointNet++ architecture). The model outputs the grasping pose pose_7d=[x,y,z,qx,qy,qz,qw], where (x,y,z) is the TCP target position of the end effector in the robot's base coordinate system, and (qx,qy,qz,qw) is the grasping pose represented by unit quaternions, with the quaternion order being [qx,qy,qz,qw]. All length units are meters.

[0096] Step 7: If the background detects multiple independent physical entities within the selected area during the point cloud processing in Step 6 using an instance segmentation algorithm (such as Mask R-CNN) or a point cloud clustering algorithm (such as DBSCAN), then the disambiguation process is initiated.

[0097] The algorithm not only identifies individual entities, but also focuses on analyzing the spatial inclusion and semantic relationships between them:

[0098] When multiple objects are detected being contained in or wrapped by a larger, identifiable container (such as a bag, box, or tray), the algorithm identifies the container itself as a single "packaged entity".

[0099] At the same time, it can penetrate the surface (if the container is transparent or open) or infer from spatial relationships to identify multiple "content entities" inside.

[0100] If multiple objects are directly stacked, scattered, or placed side by side without being enclosed by a unified container, they are all identified as independent "scattered objects".

[0101] Generate multi-dimensional feature descriptions for each of the above candidate entities (including: packaging entities, contents entities, and scattered entities), including visual features, semantic categories, three-dimensional spatial locations, and specifically annotate their organizational structure attributes (such as: "packaging container (carton)", "internal individual (bolt)", "scattered individual").

[0102] The backend sends the candidate target information back to the client and displays it hierarchically and with highlights on the static reference image:

[0103] Users need to select a single entity from all the highlighted targets as the final operation object. The options are: the entire container (e.g., grab the whole box); a specific object inside the container (e.g., take a part out of the box); or scattered independent objects.

[0104] After receiving the user's selection, the system will only perform subsequent pose calculations on the selected target.

[0105] Step 8: For the calculated capture pose_7d, the system performs multi-dimensional verification:

[0106] Kinematic reachability verification: Call the robot's inverse kinematics (IK) solver to verify whether there exists a set of joint angles that allow the end effector of the robotic arm to reach the pose in a configuration without singularities.

[0107] Collision detection: Based on the robot's URDF model and the 3D point cloud (or simplified bounding box model) of the environment, collision interference checks are performed using a motion planning library (such as FCL) to ensure that the path from the current position to the target pose will not collide with the environment or the robot itself.

[0108] Mechanical stability assessment: For gripping operations, assess whether the contact force between the gripper and the target object in this position satisfies the force closure condition, or judge whether the gripping is stable based on empirical rules.

[0109] Step 9: After verification, the backend server sends the result back to the client via MQTT.

[0110] If the verification passes, a success message containing the calculated pose_7d is returned.

[0111] The message format is as follows:

[0112] {

[0113] "id":"task_20250101_001",

[0114] "status":"success",

[0115] "action_type":"grasp",

[0116] "pose_7d":[0.321,-0.105,0.450,0.0,0.7071,0.0,0.7071]}

[0117] Where id is a unique task identifier, action_type is the operation type, and status is the final status of this verification.

[0118] If the verification fails: a failure message containing the specific error code and description is returned.

[0119] Error types and actionable feedback include, but are not limited to, "No valid XYZ obtained", "Catching service unavailable", "System busy", etc., guiding users or maintenance personnel to take appropriate measures.

[0120] Step 10: After receiving the verification message, the client automatically generates a task card and adds it to the client's ordered task queue panel. Each task card contains: task number, operation type icon, thumbnail, and simplified text display of the target pose.

[0121] The task queue is built and managed locally on the client side, which means that even when the network connection is unstable, users can still select and plan multiple tasks continuously, and all task data is temporarily stored in the local browser or application storage.

[0122] When the user has planned all tasks and clicked the "Execute" button, the client serializes the entire task queue into a JSON array and publishes it as a one-time transaction to the topic robot / {robot_id} / task_list via MQTT. This "offline planning, batch submission" mode greatly reduces the reliance on continuous high bandwidth and low latency of the network, making it particularly suitable for weak network environments. However, the environment needs to be quickly re-verified before execution.

[0123] Step 11: After receiving the task list, the bridging node on the robot side begins to execute the tasks automatically in sequence.

[0124] For each task in the queue, before execution, the system will reacquire the current sensor data (RGB-D image) and perform a quick pre-execution safety check on the target pose of the task. This step is to deal with possible environmental changes that may occur during the time from task planning to execution (such as someone entering the work area or the target being moved).

[0125] The system continuously monitors the robot's movements and operations.

[0126] Read data from the joint torque sensor to detect abnormal forces.

[0127] Obstacle approach detection is performed using distance sensors (such as LiDAR) on the robot itself or in the environment.

[0128] If any security risks are detected during pre-execution verification or process monitoring, the robot will immediately pause or terminate the current action and notify the client. Additionally, depending on the configuration, the system may automatically clear the remaining task queue to prevent error accumulation.

[0129] The system provides structured and actionable feedback for various failure scenarios, such as:

[0130] "AI service unavailable": This prompts you to check the running status of the background AI module.

[0131] "No valid XYZ values ​​obtained": This prompts the user to adjust the selected area to ensure that the target has valid values ​​in the depth map.

[0132] "System busy": This message prompts the user to wait for the current task to complete before submitting a new request.

[0133] To ensure the real-time performance of remote operations, the system has implemented the following optimizations:

[0134] Based on the real-time measured network bandwidth, the encoding bitrate and frame rate of the video stream are dynamically adjusted to prioritize the smooth operation of the command transmission channel.

[0135] Each instruction carries a sending timestamp. If the instruction received by the robot is delayed beyond a threshold (e.g., 50ms), it will be discarded to avoid executing outdated commands.

[0136] The client sends heartbeat packets periodically. The MQTTBroker is configured with a Will (LWT) message. Once the client disconnects abnormally, the Broker will automatically publish an LWT message, triggering the robot to enter a safe pause state.

[0137] Example

[0138] Taking hotel room cleaning as an example, the complete workflow of this system is demonstrated:

[0139] In the client-side video, the operator selects an empty water bottle to be recycled on the target bedside table. A command menu pops up, and the operator clicks "Grab". The backend calculates the grabbing pose of the water bottle through spatial mapping and completes the feasibility verification. A "Grab Empty Bottle" task is automatically added to the local queue.

[0140] The operator then selects the location of the full water bottle next to the mini-fridge, chooses the "place" command, and the system calculates and verifies the placement pose. A "replenish water bottle" task is added to the queue, forming an ordered sequence with the previous task.

[0141] When the operator clicks "Execute", the client packages the task sequence of "grab empty bottle - place water bottle" and sends it to the service robot all at once.

[0142] After receiving the instruction, the robot first executes the "grab empty bottle" command. Before moving, it quickly re-acquires the image of the current bedside table area for a safety check. Once it confirms that the target location has not changed and the path is safe, it completes the grab. Next, it executes the "replenish water bottle" command, similarly performing pre-execution checks before accurately placing the water bottle in the designated location. Throughout the process, if the system detects someone approaching via real-time video stream, it will immediately pause its actions and issue an alarm.

[0143] If the operator selects multiple different items within the area (such as an empty bottle and a remote control), the system will identify each object through instance segmentation, highlight and label them (such as "empty water bottle" and "remote control"). The process will continue after the operator clicks to select "empty water bottle".

[0144] The operator selects an empty water bottle to be recycled on the bedside table in the client's video feed. The system freezes the screen, displays a command menu, and the operator clicks "Grab." The backend calculates the grabbing pose of the water bottle through spatial mapping and completes a feasibility check. A "Grab Empty Bottle" task is automatically added to the local queue.

[0145] The operator then selects the location of the full water bottle next to the mini-fridge and chooses the "place" command. The system calculates the placement pose and verifies it. A "replenish water bottle" task is added to the queue, forming an ordered sequence with the previous task.

[0146] The operator clicks "Execute". The client packages the "grab empty bottle - place water bottle" task sequence and sends it to the service robot all at once.

[0147] After receiving the instruction, the robot first executes the "grab empty bottle" command. Before moving, it quickly re-acquires an image of the current bedside table area for a safety check, confirming that the target location has not changed and the path is safe before completing the grab. Next, it executes the "replenish water bottle" command, similarly performing pre-execution checks before accurately placing the water bottle in the designated location. Throughout the process, if the system detects someone entering the room via real-time video stream, it will immediately pause its actions and issue an alarm.

[0148] If the operator selects multiple different items within the area (such as an empty bottle and a remote control), the system will identify each object through instance segmentation, highlight and label them (such as "empty water bottle" and "remote control"). The process will continue after the operator clicks to select "empty water bottle".

[0149] This invention deeply integrates human intuitive visual judgment with the precise automatic execution capabilities of robots. It replaces reliance on precise verbal descriptions with visual-spatial interaction, overcoming the inherent shortcomings of traditional voice control, such as vague instructions in complex environments, ambiguity, and difficulty in accurately binding with spatial scenes. Rigorous step-by-step verification ensures operational safety and success, and ingenious architectural design reduces stringent requirements on the network environment. It is an efficient, reliable, and easy-to-use remote operation solution for human-robot collaborative robots.

[0150] Traditional voice control methods face significant challenges in scenarios like hotel cleaning. For example, when an operator issues a voice command to "grab that bottle," the robot faces multiple uncertainties: First, it needs to correctly identify the object category "bottle" within its field of vision; second, when multiple bottles exist in the scene (such as empty water bottles, full water bottles, and detergent bottles), the voice command cannot clearly specify which one; third, the command "grab" itself does not contain precise spatial location and posture information, and the robot still needs to rely on its own vision system to perform complex recognition and positioning calculations, and this process lacks direct human confirmation in the loop, which can easily lead to misoperation.

[0151] In contrast, the visual selection interaction method of the present invention directly solves the above problems: the operator directly selects a specific target water bottle in the video screen. This action simultaneously completes the specification of the target instance and the determination of the operation area, transforming ambiguous semantic instructions into clear spatial instructions, eliminating ambiguity from the source, and providing an intuitive interface for human-computer collaborative confirmation.

[0152] The present invention and its embodiments have been described above illustratively, and this description is not restrictive; the actual structure is not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A method for teleoperating a human-machine collaborative robot based on visual interaction, characterized in that: include The user interface displays the real-time video feed transmitted from the robot. In response to the user's tapping action on the video screen, the two-dimensional coordinates of the target in the image are determined; The two-dimensional coordinates are spatially mapped to the three-dimensional environmental data collected by the robot to calculate the target's execution pose in the robot's base coordinate system; Perform robot executability verification on the executed pose.

2. The method for teleoperating a human-machine collaborative robot based on visual interaction according to claim 1, characterized in that: The point selection operation includes a box selection operation, and the closed area information defined by the box selection operation is used as the input basis for triggering the background to perform target recognition and 3D pose calculation.

3. The method for teleoperating a human-machine collaborative robot based on visual interaction according to claim 1, characterized in that: After the user completes the selection operation, operable command options are immediately provided in the user interface.

4. The method for teleoperating a human-machine collaborative robot based on visual interaction according to claim 1, characterized in that: The spatial mapping of two-dimensional coordinates to three-dimensional environmental data includes: Record the timestamp of the video frame corresponding to when the selection operation occurs; The synchronously acquired depth image data is obtained based on the timestamp; Based on the depth image data obtained from the camera, the two-dimensional coordinates are back-projected onto three-dimensional spatial points in the robot's base coordinate system.

5. The method for teleoperating a human-machine collaborative robot based on visual interaction according to claim 1, characterized in that: Determining the two-dimensional coordinates of the target in the image includes: converting the display coordinates generated based on the user's click operation into the original image pixel coordinates and performing normalization processing; The step of spatially mapping two-dimensional coordinates to three-dimensional environmental data collected by the robot includes: restoring the normalized coordinates to pixel coordinates before performing the spatial mapping.

6. The method for teleoperating a human-machine collaborative robot based on visual interaction according to claim 1, characterized in that: The executability verification includes verification of at least one of the following: robot kinematic reachability; collision interference on the motion path; and mechanical stability of the operation process.

7. The method for teleoperating a human-machine collaborative robot based on visual interaction according to claim 1, characterized in that: The verified tasks are added to an ordered task queue, and the queue is displayed in the user interface; in response to the execution command, the robot is controlled to automatically and sequentially execute the tasks in the queue.

8. The method for teleoperating a human-machine collaborative robot based on visual interaction according to claim 7, characterized in that: The task queue is built and managed locally in the user interface; The response to the execution instruction includes: serializing the entire task queue and submitting it to the robot as a one-time transaction, which the robot then executes autonomously in sequence.

9. A method for teleoperating a human-machine collaborative robot based on visual interaction according to any one of claims 1-8, characterized in that: Before adding a task to the queue, if multiple candidate targets are detected within the selected area, they are displayed separately on the user interface, and the user is given secondary confirmation of the target entity.

10. A human-machine collaborative robot teleoperation system based on visual interaction, characterized in that: include The display and interaction unit is used to display the video feed transmitted in real time from the robot and to receive user input. The task processing unit is used to respond to the point selection operation, determine the target coordinates, and perform spatial mapping and executability verification; the memory stores the computer program; as well as A processor configured to execute the program to implement the method steps as described in any one of claims 1 to 9.