Open-vocabulary robotic arm semantic control system combining visual language models

Abstract

An open-vocabulary robotic arm semantic control system incorporating a visual language model includes a robotic arm and a control device. The robotic arm has multiple arm axes, one of which is fixed in place. A gripper and a 3D camera are mounted on the other arm axis, allowing the 3D camera to capture color images and depth information of the current workspace as the robotic arm moves. The control device is connected to the robotic arm and receives semantic input commands. It utilizes a visual language model to understand the intent of the input semantic commands and, in conjunction with an open-vocabulary recognition model, identifies non-preset target objects in complex scenes.

Claims

1. An open-vocabulary robotic arm semantic control system combining a visual language model, comprising: a robotic arm and a control device, wherein: The robotic arm has multiple arm axes connected by joints. The robotic arm is fixed at one end of the arm axis, and a gripper and a 3D camera are installed on the other end of the arm axis. The 3D camera can capture color images and depth information of the current workspace as the robotic arm moves. The control device is connected to the robotic arm. The control device can receive semantic input commands and use a visual language model within the control device to understand the intent of the input semantic commands. It also combines an open vocabulary recognition model to identify non-preset target objects in complex scenes. Finally, the movement path and gripping action of the robotic arm are generated through a rule-based execution module. The open vocabulary recognition model and the rule-based execution module are built into the internal database of the control device.

2. The open-vocabulary robotic arm semantic control system combining a visual language model as described in claim 1, wherein, The control device and the robotic arm are connected via wired transmission.

3. The open-vocabulary robotic arm semantic control system combining a visual language model as described in claim 1, wherein, The control device and the robotic arm are connected wirelessly.

4. The open-vocabulary robotic arm semantic control system combining a visual language model as described in claim 1, wherein, The control device is a desktop computer.

5. The open-vocabulary robotic arm semantic control system combining a visual language model as described in claim 1, wherein, The control device is a laptop computer.

6. The open-vocabulary robotic arm semantic control system combining a visual language model as described in claim 1, wherein, The control device is a tablet computer.

7. The open-vocabulary robotic arm semantic control system combining a visual language model as described in claim 1, wherein, The control device is a smartphone.

Images