Object pose recognition and grasping method and device, equipment and medium

Abstract

The present application relates to the technical field of machine vision, and discloses an object pose recognition and grasping method, device, equipment and medium, the method comprising: receiving a natural language instruction containing a target semantic, and obtaining visual information corresponding to a warehouse operation environment based on the natural language instruction, wherein the target semantic comprises a target object; inputting the visual information and the natural language instruction into a pre-trained end-to-end visual language action model for cross-modal alignment and fusion, generating an instance recognition result of the target object and a candidate grasping pose corresponding to the instance recognition result; converting the candidate grasping pose to a pre-constructed robot base coordinate system, and optimizing and solving the candidate grasping pose after the coordinate system conversion based on kinematic constraints to obtain a robot joint trajectory sequence; and controlling the warehouse robot to execute the joint trajectory sequence to complete grasping of the target object. The present application improves the accuracy of object pose recognition and grasping.

Images