Method and apparatus for robotic grasping of three-dimensional objects

Abstract

Embodiments of the present disclosure provide a method for a robot to grasp a three-dimensional object, comprising: determining a current position and pose of a vision sensor of the robot relative to the three-dimensional object; obtaining a grasping template of the three-dimensional object, the grasping template comprising a specified grasping position and pose of the vision sensor relative to the three-dimensional object; determining whether the grasping template further comprises at least one reference grasping position and pose of the vision sensor relative to the three-dimensional object, wherein the reference grasping position and pose are generated based on the specified grasping position and pose; and generating a grasping position and pose of the robot based on the grasping template and the current position and pose, according to a determination result. In the present disclosure, the reference grasping position and pose are used to optimize the motion path of the robot from the current position and pose to the grasping position and pose, thereby increasing the grasping flexibility, improving the grasping speed and efficiency, and saving energy. The method is simple and easy to implement, has a small amount of data calculation, and does not require an expensive three-dimensional camera, thereby saving time and economic cost.

Description

Technical Field

[0001] This disclosure relates to the technical field of industrial robots, and more specifically, to methods, apparatus, computing devices, computer-readable storage media, and program products for robots to grasp three-dimensional objects. Background Technology

[0002] Using robots to grasp three-dimensional objects (3D objects) is a common process in industrial workflows. When performing object grasping tasks, the relative position and pose between the 3D object and the camera are particularly important. Currently, methods for estimating the relative position and pose between a 3D object and a 2D camera typically use template matching or deep learning to directly estimate the position and pose. Alternatively, some existing technologies employ methods based on a 3D camera to estimate the relative position and pose between the 3D object and the camera. In these existing technologies, the position and pose of the 3D object in space are obtained by comparing the actual scanned 3D object with its model.

[0003] Traditional methods for acquiring the grasp point, grasp position, and attitude include traction teaching and deep learning. Furthermore, traditional methods for representing position and attitude typically use six parameters: x, y, z, roll angle, pitch angle, and yaw angle. Summary of the Invention

[0004] Traditional template matching methods based on 2D cameras cannot detect objects in complex environments, while obtaining the position and pose of a 3D object relative to the camera is extremely complex and difficult using deep learning-based methods. Position and pose estimation methods based on 3D cameras require high-precision cameras, which are expensive and very time-consuming.

[0005] Furthermore, in traction teaching methods, the grasp point, grasp position, and pose are fixed, lacking flexibility. Calculating the grasp point, grasp position, and pose using deep learning relies on massive datasets, making it equally expensive and time-consuming. Moreover, the traditional six parameters representing position and pose are redundant in some cases, increasing the computational burden.

[0006] The first embodiment of this disclosure proposes a method for a robot to grasp a three-dimensional object, comprising: determining the current position and orientation of the robot's vision sensor relative to the three-dimensional object; acquiring a grasping template of the three-dimensional object, the grasping template including a specified grasping position and orientation of the vision sensor relative to the three-dimensional object; determining whether the grasping template also includes at least one reference grasping position and orientation of the vision sensor relative to the three-dimensional object, wherein the reference grasping position and orientation are generated based on the specified grasping position and orientation; and generating the robot's grasping position and orientation using the grasping template and the current position and orientation based on the determination result.

[0007] In this embodiment, by determining whether the preset grasping template includes a reference grasping position and posture of the 3D object to be grasped, the robot's motion path from its current position and posture to the grasping position and posture can be optimized when the template includes such a position and posture. This increases the flexibility of grasping, improves grasping speed and efficiency, and saves energy. Furthermore, this method is simple and easy to implement, requires minimal data computation, and does not require an expensive 3D camera, thus saving time and economic costs.

[0008] A second embodiment of this disclosure provides an apparatus for a robot to grasp a three-dimensional object, comprising: a current position and attitude determination unit configured to determine the current position and attitude of the robot's vision sensor relative to the three-dimensional object; a grasping template acquisition unit configured to acquire a grasping template of the three-dimensional object, the grasping template including a specified grasping position and attitude of the vision sensor relative to the three-dimensional object; a reference position and attitude judgment unit configured to determine whether the grasping template also includes at least one reference grasping position and attitude of the vision sensor relative to the three-dimensional object, wherein the reference grasping position and attitude are generated based on the specified grasping position and attitude; and a grasping position and attitude generation unit configured to generate the robot's grasping position and attitude based on the judgment result, using the grasping template and the current position and attitude.

[0009] A third embodiment of this disclosure provides a computing device comprising: a processor; and a memory for storing computer-executable instructions that, when executed, cause the processor to perform the method of the first embodiment.

[0010] A fourth embodiment of this disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon for performing the method of the first embodiment.

[0011] A fifth embodiment of this disclosure provides a computer program product tangibly stored on a computer-readable storage medium and including computer-executable instructions that, when executed, cause at least one processor to perform the method of the first embodiment. Attached Figure Description

[0012] Features, advantages, and other aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description, in which several embodiments of the disclosure are illustrated by way of example and not limitation, in the drawings:

[0013] Figure 1A method for a robot to grasp a three-dimensional object according to an embodiment of the present disclosure is shown;

[0014] Figure 2 An embodiment of the present disclosure is shown for implementation. Figure 1 System architecture diagram of the method in the text;

[0015] Figure 3 It shows the use of Figure 2 The system executes a workflow for grasping 3D objects;

[0016] Figure 4 It shows in Figure 2 A schematic diagram of an Earth model built from a virtual model of an exemplary three-dimensional object in the system;

[0017] Figure 5 It shows as Figure 4 The virtual model of the exemplary 3D object in the image specifies the front view of the Earth model when the grab point is specified;

[0018] Figure 6 An apparatus for a robot to grasp a three-dimensional object according to an embodiment of the present disclosure is shown; and

[0019] Figure 7 A block diagram of a computing device for a robot to grasp a three-dimensional object is shown according to an embodiment of the present disclosure. Detailed Implementation

[0020] Various exemplary embodiments of this disclosure are described in detail below with reference to the accompanying drawings. While the exemplary methods and apparatuses described below include software and / or firmware executed on hardware among other components, it should be noted that these examples are merely illustrative and should not be considered limiting. For example, it is conceivable that any or all hardware, software, and firmware components may be implemented exclusively in hardware, exclusively in software, or in any combination of hardware and software. Therefore, although exemplary methods and apparatuses have been described below, those skilled in the art will readily understand that the examples provided are not intended to limit the ways in which these methods and apparatuses may be implemented.

[0021] Furthermore, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of the methods and systems according to various embodiments of this disclosure. It should be noted that the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0022] The terms “comprising,” “including,” and similar terms as used herein are open-ended, meaning “including / including but not limited to,” implying that other content may also be included. The term “based on” means “at least partially based on.” The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment,” and so on.

[0023] Figure 1 A method for a robot to grasp a three-dimensional object according to an embodiment of the present disclosure is shown. This method can be executed by an industrial computer in a factory. Reference Figure 1 Method 100 begins with step 101. In step 101, the current position and orientation of the robot's vision sensor relative to the three-dimensional object are determined. The vision sensor is a two-dimensional vision sensor, such as a two-dimensional camera. In this embodiment, the vision sensor is mounted at the robot's end effector (e.g., at the robot's gripper) and moves with the robot. The current position and orientation of the vision sensor relative to the three-dimensional object represent the relative position and orientation between the vision sensor and the three-dimensional object in the current state.

[0024] In some embodiments, step 101 further includes: acquiring a real image of the 3D object from the perspective of its current position and pose; acquiring an image template of the 3D object, the image template representing multiple virtual images of the virtual model of the 3D object from multiple different positions and poses; determining the similarity between the real image and each of the multiple virtual images; and generating the current position and pose based on the corresponding position and pose of the virtual image with the highest similarity. A real image of the 3D object from the perspective of the current position and pose of the visual sensor can be obtained through a visual sensor. Typically, this real image includes other objects in the environment besides the 3D object. To determine the current position and pose of the visual sensor relative to the 3D object, the real image can be matched with the image template of the 3D object. The image template represents multiple virtual images of the virtual model of the 3D object from multiple different positions and poses of the visual sensor. The image template of the 3D object can be pre-established. In this embodiment, the virtual model of the 3D object is a CAD design model of the 3D object. In the virtual environment, the position and pose changes of the visual sensor relative to the virtual model of the 3D object are simulated, thereby simulating the different relative positions and poses between the visual sensor and the 3D object in the real world. It can iterate through each position and pose within a preset range of positions and poses to obtain a virtual image of the 3D object's virtual model from the viewpoint of each position and pose. The real image of the 3D object is compared with each of the multiple virtual images, and the similarity between them is determined. The highest similarity can be compared with a preset threshold (e.g., 90% or 95%). If the highest similarity exceeds the preset threshold, the position and pose of the virtual image with the highest similarity is taken as the current position and pose of the visual sensor relative to the 3D object. The preset position and pose range and preset threshold can be set by the user via an input interface such as a touchscreen.

[0025] In some embodiments, after obtaining virtual images of the virtual model of a 3D object from the viewpoint of each position and pose, these images are converted into binary images. The features of this binary image are obtained using HoG (Histogram of Gradients) and represented by an N-dimensional vector. This N-dimensional vector represents the edge features of the virtual model of the 3D object in the virtual image. All these N-dimensional vectors of the virtual images and their corresponding positions and poses are stored in memory as image templates.

[0026] In some embodiments, determining the similarity between a real image and each of a plurality of virtual images further includes: generating a mask of a 3D object based on the real image of the 3D object using a Mask-RCNN model; and obtaining features of the real image of the 3D object based on the mask of the 3D object. The Mask-RCNN model can be trained in advance using multiple real image samples of the 3D object. The Mask-RCNN model is capable of recognizing 3D objects from real images of 3D objects and generating a mask of the 3D object. The mask of the 3D object is then converted into a binary image. Features of this binary image are also obtained using HoG, thereby representing the edge features of the 3D object in the real image with an N-dimensional vector. Compared to traditional methods that directly perform template matching using real images, extracting the mask of the 3D object using the Mask-RCNN model before template matching increases the robustness and accuracy of recognizing 3D objects in complex environments.

[0027] In some embodiments, similarity includes the similarity between features of the real image and features of each virtual image. The similarity between an N-dimensional vector derived from the real image and an N-dimensional vector derived from each virtual image is determined. Any similarity calculation method in the prior art (e.g., Euclidean distance) can be used to determine the similarity.

[0028] Continue to refer to Figure 1 In step 102, a grasping template for the 3D object is obtained. The grasping template includes a specified grasping position and orientation of the vision sensor relative to the 3D object. The specified grasping position and orientation can be pre-specified by the user and represent the position and orientation of the vision sensor relative to the 3D object when the robot grasps it. In a virtual environment, the user can determine the specified grasping position and orientation by adjusting the simulated position and orientation of the vision sensor relative to a virtual model of the 3D object.

[0029] In some embodiments, method 100 further includes ( Figure 1(Not shown in the image): Generate a grasping template. The grasping template is used to obtain the grasping position and orientation of the robot's vision sensor relative to the 3D object when grasping it. In addition to the specified grasping position and orientation of the vision sensor relative to the 3D object, the grasping template may selectively include at least one reference grasping position and orientation of the vision sensor relative to the 3D object. The reference grasping position and orientation are other grasping positions and orientations of the vision sensor relative to the 3D object that can be selected by the robot when grasping the 3D object, besides the specified grasping position and orientation. The reference grasping position and orientation can be generated based on the specified grasping position and orientation. In some embodiments, the reference grasping position and orientation can be generated using a specified virtual image of a virtual model of the 3D object from the perspective of the specified grasping position and orientation. From the perspective of the reference grasping position and orientation, the similarity between the virtual image of the virtual model of the 3D object and the specified virtual image is higher than a preset threshold.

[0030] In some embodiments, generating a grasping template further includes: acquiring a specified virtual image of a virtual model of a three-dimensional object from a specified grasping position and posture; simulating multiple different positions and postures of a visual sensor relative to the three-dimensional object to obtain multiple virtual images of the virtual model of the three-dimensional object from multiple different positions and postures; determining the similarity between each virtual image in the multiple virtual images and the specified virtual image; and using the corresponding position and posture of the virtual image with a similarity higher than a preset threshold as a reference grasping position and posture.

[0031] As described above, in a virtual environment, a user can determine a specified grasping position and posture by adjusting the position and pose of a simulated visual sensor relative to a virtual model of a 3D object. After determining the specified grasping position and posture, a virtual image of the 3D object's virtual model is obtained from the viewpoint of that specified grasping position and posture, and used as the specified virtual image. The specified virtual image may include a two-dimensional projection of the 3D object's virtual model from the viewpoint of the specified grasping position and posture. Next, in the virtual environment, the simulated visual sensor is made to change its position and posture relative to the virtual model of the 3D object, thereby simulating different relative positions and postures between the visual sensor and the 3D object in the real world. Each position and posture within a preset range of positions and postures can be traversed to obtain a virtual image of the 3D object's virtual model from the viewpoint of each position and posture. Each of these virtual images is compared with the specified virtual image, and their similarity is determined. When the similarity between a virtual image and the specified virtual image exceeds a preset threshold (that is, the difference between them is less than a certain threshold), the corresponding position and posture of that virtual image is used as the reference grasping position and posture. As mentioned above, users can set preset positions, posture ranges, and preset thresholds via input interfaces such as touchscreens.

[0032] In some embodiments, similarity includes the similarity between features of a virtual image and features of a specified virtual image. As described above, the virtual image of the virtual model of the 3D object at each position and pose is converted into a binary image, and its features are represented by an N-dimensional vector using HoG. Similarly, the specified virtual image of the virtual model of the 3D object at a specified grasping position and pose is also converted into a binary image, and its features are represented by an N-dimensional vector using HoG. The similarity between the N-dimensional vector obtained from each virtual image and the N-dimensional vector obtained from the specified virtual image is determined. Any similarity calculation method in the prior art can be used to determine the similarity. The N-dimensional vectors and corresponding positions and poses of all virtual images with similarity higher than a preset threshold, as well as the N-dimensional vector of the specified virtual image and the specified grasping position and pose, are stored in memory as grasping templates for the 3D object. When the similarity is lower than the preset threshold, only the N-dimensional vector of the specified virtual image and the specified grasping position and pose are stored in memory as grasping templates for the 3D object.

[0033] Next, in step 103, it is determined whether the grasping template also includes at least one reference grasping position and pose of the vision sensor relative to the 3D object, wherein the reference grasping position and pose are generated based on the specified grasping position and pose. As mentioned above, when generating the grasping template, the virtual image of the 3D object from the viewpoint of the specified grasping position and pose is similar to the virtual image from the viewpoint of the reference grasping position and pose. Similarly, during grasping, the image of the 3D object from the viewpoint of the specified grasping position and pose will also be similar to the image from the viewpoint of the reference grasping position and pose. Therefore, when the grasping template includes the reference grasping position and pose of the vision sensor relative to the 3D object, the robot's grasping position and pose can be generated using the reference grasping position and pose, thereby optimizing the robot's path from its current position and pose to the grasping position and pose.

[0034] In step 104, based on the judgment result, the robot's grasping position and posture are generated using the grasping template and the current position and posture. The specified grasping position and posture, and the reference grasping position and posture, are both the position and posture of the vision sensor relative to the 3D object. For the robot to grasp a 3D object, it is necessary to first generate the target grasping position and posture of the vision sensor relative to the 3D object, and then convert it into the robot's grasping position and posture.

[0035] In some embodiments, generating the robot's grasping position and posture using a grasping template and the current position and posture based on the judgment result further includes: determining the target grasping position and posture of the vision sensor relative to the three-dimensional object based on the judgment result using the grasping template and the current position and posture; and converting the target grasping position and posture into the robot's grasping position and posture through hand-eye calibration. The target grasping position and posture of the vision sensor relative to the three-dimensional object can be converted into the robot's grasping position and posture using any existing hand-eye calibration method. Furthermore, the robot's grasping position and posture can be further converted into the position and posture of the three-dimensional object in the robot's coordinate system, enabling the robot to perform the grasping of the three-dimensional object.

[0036] In some embodiments, determining the target position and orientation of the visual sensor relative to the 3D object based on the judgment result using the grasping template and the current position and orientation further includes: when the grasping template also includes at least one reference grasping position and orientation, determining the grasping position and orientation in the grasping template with the shortest movement distance from the current position and orientation, and using it as the target grasping position and orientation; and when the grasping template does not include a reference grasping position and orientation, using the specified grasping position and orientation as the target grasping position and orientation. If the grasping template includes at least one reference grasping position and orientation, it indicates that there are other grasping positions and orientations similar to the specified grasping position and orientation. In this case, the grasping position and orientation with the shortest movement distance is determined based on the current position and orientation, which can be one of at least one reference grasping position and orientation and the specified grasping position and orientation. The grasping position and orientation with the shortest movement distance from the current position and orientation is determined by calculating the distance between the current position and orientation and the position of each grasping position and orientation in the grasping template. The distance calculation can be performed using any distance calculation method in the prior art. The target grab position and posture will be the one with the shortest movement distance from the current position and posture. If the grab template does not include a reference grab position and posture, it means that there are no other grab positions and postures similar to the specified grab position and posture. In this case, the specified grab position and posture will be directly used as the target grab position and posture.

[0037] In some embodiments, method 100 further includes ( Figure 1(Not shown in the image): A globe model is created with a specified gripping point on the virtual model of the 3D object as its center. The current position and attitude, the specified gripping position and attitude, and the reference gripping position and attitude are represented by position and attitude parameters in the globe model. The virtual model of the 3D object and the sphere of the globe model can be provided to the user via a display interface. The view from the perspective of the vision sensor is set as the main view of the globe model and provided to the user. The user can drag, translate, and rotate the virtual model of the 3D object in the main view to specify the gripping position located on the two edges of the virtual model of the 3D object. The center point of the gripping position is the gripping point. The gripping point is used as the center of the globe model. At this point, the relative attitude between the simulated vision sensor and the virtual model of the 3D object is determined. The intersection of the outline of the virtual model of the 3D object and the plane passing through the center of the sphere in the current perspective is the gripping position of the robot's two fingers. Next, a side view of the globe model is provided to the user, who can adjust the gripping depth, i.e., the distance between the simulated vision sensor and the gripping point, in the side view. This specifies the relative position and orientation between the 3D object and the vision sensor when grasping a 3D object. The simulated position and orientation of the vision sensor relative to the virtual model of the 3D object are used as the specified grasping position and orientation. In the Earth model, position and orientation can be represented using four parameters: latitude, longitude, depth, and yaw angle. Using the position and orientation parameters from the Earth model accurately describes the relative position and orientation between the vision sensor and the 3D object, thus reducing data storage and computational load and improving grasping speed compared to the traditional method of using six parameters to represent position and orientation.

[0038] In the above embodiments, by determining whether the preset grasping template includes a reference grasping position and posture of the 3D object to be grasped, when the grasping template includes a reference grasping position and posture, the robot's motion path from its current position and posture to the grasping position and posture can be optimized. This increases the flexibility of grasping, improves grasping speed and efficiency, and saves energy. Furthermore, this method is simple and easy to implement, requires little data computation, and does not require an expensive 3D camera, thus saving time and economic costs. When using a CAD design model of the 3D object as a virtual model, both the image template and the grasping template are derived from the CAD design model, making the estimation of the current position and posture and the acquisition of the grasping position and posture more reliable and accurate. Moreover, using the Mask-RCNN model to extract the mask of the 3D object and then performing matching increases the robustness and accuracy of recognizing 3D objects in complex environments. Using the position and posture parameters in the Earth model can accurately describe the relative position and posture between the visual sensor and the 3D object, reducing data storage and computational load and improving grasping speed.

[0039] The following description uses a specific embodiment as an example. Figure 1 A method for robots to grasp three-dimensional objects.

[0040] Also refer to Figures 2-5 This embodiment will be explained below. Figure 2 An embodiment of the present disclosure is shown for implementation. Figure 1 The system architecture diagram of the method in the text. Figure 3 It shows the use of Figure 2 The system executes a workflow for 3D object grabbing. Figure 2 In the system 200, there are camera 20, industrial computer 21, touch screen 22, and robot ( Figure 2 (Not shown in the image). Camera 20 is a two-dimensional camera, mounted on the robot's end effector, used to capture realistic images containing three-dimensional objects. Industrial computer 21 includes a parameter adjustment module 210, a virtual model import module 211, a globe model creation module 212, an image template generation module 213, a grasping template generation module 214, a data storage device 215, an object detection module 216, a template matching module 217, a grasping generation module 218, a parameter conversion module 219, and a hand-eye calibration module 220. The virtual model import module 211, the globe model creation module 212, the image template generation module 213, and the grasping template generation module 214 together constitute the template generation module. The user can input the position and pose range of camera 20 relative to the three-dimensional object, as well as a first threshold and a second threshold for similarity comparison, from the touch screen 22. These parameters can be provided to the template generation module via the parameter adjustment module 210. The touch screen 22 can also provide a display interface for industrial computer 21, through which information such as the virtual model of the three-dimensional object and the recognition results of the three-dimensional object in the real image can be displayed.

[0041] In this embodiment, the parameter adjustment module 210, virtual model import module 211, earth model creation module 212, image template generation module 213, grasping template generation module 214, data storage 215, object detection module 216, template matching module 217, grasping generation module 218, parameter conversion module 219, and hand-eye calibration module 220 are all integrated into the industrial computer 21, while the touch screen 22 is located outside the industrial computer 21. In other embodiments, these modules in the industrial computer 21 may be located on different computing devices. For example, the entire template generation module may be located on another computing device. In other embodiments, the touch screen 22 may be part of the industrial computer 21.

[0042] exist Figure 3In workflow 300, step 301 includes the user importing a virtual model of the 3D object to be captured via virtual model import module 211. In this embodiment, the virtual model of the 3D object is a CAD design model of the 3D object. The virtual model of the 3D object can be imported via a network interface or a hardware interface.

[0043] In step 302, an earth model is created by the earth model creation module 212. The user inputs the range of distance variation between the camera 20 and the 3D object via the touch screen 22. This range of distance variation can be the working range of the camera 20. The earth model creation module 212 provides a virtual model of the 3D object and a sphere with a radius of any value within the working range, and displays it to the user via the touch screen 22.

[0044] Figure 4 It shows in Figure 2 This is a schematic diagram of an Earth model created from a virtual model of an exemplary 3D object within the system. For example, it can be seen in... Figure 4 As seen in the image, a virtual model 401 of the three-dimensional object is located inside a sphere, and a virtual camera 402 is located on the surface of the sphere, with its viewing angle directed towards the virtual model 401. The equatorial plane of the sphere is set as the horizontal plane (XY) of the Earth model 400, and the vertical axis of the sphere is set as the Z-axis of the Earth model 400. Next, the user specifies the center and radius of the Earth model. The Earth model creation module 212 sets the main view of the Earth model 400 as the view from the perspective of the virtual camera 402 and displays the main view of the Earth model 400 to the user via the touch screen 22.

[0045] Figure 5 It shows as Figure 4 The example 3D object virtual model is shown in the front view of the Earth model when specifying the gripping point. From the perspective of the virtual camera 402, the user can drag, translate, and rotate the virtual model 401 of the 3D object on the front view 500 to specify the gripping position on the virtual model 401. In this embodiment, the robot is a two-finger robot, and the two intersection points of the outline of the virtual model 401 of the 3D object and the plane 502 passing through the center of the sphere from the current perspective are the robot's gripping positions. Figure 5 In the example, the user-specified gripping locations are intersections A and B, and their center O is the gripping point, making the gripping point the center of the Earth model. This determines the relative pose between the virtual camera 402 and the virtual model 401 of the 3D object, that is, the relative pose between the camera 20 and the 3D object when the robot grips the 3D object.

[0046] After confirming the gripping point, switch to the side view of the Earth model. The side view is set to be a view perpendicular to the perspective of the virtual camera 402. The user can adjust the distance between the virtual camera 402 and the gripping point (i.e., the center of the sphere) on the side view, which determines the gripping depth when the robot grips the 3D object. This specifies the relative position and orientation between the 3D object and the camera 20 when gripping the 3D object. The position and orientation of the virtual camera 402 relative to the virtual model 401 of the 3D object at this time are used as the specified gripping position and orientation. Figure 4 As shown, in the Earth model 400, θ is the angle between the virtual camera 402 and the Z-axis, which can be used to represent latitude; Ψ is the angle between the projection point of the virtual camera 402 on the XY plane and the X-axis, which can be used to represent longitude; r is the distance of the virtual camera 402 from the grasping point, which can be used to represent grasping depth; rot is the angle of rotation of the virtual camera 402 around its line of sight (the line connecting its center and the center of the sphere), which can be used to represent yaw angle. Therefore, the position and attitude of the virtual camera 402 can be represented by four parameters: latitude, longitude, depth, and yaw angle (θ, Ψ, r, rot). Furthermore, a two-dimensional projection 501 of the virtual model 401, which specifies the virtual image, including a three-dimensional object, is projected from the viewpoint of the camera 20 at the specified grasping position and attitude, as shown in the figure. Figure 5 As shown in the diagram. In the above steps, the user determines the specified grasping position and pose of the camera 20 relative to the 3D object when the robot grasps the 3D object, and the specified virtual image 501 of the virtual model 401 of the 3D object from the viewpoint of that specified grasping position and pose. Next, the specified virtual image is converted into a binary image. In the binary image, the area where the virtual model 401 of the 3D object is located (i.e., the two-dimensional projection) is white, while other areas are black. Features of the binary image are extracted using HoG (Histogram of Gradients), and these features are represented by an N-dimensional vector. It can be understood that this N-dimensional vector represents the edge features of the virtual model 401 of the 3D object in the virtual image. Subsequently, the specified grasping position and pose, represented by the position and pose parameters in the Earth model, and the corresponding N-dimensional vector are stored in the data storage 215.

[0047] Next, in step 303 of workflow 300, the image template generation module 213 generates an image template of the 3D object. The user can input the position and orientation range of the virtual camera 402 relative to the virtual model 401 of the 3D object into the parameter adjustment module 210 via the touchscreen 22, as preset position and orientation ranges. The user can set the position and orientation range according to the actual situation of the 3D object (e.g., symmetry). As mentioned above, position and orientation can be represented by four parameters: latitude, longitude, depth, and yaw angle (θ, Ψ, r, rot). The position and orientation range can include the numerical range of each of these parameters. Furthermore, the user can set the step size of each parameter within this numerical range according to actual needs. A change in any of these four parameters will result in a new position and orientation. The image template generation module 213 iterates through each position and orientation within the preset position and orientation range of the virtual camera 402 relative to the virtual model 401 of the 3D object, obtaining a virtual image of the virtual model 401 of the 3D object from the perspective of each position and orientation. The virtual image includes a two-dimensional projection of a virtual model 401 of a three-dimensional object from that viewpoint. Similarly, each virtual image is converted into a binary image, and features of the binary image are extracted using HoG, with each feature represented by an N-dimensional vector. Subsequently, all N-dimensional vectors and their corresponding positions and poses are stored as image templates in the data storage 215.

[0048] In step 304, the grasping template generation module 214 generates a grasping template for the 3D object. The grasping template generation module 214 compares each N-dimensional vector generated in step 303 with the N-dimensional vector generated in step 302 and determines their similarity. When the similarity between an N-dimensional vector generated in step 303 and the N-dimensional vector generated in step 302 exceeds a first threshold, the position and pose corresponding to that N-dimensional vector are used as reference grasping positions and poses. All reference grasping positions and poses and their corresponding N-dimensional vectors, as well as the specified grasping position and pose and its corresponding N-dimensional vector, are stored as a grasping template for the 3D object in the data storage 215. When the similarity between all N-dimensional vectors generated in step 303 and the N-dimensional vectors generated in step 302 is below the first threshold, it indicates that there are no other grasping positions and poses similar to the user-specified grasping position and pose. In this case, the grasping template only includes the specified grasping position and pose and its corresponding N-dimensional vector. To improve computation speed, step 304 is executed after each move to a new position and pose and obtaining the N-dimensional vector from the viewpoint of that new position and pose in step 303.

[0049] When multiple 3D objects need to be grasped, steps 301-304 are performed for each 3D object, and the generated image template and grasping template are saved in the data storage 215.

[0050] In step 305, the object detection module 216 receives a real image of the 3D object to be captured from the camera 20, identifies the 3D object in the real image, and generates a mask of the 3D object. The user can activate the camera 20 via the touchscreen 22 to capture the 3D object. In the object detection module 216, a pre-trained Mask-RCNN model is used to identify the 3D object in the real image and generate a mask of the 3D object. Next, the mask is converted into a binary image, and an N-dimensional vector is generated using HoG to represent the features of the binary image. In other embodiments, the Mask-RCNN model can also be pre-trained in the object detection module 216 and stored in the data storage 215 for later retrieval.

[0051] In step 306, the template matching module 217 reads the image template from the data storage 215 and determines the similarity between the N-dimensional vector generated in step 305 (i.e., the N-dimensional vector of the real image) and each N-dimensional vector in the image template. The highest similarity among these similarities is determined and compared with a second threshold input by the user. If the N-dimensional vector of the real image has the highest similarity to some N-dimensional vector in the image template, and this highest similarity exceeds the second threshold, it indicates that the real image of the 3D object matches the virtual image corresponding to that N-dimensional vector. The corresponding position and pose of the virtual image are used as the current position and pose of the camera 20 relative to the 3D object. If the highest similarity is lower than the second threshold, it indicates that the real image of the 3D object cannot be matched with a certain virtual image. At this point, the process proceeds to step 309, where the capture generation module 218 uses the specified capture position and pose in the capture template as the target capture position and pose of the camera 20.

[0052] After determining the current position and orientation of the camera 20 relative to the 3D object, workflow 300 proceeds to step 307. In step 307, the capture generation module 218 reads the capture template from the data storage 215 and determines whether the capture template includes a reference capture position and orientation of the camera 20 relative to the 3D object. When the capture template includes a reference capture position and orientation, in step 308, the capture template generation module 214 calculates the distance from the current position and orientation to each capture position and orientation in the capture template, and selects the capture position and orientation with the shortest distance from the current position and orientation as the target capture position and orientation of the camera 20 relative to the 3D object. The capture position and orientation with the shortest distance from the current position and orientation can be a specified capture position and orientation or a reference capture position and orientation. In this embodiment, the capture position and orientation with the shortest distance from the current position and orientation is determined by calculating the Euclidean distance. When the capture template does not include a reference capture position and orientation, the process proceeds to step 309. In step 309, the capture generation module 218 uses the specified capture position and posture in the capture template as the target capture position and posture of the camera 20.

[0053] Next, in step 310, the parameter conversion module 219 converts the target grasping position and attitude, represented by four parameters—latitude, longitude, depth, and yaw angle (θ, Ψ, r, rot)—into six parameters that the robot can recognize: x, y, z, roll, pitch, and yaw. During the conversion, according to… Figure 4 In the Earth model, the values ​​of x, y, and z can be represented by r, θ, and Ψ in equation (1):

[0054]

[0055] according to Figure 4 In the Earth model, rot and yaw are defined in the same way, representing rotation around the Z-axis in the camera coordinate system, thus yielding equation (2):

[0056] [yaw]=[rot] (2)

[0057] Furthermore, the Z-axis in the camera coordinate system points towards the origin of the world coordinate system of the Earth model. Therefore, the vector [0, 0, 1] in the camera coordinate system can be converted to [0, 0, 1] in the world coordinate system. The rotation order of Euler angles is set to XYZ, thus we obtain equation (3):

[0058]

[0059] in,

[0060] The roll and pitch angles can be calculated:

[0061]

[0062] Therefore, the parameter transformation can be performed according to the following formula (5):

[0063]

[0064] In step 311, the hand-eye calibration module 220 converts the target grasping position and posture of the camera 20 relative to the 3D object into the robot's grasping position and posture. After obtaining the robot's grasping position and posture, the robot's grasping position and posture are further converted into the position and posture of the 3D object in the robot's coordinate system and sent to the robot's controller to control the robot to perform the grasping of the 3D object.

[0065] In the above embodiments, by determining whether the preset grasping template includes a reference grasping position and posture of the 3D object to be grasped, when the grasping template includes a reference grasping position and posture, the robot's motion path from its current position and posture to the grasping position and posture can be optimized. This increases the flexibility of grasping, improves grasping speed and efficiency, and saves energy. Furthermore, this method is simple and easy to implement, requires little data computation, and does not require an expensive 3D camera, thus saving time and economic costs. When using a CAD design model of the 3D object as a virtual model, both the image template and the grasping template are derived from the CAD design model, making the estimation of the current position and posture and the acquisition of the grasping position and posture more reliable and accurate. Moreover, using the Mask-RCNN model to extract the mask of the 3D object and then performing matching increases the robustness and accuracy of recognizing 3D objects in complex environments. Using the position and posture parameters in the Earth model can accurately describe the relative position and posture between the visual sensor and the 3D object, reducing data storage and computational load and improving grasping speed.

[0066] Figure 6 An apparatus for inspecting components assembled on a production line, according to one embodiment of the present disclosure, is shown. (Refer to...) Figure 6The device 600 includes a current position and attitude determination unit 601, a grasping template acquisition unit 602, a reference position and attitude judgment unit 603, and a grasping position and attitude generation unit 604. The current position and attitude determination unit 601 is configured to determine the current position and attitude of the robot's vision sensor relative to a three-dimensional object. The grasping template acquisition unit 602 is configured to acquire a grasping template of the three-dimensional object, the grasping template including a specified grasping position and attitude of the vision sensor relative to the three-dimensional object. The reference position and attitude judgment unit 603 is configured to determine whether the grasping template also includes at least one reference grasping position and attitude of the vision sensor relative to the three-dimensional object, wherein the reference grasping position and attitude are generated based on the specified grasping position and attitude. The grasping position and attitude generation unit 604 is configured to generate the robot's grasping position and attitude based on the judgment result, using the grasping template and the current position and attitude.

[0067] Figure 6 Each unit in the system can be implemented using software, hardware (such as integrated circuits, FPGAs, etc.), or a combination of both.

[0068] In some embodiments, the grasping position and pose generation unit 604 further includes ( Figure 6 (Not shown in the image): Target position and orientation determination unit, which is configured to determine the target grasping position and orientation of the vision sensor relative to the three-dimensional object based on the judgment result, using the grasping template and the current position and orientation; and hand-eye calibration unit, which is configured to convert the target grasping position and orientation into the robot's grasping position and orientation through hand-eye calibration.

[0069] In some embodiments, the target position and attitude determination unit is further configured to: when the grasping template further includes at least one reference grasping position and attitude, determine the grasping position and attitude in the grasping template that has the shortest moving distance from the current position and attitude, and use it as the target grasping position and attitude; and when the grasping template does not include a reference grasping position and attitude, specify the grasping position and attitude as the target grasping position and attitude.

[0070] In some embodiments, the device 600 further includes ( Figure 6 (Not shown in the image): Crawling template generation unit, which is configured to generate crawling templates.

[0071] In some embodiments, the grasping template generation unit further includes: a specified image acquisition unit configured to acquire a specified virtual image of a virtual model of a three-dimensional object from a specified grasping position and pose; a virtual image acquisition unit configured to simulate multiple different positions and poses of a visual sensor relative to a three-dimensional object to obtain multiple virtual images of the virtual model of the three-dimensional object from multiple different positions and poses; a virtual image comparison unit configured to determine the similarity between each virtual image in the multiple virtual images and the specified virtual image; and a reference position and pose saving unit configured to save the corresponding positions and poses of virtual images with similarity exceeding a preset threshold as reference grasping positions and poses in the grasping template.

[0072] In some embodiments, similarity includes the similarity between features of a virtual image and features of a specified virtual image.

[0073] In some embodiments, the current position and attitude determination unit further includes ( Figure 7 (Not shown in the image): Real image acquisition unit, configured to acquire a real image of the 3D object from the perspective of its current position and pose; Image template acquisition unit, configured to acquire an image template of the 3D object, the image template representing multiple virtual images of the virtual model of the 3D object from multiple different positions and poses; Real image comparison unit, configured to determine the similarity between the real image and each of the multiple virtual images; and Current position and pose generation unit, configured to generate the current position and pose based on the corresponding position and pose of the virtual image with the highest similarity.

[0074] In some embodiments, the real image comparison unit is further configured to: identify three-dimensional objects in a real image using a Mask-RCNN model and generate a mask of the three-dimensional object; and obtain features of the real image based on the mask of the three-dimensional object.

[0075] In some embodiments, similarity includes the similarity between features of the real image and features of each virtual image.

[0076] In some embodiments, the device 600 further includes ( Figure 6 (Not shown in the image): Earth model building unit, which is configured to build an earth model with a specified gripping point on the virtual model of the 3D object as the center. The current position and attitude, the specified gripping position and attitude, and the reference gripping position and attitude are represented by the position and attitude parameters in the earth model.

[0077] Figure 7 A block diagram of a computing device for inspecting components assembled in a production line, according to an embodiment of the present disclosure, is shown. Figure 7As can be seen, the computing device 700 for maintaining field equipment in a factory includes a processor 701 and a memory 702 coupled to the processor 701. The memory 702 stores computer-executable instructions, which, when executed, cause the processor 701 to perform the methods described in the above embodiments.

[0078] Alternatively, the above methods can be implemented using a computer-readable storage medium. The computer-readable storage medium carries computer-readable program instructions for executing the various embodiments of this disclosure. The computer-readable storage medium can be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combinations thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0079] Therefore, in another embodiment, this disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon for performing the methods of various embodiments of this disclosure.

[0080] In another embodiment, this disclosure provides a computer program product tangibly stored on a computer-readable storage medium and including computer-executable instructions that, when executed, cause at least one processor to perform the methods of various embodiments of this disclosure.

[0081] Generally, the various example embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of the embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or methods described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0082] Computer-readable program instructions or computer program products for executing the various embodiments of this disclosure can also be stored in the cloud. When needed, users can access the computer-readable program instructions stored in the cloud for executing an embodiment of this disclosure via mobile internet, fixed network or other networks, thereby implementing the technical solutions disclosed in the various embodiments of this disclosure.

[0083] While embodiments of this disclosure have been described with reference to several specific examples, it should be understood that the embodiments of this disclosure are not limited to the specific embodiments disclosed. The embodiments of this disclosure are intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A method for a robot to grasp a three-dimensional object, including: Determine the current position and orientation of the robot's vision sensors relative to the three-dimensional object; Obtain a grasping template of the three-dimensional object, wherein the grasping template includes a specified grasping position and posture of the vision sensor relative to the three-dimensional object; Determine whether the grasping template further includes at least one reference grasping position and pose of the visual sensor relative to the 3D object, wherein the reference grasping position and pose are generated based on the specified grasping position and pose; and Based on the judgment result, the robot's grasping position and posture are generated using the grasping template and the current position and posture; Based on the judgment result, the target grasping position and posture of the visual sensor relative to the 3D object are determined using the grasping template and the current position and posture; and The target grasping position and posture are converted into the robot's grasping position and posture through hand-eye calibration; When the grasping template also includes at least one of the reference grasping positions and postures, the grasping position and posture with the shortest movement distance from the current position and posture in the grasping template is determined and used as the target grasping position and posture; and When the reference grasping position and posture are not included in the grasping template, the specified grasping position and posture shall be used as the target grasping position and posture; Generating the crawling template includes: Obtain a specified virtual image of the virtual model of the three-dimensional object from the viewpoint of the specified grasping position and posture; Simulate multiple different positions and postures of the visual sensor relative to the three-dimensional object to obtain multiple virtual images of the virtual model of the three-dimensional object from the perspectives of the multiple different positions and postures; Determine the similarity between each of the plurality of virtual images and the specified virtual image; and The corresponding positions and poses of virtual images with similarity higher than a preset threshold are used as the reference grasping positions and poses.

2. The method according to claim 1, wherein, The similarity includes the similarity between the features of the virtual image and the features of the specified virtual image.

3. The method according to claim 1, wherein, Determining the current position and orientation of the robot's vision sensors relative to the three-dimensional object further includes: Obtain a true image of the 3D object from the current position and orientation. Obtain an image template of the three-dimensional object, wherein the image template represents multiple virtual images of the virtual model of the three-dimensional object from multiple different positions and poses. Determine the similarity between the real image and each of the plurality of virtual images; and The current position and pose are generated based on the corresponding position and pose of the virtual image with the highest similarity.

4. The method according to claim 3, wherein, Determining the similarity between the real image and each of the plurality of virtual images further includes: Using the Mask-RCNN model, a mask for the 3D object is generated based on the real image of the 3D object; and Features of the true image of the three-dimensional object are obtained based on the mask of the three-dimensional object.

5. The method according to claim 4, wherein, The similarity includes the similarity between the features of the real image and the features of the virtual image.

6. The method of claim 1, further comprising: An Earth model is established with the specified gripping point on the virtual model of the three-dimensional object as the center. The current position and attitude, the specified gripping position and attitude, and the reference gripping position and attitude are represented by the position and attitude parameters in the Earth model.

7. A device for a robot to grasp a three-dimensional object, comprising: A current position and attitude determination unit is configured to determine the current position and attitude of the robot's vision sensors relative to the three-dimensional object; A grasping template acquisition unit is configured to acquire a grasping template of the three-dimensional object, wherein the grasping template includes a specified grasping position and posture of the vision sensor relative to the three-dimensional object; A reference position and pose determination unit is configured to determine whether the grasping template further includes at least one reference grasping position and pose of the visual sensor relative to the three-dimensional object, wherein the reference grasping position and pose are generated based on the specified grasping position and pose; and A grasping position and posture generation unit is configured to generate the robot's grasping position and posture based on the judgment result, using the grasping template and the current position and posture; A target position and orientation determination unit is configured to determine, based on the determination result, the target grasping position and orientation of the vision sensor relative to the three-dimensional object using the grasping template and the current position and orientation; and A hand-eye calibration unit is configured to convert the target grasping position and posture into the robot's grasping position and posture through hand-eye calibration; The target position and attitude determination unit is further configured as follows: When the grasping template also includes at least one of the reference grasping positions and postures, the grasping position and posture with the shortest movement distance from the current position and posture in the grasping template is determined and used as the target grasping position and posture; and When the reference grasping position and posture are not included in the grasping template, the specified grasping position and posture shall be used as the target grasping position and posture; The crawling template generation unit includes: A designated image acquisition unit is configured to acquire a designated virtual image of the virtual model of the three-dimensional object from the viewpoint of the designated grasping position and posture. A virtual image acquisition unit is configured to simulate multiple different positions and poses of the visual sensor relative to the three-dimensional object, and obtain multiple virtual images of the virtual model of the three-dimensional object from the perspectives of the multiple different positions and poses. A virtual image comparison unit is configured to determine the similarity between each of the plurality of virtual images and the specified virtual image; and A reference position and pose storage unit is configured to use the corresponding position and pose of a virtual image whose similarity exceeds a preset threshold as the reference grasping position and pose.

8. The apparatus according to claim 7, wherein, The current position and attitude determination unit further includes: A real image acquisition unit is configured to acquire a real image of the three-dimensional object from the perspective of its current position and pose. An image template acquisition unit is configured to acquire an image template of the three-dimensional object, wherein the image template represents multiple virtual images of the virtual model of the three-dimensional object from multiple different positions and poses. A real image comparison unit is configured to determine the similarity between the real image and each of the plurality of virtual images; and The current position and pose generation unit is configured to generate the current position and pose based on the corresponding position and pose of the virtual image with the highest similarity.

9. The apparatus according to claim 8, wherein, The real image comparison unit is further configured as follows: The Mask-RCNN model is used to identify the 3D object in the real image and generate a mask for the 3D object; and Features of the true image of the three-dimensional object are obtained based on the mask of the three-dimensional object.

10. The apparatus according to claim 7, further comprising: The Earth model building unit is configured to build an Earth model with a specified gripping point on the virtual model of the three-dimensional object as the center. The current position and attitude, the specified gripping position and attitude, and the reference gripping position and attitude are represented by position and attitude parameters in the Earth model.

11. A computing device, including: processor; as well as A memory for storing computer-executable instructions that, when executed, cause the processor to perform the method according to any one of claims 1-6.

12. A computer-readable storage medium having computer-executable instructions stored thereon for performing the method according to any one of claims 1-6.

13. A computer program product tangibly stored on a computer-readable storage medium and comprising computer-executable instructions that, when executed, cause at least one processor to perform the method according to any one of claims 1-6.

Images