Visual grabbing method for small-size fastener robot

Abstract

The invention discloses a robot vision grabbing method and system for small-size fasteners. The robot vision grabbing method comprises the following steps that an OBB model labeling data set is constructed; a virtual environment is constructed, and a robot body path is verified; the robot body grabs parts with an uncertain number from the part box and places the parts on the flexible pad; then, the robot body moves the tail end to a detection position, data collection and OBB detection are conducted on the part located on the flexible pad through a visual detection head, the posture of the part is judged, and a proper grabbing strategy is selected for grabbing according to the posture of the part; and after being grabbed, the part is placed to a designated position, and cyclic grabbing is conducted till no part exists in the part box. According to the robot vision grabbing method and system for the small-size fasteners, the OBB vision detection model and the path planning algorithm are combined, the robot automatic grabbing algorithm process of the small fasteners is provided, and efficient grabbing of the small-size fasteners in the fixed posture can be achieved only by arranging detection points on the vision probe and the flexible pad.

Description

Technical Field

[0001] This invention relates to the field of robot manufacturing technology, specifically to a robot vision grasping method for small-sized fasteners. Background Technology

[0002] Small fasteners refer to small parts or components used for connection and fixation, including miniature screws, nuts, bolts, washers, etc. They are indispensable basic components in modern micro-engineering, and their assembly usually occurs under conditions of extremely limited space and high precision requirements. Currently, most production lines in my country still rely on manual labor to pick up, place, and tighten small fasteners.

[0003] According to invention patent publication number CN118781126A, published on October 15, 2024, a fastener assembly method, system, device, and storage medium are disclosed. The method includes: receiving an assembly request instruction and fastener specification data and / or position data; responding to the assembly request instruction, acquiring a first depth map of the assembly object; and gripping the fastener using a fastener assembly device based on the fastener specification data and / or position data; the first depth map includes the assembly hole area of ​​the assembly object; determining the 3D pose data of the center of the assembly hole based on the first depth map; and controlling the fastener assembly device to insert the fastener into the assembly hole of the assembly object based on the 3D pose data of the center of the assembly hole. This application enables the fastener assembly device to automatically grip the fastener, automatically identify the 3D pose data of the center of the assembly hole, and then insert the fastener into the assembly hole, realizing automated fastener assembly with a high degree of automation, which can significantly improve the assembly quality of fasteners and reduce the defect rate.

[0004] In mass production and continuous manufacturing, manual methods suffer from low efficiency, poor consistency, and a high risk of omissions and misassemblies. However, existing robotic systems are still immature in handling micro-sized parts. For automated handling of large fasteners, robots use 6D object detection to determine the fastener's pose and can directly pick it up from the hopper and place it. However, for automated handling of small fasteners, 6D object detection methods struggle to detect the pose of small objects, resulting in poor positioning accuracy. Furthermore, insufficient gripping space and a small contact area make it difficult to handle small fasteners individually. Additionally, the random poses of fasteners in the hopper make it difficult for the gripper to grasp them in a unique and repeatable manner. Existing solutions using flexible vibrating disc feeding systems suffer from large space requirements, high costs, and low reusability. Therefore, this paper proposes a robotic vision-based gripping method and system for small fasteners, aiming to address the problem of robots struggling to handle small fasteners in existing technologies. Summary of the Invention

[0005] The purpose of this invention is to provide a robot vision grasping method and grasping system for small-sized fasteners, aiming to solve the problem that robots have difficulty grasping small-sized fasteners in the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a robot vision grasping method for small-sized fasteners, comprising the following steps: Construct an OBB model annotation dataset; A virtual environment is constructed to verify the robot's path. After the path detection shows no collisions, the robot begins to grasp. The robot body picks up an indefinite number of parts from the parts box and places them on a flexible pad; The robot then moves its end effector to the detection position and uses a vision detection head to collect data and perform OBB detection on the part located on the flexible pad. It then determines the part's posture and selects an appropriate grasping strategy based on the part's posture to grasp it. After grabbing a part, place it in the designated location and repeat the grabbing process until there are no parts left in the parts box.

[0007] Preferably, the OBB model dataset annotation uses YOLO_v11 to train the OBB model to detect the state and pose of parts placed on a flexible pad.

[0008] As a preferred method, a dataset of photos of various types of parts is obtained by taking a certain number of photos of each type of part, and then inputting it into X-AnyLabeling software for OBB rotation box annotation. The photos in the initial dataset are divided into training and validation sets according to the proportions, and then imported into YOLO_v11 to train the OBB model, finally obtaining the trained OBB model.

[0009] Preferably, the data acquisition and OBB detection include: The robot body remains above the flexible pad and acquires color images through a vision inspection head; The visual inspection head acquires images and detects the position of the detection mark. The center coordinates ID0 of the detection mark in the upper left corner are used as the transformation origin. The color image is then subjected to perspective transformation to obtain a frontal rectangle. Input the front view rectangle into the OBB model, and draw the angled rotating rectangle of the part through the OBB model to obtain the position and orientation of the part.

[0010] Preferably, the pixel scale between the pixel length of the flexible pad after perspective transformation and the actual length is calculated. Specifically: ; Transform the 2D pixel coordinates (u, v) of the target point relative to the transformation origin ID0 into the actual physical coordinates. ; Specifically: ; Physical coordinates in the image coordinate system Transformed into physical coordinates in the robot's base coordinate system ; Specifically: ; Where R is a 3×3 pose matrix and t is a 3×1 translation matrix.

[0011] Preferably, the grasping pose matrix is ​​calculated by drawing the rotating rectangle of the part using an OBB model. ; Specifically: ; in, The rotation angle of the part. It is an antisymmetric matrix; ; in, Let be the normal unit vector of the flexible pad (3). ; Preferably, based on the OBB model identification, the part posture includes a single upright position, multiple stacked positions, and a single independent position; The crawling strategy includes: For a single, independent object, it can be grasped directly by the robot itself; For multiple stacked objects, the robot body, in conjunction with the vision inspection head, separates the stacked objects and then selects a new grasping strategy based on the posture of the parts. For a single upright object, the robot pushes the object over and then selects a gripping strategy based on the part's posture.

[0012] A robot gripping system for small-sized fasteners is characterized by comprising a robot body, a vision inspection head, a worktable, and a flexible pad. Detection marks are provided at the four corners of the flexible pad, and the four detection marks form an imaging plane. The flexible pad and the parts box are both disposed on the worktable, and the vision inspection head is disposed on the robot body.

[0013] The robotic vision grasping method for small-sized fasteners provided by the present invention, as described above, has the following beneficial effects: This invention combines the OBB vision inspection model and path planning algorithm to propose an automated robotic grasping algorithm for small fasteners. It only requires setting detection points on the vision probe and flexible pad to achieve efficient grasping of small fasteners in a fixed posture. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0015] Figure 1 This is a schematic diagram of the overall structure provided for an embodiment of the present invention; Figure 2 This is a flowchart provided for an embodiment of the present invention.

[0016] Explanation of reference numerals in the attached figures: 1. Robot body; 2. Parts box; 3. Flexible pad; 4. Vision inspection head; 5. Inspection mark; 6. Worktable. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 and Figure 2 As shown, a robot vision grasping method for small-sized fasteners includes the following steps: Construct an OBB model annotation dataset; Specifically, the OBB model dataset annotation uses YOLO_v11 to train the OBB model to detect the state and pose of parts placed on the flexible pad 3.

[0019] A dataset of photos of various types of parts is obtained by taking a certain number of photos of each type of part. The dataset is then input into the X-AnyLabeling software for OBB rotation box annotation. The photos in the initial dataset are divided into training and validation sets according to the proportions, and then imported into YOLO_v11 to train the OBB model, finally obtaining the trained OBB model.

[0020] A virtual environment is constructed to verify the path of robot body 1. After the path detection shows no collision, robot body 1 begins to grasp. A virtual world is constructed, loading models of robot body 1, grippers mounted on robot body 1, flexible pads 3, and fasteners such as screws, nuts, and washers; a virtual camera is set up, and collision detection logic is loaded. Preset gripping / placement parameters are input, and a path planning algorithm is run in the virtual environment. After verifying that the robot's movement is collision-free and the gripper's posture is adapted, the actual execution phase begins.

[0021] The robot body 1 picks up an indefinite number of parts from the parts box 2 and places them on the flexible pad 3; The robot body 1 moves above the parts box 2, the end effector grippers open, the robot end effector descends into the parts box 2, the grippers close, and after confirming clamping through gripper force feedback, it rises to a safe height. The robot body 1 moves along the planned path to above the flexible pad 3, the grippers slowly release, and the fasteners fall freely onto the flexible pad 3.

[0022] The visual inspection head 4 collects data and performs OBB detection on the part located on the flexible pad 3, determines the part's posture, and selects an appropriate gripping strategy to grip the part based on its posture. Data acquisition and OBB testing include: The robot body 1 is held above the flexible pad 3, and color images are acquired through the vision inspection head 4; The visual inspection head 4 acquires images and detects the position of the detection mark 5. The center coordinates ID0 of the detection mark 5 in the upper left corner are used as the transformation origin. The color image is then subjected to perspective transformation to obtain a frontal rectangle. Input the front view rectangle into the OBB model, and draw the angled rotating rectangle of the part through the OBB model to obtain the position and orientation of the part.

[0023] Calculate the pixel scale between the pixel length of the flexible pad 3 after perspective transformation and its actual length; Specifically: ; Transform the 2D pixel coordinates (u, v) of the target point relative to the transformation origin ID0 into the actual physical coordinates. ; Specifically: ; Physical coordinates in the image coordinate system Transformed into physical coordinates in the robot's base coordinate system ; Specifically: ; Where R is a 3×3 pose matrix and t is a 3×1 translation matrix.

[0024] Based on the OBB model identification, the part postures include single upright, multiple stacked, and single independent; The crawling strategy includes: For a single, independent object, it can be grasped directly by the robot body 1; For multiple stacked objects, the robot body 1, together with the vision detection head 4, separates the multiple stacked objects, and then selects a new grasping strategy based on the posture of the parts. For a single upright object, the robot body 1 pushes the object over, then selects a gripping strategy based on the part's posture to grip the part and places it in the designated position. This process is repeated until there are no parts in the parts box 2.

[0025] The gripping part is obtained by drawing a rotating rectangle of the part using an OBB model and calculating the gripping pose matrix. ; As an embodiment of the present invention, the parts include a nut, a washer, and a screw; Specifically: ; in, The rotation angle of the screw. It is an antisymmetric matrix; ; in, Let 3 be the normal unit vector of the flexible pad 3. For a single nut and washer, the gripping position is the center point of the rotating rectangle, and the gripping pose matrix is... The identity matrix is ​​E; For a single screw, the screw head position is determined by the rule that the gray value of the screw head is greater than that of the screw tail. The gripping position is selected from the screw head and the gripping direction is selected from the short side of the rectangle to ensure stable gripping. The robot body 1 descends from 10mm above the gripping position to the gripping position through path planning, the gripper closes, and after confirming that the gripping is tight, it rises to a safe height to complete the gripping of a single part.

[0026] A robot gripping system for small fasteners includes a robot body 1, a vision inspection head 4, a worktable 6, a parts box 2, and a flexible pad 3. Detection marks 5 are provided at the four corners of the flexible pad 3. The flexible pad 3 and the parts box 2 are both set on the worktable 6, and the vision inspection head 4 is set on the robot body 1.

[0027] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0028] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0029] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0030] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0031] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

[0032] The embodiments of this application also provide a specific implementation of an electronic device capable of implementing all the steps in the methods described above, wherein the electronic device specifically includes the following: Processor, memory, communications interface, and bus; The processor, memory, and communication interface communicate with each other through the bus. The processor is used to invoke a computer program stored in the memory. When the processor executes the computer program, it implements all the steps in the method described in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: Construct an OBB model annotation dataset; A virtual environment is constructed to verify the robot's path. After the path detection shows no collisions, the robot begins to grasp. The robot body picks up an indefinite number of parts from the parts box and places them on a flexible pad; The robot then moves its end effector to the detection position and uses a vision detection head to collect data and perform OBB detection on the part located on the flexible pad. It then determines the part's posture and selects an appropriate grasping strategy based on the part's posture to grasp it. After grabbing a part, place it in the designated location and repeat the grabbing process until there are no parts left in the parts box.

[0033] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the methods in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the methods in the above embodiments. For example, when the processor executes the computer program, it implements the following steps: Construct an OBB model annotation dataset; A virtual environment is constructed to verify the robot's path. After the path detection shows no collisions, the robot begins to grasp. The robot body picks up an indefinite number of parts from the parts box and places them on a flexible pad; The robot then moves its end effector to the detection position and uses a vision detection head to collect data and perform OBB detection on the part located on the flexible pad. It then determines the part's posture and selects an appropriate grasping strategy based on the part's posture to grasp it. After grabbing a part, place it in the designated location and repeat the grabbing process until there are no parts left in the parts box.

[0034] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, for hardware + program embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Although the embodiments in this specification provide the method operation steps as shown in the embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible execution order among many steps and does not represent the only execution order. In actual device or terminal product execution, the methods can be executed in the order shown in the embodiments or drawings or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded. For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing the embodiments of this specification, the functions of each module can be implemented in one or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings or direct couplings or communication connections may be through some interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms. This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which are executable by the processor of the computer or other programmable data processing device, produce instructions for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0035] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The various embodiments in this specification are described in a progressive manner, and similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the description of the method embodiments. In the description of this specification, the reference to the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., means that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the embodiments of this specification.

[0036] In this specification, the illustrative expressions of the terms used do not necessarily refer to the same embodiments or examples. Furthermore, those skilled in the art can combine and integrate different embodiments or examples described in this specification, as well as features of different embodiments or examples, without contradiction. The above descriptions are merely embodiments of this specification and are not intended to limit the embodiments of this specification. Various modifications and variations can be made to the embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of this specification should be included within the scope of the claims of the embodiments of this specification.

Claims

1. A robotic vision grasping method for small size fasteners, characterized in that, The method comprises the following steps: constructing an OBB model labeling data set; constructing a virtual environment, verifying the path of the robot body (1), and after detecting that there is no collision in the path, the robot body (1) starts to grab; the robot body (1) grabs an indefinite number of parts from the part box (2) and places them on the flexible pad (3); then the robot body (1) moves the end to the detection position, and the vision detection head (4) collects data and performs OBB detection on the parts located on the flexible pad (3), judges the part posture, and selects an appropriate grabbing strategy according to the part posture to grab; after grabbing the parts, they are placed in the designated position and the cycle of grabbing is repeated until there are no parts in the part box (2).

2. The robotic vision grasping method for small fastener according to claim 1, wherein, The OBB model data set labeling trains the OBB model to detect the state and posture of the parts placed on the flexible pad (3) through YOLO_v11.

3. The robotic vision grasping method for small fastener according to claim 2, wherein, The vision detection head (4) shoots a data set of photos of each type of part, shoots a certain number of photos of each type of part to obtain an initial data set, and then inputs the initial data set into X-AnyLabeling software for OBB rotating frame labeling; the photos in the initial data set are divided into a training set and a verification set according to a proportion, and are imported into YOLO_v11 for training of the OBB model, and finally a trained OBB model is obtained.

4. The robotic vision grasping method for small fasteners according to claim 1, wherein, The data collection and OBB detection comprise: the robot body (1) is kept above the flexible pad (3), and a color image is collected by the vision detection head (4); the vision detection head (4) collects the image and detects the position of the detection marker (5), takes the marker center coordinate ID0 of the upper left detection marker (5) as the transformation origin, performs perspective transformation on the color image, and obtains an orthographic rectangular image; the orthographic rectangular image is input into the OBB model, and the position and posture of the part are obtained by drawing a rotating rectangular frame with an angle of the part through the OBB model.

5. The robotic vision grasping method of small size fastener according to claim 4, wherein, the pixel-to-actual scale of the flexible pad (3) after perspective transformation is calculated. Specifically, ； Transforming the 2D pixel coordinates (u, v) of the target point relative to the transformation origin ID0 into actual physical coordinates ; Specifically, ； transforming physical coordinates in an image coordinate system into physical coordinates in a robot base coordinate system ; Specifically, ； wherein R is a 3*3 pose matrix, and t is a 3*1 translation matrix.

6. The robotic vision grasping method for small fasteners according to claim 1, wherein, The grasping pose matrix is calculated by drawing a rotating rectangular frame of the part through the OBB model ; Specifically, ； wherein is the angle of rotation of the part, is an antisymmetric matrix; ； wherein is the normal unit vector of the flexible mat (3), ; for a single independent screw, the position of the screw head is determined by the rule that the gray value of the screw head is greater than the gray value of the screw tail, the grabbing position is selected as the screw head, and the grabbing direction is selected as the short side direction of the rectangular frame to stabilize the grabbing.

7. The robotic vision grasping method for small fastener according to claim 1, wherein, After OBB model recognition, the part posture comprises a single upright, multiple stacking, and a single independent; the grabbing strategy comprises: for a single independent object, the robot body (1) directly grabs the object; for multiple stacked objects, the robot body (1) cooperates with the vision detection head (4) to split the multiple stacked objects, and then selects a grabbing strategy according to the part posture after splitting; for a single upright object, the robot body (1) pushes the object down, and then selects a grabbing strategy according to the part posture.

8. A robotic pick system for small fasteners, characterized by, Including robot body (1), visual detection head (4), workbench (6), spare box (2) and flexible pad (3), the flexible pad (4) is provided with detection mark (5) at four corners, the flexible pad (3), spare box (2) are set on workbench (6), and the visual detection head (4) is set on robot body (1).

Images