Deep frame workpiece grabbing method based on 3D vision

Abstract

The invention discloses a deep frame workpiece grabbing method based on 3D vision. The deep frame workpiece grabbing method comprises the steps that a pose relational expression of a mechanical arm in a deep frame is constructed based on a virtual environment; performing primary processing on the successfully matched pose set based on the pose relational expression to generate a grabbable pose set and an alternative pose set; and performing normalization processing on poses in the grabbable pose set and the alternative pose set, calculating a corresponding objective function value, and sorting to take out the most suitable grabbing pose. According to the method, the deep-frame obstacle can be removed in the subsequent path planning, so that the path planning efficiency is higher, whether the successfully matched workpiece can be grabbed or not can be quickly judged, and the frame cleaning rate can be improved.

Description

Technical Field

[0001] The invention belongs to the technical field of deep-frame workpiece grasping by a robot arm, and in particular relates to a deep-frame workpiece grasping method based on 3D vision. Background Art

[0002] The grasping of scattered and stacked workpieces in a deep frame plays a very important role in the field of automation. It involves how to achieve stable and effective grasping of workpieces in a given deep frame environment. In the field of industrial robots, the traditional method for grasping workpieces in deep frames is basically to directly perform path planning after determining the matching posture successfully through vision, but this method does not take into account the problem that workpieces at the edge of the deep frame are difficult to grasp and the subsequent path planning is not efficient in the deep frame, resulting in a low frame clearance rate. Therefore, it is necessary to think of a new grasping idea to improve grasping efficiency. Summary of the invention

[0003] In order to solve the above problems, the present invention first provides a deep frame workpiece grasping method based on 3D vision, comprising:

[0004] S1. Construct a robot arm and a deep frame model in a virtual environment according to the actual robot arm and deep frame, divide the space in the deep frame into equal parts, simulate the robot arm posture for each equally divided space, and thus construct a posture relationship equation;

[0005] S2. Based on 3D vision, the posture of each workpiece to be grasped in the deep frame is obtained, and the posture of each workpiece that can be grasped by the robot arm is matched; a posture edge threshold is set, and the posture relationship formula is used to filter the posture of the workpiece that can be grasped by the robot arm based on the edge threshold to generate a set of graspable postures of the workpiece;

[0006] S3, normalizing the postures in the grasping posture set of each workpiece and calculating the corresponding objective function value, sorting the postures according to the corresponding objective function value and selecting the optimal grasping posture, and grasping the workpiece according to the selected optimal grasping posture;

[0007] S4. After the previous workpiece is grasped, execute the operation of S2 again to generate a set of graspable poses for the next workpiece, and determine whether the set of graspable poses is empty. If it is not empty, return to step S3; if it is empty, the grasping is completed.

[0008] In a second aspect, the present invention provides a system for implementing the above-mentioned deep frame workpiece grasping method, comprising:

[0009] A pose relation building module is used to build a robot arm and a deep frame model in a virtual environment according to the actual robot arm and the deep frame, and simulate the robot arm posture based on the robot arm and the deep frame model to build a pose relation;

[0010] The graspable posture generation module is used to obtain the position of the grasped workpiece in the deep frame based on 3D vision, and match the posture of the robot arm to grasp each workpiece; set a posture edge threshold, use the posture relationship formula to filter the posture of the robot arm that can grasp the workpiece based on the edge threshold, and generate a graspable posture set;

[0011] The grasping posture selection module is used to normalize the postures in the graspable posture set and calculate the corresponding objective function value, sort the postures according to the corresponding objective function value, and select the optimal grasping posture;

[0012] The graspable pose set judgment module is used to judge whether the graspable pose set is empty.

[0013] In a third aspect, the present invention provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the above-mentioned deep frame workpiece grasping method.

[0014] In a fourth aspect, the present invention provides a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by a processor, they are used to implement the above-mentioned deep frame workpiece grasping method.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The present invention utilizes virtual simulation technology, and under the constraints of the robot arm joint angle and deep frame space, simulates the extreme posture of the robot arm at each equally divided position in the deep frame on a virtual simulation platform, thereby constructing an extreme posture relationship. This method solves the problem of real-time collision interaction between the deep frame and the robot arm in the prior art, thereby improving the efficiency of path planning after the subsequent grasping posture is determined. At the same time, based on the posture of the workpiece to be grasped and the extreme posture relationship solved by 3D vision, a feasible grasping posture set of the workpiece is screened out, and finally the posture in the grasping posture set is normalized and the most suitable grasping posture is solved through the objective function to perform grasping, thereby avoiding the problem of low grasping success rate caused by blind grasping of the robot arm after obtaining multiple grasping postures, thereby improving the frame clearing rate of grasping stacked workpieces in the deep frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.

[0018] Figure 1 A diagram of a virtual simulation platform for grabbing deep-frame workpieces;

[0019] Figure 2 Construct a flow chart for the relationship between the extreme poses of the manipulator in the deep frame;

[0020] Figure 3 Flowchart for determining the set of graspable poses;

[0021] Figure 4 This is the flowchart for grasping deep frame workpieces. DETAILED DESCRIPTION

[0022] The present invention is further described and illustrated below in conjunction with specific operation examples. The examples are merely exemplary of the present disclosure and do not define the scope of limitation. The technical features of each embodiment of the present invention may be combined accordingly without conflicting with each other.

[0023] The present invention will now be described in further detail with reference to the accompanying drawings.

[0024] Figure 1 It is a virtual simulation platform for deep frame workpiece grasping. Figure 2 The flowchart for constructing the robot arm posture relationship is as follows: This method embodiment constructs the robot arm and deep frame model in the virtual environment according to the actual relative position of the robot arm and the deep frame, and divides the space in the deep frame into equal parts, and simulates the robot arm posture for each equally divided space, thereby constructing the posture relationship, which is specifically described as follows:

[0025] Determine the relative position of the robot arm and the deep frame in space, take the center of the robot arm base as the world coordinate origin, build a virtual model of the robot arm and the deep frame, and build a deep frame coordinate system with the midpoint of the bottom surface of the deep frame as the origin of the deep frame, and determine the limit point p extending in the x, y, and z directions in the world coordinate system x 、p y 、p z , and divide the area from the center of the bottom surface of the deep frame to each limit point into n equal parts to ensure that the space of the deep frame is divided into evenly distributed equal grids;

[0026] Move the robot model to the origin of the deep frame, use the repositioning method to determine the limit pitch angle and limit roll angle of this position under the fixed yaw angle, and record the value of the yaw angle and the maximum pitch angle at this time. max and minimum pitch min And the maximum roll angle roll max and minimum roll min The repositioning method refers to a method in which only the posture is adjusted to determine the limit pitch angle and the limit roll angle under a fixed yaw angle while the position of the end point of the robot arm remains unchanged;

[0027] Move the robot arm to each equally divided grid in turn to perform the above operations, and obtain the maximum and minimum values ​​of the pitch angle and the maximum and minimum values ​​of the roll angle on each equally divided grid in the spatial coordinate system;

[0028] Then the pitch angle is the maximum value. max is the function value, the y-axis coordinate and the z-axis coordinate are variables, and the pitch is constructed max The functional relationship F between y and z max (pitch max ,y,z)=0, and similarly, the minimum pitch angle is obtained. min The functional relationship F between y and z min (pitch min ,y,z)=0, the maximum roll angle roll max The functional relationship G between y and z max (roll max ,y,z)=0 and the minimum value of the yaw angle roll min The functional relationship G between y and z min (roll min ,y,z)=0;

[0029] Divide the yaw angle into m equal parts within the range of (-90°, 90°), and determine the pose relationship F for each yaw angle after equal division max (pitch max ,y,z,r)=0,F min (pitch min ,y,z,r)=0,G max (roll max ,y,z,r)=0,G min (roll min ,y,z,r)=0, where r represents the yaw angle value.

[0030] like Figure 3 As shown in FIG. 1 , it is a flow chart for determining a graspable pose set. The method of the present invention obtains the pose of each workpiece to be grasped in the deep frame based on 3D vision, matches the pose that the robot arm can grasp each workpiece; sets a pose edge threshold, uses the pose relationship and based on the edge threshold, screens the pose that the robot arm can grasp the workpiece, and generates a graspable pose set of the workpiece, which is specifically described as follows:

[0031] Use a 3D point cloud camera to shoot the workpiece in the deep frame to determine the position of the workpiece, and based on the position of the workpiece, obtain the grasping posture set M that the robot arm can grasp the workpiece 匹配位姿 ;

[0032] Set a posture edge threshold Q, and then匹配位姿 The grasping posture in the position relationship is used to detect it. If the grasping posture exceeds the limit posture T at the position of the workpiece 极限 , the posture is adjusted and collision detection is performed under the condition that the position remains unchanged. The adjusted posture is divided into two types. The first type is adjusted to (T 极限min , T 极限min +Q) or (T 极限max -Q, T 极限max ) range, the second is adjusted to (T 极限min +Q,T 极限max -Q) range, if the posture cannot be adjusted to the above two ranges, that is, collision still occurs after adjustment, then the posture is discarded, where T 极限min Includes roll min and pitch min , T 极限max Includes roll max and pitch max , Q contains the edge thresholds corresponding to roll and pitch;

[0033] When the adjusted posture is the first case above, the posture is included in the candidate posture set M 备选位姿 If the adjusted posture is the second case, then the posture is included in the priority grasping posture set M 优先位姿 ;

[0034] If the gripping posture does not exceed the limit posture T at the position of the workpiece 极限 , then determine the range of the posture according to the workpiece grasping position and posture relationship. If the posture is in the first case, then include the grasping posture in the candidate posture set M 备选位姿 If the posture is in the second case, then the posture is included in the priority grasping posture set M 优先位姿 .

[0035] like Figure 4 As shown, in an embodiment of the present invention, the postures in the grasping posture set of each workpiece are normalized and the corresponding objective function value is calculated, the postures are sorted according to the corresponding objective function value, and the optimal grasping posture is selected, and the workpiece is grasped according to the selected optimal grasping posture, as follows:

[0036] According to the set weights, the objective function is constructed to prioritize the pose set M 优先位姿 The grasping posture of is normalized using the Z-score normalization method, and the objective function value of the grasping posture after normalization is calculated; if the priority grasping posture set M 优先位姿 If there is no grasping pose in the set of candidate poses M 备选位姿Repeat the above operation for the grasping posture in ; select the grasping posture with the largest objective function value to grasp the workpiece. Construct the objective function as: F = a*z a +b*z b +c*z c

[0037] Among them, F is the objective function value; z a The normalized value of the absolute difference between the optimal pitch angle at the position to be grasped and the current pitch angle. The optimal pitch angle is the pitch calculated based on the current position using the posture relationship. max and pitch min The mean value of z b is the normalized value of the absolute difference between the optimal roll angle and the current roll angle at the position to be grasped. The optimal roll angle is the roll angle calculated according to the current position using the posture relationship. max and roll min The mean value of z c is the normalized value of the absolute difference between the global optimal position of the deep frame and the position to be grasped. The global optimal position of the deep frame is the center point of the upper opening of the deep frame. a, b, c are weight coefficients.

[0038] Furthermore, after the previous workpiece is grasped, the next workpiece graspable pose set is generated again based on 3D vision, and it is determined whether the graspable pose set is empty. If it is not empty, the optimal graspable workpiece is solved according to the above steps to perform grasping; if it is empty, the grasping is ended, as follows:

[0039] After the current workpiece is grasped, the M′ of the remaining workpieces is obtained again based on 3D vision 优先位姿 and M′ 备选位姿 ;

[0040] Determine M′ 优先位姿 Is it empty? If not, solve M′ 优先位姿 The optimal grasping workpiece in is grasped, if it is empty, then judge M′ 备选位姿 Is it empty? If not, solve M′ 备选位姿 The optimal grasping workpiece in is grasped, if it is empty, the grasping ends.

[0041] Based on the same inventive concept, a system for implementing the above-mentioned deep frame workpiece grasping method is also provided in this embodiment, including:

[0042] A pose relation building module is used to build a robot arm and a deep frame model in a virtual environment according to the actual robot arm and the deep frame, and simulate the robot arm posture based on the robot arm and the deep frame model to build a pose relation;

[0043] The graspable posture generation module is used to obtain the position of the grasped workpiece in the deep frame based on 3D vision, and match the posture of the robot arm to grasp each workpiece; set a posture edge threshold, use the posture relationship formula to filter the posture of the robot arm that can grasp the workpiece based on the edge threshold, and generate a graspable posture set;

[0044] The grasping posture selection module is used to normalize the postures in the graspable posture set and calculate the corresponding objective function value, sort the postures according to the corresponding objective function value, and select the optimal grasping posture;

[0045] The graspable pose set judgment module is used to judge whether the graspable pose set is empty.

[0046] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0047] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0048] An embodiment of the present invention further provides an electronic device, including a memory and a processor;

[0049] The memory is used to store computer programs;

[0050] The processor is used to implement the above-mentioned deep frame workpiece grasping method based on 3D vision when executing the computer program.

[0051] An embodiment of the present invention further provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the above-mentioned deep-frame workpiece grasping method based on 3D vision is implemented.

[0052] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit and an external storage device of any device with data processing capability. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0053] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that, although the work done in this development process may be complicated and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application. Without departing from the concept of the present application, several variations and improvements can also be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A deep frame workpiece grasping method based on 3D vision, It is characterized in that include: S1. Construct a robot arm and a deep frame model in a virtual environment according to the actual robot arm and deep frame, divide the space in the deep frame into equal parts, simulate the robot arm posture for each equally divided space, and thus construct a posture relationship equation; S2. Based on 3D vision, the posture of each workpiece to be grasped in the deep frame is obtained, and the posture of each workpiece that can be grasped by the robot arm is matched; a posture edge threshold is set, and the posture relationship formula is used to filter the posture of the workpiece that can be grasped by the robot arm based on the edge threshold to generate a set of graspable postures of the workpiece; S3, normalizing the postures in the grasping posture set of each workpiece and calculating the corresponding objective function value, sorting the postures according to the corresponding objective function value and selecting the optimal grasping posture, and grasping the workpiece according to the selected optimal grasping posture; S4. After the last workpiece is grasped, execute the operation of S2 again to generate the graspable pose set of the remaining workpieces, and determine whether the graspable pose set is empty. If it is not empty, return to S3; if it is empty, the grasping is completed.

2. The deep frame workpiece grasping method according to claim 1, It is characterized in that The S1 comprises the following sub-steps: S1.

1. Determine the relative position of the robot arm and the deep frame in space, construct a virtual model of the robot arm and the deep frame, and construct a spatial rectangular coordinate system with the midpoint of the bottom surface of the deep frame as the origin, and divide the x-axis, y-axis and z-axis of the spatial rectangular coordinate system into equal parts according to the specified spacing to ensure that the space of the deep frame is divided into evenly distributed equal grids; S1.2, move the robot model to the origin, use the repositioning method to determine the limit pitch angle and limit roll angle of this position under a fixed yaw angle, and record the value of the yaw angle, the maximum and minimum values ​​of the pitch angle, and the maximum and minimum values ​​of the roll angle at this time; S1.3, move the robot arm to each equally divided grid in turn to perform the operation of S1.2, and obtain the maximum and minimum values ​​of the pitch angle and the maximum and minimum values ​​of the roll angle on each equally divided grid of the spatial rectangular coordinate system; S1.4, then pitch at the maximum pitch angle max is the function value, the y-axis coordinate and the z-axis coordinate are variables, and the pitch is constructed max The functional relationship F between y and z max (pitch max , y, z) = 0, and the minimum pitch angle is obtained in the same way min The functional relationship F between y and z min (pitch min , y, z) = 0, the maximum roll angle roll max The functional relationship G between y and z max (roll max , y, z) = 0 and the minimum value of the yaw angle roll min The functional relationship G between y and z min (roll min , y, z) = 0; S1.5, divide the yaw angle into equal parts within the range of (-90°, 90°) with the specified accuracy, and execute steps S1.2 to S1.4 for each yaw angle after equal division, so as to determine the posture relationship F max (pitch max , y,z,r)=0,F min (pitch min ,y,z,r)=0,G max (roll max ,y,z,r)=0,G min (roll min , y,z,r)=0, where r represents the yaw angle value.

3. The deep frame workpiece grasping method according to claim 2, It is characterized in that The S2 comprises the following sub-steps: S2.

1. Use a 3D point cloud camera to shoot the workpiece in the deep frame to determine the position of the workpiece, and based on the position of the workpiece, obtain the set of gripping postures M in which the robot arm can successfully grip the workpiece. 匹配位姿 ; S2.2, set a posture edge threshold, and then 匹配位姿 The grasping posture in the position relationship is used to detect it. If the grasping posture exceeds the limit posture T at the position of the workpiece 极限 , the posture is adjusted and collision detection is performed under the condition that the position remains unchanged. The adjusted posture is divided into two types. The first type is adjusted to (T 极限min , T 极限min +Q) or (T 极限max -Q, T 极限max ) range, the second is adjusted to (T 极限min +Q,T 极限max -Q) range, if the posture cannot be adjusted to the above two ranges, that is, collision still occurs after adjustment, then the posture is discarded, where T 极限min Including roll min and pitch min , T 极限max Including roll max and pitch max , Q includes the edge thresholds corresponding to the roll angle and pitch angle; S2.

3. When the adjusted posture is the first case in S2.2, the posture is included in the candidate posture set M 备选位姿 If the adjusted posture is the second case in S2.2, then the posture is included in the priority grasping posture set M 优先位姿 ; S2.

4. If the gripping posture does not exceed the limit posture T of the workpiece 极限 , then determine the range of the posture according to the workpiece grasping position and posture relationship. If the posture is in the first case in S2.2, then include the grasping posture in the candidate posture set M 备选位姿 If the posture is in the second case in S2.2, then the posture is included in the priority grasping posture set M 优先位姿 .

4. The deep frame workpiece grasping method according to claim 3, It is characterized in that The S3 is specifically as follows: constructing an objective function according to the set weights, giving priority to grasping the pose set M 优先位姿 The grasping posture of is normalized using the Z-score normalization method, and the objective function value of the grasping posture after normalization is calculated; if the priority grasping posture set M 优先位姿 If there is no grasping pose in the set of candidate poses M 备选位姿 Repeat the above operation for the grasping posture in ; select the grasping posture with the largest objective function value to grasp the workpiece.

5. The deep frame workpiece grasping method according to claim 4, It is characterized in that The objective function is: F=a*z a +b*z b +c*z c Among them, F is the objective function value; a, b, c are weight coefficients; z a The normalized value of the absolute difference between the optimal pitch angle at the position to be grasped and the current pitch angle. The optimal pitch angle is the pitch calculated based on the current position using the posture relationship. max and pitch min The mean of z b is the normalized value of the absolute difference between the optimal roll angle and the current roll angle at the position to be grasped. The optimal roll angle is the roll angle calculated according to the current position using the posture relationship. max and roll min The mean of z c It is the normalized value of the absolute difference between the global optimal position of the deep frame and the position to be grasped. The global optimal position of the deep frame is the center point position of the upper opening of the deep frame.

6. The deep frame workpiece grasping method according to claim 4, It is characterized in that The step 4 comprises the following steps: S4.

1. After the current workpiece is grabbed, execute S2 again to obtain M′ of the remaining workpieces. 优先位姿 and M′ 备选位姿 ; S4.

2. Determine M′ 优先位姿 Is it empty? If not, execute the operations of S3 and S4.

1. If it is empty, determine M' 备选位姿 Is it empty? If it is not empty, execute the operations of S3 and S4.

1. If it is empty, the crawling ends.

7. The deep frame workpiece grasping method according to claim 1, It is characterized in that In S4, when it is determined that the graspable pose set is empty and the grasping is completed, the remaining workpieces in the deep frame are counted and the frame clearing rate is calculated.

8. A system for implementing the deep frame workpiece grasping method of claim 1, It is characterized in that include: A pose relation building module is used to build a robot arm and a deep frame model in a virtual environment according to the actual robot arm and the deep frame, and simulate the robot arm posture based on the robot arm and the deep frame model to build a pose relation; The graspable posture generation module is used to obtain the position of the grasped workpiece in the deep frame based on 3D vision, and match the posture of the robot arm to grasp each workpiece; set a posture edge threshold, use the posture relationship formula to filter the posture of the robot arm that can grasp the workpiece based on the edge threshold, and generate a graspable posture set; The grasping posture selection module is used to normalize the postures in the graspable posture set and calculate the corresponding objective function value, sort the postures according to the corresponding objective function value, and select the optimal grasping posture; The graspable pose set judgment module is used to judge whether the graspable pose set is empty.

9. An electronic device, It is characterized in that The invention comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement the deep frame workpiece grasping method according to any one of claims 1 to 7.

10. A machine-readable storage medium, It is characterized in that The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are called and executed by the processor, they are used to implement the deep frame workpiece grasping method according to any one of claims 1 to 7.

Images