U-shaped rib placing method and device

Abstract

The invention belongs to the technical field of U-bar production, and provides a U-bar placing method and device, and the method comprises the steps: obtaining U-bar RGB-D data, and calculating U-bar point cloud data; generating U-rib mask data according to the U-rib point cloud data, and calculating a U-rib pose; the U-shaped ribs are grabbed according to the U-shaped rib poses; obtaining the RGB-D image of the side surface of the clamp without the U rib, determining the initial position of the clamp, and calculating the pose of the clamp; placing the U-shaped ribs according to the U-shaped rib poses and the clamp poses; according to the method, the position and posture of the U rib are calculated through the obtained RGB-D data of the U rib, depth features can be increased through the RGB-D data of the U rib, data information is richer, and the position and posture can be recognized more accurately.

Description

Technical Field

[0001] The invention belongs to the technical field of U-rib production, and in particular relates to a U-rib swinging method and device. Background Art

[0002] U-bar production is a crucial step in the production of prefabricated small box girders for highways. After being straightened, bent, and welded using a positioning mesh, the U-bar is placed on a conveyor belt. A clamp at the end of a robotic arm grabs the horizontally positioned U-bar from the conveyor belt and places it vertically onto the base plate fixture for the next step.

[0003] The traditional solution uses manual material placement, which is labor-intensive, dangerous, and has a low degree of neatness in material placement.

[0004] In recent years, industrial robots have developed rapidly, and robots are often used for operations in fixed factory scenes. The existing material placement solution is to set the coordinates of two fixed points on the robot arm through the teach pendant, combined with the motion control of the conveyor belt to achieve the grasping and placement of the U-ribs. However, although there are auxiliary positioning devices for the U-ribs from the welding table to the conveyor belt, the positioning accuracy is insufficient, resulting in deviations in the position of the U-ribs on the conveyor belt. The robot arm often fails to grasp them, requiring manual intervention. When placing, the base plate tooling is mounted on the AGV cart. Each time the material is transported, the position relative to the robot arm is not fixed, and manual intervention is also required for the placement of the U-ribs.

[0005] With the advancement of computer vision and artificial intelligence, industrial robots are rapidly developing in the field of workpiece grasping. The workpiece grasping process involves capturing an image of the workpiece with a camera, identifying the grasping points, calculating the grasping pose, and then performing the grasping action using a six-axis robotic arm. Recognizing the workpiece's pose is crucial, as recognition accuracy impacts the robotic arm's ability to accurately grasp the workpiece.

[0006] However, due to the complex on-site conditions of the U-rib production line, there are scenes such as welding and cutting, which will cause strong light interference. In addition, the conveyor belt surface is smooth, and there is serious reflection when taking pictures of the U-ribs. This will cause some data to be missing and affect the subsequent grasping and placement of the U-ribs. Summary of the Invention

[0007] In view of the problems in the background technology, the present invention proposes a U-rib swinging method.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention discloses a U-rib material swinging method, comprising:

[0010] Obtain U-rib RGB-D data and calculate U-rib point cloud data;

[0011] Generate U-rib mask data based on the U-rib point cloud data and calculate the U-rib pose;

[0012] Grasping the U-rib according to the U-rib posture;

[0013] Obtain an RGB-D image of the side of the fixture, identify the fixture without U-ribs, determine the initial position of the fixture, and calculate the fixture pose;

[0014] The U-ribs are placed according to the U-rib posture and the fixture posture.

[0015] Furthermore, the obtaining of U-rib RGB-D data and the calculation of U-rib point cloud data include:

[0016] Obtain U-rib RGB-D data through a 3D camera;

[0017] According to the camera internal parameters, the U-rib RGB-D data is converted into U-rib point cloud data.

[0018] Furthermore, generating U-rib mask data and calculating U-rib pose based on the U-rib point cloud data includes:

[0019] Using the Ransac plane detection algorithm, plane regression is performed on the U-rib point cloud data to obtain the conveyor belt plane point cloud;

[0020] Based on the conveyor belt plane point cloud, extract U-rib point cloud data and generate U-rib mask data;

[0021] Generate a U-rib RGB-D image and a U-rib mask image according to the U-rib RGB-D data and the U-rib mask data;

[0022] Preprocessing the U-rib RGB-D image and the U-rib mask image;

[0023] Using a feature extraction network to extract features from the U-rib RGB-D image and the U-rib mask image to obtain a feature map;

[0024] Using a first detection network to detect the feature map, obtaining the center point coordinates and angles of the U rib in a pixel coordinate system; wherein the angles include a pitch angle, a yaw angle, and a roll angle, and the pitch angle and the yaw angle are both 0;

[0025] The U-rib pose is calculated based on the center point coordinates and angle of the U-rib in the pixel coordinate system.

[0026] Furthermore, the feature extraction network is a Resnet-18 or Hourglass network.

[0027] Furthermore, the first detection network includes a first detection head and a second detection head, the first detection head is used to detect the coordinates of the center point of the U rib, and the second detection head is used to detect the rolling angle of the U rib;

[0028] The first detection head detects the coordinates of the center point of the U rib, including:

[0029] Marking a U-rib bounding box, and calculating the coordinates of key points of the U-rib bounding box based on the U-rib bounding box;

[0030] Downsampling the coordinates of the key points of the U-rib bounding box;

[0031] The downsampled U-rib bounding box key point coordinates are dispersed onto the heat map through a Gaussian kernel to obtain the U-rib center point coordinates;

[0032] According to the U-rib bounding box, the width and height of the U-rib bounding box are calculated.

[0033] Furthermore, the second detection head detects the U-rib rolling angle, including:

[0034] Obtain the ROI region features near the coordinates of the U-rib center point on the feature map;

[0035] The U-rib rolling angle range is divided into multiple categories using a preset step size to determine the category to which the U-rib rolling angle belongs;

[0036] The regression network is used to adjust the U-rib roll angle of each category to obtain the U-rib roll angle.

[0037] Furthermore, the calculating of the U-rib pose according to the center point coordinates and angles of the U-rib in the pixel coordinate system includes:

[0038] The camera intrinsic parameter matrix is ​​used to convert the center point coordinates of the U rib in the pixel coordinate system into the three-dimensional coordinates in the camera coordinate system;

[0039] Constructing a translation matrix T according to the three-dimensional coordinates of the U rib in the camera coordinate system;

[0040] Calculate the rotation matrix R according to the angle of the U rib in the pixel coordinate system;

[0041] The U-rib pose is calculated based on the translation matrix T and the rotation matrix R.

[0042] Furthermore, the acquisition of the RGB-D image of the side of the fixture without the U-ribs, determination of the initial position of the fixture, and calculation of the fixture posture include:

[0043] Use the robotic arm to take a picture of the side of the fixture to obtain an RGB-D image of the side of the fixture;

[0044] The second detection network is used to process the RGB-D image of the side of the fixture, obtain the target center point and size of the fixture, and determine the initial fixture position.

[0045] Furthermore, the calculation of the fixture posture includes:

[0046] Use the robotic arm to take a photo of the front of the fixture to obtain several feature points of the fixture;

[0047] Using a recognition algorithm network to identify the fixture feature points, and obtain the coordinates of the fixture feature points in a pixel coordinate system;

[0048] Using the camera intrinsic parameter matrix, the coordinates of the fixture feature points in the pixel coordinate system are converted into three-dimensional coordinates in the camera coordinate system;

[0049] The plane fits the three-dimensional coordinates in the camera coordinate system to obtain the plane normal vector n, and the yaw angle is calculated by the plane normal vector n Get the fixture pose.

[0050] In a second aspect, the present invention discloses a U-rib swinging device, comprising:

[0051] Acquisition module, used to obtain U-rib RGB-D data and calculate U-rib point cloud data;

[0052] A first calculation module is used to generate U-rib mask data and calculate U-rib pose according to the U-rib point cloud data;

[0053] A grasping module, used for grasping the U-rib according to the U-rib posture;

[0054] The second calculation module is used to obtain the RGB-D image of the side of the fixture without the U-rib, determine the initial position of the fixture, and calculate the fixture pose;

[0055] The material placement module is used to place the U-ribs according to the U-rib posture and the fixture posture.

[0056] Beneficial effects of the present invention:

[0057] 1. In response to the problem of strong reflection in the actual material placement environment, the present invention obtains RGB-D data of U-ribs and adds depth features to enrich the data information, solving the problem of partial data loss caused by the reflection phenomenon when photographing U-ribs. A detection network is designed to identify the key points of U-ribs, obtain the precise position and posture of U-ribs, and realize automatic and accurate capture.

[0058] 2. The method of the present invention realizes the detection of the coordinates and angles of the center point of the U-rib through the first detection network, can obtain the precise position of the U-rib, realize automatic and accurate grasping, and solves the problem of strong reflection in the actual material placement environment.

[0059] 3. Use the second detection network to identify the initial fixture position, then identify the fixture feature points, calculate the fixture posture, and realize the automatic and accurate placement of U-ribs. This solves the problems of unstable fixture position and weak texture during placement, making the placement accuracy reach 0.5 cm and the angle reach 1 degree.

[0060] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 A flow chart of the U-rib swinging method of the present invention is shown;

[0063] Figure 2 The schematic diagram of the material placing site structure of the present invention is shown;

[0064] Figure 3 An RGB diagram of a U-rib of the present invention is shown;

[0065] Figure 4 A flow chart of processing an RGB image of a U-rib according to the present invention is shown;

[0066] Figure 5 shows a schematic diagram of the second detection network structure of the present invention;

[0067] Figure 6 Shows a schematic structural diagram of the U-rib clamp of the present invention;

[0068] Figure 7 Shows a device framework diagram of the present invention;

[0069] Figure 8 Shown is a schematic diagram of the device structure of the present invention. DETAILED DESCRIPTION

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0071] Intelligent U-rib placement includes grasping and placing the U-ribs. After being welded to the positioning net, the U-ribs are placed horizontally on a conveyor belt and automatically grasped by a robotic arm. Grasping the U-ribs requires high-precision pose estimation. Pose estimation involves calculating the coordinates and angles of the target in three-dimensional space. Angles include pitch, yaw, and roll.

[0072] Refer to the instruction manual Figure 1 and attached Figure 2 As shown, a U-rib material swinging method specifically includes the following steps:

[0073] S1. Obtain U-rib RGB-D data and calculate U-rib point cloud data;

[0074] S11, obtain U-rib RGB-D data through a 3D camera;

[0075] S12. Convert the U-rib RGB-D data into U-rib point cloud data according to the camera internal parameters.

[0076] The U-rib pose is calculated through U-rib RGB-D data. The U-rib RGB-D data can add depth features, making the data information richer and enabling more accurate pose identification.

[0077] S2. Generate U-rib mask data based on U-rib point cloud data and calculate U-rib pose;

[0078] S21. Use the Ransac plane detection algorithm to perform plane regression on the U-rib point cloud data to obtain the conveyor belt plane point cloud;

[0079] S22. Extracting U-rib point cloud data based on the conveyor belt plane point cloud and generating U-rib mask data;

[0080] S23, generating a U-rib RGB-D image and a U-rib mask image according to the U-rib RGB-D data and the U-rib mask data;

[0081] S24, preprocessing U-rib RGB-D image and U-rib mask image;

[0082] Specifically, the preprocessing steps include random cropping, flipping, etc.

[0083] S25, using a feature extraction network to extract features from the U-rib RGB-D image and the U-rib mask image to obtain a feature map;

[0084] The feature extraction network can use networks such as Resnet-18 and Hourglass.

[0085] S26. Detect the feature map using the first detection network to obtain the center coordinates and angles of the U rib in the pixel coordinate system; wherein the angles include pitch angle, yaw angle, and roll angle, and the pitch angle and yaw angle are both 0;

[0086] Because the conveyor belt is heavily reflective when the U-rib is being captured, the image contains distinct illuminated areas. Classic image processing algorithms based on gradients and grayscale tend to misidentify illuminated areas as targets when detecting the key points of the U-rib, resulting in poor anti-interference capabilities. Therefore, the present invention employs a first detection network to calculate the coordinates and angle of the center point of the U-rib in the pixel coordinate system.

[0087] The first detection network uses two detection heads to detect the coordinates and angles of the U-rib center points. The first detection head detects the coordinates of the U-rib center points, while the second detection head detects the angles. Both the first and second detection heads are composed of 2D convolution, batchnorm, ReLU, and maxpool.

[0088] Refer to the instruction manual Figure 3 And attached Figure 4 As shown in the figure, the coordinates of the center point of the U-rib are the coordinates of the center point P of line L. θ is the angle of line L, which is the roll angle. Line L is the part of the U-rib that contacts the conveyor belt. This line describes the position and direction of the U-rib on the conveyor belt.

[0089] As an easy-to-implement supervised learning method, heatmap helps accelerate network convergence. Therefore, the first detection head performs coordinate detection of the center point P of the U-rib based on the heatmap method. The specific steps include:

[0090] S261, marking the U-rib boundary box, and calculating the coordinates of the key points of the U-rib boundary box based on the U-rib boundary box;

[0091] Marking the U-rib boundary box is to mark the boundary box of the straight line L, and the coordinates of the center point of the U-rib boundary box are the coordinates of the center point P of the straight line L.

[0092] Specifically, if the center point coordinates of the marked U-rib bounding box are (x1, y1), (x2, y2), then the key point coordinates are:

[0093] S262, downsampling the coordinates of the key points of the U-rib bounding box;

[0094] The coordinates of the key points after downsampling are: Among them, R is the downsampling factor, which determines the resolution of the heat map.

[0095] Exemplarily, the downsampling factor R is set to 4. When R is 4, the resolution of the heat map is higher.

[0096] S263, the coordinates of the key points of the U-rib bounding box after downsampling are dispersed onto the heat map through the Gaussian kernel to obtain the coordinates of the center point of the U-rib, which is expressed as:

[0097]

[0098] Where, σ p is the target scale-adaptive standard deviation.

[0099] The heat map range is 0-1, and the peak in the heat map represents the position of the target center point. That is, when the heat map is 1, it represents the position of the target center point, that is, the coordinates (x0, y0) of the center point P of the line L to be detected.

[0100] S264. Calculate the width and height of the U-rib bounding box according to the U-rib bounding box.

[0101] After the above steps, the first detection network outputs four parameters, namely x0, y0, w, and h, where x0 and y0 represent the pixel coordinates of the center point P of the straight line, and w and h are the width and height of the regressed center point detection box.

[0102] Since the U-rib is always placed horizontally on the conveyor belt, in actual grasping, the robotic arm can be adjusted so that the pitch angle and yaw angle of the U-rib in the camera coordinate system are both 0, so only the roll angle θ needs to be detected.

[0103] Since the angle feature is contained near the straight line area on the image, the closer it is to the straight line area, the greater the weight factor for angle estimation. In addition to the straight line area, other features on the global feature map have a greater impact on the angle estimation noise.

[0104] Therefore, the second detection head is used to detect the roll angle θ, which specifically includes the following steps:

[0105] S265, obtaining ROI area features near the center point coordinates on the feature map;

[0106] Specifically, based on the center point coordinates detected by the first detection head, the ROI (Region of Interest) area near the center point P on the feature map is selected as the input of the second detection head. The region feature size is preferably 64*64*4.

[0107] By selecting the ROI area features near the center point coordinate P on the feature map as the input of the second detection head, irrelevant information can be reduced and the accuracy of angle estimation can be improved.

[0108] Since direct regression of angles is a nonlinear problem and difficult, a piecewise linear regression method is used, which includes the following steps:

[0109] S266, dividing the angle range into multiple categories using a preset step size;

[0110] Specifically, the features in the ROI region are classified by a classification network to determine the category to which the rolling angle θ of the U-rib belongs.

[0111] For example, the angle range is -90°-90°, and the step size is set to 3°, so there are 60=(90-(-90) / 3) categories.

[0112] S267, using a regression network to adjust the angle of each category to obtain a more accurate roll angle θ;

[0113] Among them, the regression network adopts convolutional neural network.

[0114] After the above steps, the first detection network finally outputs the coordinates (x0, y0) and angle of the center point P.

[0115] The first detection network is used to detect the coordinates and angles of the U-rib center point, obtain the precise position of the U-rib, and achieve automatic and accurate grasping, solving the problem of strong reflection in the actual material placement environment.

[0116] S27. Calculate the U-rib pose based on the center point coordinates and angle of the U-rib in the pixel coordinate system;

[0117] S271, using the camera intrinsic parameter matrix, converting the center point coordinates of the U rib in the pixel coordinate system into three-dimensional coordinates in the camera coordinate system;

[0118] S272. Construct a translation matrix T according to the three-dimensional coordinates of the U rib in the camera coordinate system;

[0119] S273, calculating the rotation matrix R according to the angle of the U rib in the pixel coordinate system;

[0120] Specifically, the angles include pitch angle, yaw angle, and roll angle, that is, the rotation matrix R is calculated according to the pitch angle, yaw angle, and roll angle.

[0121] S274. Calculate the U-rib pose based on the translation matrix T and the rotation matrix R.

[0122] For example, the camera intrinsic parameter matrix K is used to convert the coordinates of point P into three-dimensional coordinates (x, y, z) in the camera coordinate system. The conversion expression is:

[0123]

[0124] Where, f x 、f y 、c x 、c y All are camera internal parameters.

[0125] Since the camera z-axis can be adjusted to be perpendicular to the pixel coordinate systems u and v by adjusting the robotic arm, the depth value of the RGB-D data represents the z-axis coordinate.

[0126] It is known that the pitch angle and yaw angle are both 0, then the rotation matrix R can be obtained according to the roll angle θ, and its expression is:

[0127] R=Rz*(Ry*Rx)

[0128] Where,

[0129] S3. Grab the U-rib according to its position and posture;

[0130] Specifically, the robotic arm is controlled to grasp the U-rib according to the U-rib posture.

[0131] After grabbing the U-rib using the end-of-arm gripper, the U-ribs need to be placed in the free fixtures on the base plate tooling. When placing the U-ribs, due to the small spacing between the fixtures, several fixtures will appear in one image, and the tooling has two rows of fixtures. The image background is complex, making it difficult to calculate the pose using the template matching method. In addition, due to the limitations of the on-site environment, it is difficult to use traditional feature methods for pose estimation.

[0132] After grabbing the U-rib using the end-of-arm gripper, the U-ribs need to be placed sequentially in the available fixtures on the baseplate tooling. Due to the small spacing between the fixtures, multiple fixtures appear in a single image. Furthermore, the fixtures have two rows of fixtures, creating a complex image background. This makes pose calculation difficult using template matching, and pose estimation using traditional feature-based methods is difficult due to the limitations of the on-site environment.

[0133] The specific steps for placing U-ribs are as follows:

[0134] S41. Identify the position of the initial fixture; wherein the dividing point between the fixture where the U-bar is placed and the fixture where the U-bar is not placed is the position of the initial fixture.

[0135] S411, using the robotic arm to obtain RGB-D data of the side of the fixture without the U-ribs placed;

[0136] Specifically, the robotic arm was adjusted to take a photo of the side of the fixture. Due to the constraints of the on-site base plate tooling and placement, only the yaw angle of the fixture relative to the camera would change. Therefore, the robotic arm's position was adjusted when taking the photo so that the pitch and roll angles of the fixture in the camera coordinate system were 0, obtaining the fixture's RGB-D data.

[0137] S412: Process the RGB-D data of the fixture using the second detection network to obtain the target center point (x, y) and size (w, h) of the fixture to determine the initial position of the fixture.

[0138] The fixture surface texture features are weak, the fixtures are densely arranged, and each U-shaped rib requires two fixtures to support it. When photographing the side of the fixture, the front fixture is easily obscured by the rear fixture, making it difficult to identify the fixture using RGB images. Using depth information can remove interference from surrounding U-shaped ribs and filter out the rear fixtures. Therefore, the fixture's RGB-D data is used as the input to the secondary detection network.

[0139] Refer to the instruction manual Figure 5 As shown, in some embodiments, the second detection network is a Yolov5 model, and the Yolov5 model is mainly composed of Backbone, Neck and Prediction. Among them, Backbone adopts the BottleNeckCSP structure, which is composed of a Focus structure, three groups of CBL, CSP1_x, and SPP serially. Focus is to slice the image, and collect a value for every other pixel in each picture, so that complementary and information-free pictures can be formed. At the same time, the width and height information are concentrated in the channel space and then spliced. CBL is composed of a Conv layer, a BN layer, and a Relu layer. CSP1_x is used for the network backbone, in which a residual structure is added to avoid the gradient disappearance problem caused by the deepening of the network depth, so that fine-grained features can be extracted. Neck adopts the FPN+PAN structure. The FPN layer can capture strong semantic features, while PAN can convey strong positioning features. By combining these two modules, the target positioning function can be well completed.

[0140] Refer to the instruction manual Figure 6 As shown, S42, calculating the position and posture of the fixture. After obtaining the initial position of the fixture to be placed, it is necessary to estimate the precise position and posture of the fixture. It is necessary to identify key points on the two-dimensional graph, which specifically includes the following steps:

[0141] S421, taking a photo of the front of the fixture by the robotic arm to obtain several feature points of the fixture;

[0142] Specifically, the 3D camera is mounted on the end fixture of the robotic arm in an eye-in-hand configuration. The robotic arm is moved directly in front of the fixture to take a photo and obtain at least four feature points of the fixture. Feature point recognition is performed on the fixture, and the selected feature points are the centers of the four screws on the fixture. The coordinates of the feature points in the pixel coordinate system are obtained.

[0143] S422, using a recognition algorithm network to identify feature points and obtain the coordinates of the feature points in a pixel coordinate system;

[0144] S423, using the camera intrinsic parameter matrix, converting the coordinates of the feature point in the pixel coordinate system into three-dimensional coordinates in the camera coordinate system;

[0145] S424: Fit the plane to the three-dimensional coordinates in the camera coordinate system to obtain the plane normal vector n, and calculate the yaw angle using the plane normal vector n. Get the 6D pose of the fixture.

[0146] S43. Convert the position of the fixture from the camera coordinate system to the tool coordinate system of the end-of-arm robot to place the U-rib in the fixture.

[0147] Specifically, external parameter calibration is used to transform the position of the fixture into the tool coordinate system of the end-of-arm robot to place the U-rib in the fixture.

[0148] In some embodiments, the coordinates of the four feature points obtained are X1(x1, y1, z1), X2(x2, y2, z2), X3(x3, y3, z3), and X4(x4, y4, z4), where X1 and X2 are the upper two points of the four feature points, and X3 and X4 are the lower two points of the four feature points. The coordinates of the four feature points obtained are projected into the camera coordinate system, and the fixture coordinates are Where L is the distance between the two fixtures used for a U-bar. The translation matrix T can be calculated based on the fixture coordinates. If the pitch angle and roll angle are both 0, the rotation matrix R can be obtained, which is expressed as: R = Rz*(Ry*Rx), where

[0149] By using the second detection network to identify the initial fixture position, then identify the fixture feature points, and calculate the fixture posture, the U-rib is automatically and accurately placed, solving the problems of unstable fixture position and weak texture during placement.

[0150] S5. Place the U-bar according to the U-bar posture and fixture posture.

[0151] Specifically, the robot arm is controlled to place the U-rib according to the U-rib posture and the fixture posture.

[0152] Refer to the instruction manual Figure 7 As shown, based on the same inventive concept, the present invention also proposes a U-rib swinging device, comprising:

[0153] An acquisition module 110 is used to acquire U-rib RGB-D data and calculate U-rib point cloud data;

[0154] The first calculation module 120 is used to generate U-rib mask data and calculate U-rib pose based on U-rib point cloud data;

[0155] A grasping module 130 is used to grasp the U-rib according to its position and posture;

[0156] The second calculation module 140 is used to obtain an RGB-D image of the side of the fixture without the U-ribs, determine the initial position of the fixture, and calculate the fixture posture;

[0157] The material placement module 150 is used to place the U-ribs according to the U-rib posture and the fixture posture.

[0158] Refer to the instruction manual Figure 8 As shown, based on the same inventive concept, the present invention also proposes a device, including a memory and a processor, wherein the memory stores computer instructions that can be run on the processor, and the processor executes the above-mentioned U-rib swinging method when running the computer instructions.

[0159] Based on the same inventive concept, the present invention also proposes a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed, the above-mentioned U-rib swinging method can be implemented.

[0160] Any reference to memory, storage, database, or other media used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0161] Any reference to memory, storage, database, or other media used in the embodiments provided herein may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory.

[0162] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements that are not explicitly listed, or elements that are inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0163] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A U-rib swing material method, characterized in that: include: Get U-rib RGB-D data and calculate U-rib point cloud data; Generate U-rib mask data based on the U-rib point cloud data and calculate the U-rib pose; Grasping the U-rib according to the U-rib posture; Obtain the RGB-D image of the side of the fixture without U-ribs, determine the initial position of the fixture, and calculate the fixture posture; The U-rib is placed according to the U-rib posture and the fixture posture.

2. The U-rib swinging method according to claim 1, characterized in that: The obtaining of U-rib RGB-D data and the calculation of U-rib point cloud data include: Get U-rib RGB-D data through 3D camera; According to the camera internal parameters, the U-rib RGB-D data is converted into U-rib point cloud data.

3. The U-rib swinging method according to claim 1, characterized in that: The method of generating U-rib mask data and calculating U-rib pose according to the U-rib point cloud data includes: Using the Ransac plane detection algorithm, plane regression is performed on the U-rib point cloud data to obtain a conveyor belt plane point cloud; Based on the conveyor belt plane point cloud, extract U-rib point cloud data and generate U-rib mask data; Generate a U-rib RGB-D image and a U-rib mask image according to the U-rib RGB-D data and the U-rib mask data; Preprocessing the U-rib RGB-D image and the U-rib mask image; Using a feature extraction network to extract features from the U-rib RGB-D image and the U-rib mask image to obtain a feature map; The feature map is detected by a first detection network to obtain the center point coordinates and angles of the U rib in a pixel coordinate system; wherein the angles include a pitch angle, a yaw angle, and a roll angle, and the pitch angle and the yaw angle are both 0; The U-rib pose is calculated based on the center point coordinates and angle of the U-rib in the pixel coordinate system.

4. The U-rib swinging method according to claim 3, characterized in that: The feature extraction network is Resnet-18 or Hourglass network.

5. The U-rib swinging method according to claim 3, characterized in that: The first detection network includes a first detection head and a second detection head, the first detection head is used to detect the coordinates of the center point of the U rib, and the second detection head is used to detect the rolling angle of the U rib; The first detection head detects the coordinates of the center point of the U rib, including: Marking a U-rib boundary box, and calculating the coordinates of key points of the U-rib boundary box according to the U-rib boundary box; Downsampling the coordinates of the key points of the U-rib bounding box; The downsampled U-rib boundary box key point coordinates are dispersed onto the heat map through a Gaussian kernel to obtain the U-rib center point coordinates; According to the U-rib bounding box, the width and height of the U-rib bounding box are calculated.

6. The U-rib swinging method according to claim 5, characterized in that: The second detection head detects the U-rib rolling angle, including: Obtaining ROI area features near the coordinates of the U-rib center point on the feature map; The U-rib rolling angle range is divided into a plurality of categories using a preset step length to determine the category to which the U-rib rolling angle belongs; The regression network is used to adjust the U-rib roll angle of each category to obtain the U-rib roll angle.

7. The U-rib swinging method according to claim 3, characterized in that: The calculating the U-rib pose according to the center point coordinates and angle of the U-rib in the pixel coordinate system includes: The camera intrinsic parameter matrix is ​​used to convert the center point coordinates of the U rib in the pixel coordinate system into the three-dimensional coordinates in the camera coordinate system; Constructing a translation matrix T according to the three-dimensional coordinates of the U rib in the camera coordinate system; Calculate the rotation matrix R according to the angle of the U rib in the pixel coordinate system; According to the translation matrix T and the rotation matrix R, the U-rib pose is calculated.

8. The U-rib swinging method according to claim 1, characterized in that: The step of obtaining the RGB-D image of the side surface of the fixture without placing the U-ribs and determining the initial position of the fixture includes: Use the robotic arm to take a photo of the side of the fixture to obtain an RGB-D image of the side of the fixture; The second detection network is used to process the side RGB-D image of the fixture, obtain the target center point and size of the fixture, and determine the initial fixture position.

9. The U-rib swinging method according to claim 8, characterized in that: The calculation of the fixture posture includes: Use the robotic arm to take a photo of the front of the fixture to obtain several feature points of the fixture; Using a recognition algorithm network to identify the fixture feature points, and obtaining the coordinates of the fixture feature points in a pixel coordinate system; The camera intrinsic parameter matrix is ​​used to convert the coordinates of the fixture feature points in the pixel coordinate system into three-dimensional coordinates in the camera coordinate system; The plane is fitted with the three-dimensional coordinates in the camera coordinate system to obtain a plane normal vector n, and the yaw angle φ is calculated by the plane normal vector n to obtain the fixture posture.

10. A U-rib swinging device, characterized in that: include: The acquisition module is used to obtain U-rib RGB-D data and calculate U-rib point cloud data; A first calculation module is used to generate U-rib mask data and calculate U-rib pose according to the U-rib point cloud data; A grasping module, used for grasping the U-rib according to the U-rib posture; The second calculation module is used to obtain the RGB-D image of the side of the fixture without the U-ribs, determine the initial position of the fixture, and calculate the fixture posture; The material placement module is used to place the U-ribs according to the U-rib posture and the fixture posture.

Images