A robot arm 6D pose estimation method and system based on a fuzzy perspective prior

Abstract

The application discloses a robot arm 6D pose estimation method and system based on a fuzzy view angle prior, relates to the technical field of robot intelligent manufacturing, and comprises the following steps: obtaining steel bar point cloud data based on a color image and a depth image and in combination with a camera internal parameter; performing plane fitting segmentation processing on the steel bar point cloud data to obtain a current steel bar layer point cloud plane; mapping the current steel bar layer point cloud plane into a current layer steel bar image; performing target detection and recognition and pixel binarization processing on the current layer steel bar image to obtain a frame mask image; taking the frame mask image, the color image and the depth image as inputs, performing translation part processing and rotation part processing based on a zero sample pose estimation model, and coupling the processing results to obtain a 6D estimated pose corresponding to a steel bar node frame image. The method provided by the application simplifies the processing process when precisely obtaining the 6D estimated pose corresponding to each steel bar node, and is conducive to realizing local deployment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent work of industrial robots, and in particular to a robot arm 6D pose estimation method and system based on fuzzy visual angle prior. BACKGROUND

[0002] In the construction process of reinforced concrete structures, steel binding and steel welding are one of the most basic and key processes, and the quality thereof is directly related to the overall stability of the steel reinforcement and the stress performance after concrete pouring. In existing construction, steel binding mainly relies on manual operation, and workers need to complete the winding and fixing of the binding wire at complex steel intersection nodes. This method not only has high labor intensity and a poor working environment, but also is prone to cause quality problems such as node loosening and steel displacement due to non-standard operation or omission. With the development of intelligent construction and industrial robot technology, steel binding automation has gradually become a research hotspot. One important problem is how to realize six-degree-of-freedom (6D) pose estimation of the binding robot arm when executing the binding action. 6D pose estimation refers to determining the position and orientation of an object in three-dimensional space. 6D represents six degrees of freedom, including translation along X, Y, and Z directions, and rotation around X, Y, and Z directions.

[0003] Traditional pose estimation methods (such as point cloud registration) rely on point cloud geometric features (surface points, normal vectors, feature descriptors, etc.), and obtain the relative pose through iterative alignment or feature matching. The method has high requirements for camera quality and poor robustness under occlusion and noise conditions. Although the existing zero-shot deep learning 6D pose estimation network has strong adaptability, it has a large dependence on computing hardware (GPU) and relatively slow running speed, which is difficult to meet the real-time and low-cost deployment requirements on site.

[0004] Therefore, it is necessary to provide a robot arm 6D pose estimation method and system based on fuzzy visual angle prior. When performing 6D pose estimation based on a zero-shot deep learning 6D pose estimation network, multi-source information is fused to realize processing based on fuzzy visual angle prior, improve the calculation speed, and reduce the dependence on hardware, thereby facilitating the implementation of local deployment on construction robots. SUMMARY

[0005] The main purpose of the present application is to provide a robot arm 6D pose estimation method and system based on fuzzy visual angle prior, which aims to solve the technical problems of complex processing and difficulty in realizing local deployment when performing 6D pose estimation using a zero-shot deep learning 6D pose estimation network in the prior art.

[0006] To achieve the above object, the application provides a robot arm 6D pose estimation method based on a fuzzy perspective prior, comprising the following steps: S10, obtaining steel bar point cloud data based on a target workpiece corresponding RGB image and depth image and combining camera intrinsic parameters;

[0007] S20, performing plane fitting segmentation processing on the steel bar point cloud data to obtain a current steel bar layer point cloud plane;

[0008] S30, mapping the current steel bar layer point cloud plane into a current layer steel bar image;

[0009] S40, performing target detection and recognition and pixel binarization processing on the current layer steel bar image to obtain a frame mask image corresponding to each steel bar node in the current layer steel bar image;

[0010] S50, taking the frame mask image, the RGB image and the depth image as inputs, performing translation part processing and rotation part processing based on a zero sample pose estimation model, and coupling the processing results to obtain a 6D estimated pose corresponding to the steel bar node frame image;

[0011] The rotation part processing comprises rotation pose initialization, rotation network optimization and rotation network scoring processing, and the rotation pose initialization is used to generate a rotation candidate pose photographing point and a candidate photographing pose corresponding to each rotation candidate pose photographing point.

[0012] The rotation pose initialization specifically comprises: selecting a point on a target circle on a regular polyhedral spherical subdivision grid as a rotation candidate pose photographing point, and generating a candidate photographing pose based on each rotation candidate pose photographing point.

[0013] The angle between the camera optical axis corresponding to the rotation candidate pose photographing point and the axis of the object coordinate system is , and the difference between and is not greater than a preset angle, wherein is the vector expression of the camera optical axis, is the expression of the axis in the object coordinate system.

[0014] Further, the step of generating a candidate photographing pose based on each rotation candidate pose photographing point specifically comprises:

[0015] Within a rotation range of -β to β, a preset rotation step is adopted to rotate around the camera optical axis to obtain a plurality of candidate photographing poses based on each rotation candidate pose photographing point, and β is not greater than 180 degrees.

[0016] Further, the preset rotation step is 60 degrees, and β is 60 degrees or 100 degrees.

[0017] One rotation candidate pose photographing point corresponds to three candidate photographing poses.

[0018] Further, the steel bar point cloud data is subjected to plane fitting segmentation processing to obtain a plurality of target steel bar layer point cloud planes, the optimal target steel bar layer point cloud plane is determined as the current steel bar layer point cloud plane, and the normal vector of the target steel bar layer point cloud plane is determined as the axis of the object coordinate system.

[0019] Further, in step S30, the current steel bar layer point cloud plane is mapped into a current layer steel bar image by using an inflation kernel operation.

[0020] Further, in step S40, a steel bar node frame image in the current layer steel bar image is obtained by performing target detection and recognition on the current layer steel bar image based on a steel bar node target detection model; wherein the establishment of the steel bar node target detection model comprises the following steps:

[0021] Collecting training data, collecting a steel bar node data set by using a 3D camera;

[0022] Processing training data, segmenting and filtering the multi-layer steel bar images in the steel bar node data set to obtain single-layer steel bar images;

[0023] Labeling training data, manually labeling the single-layer steel bar images, and the labeling of one steel bar intersection object consists of one bounding box and one key point;

[0024] Converging and training the initial steel bar node target detection model to obtain an updated steel bar node target detection model.

[0025] Further, pixel binarization processing is performed on each steel bar node frame image, non-white pixels in the steel bar node frame image are set to white, and a corresponding frame mask image is generated.

[0026] Further, in step S40, the steel bar node frame images are sequentially sorted based on an S-shaped path working strategy, which specifically comprises:

[0027] Based on the size of the axis of the image coordinate system of the current layer steel bar image, the steel bar node frame images are subjected to clustering analysis to obtain a plurality of rows of steel bar node frame image rows arranged along the axis, wherein each row of steel bar node frame image rows has at least one steel bar node frame image arranged along the horizontal direction;

[0028] It is judged whether the current row of steel bar node frame images is an odd row;

[0029] If the current row of steel bar node frame images is an odd row, the steel bar node frame images are sequentially sorted based on the image coordinate system of the current layer steel bar image. The size of the axis coordinate is along The first direction of the axis sequentially sorts the steel bar node framework images in the row of steel bar node framework images;

[0030] If the current row of steel bar node framework images is an even row, the image coordinate system is based on The size of the axis coordinate is along The second direction of the axis sequentially sorts the steel bar node framework images in the row of steel bar node framework images; wherein the second direction is opposite to the first direction;

[0031] The sorted steel bar node framework images are sequentially connected to form an S shape.

[0032] The application also provides a robot arm 6D pose estimation system based on a fuzzy perspective prior, comprising a welding robot and a visual perception device, the welding robot has a base and a robot hand, and the visual perception device is arranged on the robot hand.

[0033] A processing device is arranged on the welding robot, and the processing device is used to realize the steps of the robot arm 6D pose estimation method based on the fuzzy perspective prior.

[0034] The application also provides a computer readable storage medium, which stores a computer program, and the computer program realizes the steps of the robot arm 6D pose estimation method based on the fuzzy perspective prior when executed by a processor.

[0035] Compared with the prior art, the robot arm 6D pose estimation method based on the fuzzy perspective prior has the following beneficial effects:

[0036] The robot arm 6D pose estimation method based on the fuzzy perspective prior provided by the application firstly acquires an RGB image and a depth image of a target workpiece, combines and processes the RGB image, the depth image and camera intrinsic parameters to acquire steel point cloud data; then performs plane fitting segmentation processing on the steel point cloud data to obtain a plurality of target steel layer point cloud planes, and determines an optimal target steel layer point cloud plane as a current steel layer point cloud plane; after the current steel layer point cloud plane is mapped into a current layer steel image, target detection and recognition and pixel binarization processing are performed to acquire a framework mask image corresponding to the current layer steel image; finally, the framework mask image, the RGB image and the depth image are taken as inputs, and a zero sample pose estimation model is used for recognition to obtain a 6D estimated pose corresponding to each steel node; in the scheme of the application, when determining a rotation candidate pose shooting point and a candidate shooting pose, instead of using each vertex of a special icosphere as a shooting point, only the included angle between the optical axis of the camera and the axis is The corresponding circle on several points is a rotation candidate pose photographing point, which simplifies the process of accurately obtaining the 6D estimated pose of each steel bar node, and is beneficial to the implementation of local deployment. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained according to the structures shown in the drawings without creative labor for those skilled in the art.

[0038] Figure 1 is a flowchart of a robot arm 6D pose estimation method based on a fuzzy view angle prior in an embodiment of the present application;

[0039] Figure 2 is an icosphere (special sphere) diagram under different subdivision values in the prior art, wherein a is an icosphere diagram when subdivision=1, and b is an icosphere diagram when subdivision=2;

[0040] Figure 3 is a pose initialization processing principle diagram of a zero-sample pose estimation model in an embodiment of the present application;

[0041] Figure 4 is an icosphere pose initialization diagram of a FoundationPose model in the prior art;

[0042] Figure 5 is an icosphere pose initialization diagram of a FoundationPose model in an embodiment of the present application, wherein a is an icosphere pose initialization diagram when the included angle between the camera optical axis and the axis of the object coordinate system is 166.40°, b is an icosphere pose initialization diagram when the included angle between the camera optical axis and the axis of the object coordinate system is 93.69°, and c is an icosphere pose initialization diagram when the included angle between the camera optical axis and the axis of the object coordinate system is 12.01°;

[0043] Figure 6 is a current layer steel bar image diagram obtained after the inflation operation in an embodiment of the present application.

[0044] ​​​The objectives, functional features and advantages of the present application will be further described with reference to the embodiments in combination with the accompanying drawings. DETAILED DESCRIPTION

[0045] It should be understood that the specific embodiments described herein merely set forth preferred combinations of components and / or other features, and that the scope of the application is not limited to such specific embodiments.

[0046] The technical solutions in the embodiments of the present application will be apparently and completely described with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0047] It should be noted that all the directionality indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present application are only used to explain the relative position relationship, movement condition, etc. between components in a certain posture (as shown in the drawings). If the certain posture changes, the directionality indications also change accordingly.

[0048] In addition, the descriptions of “first”, “second” and the like in the present application are only for the purpose of description, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features. Therefore, the features limited by “first”, “second” can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the realization of a person of ordinary skill in the art. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist, and is not within the protection scope required by the present application.

[0049] Please refer to the accompanying drawings Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 and Figure 6 , the present application provides a robot arm 6D pose estimation method based on fuzzy view prior, comprising the following steps:

[0050] S10, based on the corresponding RGB image and depth image of the target workpiece and combining the camera intrinsic parameter to obtain the steel bar point cloud data;

[0051] S20, the steel bar point cloud data is subjected to plane fitting segmentation processing to obtain the current steel bar layer point cloud plane;

[0052] S30, the current steel bar layer point cloud plane is mapped into the current layer steel bar image;

[0053] S40, target detection and recognition and pixel binarization processing are performed on the current layer steel bar image to obtain a frame mask image corresponding to each steel bar node in the current layer steel bar image;

[0054] S50, the frame mask image, the RGB image and the depth image are taken as inputs, translation part processing and rotation part processing are performed based on a zero sample pose estimation model, and a 6D estimated pose corresponding to the steel bar node frame image is obtained by coupling the processing results;

[0055] The rotation part processing includes rotation pose initialization, rotation network optimization and rotation network scoring processing, and the rotation pose initialization is used to generate a rotation candidate pose photographing point and a candidate photographing pose corresponding to each rotation candidate pose photographing point.

[0056] The rotation pose initialization specifically includes: selecting a point on a target circle on a regular polyhedral spherical subdivision grid as a rotation candidate pose photographing point, and generating a candidate photographing pose based on each rotation candidate pose photographing point.

[0057] The angle between the camera optical axis corresponding to the rotation candidate pose photographing point and the axis of the object coordinate system is , , , The difference between and is not greater than a preset angle, wherein , is the vector expression of the camera optical axis, is the axis expression of the object coordinate system, , , and are the components of the vector in the x-axis direction, the y-axis direction and the z-axis direction respectively.

[0058] Through research, it is found that the existing robot binding, robot welding 6D estimated pose methods mainly include the following: a method based on traditional geometry, mainly based on point cloud registration, for example, using an ICP algorithm, iteratively aligning the actual collected reinforcement node point cloud with the standard model point cloud, so that the two sets of point clouds coincide as much as possible, solving the spatial position and attitude of the node, the principle is simple, and it does not depend on a large amount of data, but the initial position requirement is high, and the point cloud is prone to failure when there is noise or occlusion, and the stability is poor; a method based on template matching, the first is image template matching, rendering the RGB image template of the reinforcement node at different viewing angles in advance, which can also combine gradient, normal vector and other features, and in the image taken on the construction site, the templates are used to match pixel by pixel, the closest viewing angle is found, and the pose is estimated; the second is to generate a point cloud subset template of different attitudes of the object offline in advance (for example, rendering point clouds at multiple angles from a CAD model), and then comparing the point cloud collected on site with these templates one by one, finding the most similar one, and directly taking it as the estimated result, in the actual construction site environment with many reinforcements and high appearance similarity, confusion is easy to occur, and the template quantity is large, and the expansibility is poor.

[0059] The robot arm 6D pose estimation method based on the fuzzy visual angle prior provided by the application first acquires an RGB image and a depth image of a target workpiece, combines and processes the RGB image, the depth image and a camera intrinsic parameter to obtain reinforcement point cloud data; then, the reinforcement point cloud data is subjected to plane fitting segmentation processing to obtain a plurality of target reinforcement layer point cloud planes, and the optimal target reinforcement layer point cloud plane is determined as a current reinforcement layer point cloud plane; next, after the current reinforcement layer point cloud plane is mapped into a current layer reinforcement image, target detection and recognition and pixel binarization processing are performed to obtain a frame mask image corresponding to the current layer reinforcement image; finally, taking the frame mask image, the RGB image and the depth image as inputs, a zero-shot pose estimation model is used for recognition to obtain a 6D estimated pose corresponding to each reinforcement node; in the scheme of the application, when determining a rotation candidate pose shooting point and a candidate shooting pose, instead of using each vertex of a special icosphere as a shooting point, only a few points on a corresponding circle with an included angle between the camera optical axis and the object coordinate system as the rotation candidate pose shooting point, the processing process is simplified when accurately obtaining the 6D estimated pose corresponding to each reinforcement node, and this is conducive to realizing local deployment. When accurately obtaining the 6D estimated pose corresponding to each reinforcement node, the processing process is simplified, and this is conducive to realizing local deployment.

[0060] It can be understood that in the scheme of the application, the preset angle is 5 degrees, and in a preferred embodiment of the application, the preset angle is 5 degrees. .

[0061] It can be understood that in the scheme of the application, the target workpiece is a reinforcement combined component to be bound or welded, and the target workpiece has a reinforcement intersection point.​

[0062] It can be understood that the translation and rotation of the pose estimation are both relative to the camera position and direction of shooting the object, and in the steel binding scene, through the 6D pose estimation, the robot can accurately know the spatial position and direction of each steel cross node relative to the camera, so as to realize accurate and safe binding operation. Through the calibration of the camera coordinate system and the object coordinate system corresponding to the object in the space when the camera is shooting, the 6D pose estimation gives a conversion matrix, which expresses the meaning that the object coordinate system can be completely coincided with the camera coordinate system through what movement.

[0063] Specifically, the conversion matrix expression is The three elements in the upper right corner of the matrix are the translation of the object relative to the camera ; wherein the rotation matrix can be converted into the Euler angle (roll, pitch, yaw) around the X, Y and Z axes, if the order is YXZ, that is, the rotation order is first around Y, then around X, and finally around Z, , .

[0064] It can be understood that in the scheme of the application, Icosphere (icosphere) is mainly researched and applied. Icosphere is a geometric primitive in computer graphics, which approximates a sphere by twenty equilateral triangles. A parameter of Icosphere is subdivision. When subdivision = 1, each edge of the equilateral triangle is bisected, that is, each triangle is cut into four small equilateral triangles, as shown by a and b in Figure 2 .

[0065] Further, the step of generating a candidate photographing pose based on each rotation candidate pose photographing point specifically comprises: rotating around the camera optical axis with a preset rotation step within a rotation range of -β to β to obtain a plurality of candidate photographing poses corresponding to each rotation candidate pose photographing point, β being not greater than 180 degrees.

[0066] Further, the preset rotation step is 60 degrees, β is 60 degrees or 100 degrees, and one rotation candidate pose photographing point corresponds to three candidate photographing poses.

[0067] Further, the steel point cloud data is subjected to plane fitting segmentation processing to obtain a plurality of target steel layer point cloud planes, the optimal target steel layer point cloud plane is determined as the current steel layer point cloud plane, and the normal vector of the target steel layer point cloud plane is determined as the axis of the object coordinate system.

[0068] In the scheme of the present application, after determining the rotation candidate pose photographing points, the corresponding candidate photographing poses of each rotation candidate pose photographing point are further determined, which not only reduces the number of rotation candidate pose photographing points, but also solves the technical problem that mirror images exist when obtaining pictures in the existing 360-degree circumferential range, and further identification processing of the mirror images is required to determine the state of the robot arm.

[0069] It can be understood that the zero-shot network is a commonly used end-to-end identification network in the current 6D pose estimation network, and the zero-shot network can be trained based on objects such as water cups and pen containers to estimate the pose of the steel bar node, and the zero-shot network can still complete the pose estimation task of the steel bar node without training the steel bar node as a training object, which shows strong generalization. The representative network with good performance and high accuracy in the zero-shot network is the FoundationPose network, and the main structure of the FoundationPose network includes three parts of pose initialization, optimization network and scoring network.

[0070] Please refer to Figure 3 In a preferred embodiment of the present application, the Foundationpose pose estimation network decouples the rotation part and the translation part into two parts, and the Foundationpose pose estimation network includes translation part estimation and rotation part estimation. In the scheme of the present application, the translation part pose estimation includes translation initialization and fine alignment, which mainly improves the rotation pose initialization of the rotation part. The rotation part initialization includes candidate pose generation, candidate pose optimization (rotation network optimization), and optimization pose (rotation network scoring) to select the final rotation part estimation.

[0071] In specific practice, one of the improvement points of the scheme of the present application is the initialization strategy of the rotation part, and the final result is to reduce the generation of irrelevant rotation candidate pose photographing points and improve the speed and accuracy. In the prior art, the translation part initialization of the Foundationpose pose estimation network mainly refers to initializing the translation using the 3D point located in the detected 2D bounding box depth, and the rotation part initialization refers to uniformly sampling viewpoints from the Icosphere on the object facing the center of the camera, and the camera pose will be further enhanced by discrete in-plane rotations, resulting in global pose initialization, which is sent as the input of the pose optimization network. Please refer to Figure 4Before its improvement, the Foundationpose pose estimation network initializes the object by placing it at the center of a special sphere (icosphere) with a radius of 1. It then assumes the camera will take pictures at each vertex of the icosphere. At each vertex, the camera rotates 360 degrees around the z-axis of the camera coordinate system, taking a picture every 60 degrees. In practice, 42 vertices are captured, each rotating 6 times, resulting in 6 × 42 = 252 possible poses. Please refer to [reference needed]. Figure 5 In Figures a, b, and c, the solution of this invention, based on a rough perspective, uses a target circle corresponding to the shooting angle. Rotating the candidate pose shooting point onto this target circle helps reduce the camera's search range. For actual binding tasks, when segmenting the rebar layers to obtain the target rebar layer point cloud plane, all rebar nodes corresponding to the binding layer are on the current rebar layer point cloud plane. For each rebar node, its object coordinate system... The plane is approximately parallel to the dividing surface; the normal vector of the current reinforcement layer point cloud plane and the coordinate system of the object are... The axes are represented the same in the camera coordinate system; assuming the normal vector of the target rebar layer point cloud plane is (A, B, C), the condition that can be obtained is that the coordinates of the object system are the same in the camera coordinate system. The axis expression is In the actual algorithm, since the camera is known to be facing the steel cage, and since the plane normal vector has two directions, it will determine whether to take the direction of the camera. For a normal vector whose axis angle is greater than 90 degrees, the camera's optical axis is the coordinate system of the camera. Axis, expressed as From this, it can be deduced that when the camera is photographing an object, the camera's optical axis... and the object coordinate system Angle between axes ,

[0072]

[0073] .

[0074] Understandably, after obtaining the camera's optical axis and the object coordinate system Angle between axes After considering the conditions, returning to the initialization in the Foundationpose pose estimation network, we can see that the reference coordinate system becomes the object coordinate system. Furthermore, in the object coordinate system, the camera, because it is looking from a vertex towards the origin of the object coordinate system, assumes the camera's position is... (Assuming the icosphere radius is 1, then the distance from the vertex to the origin is 1, i.e.) =1, the optical axis vector expression is , the object coordinate system Z axis expression ), at this time the camera optical axis and the object coordinate system The expression of the axis is:

[0075]

[0076] .

[0077] Understandably, since the selection of the coordinate system does not change the angle between the two vectors, the camera shooting point under the correct pose , thus, in the initialization process, the vector of the camera shooting point pointing to the origin of the object coordinate system and the object coordinate system The angle between the axes is At this time, the candidate pose camera shooting point is a circle on the icosphere rather than the entire spherical surface, and then the point selected on the target circle on the regular polyhedral spherical surface subdivision grid is set as the rotation candidate pose shooting point.

[0078] Using the scheme of the application, even if the icosphere sphere takes subdivision=2, the maximum number of shooting point positions searched is 24 at about 90°, which is much lower than the 42 shooting point positions before the improvement of subdivision=1 of FoundationPose.

[0079] Further, another improvement point of the scheme of the application is the pose initialization based on the rough view angle, which solves the geometric symmetry problem when the rotation candidate pose shooting point is photographed. The existing scheme of rotating the rotation candidate pose shooting point around the camera optical axis by 360 degrees cannot distinguish the accurate direction of an object (cannot distinguish whether the head of an object is up or down) by using a deep learning network due to the up-down symmetry of the steel bar node. It has an impact on accurately determining the pose in actual engineering, for example, if a mechanical arm is required to hold a chopstick to touch the steel bar node from the axis direction, without improvement, for the three nodes in the middle row estimated to be upside down, the mechanical arm needs to complete the task from bottom to top, and for the steel bar nodes estimated to be right side up (left upper and left lower), the mechanical arm needs to complete the task from top to bottom, which will cause the mechanical arm to collide during the movement from bottom to top. The two pose estimation results have different effects on the actual engineering application.

[0080] Through analysis, it is known that the existing upside-down pose is generated because each camera at a photographing position is rotated 360 degrees around the optical axis during initialization, so that the upside-down pose is generated. Since the prior condition is known, it is impossible to take a photograph upside down during photographing. Therefore, when rotating around the optical axis, 360-degree rotation is no longer adopted, but each of left and right is rotated 100 degrees or 60 degrees.

[0081] Please refer to Figure 6 Further, in step S30, an inflation kernel operation is adopted to map the current layer reinforcement image.

[0082] In another optional embodiment of the present application, since the intersection points of the multi-layer reinforcement structure have feature similarities on the image, the points of different layers are difficult to distinguish, and there are false points. When identifying the intersection points of the current layer reinforcement, the existing image recognition algorithm is easily disturbed by the intersection points of the background layer reinforcement due to the lack of depth information, and cannot determine which layer the identified target belongs to. Therefore, excluding the disturbance of the background layer reinforcement is a prerequisite for detecting the current layer reinforcement node. Specifically, the present application proposes a single-layer reinforcement segmentation technology to filter the background and background layer reinforcement in the image, and to effectively extract only the current layer reinforcement pixels. First, a 3D camera is used to capture color and depth images. By using the built-in function geometry.RGBDImage.create_from_color_and_depth of the open source library open3d and combining the camera intrinsic parameters, the color and depth images can be converted into a reinforcement point cloud model. The reinforcement point cloud model not only contains reinforcement point cloud data, but also includes point cloud data of the surrounding environment or other objects and a certain degree of noise. Then, the straight-through filter is used to preliminarily filter and denoise the reinforcement point cloud data. By specifying the range on one or more axes, the region of interest containing the reinforcement point cloud is retained, and the unnecessary data and noise are deleted to obtain the processed reinforcement point cloud data (denoised point cloud). In order to obtain the point cloud data of the current reinforcement layer point cloud plane in the processed reinforcement point cloud data, the RANSAC algorithm is used to perform multi-plane fitting segmentation on the denoised point cloud to obtain a plurality of initial target reinforcement point cloud planes and their corresponding plane equations In the process of RANSAC fitting the point cloud plane, first, three points are randomly selected from the denoised point cloud set M, respectively , , The plane model parameters fitted by the three points are calculated , , , Then, the distances of the remaining point set to the fitted plane are calculated If If the distance is less than a predetermined threshold, the inner point set is added, otherwise iteration is performed. When the inner points meet the quantity requirement, the model parameters of the point cloud fitting plane are output, and if the inner point quantity requirement is not met, iteration is continued until the point cloud fitting plane model parameters of the point cloud fitting plane with the largest number of inner points are found, and a plurality of initial target reinforcement point cloud planes are obtained after plane fitting segmentation; further, the distance dis from the origin (0, 0, 0) to each initial target reinforcement point cloud plane is calculated, , and the initial target reinforcement point cloud plane with the shortest distance is the current reinforcement layer point cloud plane.

[0083] Further, since there is a discontinuity phenomenon when the point cloud is mapped to an image, the method uses an inflation kernel kernel operation to expand the display range of the reinforcement pixels, , and the image processed by the inflation kernel kernel can effectively avoid the interference of other layer reinforcements.

[0084] Further, in step S40, the reinforcement node target detection model is used to detect and identify the current layer reinforcement image to obtain a reinforcement node framework image in the current layer reinforcement image; wherein the establishment of the reinforcement node target detection model includes the following steps:

[0085] Training data collection: using a 3D camera to collect reinforcement node data sets;

[0086] Training data processing: segmenting and filtering the multi-layer reinforcement images in the reinforcement node data set to obtain single-layer reinforcement images;

[0087] Training data labeling: manually labeling the single-layer reinforcement images, and the labeling of one reinforcement intersection object consists of one bounding box and one key point;

[0088] Convergent training of the initial reinforcement node target detection model is performed, and the reinforcement node target detection model is updated.

[0089] Further, pixel binary processing is performed on each reinforcement node framework image, non-white pixels in the reinforcement node framework image are set to white, and a corresponding framework mask image is generated.

[0090] In step S40, the reinforcement node framework image is sequentially sorted based on the S-shaped path working strategy, which specifically includes:

[0091] Based on the size of the axis of the image coordinate system of the current layer reinforcement image, the reinforcement node framework image is clustered and analyzed to obtain the reinforcement node framework image along the a plurality of rows of steel bar node framework image rows arranged in the axial direction, wherein each row of steel bar node framework image rows has at least one steel bar node framework image arranged in the transverse direction;

[0092] determining whether the current row of steel bar node framework images is an odd row;

[0093] if the current row of steel bar node framework images is an odd row, then the steel bar node framework images in the row are sequentially sorted based on the axial coordinate size of the image coordinate system along the first direction of the axial direction; if the current row of steel bar node framework images is an even row, then the steel bar node framework images in the row are sequentially sorted based on the axial coordinate size of the image coordinate system along the second direction of the axial direction; and

[0094] if the current row of steel bar node framework images is an even row, then the steel bar node framework images in the row are sequentially sorted based on the axial coordinate size of the image coordinate system along the second direction of the axial direction; and if the current row of steel bar node framework images is an even row, then the steel bar node framework images in the row are sequentially sorted based on the axial coordinate size of the image coordinate system along the second direction of the axial direction; and

[0095] the sorted steel bar node framework images are sequentially connected to form an S shape.

[0096] ​​In another optional embodiment of the application, the training of the steel bar node target detection model specifically includes: data collection, collecting a steel bar node dataset using a 3D camera, obtaining a color image and a corresponding depth image, considering various backgrounds, lighting, different shooting angles and distances, different diameters, spacings, layers and rusted reinforcement cages when collecting the dataset, collecting data using different multiple 3D cameras, considering the difference in the accuracy of their depth information to consider the goodness of the single-layer steel bar segmentation effect, thereby increasing the diversity of the dataset; data processing, processing the collected dataset, for multi-layer steel bar images, using the above single-layer steel bar segmentation technology, filtering the background layer steel bar to obtain a white background image containing only the current layer steel bar; for single-layer steel bar images, no filtering operation is performed, if the steel bar in the image is not approximately parallel to the x and y axes, the minimum angle rotation technology is used to automatically rotate the image into a steel bar image approximately parallel to the x and y axes, and both the pre-rotation and post-rotation images are used as the dataset, thereby increasing the diversity of the dataset; data labeling: manually labeling the processed images using the labeling tool Labelme, the labeling of a steel bar intersection object consists of a bounding box and a key point, when labeling the steel bar intersection point with a bounding box, since the steel bar is continuous, the steel bar intersection point is not an independent individual with a clear boundary, therefore, the part where the two steel bars intersect and the part where the four ends of the steel bar extend a small amount are regarded as the steel bar intersection point and are labeled with a bounding box. The key point is labeled as close as possible to the center of the two steel bar intersection parts. Finally, a.json format labeling file is obtained. Convert the labeling file into a.txt format labeling file required for training the key point detection model. Steel bar node target detection model training: using the labeled steel bar node dataset, divide it into a training set and a validation set in a ratio of 8:2, then train the target detection model in deep learning to learn the features of the steel bar nodes in the image. During the training process, monitor the performance of the model on the training set and the validation set, and make adjustments and optimizations to ensure that the model has good generalization ability and accuracy.

[0097] In an optional embodiment of the application, the framework mask image is obtained and sorted on the current layer steel bar image based on the binding path planning. Specifically, pixel binary processing is performed on each node frame, non-white pixels in the detection frame are set to white, and one mask image is generated for each node. If a picture has three nodes, three mask images will be generated. It is found through research that the S-shaped binding path is more economical than the Z-shaped binding path during the binding of steel bars, the path of the S-shaped binding path is shorter, and the running time of the mechanical arm is shorter. The pixel of the color image detection frame (steel bar node framework image) is used for binding path planning, and the geometric center of each detection frame can be known to know the approximate pixel position of the steel bar intersection point. The color image pixel coordinates are defined as the origin at the top left corner of the image, the positive direction of the axis is to the right, The positive direction of the axis is downward; specifically, first, for the coordinates of all pixel intersection points... The values ​​are clustered to determine which steel bar intersections are in the same row; then the pixels in each row are sorted, as defined by the coordinate system. The category with the smallest x value is the top row of rebar nodes in the image. In this case, the binding order is from left to right, so the smaller the x value, the earlier the node in this category. However, the second category is in the second row, and the binding should be from right to left. Therefore, the larger the x value, the earlier the node in this category. In summary, the entire process is as follows: First, all pixel coordinates are... The values ​​can be roughly divided into several categories. The smallest value belongs to the first category, the second smallest to the second category, and so on; in the odd-numbered category (also representing odd-numbered rows, rows 1, 3, and 5)... Sort the pixels from smallest to largest, and then sort even-numbered pixels from largest to smallest. For example, if the obtained pixel information is [(345,435), (50,100), (280,230), (70,240), (260,120), (100,430)], a total of 6 binding points, first... Axis clustering, Clusters with values ​​differing by no more than 50 are grouped into one class. The clustering results are: [(345,435), (100,430)], [(50,100), (260,120)], [(280,230), (70,240)]. Sort each class by y-values ​​from smallest to largest. The class order is: [(50,100), (260,120)], [(280,230), (70,240)], [(345,435), (100,430)]. Here, the i-th class represents the i-th row from top to bottom of the image. Then, within each class, sorting is performed. When there are odd-numbered rows, the clusters are arranged from left to right. Sort from smallest to largest, so the internal sorting of the first and third categories is [(50,100), (260,120)] and [(100,430), (345,435)] respectively; when there are even-numbered rows, the sorting is from right to left, and should follow the order... Sort from largest to smallest, so the second category is sorted as [(280,230), (70,240)]. The final sorted result is: [(50,100), (260,120), (280,230), (70,240), (100,430), (345,435)]. These pixels correspond to bounding boxes, which in turn correspond to masks. The mask order corresponds to the order of each specific rebar node. Therefore, sorting the pixels is equivalent to sorting the poses of the rebar nodes, and the robot will tie them according to this pose order.

[0098] The present invention also provides a 6D pose estimation system for a robotic arm based on fuzzy perspective prior, including a welding robot and a visual perception device. The welding robot has a base and a robotic arm, and the visual perception device is located on the robotic arm.

[0099] The welding robot is equipped with a processing device, which is used to implement the steps of the above-mentioned 6D pose estimation method for the robot arm based on fuzzy view prior.

[0100] Optionally, in order to accurately acquire two-dimensional images and three-dimensional information of the target workpiece, an industrial-grade 3D structured light camera (Mech-Eye NANO) installed at the end of the robotic arm is selected as a visual perception device.

[0101] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-described method for estimating the 6D pose of a robotic arm based on fuzzy view priors.

[0102] The beneficial effects of the 6D pose estimation method and system for a robotic arm based on fuzzy view prior provided by this invention include:

[0103] A multimodal fusion method using color image information (RGB image) and depth information (depth image) is employed to achieve 6D pose estimation of rebar nodes. The zero-sample pose estimation model handles the rotation component based on fuzzy viewpoint priors, utilizing angle... The prior conditions reduce the search range of the icosphere virtual camera's image capture position, enabling high-efficiency, low-computational-requirement end-to-end 6D pose estimation of rebar nodes. Candidate image capture poses are generated by rotating candidate pose capture points within a preset rotation range (-β to β), which helps avoid generating mirrored images. The current layer rebar image is used to sort the rebar nodes using an S-shaped path, achieving computationally lightweight rebar tying path planning. Single-layer rebar segmentation technology is used to obtain the current layer rebar image, effectively achieving segmentation and rotation of the current layer rebar image without being limited by camera distance and angle, exhibiting good robustness. This scheme pre-acquires the current layer rebar image and identifies the rebar node frame image based on the rebar node target detection model, generating a coarse rebar node mask in multi-layer rebar overlapping scenes without affecting the accuracy of the final 6D pose estimation.

[0104] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for estimating the 6D pose of a robotic arm based on fuzzy view priors, characterized in that, Includes the following steps: S10: Obtain steel bar point cloud data based on the RGB image and depth image corresponding to the target workpiece and combined with camera intrinsic parameters; S20, Perform plane fitting and segmentation processing on the rebar point cloud data to obtain the current rebar layer point cloud plane; S30, map the current reinforcement layer point cloud plane to the current layer reinforcement image; S40, Perform target detection and recognition and pixel binarization processing on the current layer steel reinforcement image to obtain the frame mask image corresponding to each steel reinforcement node in the current layer steel reinforcement image; S50, taking the frame mask image, the RGB image and the depth image as input, performing translation and rotation processing based on the zero-sample pose estimation model, and coupling the processing results to obtain the 6D estimated pose corresponding to the steel bar node frame image; The rotation process includes rotation pose initialization, rotation network optimization, and rotation network scoring. The rotation pose initialization is used to generate rotation candidate pose imaging points and candidate imaging poses corresponding to each rotation candidate pose imaging point. Specifically, the rotation pose initialization includes: selecting a point on a target circle on a regular polyhedral spherical subdivision mesh as the rotation candidate pose imaging point, and generating the candidate imaging pose based on each of the rotation candidate pose imaging points; The camera optical axis and object coordinate system corresponding to the rotated candidate pose imaging point The included angle between the axes is , and The difference between them is no greater than a preset angle, where, , Let be the vector expression for the optical axis of the camera. In the coordinate system of the object Axis expression.

2. The 6D pose estimation method for a robotic arm based on fuzzy view priors as described in claim 1, characterized in that, The step of generating the candidate image pose based on each of the rotated candidate pose image points specifically includes: exist to Within the rotation range, a preset rotation step size is used to rotate around the camera's optical axis to obtain multiple candidate image poses corresponding to each of the rotation candidate pose image points. No more than 180 degrees.

3. The 6D pose estimation method for a robotic arm based on fuzzy view priors according to claim 2, characterized in that, The preset rotation step size is 60 degrees. The angle is 60 degrees or 100 degrees, and one of the rotating candidate poses for taking pictures corresponds to three candidate poses for taking pictures.

4. The 6D pose estimation method for a robotic arm based on fuzzy view priors according to claim 1, characterized in that, The rebar point cloud data is subjected to planar fitting and segmentation to obtain multiple target rebar layer point cloud planes. The optimal target rebar layer point cloud plane is determined as the current rebar layer point cloud plane, and the normal vector of the target rebar layer point cloud plane is determined as the coordinate of the object coordinate system. axis.

5. The 6D pose estimation method for a robotic arm based on fuzzy view priors according to any one of claims 1 to 4, characterized in that, In step S30, the point cloud plane of the current rebar layer is mapped to the image of the current rebar layer using an expansion kernel operation.

6. The 6D pose estimation method for a robotic arm based on fuzzy view priors according to any one of claims 1 to 4, characterized in that, In step S40, target detection and recognition are performed on the current layer of rebar image based on the rebar node target detection model to obtain the rebar node frame image in the current layer of rebar image; wherein, the establishment of the rebar node target detection model includes the following steps: Training data collection: A 3D camera was used to collect a dataset of rebar nodes; Training data processing involves segmenting and filtering multi-layer rebar images in the rebar node dataset to obtain single-layer rebar images. Training data annotation involves manually annotating the single-layer rebar image. The annotation of a rebar intersection object consists of a bounding box and a key point. The initial rebar node target detection model is trained to convergence, and then updated to obtain the rebar node target detection model.

7. The 6D pose estimation method for a robotic arm based on fuzzy view priors according to claim 6, characterized in that, Each of the rebar node frame images is subjected to pixel binarization processing, and non-white pixels in the rebar node frame image are set to white to generate a corresponding frame mask image.

8. The 6D pose estimation method for a robotic arm based on fuzzy view priors according to claim 6, characterized in that, In step S40, the rebar node frame images are sequentially sorted based on the S-shaped path working strategy, specifically including: Based on the image coordinate system of the current layer of reinforcement image Cluster analysis was performed on the image of the steel reinforcement node frame based on the size of the axis to obtain the result along the axis. A series of images of multiple rows of reinforcing steel node frames arranged along an axial direction, wherein each row of the reinforcing steel node frame images has at least one reinforcing steel node frame image arranged laterally. Determine whether the current row of the rebar node frame image is an odd number of rows; If the current row of the rebar node frame image is an odd number of rows, then based on the image coordinate system axial coordinates along The first direction of the axis sorts the rebar node frame images in the row of rebar node frame images sequentially; If the current row of the rebar node frame image is an even number of rows, then based on the image coordinate system... axial coordinates along The second direction of the axis is used to sequentially sort the rebar node frame images in the row of rebar node frame images; wherein, the second direction is opposite to the first direction; The sorted rebar node frame images are connected sequentially to form an S-shape.

9. A 6D pose estimation system for a robotic arm based on fuzzy view priors, characterized in that, It includes a welding robot and a vision sensing device. The welding robot has a base and a robotic arm, and the vision sensing device is mounted on the robotic arm. The welding robot is equipped with a processing device, which is used to implement the steps of the 6D pose estimation method for a robot arm based on fuzzy perspective prior as described in any one of claims 1 to 8.

Images