Vision-guided welding method, device, medium and robot arm for steel frame

Abstract

The present application discloses a visually guided welding method, device, medium and robot arm of a steel bar skeleton, belonging to the field of welding technology, the method comprising: in the global image of the target steel bar skeleton, identifying the target tie bar, and determining the position coordinates of the target tie bar in the world coordinate system, and obtaining the target position coordinates; using the target position coordinates to guide the movement of the three-dimensional camera carried by the robot arm, and using the three-dimensional camera to shoot the image of the target tie bar to determine the point cloud data of the target tie bar location area in the world coordinate system, and obtain the target point cloud data; processing the target point cloud data to determine the intersection between the target tie bar and the straight steel bar carrying the target tie bar, obtain the target welding point, and control the robot arm to weld the target welding point. Through this method, the tie bar nodes with bending degree in the multi-layer complex steel bar skeleton can be welded.

Description

Technical Field

[0001] The present invention relates to the field of welding technology, and in particular to a visually guided welding method, device, medium and robot arm for a steel bar skeleton. Background Art

[0002] Steel bar skeletons are widely used in the construction industry. In order to ensure the stability of steel bar skeletons during use, it is usually necessary to weld a tie bar every 10 to 20 centimeters in the steel bar skeleton. In the existing steel bar skeleton welding method, generally only the cross-shaped tie bars in a single-layer steel bar skeleton can be welded, but the tie bars with bending in a multi-layer complex steel bar skeleton cannot be welded. At present, there is no more effective solution to this technical problem.

[0003] It can be seen that how to weld tie bars with bending degrees in a multi-layer complex steel bar skeleton is a technical problem that needs to be solved urgently by technical personnel in this field. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a visually guided welding method, device, medium and robot arm for a steel bar skeleton, so as to solve the technical problem that the prior art cannot weld tie bars with curvature in a multi-layer complex steel bar skeleton. The specific scheme is as follows:

[0005] In order to solve the above technical problems, the present invention provides a visually guided welding method for a steel bar skeleton, comprising:

[0006] In a global image of a target steel bar skeleton, a target tie bar is identified, and the position coordinates of the target tie bar in a world coordinate system are determined to obtain the target position coordinates; wherein the tie bars in the target steel bar skeleton all have bending degrees; the global image includes RGB information and depth information of the target steel bar skeleton; the target tie bar is any tie bar in the target steel bar skeleton;

[0007] The target position coordinates are used to guide the movement of a three-dimensional camera carried on a robot arm, and the image of the target tie bar taken by the three-dimensional camera is used to determine the point cloud data of the target tie bar location area in the world coordinate system to obtain the target point cloud data;

[0008] The target point cloud data is processed to determine the intersection between the target tie bar and the straight steel bar that carries the target tie bar, obtain the target welding point, and control the robot arm to weld the target welding point.

[0009] Preferably, the process of identifying the target tie bar in the global image of the target steel bar skeleton, and determining the position coordinates of the target tie bar in the world coordinate system to obtain the target position coordinates includes:

[0010] Inputting the global image of the target steel bar skeleton into a target recognition network;

[0011] The target tie bar is identified, and the position coordinates of the target tie bar in the world coordinate system are determined by hand-eye calibration to obtain the target position coordinates.

[0012] Preferably, the process of using the target position coordinates to guide the movement of a three-dimensional camera carried by a robot arm, and using the three-dimensional camera to take an image of the target tie bar to determine point cloud data of the target tie bar location area in a world coordinate system, to obtain the target point cloud data includes:

[0013] The target position coordinates are used to guide the three-dimensional camera carried by the end of the robot arm to move, and the RGB image and depth map of the target tie bar location area are determined by using the image taken by the three-dimensional camera of the target tie bar, and converted into original point cloud data;

[0014] The original point cloud data is subjected to coordinate transformation to obtain point cloud data of the target tie bar location region in a world coordinate system, thereby obtaining the target point cloud data.

[0015] Preferably, the process of processing the target point cloud data to determine the intersection between the target tie bar and the straight steel bar carrying the target tie bar to obtain the target welding point includes:

[0016] In the RGB image and the depth map of the target tie bar location area, the target tie bar is identified and segmented by using a target recognition and segmentation algorithm, and the depth information of the target tie bar is removed, and the background depth information of the target steel bar skeleton is removed by using a depth filtering algorithm, so as to retain the outer layer steel bar point cloud data after removing the target tie bar, and obtain ROI point cloud data;

[0017] Downsampling the ROI point cloud data to obtain downsampled point cloud data, and determining cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the downsampled point cloud data to obtain a first surface equation and a second surface equation; wherein the first straight steel bar and the second straight steel bar are straight steel bars that carry the target tie bars;

[0018] The tie bar template library is used to perform template matching on the tie bars in the downsampled point cloud data, and the intersection point of the target tie bar and the first straight steel bar, and / or the intersection point of the target tie bar and the second straight steel bar are determined according to the first surface equation and the second surface equation to obtain the target welding point.

[0019] Preferably, the process of downsampling the ROI point cloud data to obtain downsampled point cloud data, and determining cylindrical surface equations corresponding to the first linear steel bar and the second linear steel bar respectively according to the downsampled point cloud data to obtain the first surface equation and the second surface equation comprises:

[0020] Projecting the ROI point cloud data onto the X-axis of the camera coordinate system, and establishing a point cloud density spectrum with the Y-axis of the camera coordinate system as an independent variable, to obtain a first point cloud density spectrum;

[0021] Selecting point cloud data that meets a first value range in the first point cloud density spectrum to obtain first point cloud data;

[0022] Projecting the first point cloud data onto the X-axis of the camera coordinate system, and establishing a point cloud density spectrum with the Z-axis of the camera coordinate system as an independent variable, to obtain a second point cloud density spectrum;

[0023] Selecting point cloud data that conforms to a second value range in the second point cloud density spectrum to obtain second point cloud data;

[0024] Projecting the ROI point cloud data onto the Y axis of the camera coordinate system, and establishing a point cloud density spectrum with the X axis of the camera coordinate system as an independent variable, to obtain a third point cloud density spectrum;

[0025] Selecting point cloud data that conforms to a third value range in the third point cloud density spectrum to obtain third point cloud data;

[0026] Projecting the third point cloud data on the Y axis of the camera coordinate system, and establishing a point cloud density spectrum with the Z axis of the camera coordinate system as an independent variable, to obtain a fourth point cloud density spectrum;

[0027] Selecting point cloud data that conforms to a fourth value range in the fourth point cloud density spectrum to obtain fourth point cloud data;

[0028] The cylindrical surface equations corresponding to the first linear steel bar and the second linear steel bar are respectively fitted according to the second point cloud data and the fourth point cloud data to obtain the first surface equation and the second surface equation.

[0029] Preferably, the process of performing template matching on the tie bars in the downsampled point cloud data using the tie bar template library, and determining the intersection of the target tie bar and the first straight steel bar, and / or the intersection of the target tie bar and the second straight steel bar according to the first surface equation and the second surface equation to obtain the target welding point includes:

[0030] Reading point cloud data in the tie bar template library to obtain template point cloud data;

[0031] Processing the template point cloud data and the downsampled point cloud data respectively to obtain FPFH features, thereby obtaining first processed point cloud data and second processed point cloud data;

[0032] Using a Ransac algorithm to roughly match the first processed point cloud data and the second processed point cloud data, and using an ICP algorithm to precisely match the first processed point cloud data and the second processed point cloud data to obtain matching point cloud data;

[0033] The intersection point of the target tie bar and the first straight steel bar, and / or the intersection point of the target tie bar and the second straight steel bar are determined according to the matching point cloud data, the first surface equation and the second surface equation to obtain the target welding point.

[0034] Preferably, the process of controlling the robot arm to weld the target welding point includes:

[0035] Obtaining the position coordinates of the target welding point in the world coordinate system to obtain the six-degree-of-freedom coordinates of the welding point position;

[0036] Based on the six-degree-of-freedom coordinates of the welding point position, and using a robot arm inverse solution algorithm, each joint angle of the robot arm is calculated to obtain a joint angle set;

[0037] The robot arm is controlled to weld the target welding point according to the joint angle set.

[0038] In order to solve the above technical problems, the present invention also provides a visually guided welding device for a steel bar skeleton, comprising:

[0039] A tie bar identification module is used to identify the target tie bar in the global image of the target steel bar skeleton, and determine the position coordinates of the target tie bar in the world coordinate system to obtain the target position coordinates; wherein the tie bars in the target steel bar skeleton all have bending degrees; the global image includes RGB information and depth information of the target steel bar skeleton; the target tie bar is any tie bar in the target steel bar skeleton;

[0040] A robot arm guiding module is used to guide the movement of a three-dimensional camera carried on the robot arm using the target position coordinates, and to determine point cloud data of the target tie bar location area in a world coordinate system using the image taken by the three-dimensional camera of the target tie bar, thereby obtaining target point cloud data;

[0041] The welding point welding module is used to process the target point cloud data to determine the intersection between the target tie bar and the straight steel bar bearing the target tie bar, obtain the target welding point, and control the robot arm to weld the target welding point.

[0042] In order to solve the above technical problems, the present invention further provides a mechanical arm, comprising:

[0043] Memory for storing computer programs;

[0044] The processor is used to implement the steps of the visually guided welding method of a steel bar skeleton as disclosed above when executing the computer program.

[0045] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a visually guided welding method for a steel skeleton as disclosed above are implemented.

[0046] Beneficial effect: In the present invention, first, in the global image of the target steel bar skeleton, the target tie bar is identified, and the position coordinates of the target tie bar in the world coordinate system are determined to obtain the target position coordinates; wherein, the tie bars in the target steel bar skeleton all have bending degrees; the global image includes the RGB information and depth information of the target steel bar skeleton; the target tie bar is any tie bar in the target steel bar skeleton; since the RGB information and depth information in the global image of the target steel bar skeleton contain the three-dimensional information of the target steel bar skeleton, by identifying the global image of the target steel bar skeleton, the tie bars with bending degrees can be roughly identified from the multi-layer complex steel bar skeleton. Afterwards, the target position coordinates are used to guide the three-dimensional camera carried by the mechanical arm to move to the target tie bar. Through this method, the target tie bar can be photographed at a close distance by the three-dimensional camera carried by the mechanical arm, and the target tie bar can be accurately identified. At this time, the point cloud data of the target tie bar location area in the world coordinate system can be obtained by the three-dimensional camera carried by the mechanical arm, and the target point cloud data is obtained. Finally, the target point cloud data is processed again, and the intersection between the target tie bar and the straight steel bar that carries the target tie bar is determined based on the processing result, so that the target welding point where the target tie bar needs to be welded can be determined. After the target welding point is determined, the robot arm can be controlled to weld the target welding point. Obviously, this method can be used to weld tie bars with curvature in multi-layer complex steel skeletons. Correspondingly, the visually guided welding device, medium and robot arm of a steel skeleton provided by the present invention also have the above-mentioned beneficial effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A flow chart of a visually guided welding method for a steel bar skeleton provided by an embodiment of the present invention;

[0049] Figure 2 A hardware architecture diagram for welding a steel bar skeleton provided by an embodiment of the present invention;

[0050] Figure 3 This is a diagram showing the effect of using the target recognition network to identify the global image of the target steel bar skeleton;

[0051] Figure 4A flow chart of another visually guided welding method for a steel bar skeleton provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of obtaining ROI point cloud data after preprocessing the target point cloud data;

[0053] Figure 6 A flow chart of downsampling ROI point cloud data provided by an embodiment of the present invention;

[0054] Figure 7 is a schematic diagram of the first point cloud density spectrum;

[0055] Figure 8 is a schematic diagram of the first point cloud data;

[0056] Fig. 9 is a schematic diagram of the second point cloud density spectrum;

[0057] Fig.10 is a schematic diagram of the second point cloud data;

[0058] Fig.11 This is a schematic diagram of setting the local coordinate system on the tie bar;

[0059] Fig.12 is a schematic diagram of the second processing point cloud data;

[0060] Fig.13 This is a schematic diagram for calculating the target welding point;

[0061] Fig.14 A structural diagram of a visually guided welding device for a steel bar skeleton provided by an embodiment of the present invention;

[0062] Fig.15 A structural diagram of a robotic arm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] See also Figure 1 , Figure 1 A flowchart of a visually guided welding method for a steel bar skeleton provided by an embodiment of the present invention, the method comprising:

[0065] Step S11: in the global image of the target steel bar skeleton, the target tie bar is identified, and the position coordinates of the target tie bar in the world coordinate system are determined to obtain the target position coordinates;

[0066] The tie bars in the target steel skeleton all have bending degrees; the global image includes RGB information and depth information of the target steel skeleton; the target tie bar is any tie bar in the target steel skeleton;

[0067] Step S12: using the target position coordinates to guide the three-dimensional camera carried on the robot arm to move, and using the three-dimensional camera to take pictures of the target tie bar to determine the point cloud data of the target tie bar location area in the world coordinate system, and obtain the target point cloud data;

[0068] Step S13: Process the target point cloud data to determine the intersection between the target tie bar and the straight steel bar that carries the target tie bar, obtain the target welding point, and control the robot arm to weld the target welding point.

[0069] The steel bar skeleton welding method provided in the present application is capable of welding tie bar nodes with curvature in multi-layer complex steel bar skeletons. In this method, the global image of the target steel bar skeleton is first obtained, wherein the global image of the target steel bar skeleton includes the RGB (Red Green Blue) information and depth information of the target steel bar skeleton. Moreover, the tie bars in the target steel bar skeleton all have curvature, and the target tie bar is any tie bar in the target steel bar skeleton. It should be noted that in the present application, the bending angle of the tie bars in the target steel bar skeleton is not specifically limited.

[0070] See also Figure 2 , Figure 2 A hardware architecture diagram for welding a steel bar skeleton provided by an embodiment of the present invention. Figure 2 In the figure, 100 represents a target steel bar skeleton, 101 and 102 respectively represent a first three-dimensional camera and a second three-phase camera built on the guide rail above the target steel bar skeleton, 103 represents a three-dimensional camera built on a robot arm, and the robot arm is fixed on the guide rail above the target steel bar skeleton by welding or screws, and 104 represents a tie bar in the target steel bar skeleton.

[0071] In practical applications, the target steel bar skeleton 100 can be photographed by the first three-dimensional camera 101 and the second three-dimensional camera 102 built on the guide rail above the target steel bar skeleton 100, and the global image of the target steel bar skeleton 100 can be spliced ​​and synthesized according to the photographing results.

[0072] It is understandable that, because the global image of the target steel bar skeleton contains the three-dimensional information of the target steel bar skeleton, by identifying the global image of the target steel bar skeleton, the tie bars with bending degrees can be roughly identified from the multi-layer complex steel bar skeleton. After the target tie bars are identified from the target steel bar skeleton, it is equivalent to obtaining the position coordinates of the target tie bars in the camera coordinate system. At this time, through coordinate conversion, the position coordinates of the target tie bars in the world coordinate system can be determined to obtain the target position coordinates.

[0073] After the target position coordinates of the target tie bar in the world coordinate system are determined, the rotational posture of the robot arm is adjusted according to the target position coordinates of the target tie bar in the world coordinate system, and the three-dimensional camera carried on the robot arm can be guided to move. At this time, the three-dimensional camera carried on the robot arm is controlled to move to the position above the target tie bar, and the image of the target tie bar is photographed by the three-dimensional camera. According to the image of the target tie bar photographed by the three-dimensional camera, the point cloud data of the target tie bar location area in the world coordinate system can be determined to obtain the target point cloud data.

[0074] It can be understood that because the target point cloud data contains the RGB information, depth information and spatial position information of the target tie bar, when the target point cloud data of the target tie bar location area in the world coordinate system is obtained, the point cloud data of the target tie bar and the point cloud data of the straight steel bars carrying the target tie bar can be separated from the target point cloud data by filtering, downsampling, template matching and other operations on the target point cloud data. At this time, the intersection between the target tie bar and the straight steel bar carrying the target tie bar can be determined by calculation.

[0075] When the intersection between the target tie bar and the straight steel bar carrying the target tie bar is determined, it is equivalent to finding the welding point for fixing the target tie bar from the target steel bar skeleton, that is, finding the target welding point. In this case, the robot arm is controlled to move to the target welding point, and the robot arm can be used to weld the target welding point.

[0076] Obviously, compared with manual welding of the target steel bar skeleton, the technical solution provided by this embodiment can not only improve the welding efficiency when welding the welding points in the target steel bar skeleton, but also reduce the cost investment when welding the welding points in the target steel bar skeleton.

[0077] In this embodiment, first, the target tie bar is identified in the global image of the target steel bar skeleton, and the position coordinates of the target tie bar in the world coordinate system are determined to obtain the target position coordinates; since the RGB information and depth information in the global image of the target steel bar skeleton contain the three-dimensional information of the target steel bar skeleton, by identifying the global image of the target steel bar skeleton, the tie bar with bending degree can be roughly identified from the multi-layer complex steel bar skeleton. After that, the target position coordinates are used to guide the three-dimensional camera carried by the robot arm to move to the target tie bar. Through this method, the three-dimensional camera carried by the robot arm can be used to shoot the target tie bar at a close distance, and the target tie bar can be accurately identified. At this time, the three-dimensional camera carried by the robot arm can obtain the point cloud data of the target tie bar location area in the world coordinate system, and obtain the target point cloud data. Finally, the target point cloud data is processed, and the intersection between the target tie bar and the straight steel bar carrying the target tie bar is determined according to the processing result, so that the target welding point where the target tie bar needs to be welded can be determined. After the target welding point is determined, the robot arm can be controlled to weld the target welding point. Obviously, this method can be used to weld the tie bars with bending in the multi-layer complex steel skeleton.

[0078] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. As a preferred implementation mode, the above steps: in the global image of the target steel bar skeleton, the target tie bar is identified, and the position coordinates of the target tie bar in the world coordinate system are determined to obtain the target position coordinates, including:

[0079] Inputting the global image of the target steel bar skeleton into the target recognition network;

[0080] The target tie bar is identified, and the position coordinates of the target tie bar in the world coordinate system are determined by hand-eye calibration to obtain the target position coordinates.

[0081] When using the target recognition network to identify the target tie bar in the global image of the target steel bar skeleton, because the target recognition network can clearly display the recognition result in the global image of the target steel bar skeleton through a rectangular frame, which is convenient for the processing of subsequent processes, in this embodiment, the target tie bar is identified by inputting the global image of the target steel bar skeleton into the target recognition network, and the position coordinates of the target tie bar in the world coordinate system are determined by hand-eye calibration to obtain the target position coordinates. Specifically, the target recognition network can be set to a CenterNet network (Objectas Points, a target detection algorithm).

[0082] When using hand-eye calibration to determine the position coordinates of the target tie bar in the world coordinate system, the hand-eye calibration will obtain the homogeneous transformation matrix from the camera to the end of the robot arm. Combined with the homogeneous transformation matrix from the end of the robot arm to the base of the robot arm and the homogeneous transformation matrix from the base of the robot arm to the world coordinate system, the target recognition result of the target recognition network can be converted into the position coordinates of the target tie bar in the world coordinate system. Among them, the target recognition result of the target recognition network includes the pixel coordinates of the target tie bar and the depth information corresponding to the pixel coordinates.

[0083] See also Figure 3 , Figure 3 This is a diagram showing the effect of using the target recognition network to identify the global image of the target steel bar skeleton. Figure 3 In , each node center represents the tie bar identified from the global image of the target reinforcement skeleton. Figure 3 It can be clearly seen in the figure that the target recognition network can be used to identify the tie bars in the target steel bar skeleton from the global image of the target steel bar skeleton.

[0084] When using hand-eye calibration to determine the target position coordinates of the target tie bar in the world coordinate system, the target position coordinates of the target tie bar in the world coordinate system are mainly determined through the following transformation matrix.

[0085] Among them, the expression of the transformation matrix is:

[0086] ;

[0087] In the formula, are pixel coordinates in the global image, is the center depth value of the target tie bar, is the intrinsic parameter matrix of the 3D camera corresponding to the global image, It is the inverse matrix of the homogeneous transformation matrix from the camera coordinate system to the world coordinate system corresponding to the global image. This matrix can be obtained through hand-eye calibration.

[0088] Obviously, through the technical solution provided in this embodiment, the target tie bars can be roughly identified from the target steel bar skeleton, and the position coordinates of the target tie bars in the world coordinate system can be determined.

[0089] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. As a preferred implementation mode, the above steps: using the target position coordinates to guide the three-dimensional camera carried by the robot arm to move, and using the three-dimensional camera to take pictures of the target tie bar to determine the point cloud data of the target tie bar location area in the world coordinate system, and obtain the target point cloud data, including:

[0090] The target position coordinates are used to guide the movement of the 3D camera at the end of the robot arm, and the RGB image and depth map of the target tie bar location area are determined by using the image taken by the 3D camera, and then converted into the original point cloud data;

[0091] The original point cloud data is subjected to coordinate transformation to obtain the point cloud data of the target tie bar location area in the world coordinate system, thereby obtaining the target point cloud data.

[0092] When using the three-dimensional camera carried on the robotic arm to accurately locate the target tie bar, the three-dimensional camera carried on the end of the robotic arm is first guided to move using the target position coordinates, and the three-dimensional camera carried on the end of the robotic arm is controlled to move to a position above the target tie bar, and the three-dimensional camera is used to obtain a photographic image of the target tie bar.

[0093] It should be noted that in this embodiment, in order to facilitate the calculation of subsequent processes and simplify the process of robot arm coordinate conversion, a three-dimensional camera is set at the end of the robot arm. In this case, the rotational posture of the end of the robot arm is adjusted according to the target position coordinates of the target tie bar in the world coordinate system, and the robot arm can be controlled to move to the position area of ​​the target tie bar.

[0094] Afterwards, the 3D camera carried at the end of the robotic arm is used to capture the image of the target tie bar, and the RGB image and depth map of the target tie bar location area are obtained based on the captured image of the target tie bar, and then converted into original point cloud data.

[0095] When the original point cloud data is obtained, it is necessary to first use the camera intrinsic parameter matrix of the three-dimensional camera to convert the original point cloud data of the target tie bar location area into the point cloud data of the target tie bar location area in the camera coordinate system; then, use hand-eye calibration to convert the point cloud data of the target tie bar location area in the camera coordinate system into the point cloud data of the target tie bar location area under the end of the robotic arm; then, use the posture information of the end of the robotic arm to convert the point cloud data of the target tie bar location area under the end of the robotic arm into the point cloud data of the target tie bar location area in the robotic arm base coordinate system; finally, convert the point cloud data of the target tie bar location area in the robotic arm base coordinate system into the point cloud data of the target tie bar location area in the world coordinate system.

[0096] It should be noted that in actual applications, when using the 3D camera mounted on the end of the robotic arm to capture the image of the target tie bar, there may be images of other tie bars in the shooting field of view of the 3D camera. In this case, it is only necessary to focus on the captured image of the target tie bar.

[0097] Obviously, through the technical solution provided in this embodiment, the point cloud data of the target tie bar location area in the world coordinate system can be accurately obtained.

[0098] Based on the above embodiments, this embodiment further explains and optimizes the technical solution. Figure 4 , Figure 4 A flowchart of another visually guided welding method for a steel bar skeleton provided by an embodiment of the present invention. As a preferred implementation, the above step S13: processing the target point cloud data to determine the intersection between the target tie bar and the straight steel bar carrying the target tie bar, and obtaining the target welding point, includes:

[0099] Step S301: in the RGB image and depth map of the target tie bar location area, the target tie bar is identified and segmented using a target recognition and segmentation algorithm, and the depth information of the target tie bar is removed, and the background depth information of the target steel bar skeleton is removed using a depth filtering algorithm, so as to retain the outer layer steel bar point cloud data after removing the target tie bar, and obtain ROI point cloud data;

[0100] Step S302: downsampling the ROI point cloud data to obtain downsampled point cloud data, and determining cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the downsampled point cloud data to obtain the first surface equation and the second surface equation; wherein the first straight steel bar and the second straight steel bar are straight steel bars that carry the target tie bars;

[0101] Step S303: Use the tie bar template library to perform template matching on the tie bars in the downsampled point cloud data, and determine the intersection of the target tie bar and the first straight steel bar, and / or the intersection of the target tie bar and the second straight steel bar according to the first surface equation and the second surface equation to obtain the target welding point.

[0102] It is understandable that the target point cloud data of the target tie bar location area in the world coordinate system contains a large amount of redundant information. If the target point cloud data is directly processed, it will greatly consume the resource overhead of the computer processor. In this embodiment, in order to avoid the above problem, the target point cloud data is also pre-processed.

[0103] When preprocessing the target point cloud data, the target tie bar is first identified and segmented using the target recognition and segmentation algorithm in the RGB image and depth map of the target tie bar location area; then, the depth information of the target tie bar is removed in the depth map of the target tie bar location area, and the background depth information of the target steel bar skeleton is removed using the depth filtering algorithm, thereby retaining the outer layer steel bar point cloud data after removing the target tie bar, and thus obtaining the ROI (Region Of Internet, region of interest) point cloud data. In practical applications, the target recognition and segmentation algorithm can be set to the Yolov9 algorithm or other types of recognition and segmentation algorithms, which will not be described in detail here.

[0104] For details, please see Figure 5 , Figure 5 This is a schematic diagram of obtaining ROI point cloud data after preprocessing the target point cloud data. Obviously, this processing method can significantly reduce the amount of data that the computer processor needs to process the target point cloud data.

[0105] In order to further reduce the amount of point cloud data processing in the subsequent process, the ROI point cloud data can be further downsampled to obtain downsampled point cloud data. Figure 5 The linear reinforcement on the X-axis of the camera coordinate system is called the first linear reinforcement, and the linear reinforcement on the Y-axis of the camera coordinate system is called the second linear reinforcement.

[0106] It is understandable that, because the downsampled point cloud data contains the RGB information, depth information and geometric position information of the first straight steel bar, the second straight steel bar and the target tie bar, the cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar can be fitted respectively using the downsampled point cloud data to obtain the first surface equation and the second surface equation. It should be noted that fitting the cylindrical surface equation of the straight steel bar according to the point cloud data is common knowledge of those skilled in the art and will not be described in detail here.

[0107] When the first surface equation and the second surface equation corresponding to the first straight steel bar and the second straight steel bar are fitted respectively, the tie bar template library is used to perform template matching on the tie bars in the downsampled point cloud data, and the point cloud data corresponding to the target tie bar can be found from the downsampled point cloud data.

[0108] At this time, the intersection of the target tie bar and the first straight steel bar can be determined according to the template matching result and the first surface equation. Similarly, the intersection of the target tie bar and the second straight steel bar can be determined according to the template matching result and the second surface equation. When the intersection of the target tie bar and the first straight steel bar and / or the intersection of the target tie bar and the second straight steel bar are determined, it is equivalent to obtaining the node to be welded in the target steel bar skeleton, that is, obtaining the target welding point.

[0109] Obviously, the technical solution provided in this embodiment can significantly reduce the amount of data required by a computer processor when processing point cloud data.

[0110] See also Figure 6 , Figure 6 A flowchart of downsampling ROI point cloud data provided by an embodiment of the present invention. As a preferred implementation, the above step S302: downsampling ROI point cloud data to obtain downsampled point cloud data, and determining cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the downsampled point cloud data, to obtain the first surface equation and the second surface equation, includes:

[0111] Step S01: Project the ROI point cloud data on the X-axis of the camera coordinate system, and establish a point cloud density spectrum with the Y-axis of the camera coordinate system as an independent variable to obtain a first point cloud density spectrum;

[0112] Step S02: selecting point cloud data that conforms to a first value range in the first point cloud density spectrum to obtain first point cloud data;

[0113] Step S03: Projecting the first point cloud data on the X-axis of the camera coordinate system, and establishing a point cloud density spectrum with the Z-axis of the camera coordinate system as an independent variable, to obtain a second point cloud density spectrum;

[0114] Step S04: selecting point cloud data that conforms to a second value range in the second point cloud density spectrum to obtain second point cloud data;

[0115] Step S05: Projecting the ROI point cloud data on the Y axis of the camera coordinate system, and establishing a point cloud density spectrum with the X axis of the camera coordinate system as an independent variable, to obtain a third point cloud density spectrum;

[0116] Step S06: selecting point cloud data that conforms to a third value range in the third point cloud density spectrum to obtain third point cloud data;

[0117] Step S07: Projecting the third point cloud data on the Y axis of the camera coordinate system, and establishing a point cloud density spectrum with the Z axis of the camera coordinate system as an independent variable, to obtain a fourth point cloud density spectrum;

[0118] Step S08: selecting point cloud data that conforms to a fourth value range in the fourth point cloud density spectrum to obtain fourth point cloud data;

[0119] Step S09: fitting cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar respectively according to the second point cloud data and the fourth point cloud data to obtain the first surface equation and the second surface equation.

[0120] In this embodiment, in the process of downsampling the ROI point cloud data, the ROI point cloud data is first projected on the X-axis of the camera coordinate system, and a point cloud density spectrum with the Y-axis of the camera coordinate system as an independent variable is established to obtain a first point cloud density spectrum. After obtaining the first point cloud density spectrum, point cloud data that meets the first value range is screened in the first point cloud density spectrum to obtain the first point cloud data.

[0121] For details, please see Figure 7 and Figure 8 , Figure 7 is a schematic diagram of the first point cloud density spectrum; Figure 8 is a schematic diagram of the first point cloud data. Figure 7 In the example, the horizontal axis represents the Y-axis (expressed as Y-axis), and the vertical axis represents the point cloud density (expressed as Density). In practical applications, the first value range can be set to: Density ≥ 0.03, then Figure 7 Point cloud data with a value less than Y1 and greater than Y2 are removed. Figure 8 It can be seen from the schematic diagram of the first point cloud data that the point cloud data corresponding to the first linear steel bar can be roughly extracted from the ROI point cloud data through step S01 and step S02.

[0122] For details, please see Fig. 9 and Fig.10 , Fig. 9 is a schematic diagram of the second point cloud density spectrum; Fig.10 is a schematic diagram of the second point cloud data. Fig. 9 In the figure, the horizontal axis represents the Z axis (expressed as Z-axis), and the vertical axis represents the point cloud density (expressed as Density). In practical applications, the second value range can be set to: Density ≥ 0.06, at which point cloud data that does not belong to the longitudinal and transverse straight steel bars in the first point cloud data can be filtered out to the greatest extent. Fig.10 It can be seen from the schematic diagram of the second point cloud data that the point cloud data corresponding to the first linear steel bar can be more accurately extracted from the ROI point cloud data through steps S03 and S04. That is, the point cloud data where the first linear steel bar intersects with the target tie bar can be removed from the point cloud data corresponding to the first linear steel bar through steps S03 and S04.

[0123] Based on the same method, the point cloud data of the second linear steel bar can also be extracted from the ROI point cloud data. That is, first, the ROI point cloud data is projected on the Y axis of the camera coordinate system, and a point cloud density spectrum with the X axis of the camera coordinate system as the independent variable is established to obtain the third point cloud density spectrum; then, the point cloud data that meets the third value range is screened in the third point cloud density spectrum to obtain the third point cloud data; then, the third point cloud data is projected on the Y axis of the camera coordinate system, and a point cloud density spectrum with the Z axis of the camera coordinate system as the independent variable is established to obtain the fourth point cloud density spectrum; finally, the point cloud data that meets the fourth value range is screened in the fourth point cloud density spectrum to obtain the fourth point cloud data. After obtaining the fourth point cloud data, it is equivalent to extracting the point cloud data corresponding to the second linear steel bar from the ROI point cloud data, and through these operations, the point cloud data where the second linear steel bar intersects with the target tie bar can be eliminated from the point cloud data corresponding to the second linear steel bar.

[0124] When the second point cloud data and the fourth point cloud data are obtained, it is equivalent to extracting the point cloud data corresponding to the first straight steel bar and the second straight steel bar from the ROI point cloud data. At this time, the cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar can be fitted using the second point cloud data and the fourth point cloud data to obtain the first surface equation and the second surface equation.

[0125] Obviously, through the technical solution provided in this embodiment, the cylindrical surface equation corresponding to the first straight steel bar and the second straight steel bar can be accurately fitted.

[0126] As a preferred implementation, the above step: using the tie bar template library to perform template matching on the tie bars in the downsampled point cloud data, and determining the intersection of the target tie bar and the first straight steel bar, and / or the intersection of the target tie bar and the second straight steel bar according to the first surface equation and the second surface equation, to obtain the target welding point, includes:

[0127] Read the point cloud data in the tie bar template library to obtain the template point cloud data;

[0128] The template point cloud data and the downsampled point cloud data are processed respectively to obtain FPFH features, thereby obtaining first processed point cloud data and second processed point cloud data;

[0129] The first processed point cloud data and the second processed point cloud data are roughly matched by using the Ransac algorithm, and the first processed point cloud data and the second processed point cloud data are precisely matched by using the ICP algorithm to obtain matched point cloud data;

[0130] The intersection point of the target tie bar and the first straight steel bar and / or the intersection point of the target tie bar and the second straight steel bar is determined according to the matching point cloud data, the first surface equation and the second surface equation to obtain the target welding point.

[0131] In this embodiment, in order to accurately identify the target tie bars from the downsampled point cloud data, a pre-created tie bar template library is used to perform template matching on the point cloud data in the downsampled point cloud data to identify the target tie bars.

[0132] When creating a tie bar template library, it is necessary to pre-select tie bars with different diameters and bending degrees to make the tie bar template library. Here, the process of making the tie bar template library is explained in detail by taking a tie bar with a diameter of 10 mm and a bending angle of 45° as an example.

[0133] Specifically, first draw the three-dimensional model of the tie bar in the CAD (Computer Aided Design) software, and convert the three-dimensional model of the tie bar into the stl (STereoLithography) format; then, use the API (Application Programming Interface) interface provided by the software open3d (an open source 3D visual algorithm library) to downsample the three-dimensional model of the tie bar to obtain the point cloud model of the tie bar. When the point cloud model of the tie bar is obtained, it is equivalent to creating a matching template for the tie bar. The same method can also be used to create corresponding matching templates for tie bars of other sizes and bending degrees, and then create a tie bar template library, which will not be described in detail here.

[0134] It should be noted that in actual operation, in order to facilitate the coordinate transformation of the target welding point in the subsequent process, a local coordinate system can also be established with the midpoint of the bend of the tie bar as the origin. In addition, when setting the local coordinate system on the tie bar, the Z axis of the local coordinate system needs to be set away from the bend of the tie bar. Here, we can also assume that the homogeneous transformation matrix of the local coordinate system on the tie bar relative to the tie bar in the camera coordinate system is H0. For details, please refer to Fig.11 , Fig.11 This is a schematic diagram of setting the local coordinate system on the tie bar. Fig.11 In the figure, X, Y, and Z represent the X-axis, Y-axis, and Z-axis of the local coordinate system on the tie bar, respectively.

[0135] After creating the tie bar template library, in order to accurately identify the target tie bar from the downsampled point cloud data, first read the point cloud data in the tie bar template library to obtain the template point cloud data, and then process the template point cloud data and the downsampled point cloud data respectively to obtain FPFH features (Fast Persistent Feature Histograms, point cloud feature descriptors) to extract the corresponding image feature descriptors from the template point cloud data and the downsampled point cloud data to obtain the first processed point cloud data and the second processed point cloud data. For details, please refer to Fig.12 , Fig.12 A schematic diagram of the second processing point cloud data.

[0136] After that, the Ransac (Random Sample Consensus) algorithm is used to roughly match the first processed point cloud data and the second processed point cloud data, and the ICP algorithm (Iterative Closest Point) is used to precisely match the first processed point cloud data and the second processed point cloud data, so that the overlap between the second processed point cloud data and the first processed point cloud data is maximized, and the matched point cloud data is obtained. When the matched point cloud data is obtained, it is equivalent to accurately finding the point cloud data corresponding to the target tie bar from the downsampled point cloud data.

[0137] At this time, by obtaining the common solution of the matching point cloud data and the first surface equation, the intersection point of the target tie bar and the first straight steel bar can be determined. Similarly, by obtaining the common solution of the matching point cloud data and the second surface equation, the intersection point of the target tie bar and the second straight steel bar can be determined, and finally the target welding point can be obtained.

[0138] See also Fig.13 , Fig.13 This is a schematic diagram for calculating the target welding point. Fig.13 , 201 represents the cylindrical surface corresponding to the first straight steel bar, 202 represents the point cloud data of the tie bar template after matching, intersection A represents the intersection of the target tie bar and the first straight steel bar, and intersection B represents the intersection of the target tie bar and the second straight steel bar.

[0139] It should be noted that, in actual applications, the robot arm only needs to weld one of the intersection points A and B to achieve the purpose of stabilizing the target steel bar skeleton. Moreover, in order to facilitate the operation of the robot arm, it is usually only necessary to weld the welding point at the position closest to the robot arm. Of course, in order to further improve the stability of the target steel bar skeleton, the robot arm can also be controlled to weld both the intersection points A and B, which will not be described in detail here.

[0140] Obviously, through the technical solution provided in this embodiment, the welding points that need to be welded by the robot arm can be accurately located from the target steel bar skeleton.

[0141] As a preferred implementation, the above step: the process of controlling the robot arm to weld the target welding point includes:

[0142] Obtain the position coordinates of the target welding point in the world coordinate system, and obtain the six-degree-of-freedom coordinates of the welding point position;

[0143] Based on the six-degree-of-freedom coordinates of the welding point position, the angles of each joint of the robot arm are calculated using the robot arm inverse solution algorithm to obtain a set of joint angles;

[0144] The robot arm is controlled to weld the target welding point according to the joint angle set.

[0145] It is understandable that the above method cannot directly obtain the position coordinates of the target welding point in the world coordinate system. In order for the robot arm to reach the target welding point and weld the target welding point, it is necessary to calculate the position coordinates of the target welding point in the world coordinate system.

[0146] When determining the position coordinates of the target welding point in the world coordinate system, we can assume that the homogeneous transformation matrix corresponding to the matching of the first processed point cloud data and the second processed point cloud data is H1, and assume that the transformation matrix from the camera coordinate system at the end of the robot arm to the world coordinate system is H3. At the same time, assume that the homogeneous transformation matrix between the local coordinate system on the tie bar and the target welding point in the camera coordinate system is H2. Then, the homogeneous transformation matrix of the target welding point in the world coordinate system is: H3*H2*H1*H0. At this time, the homogeneous transformation matrix of the target welding point in the world coordinate system can be used to obtain the six-degree-of-freedom coordinates of the welding point position of the target welding point in the world coordinate system.

[0147] After that, based on the six-degree-of-freedom coordinates of the welding point, the robot arm inverse solution algorithm is used to calculate the corresponding joint angles when the robot arm needs to perform the welding operation, and obtain the joint angle set. When the joint angle set is obtained, the joint angle set is used to adjust the joint angles of the robot arm, so that the posture of the robot arm can be adjusted, and the robot arm can be used to weld the target welding point.

[0148] Obviously, through the technical solution provided in this embodiment, the robot arm can be controlled to weld the target welding point.

[0149] See also Fig.14 , Fig.14 A structural diagram of a visually guided welding device for a steel bar skeleton provided by an embodiment of the present invention, the device comprising:

[0150] The tie bar identification module 21 is used to identify the target tie bar in the global image of the target steel bar skeleton, and determine the position coordinates of the target tie bar in the world coordinate system to obtain the target position coordinates; wherein the tie bars in the target steel bar skeleton all have bending degrees; the global image includes RGB information and depth information of the target steel bar skeleton; the target tie bar is any tie bar in the target steel bar skeleton;

[0151] The robot arm guiding module 22 is used to guide the three-dimensional camera carried by the robot arm to move to the target tie bar using the target position coordinates, and determine the point cloud data of the target tie bar position area in the world coordinate system to obtain the target point cloud data;

[0152] The welding point welding module 23 is used to process the target point cloud data to determine the intersection between the target tie bar and the straight steel bar that carries the target tie bar, obtain the target welding point, and control the robot arm to weld the target welding point.

[0153] Preferably, the tie bar identification module 21 includes:

[0154] A network input submodule, used for inputting the global image of the target steel bar skeleton into a target recognition network;

[0155] The tie bar identification submodule is used to identify the target tie bar and determine the position coordinates of the target tie bar in the world coordinate system by hand-eye calibration to obtain the target position coordinates.

[0156] Preferably, the robot arm guiding module 22 comprises:

[0157] A robot arm guiding submodule, used to guide the three-dimensional camera carried by the end of the robot arm to move using the target position coordinates, and to determine the RGB image and depth map of the target tie bar location area using the image taken by the three-dimensional camera, and to convert them into original point cloud data;

[0158] The coordinate conversion submodule is used to perform coordinate conversion on the original point cloud data to obtain the point cloud data of the target tie bar location area in the world coordinate system, and obtain the target point cloud data.

[0159] Preferably, the welding point welding module 23 comprises:

[0160] The ROI selection submodule is used to identify and segment the target tie bars in the RGB image and depth map of the target tie bar location area using a target recognition and segmentation algorithm, and remove the depth information of the target tie bars, and remove the background depth information of the target steel bar skeleton using a depth filtering algorithm, thereby retaining the outer layer steel bar point cloud data after removing the target tie bars, and obtaining ROI point cloud data;

[0161] A downsampling submodule is used to downsample the ROI point cloud data to obtain downsampled point cloud data, and determine the cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the downsampled point cloud data to obtain the first surface equation and the second surface equation; wherein the first straight steel bar and the second straight steel bar are straight steel bars that carry the target tie bars;

[0162] The template matching submodule is used to perform template matching on the tie bars in the downsampled point cloud data using the tie bar template library, and determine the intersection of the target tie bar and the first straight steel bar, and / or the intersection of the target tie bar and the second straight steel bar according to the first surface equation and the second surface equation, to obtain the target welding point.

[0163] Preferably, the downsampling submodule includes:

[0164] A first projection unit, used to project the ROI point cloud data on the X-axis of the camera coordinate system, and establish a point cloud density spectrum with the Y-axis of the camera coordinate system as an independent variable to obtain a first point cloud density spectrum;

[0165] A first screening unit, configured to screen point cloud data that conforms to a first value range in the first point cloud density spectrum to obtain first point cloud data;

[0166] A second projection unit, used to project the first point cloud data onto the X-axis of the camera coordinate system, and establish a point cloud density spectrum with the Z-axis of the camera coordinate system as an independent variable, to obtain a second point cloud density spectrum;

[0167] A second screening unit, configured to screen point cloud data that conforms to a second value range in the second point cloud density spectrum to obtain second point cloud data;

[0168] A third projection unit is used to project the ROI point cloud data on the Y axis of the camera coordinate system, and establish a point cloud density spectrum with the X axis of the camera coordinate system as an independent variable to obtain a third point cloud density spectrum;

[0169] A third screening unit, configured to screen point cloud data that conforms to a third value range in the third point cloud density spectrum to obtain third point cloud data;

[0170] a fourth projection unit, configured to project the third point cloud data onto the Y axis of the camera coordinate system, and establish a point cloud density spectrum with the Z axis of the camera coordinate system as an independent variable, to obtain a fourth point cloud density spectrum;

[0171] a fourth screening unit, configured to screen point cloud data that conforms to a fourth value range in the fourth point cloud density spectrum to obtain fourth point cloud data;

[0172] An equation fitting unit is used to fit the cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the second point cloud data and the fourth point cloud data, respectively, to obtain the first surface equation and the second surface equation.

[0173] Preferably, the template matching submodule includes:

[0174] A point cloud reading unit, used for reading point cloud data in the tie bar template library to obtain template point cloud data;

[0175] A point cloud processing unit, used to process the template point cloud data and the downsampled point cloud data respectively, obtain FPFH features, and obtain first processed point cloud data and second processed point cloud data;

[0176] a point cloud matching unit, configured to perform a rough match on the first processed point cloud data and the second processed point cloud data using a Ransac algorithm, and perform a fine match on the first processed point cloud data and the second processed point cloud data using an ICP algorithm to obtain matching point cloud data;

[0177] A welding point acquisition unit is used to determine the intersection point of the target tie bar and the first straight steel bar, and / or the intersection point of the target tie bar and the second straight steel bar according to the matching point cloud data, the first surface equation and the second surface equation, to obtain the target welding point.

[0178] Preferably, the welding point welding module 23 comprises:

[0179] A welding point acquisition submodule is used to acquire the position coordinates of the target welding point in the world coordinate system to obtain the six-degree-of-freedom coordinates of the welding point position;

[0180] A joint angle calculation submodule, used to calculate each joint angle of the robotic arm based on the six-degree-of-freedom coordinates of the welding point position and using a robotic arm inverse solution algorithm to obtain a joint angle set;

[0181] The welding point welding submodule is used to control the robot arm to weld the target welding point according to the joint angle set.

[0182] A visually guided welding device for a steel frame provided by an embodiment of the present invention has the beneficial effects of the visually guided welding method for a steel frame disclosed above.

[0183] See also Fig.15 , Fig.15 A structural diagram of a mechanical arm provided by an embodiment of the present invention, the mechanical arm comprises:

[0184] A memory 31, used for storing computer programs;

[0185] The processor 32 is used to implement the steps of the visually guided welding method of a steel bar skeleton as disclosed above when executing the computer program.

[0186] A robotic arm provided by an embodiment of the present invention has the beneficial effects of the aforementioned visually guided welding method for a steel bar skeleton.

[0187] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of a visually guided welding method for a steel bar skeleton as disclosed above are implemented.

[0188] A computer-readable storage medium provided by an embodiment of the present invention has the beneficial effects of the visually guided welding method for a steel bar skeleton disclosed above.

[0189] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0190] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0191] The above is a detailed introduction to the visually guided welding method, device, medium and robotic arm of a steel frame provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A visually guided welding method for a steel bar skeleton, characterized in that: include: In a global image of a target steel bar skeleton, a target tie bar is identified, and the position coordinates of the target tie bar in a world coordinate system are determined to obtain the target position coordinates; wherein the tie bars in the target steel bar skeleton all have bending degrees; the global image includes RGB information and depth information of the target steel bar skeleton; the target tie bar is any tie bar in the target steel bar skeleton; The target position coordinates are used to guide the movement of a three-dimensional camera carried on a robot arm, and the image of the target tie bar taken by the three-dimensional camera is used to determine the point cloud data of the target tie bar location area in the world coordinate system to obtain the target point cloud data; Processing the target point cloud data to determine the intersection between the target tie bar and the straight steel bar carrying the target tie bar, obtaining a target welding point, and controlling the robot arm to weld the target welding point; The process of guiding the movement of the three-dimensional camera carried by the robot arm by using the target position coordinates, and determining the point cloud data of the target tie bar location area in the world coordinate system by using the three-dimensional camera to take an image of the target tie bar, and obtaining the target point cloud data includes: The target position coordinates are used to guide the three-dimensional camera carried by the end of the robot arm to move, and the RGB image and depth map of the target tie bar location area are determined by using the image taken by the three-dimensional camera of the target tie bar, and converted into original point cloud data; Performing coordinate transformation on the original point cloud data to obtain point cloud data of the target tie bar location area in a world coordinate system, thereby obtaining the target point cloud data; The process of processing the target point cloud data to determine the intersection point between the target tie bar and the straight steel bar carrying the target tie bar to obtain the target welding point includes: In the RGB image and the depth map of the target tie bar location area, the target tie bar is identified and segmented by using a target recognition and segmentation algorithm, and the depth information of the target tie bar is removed, and the background depth information of the target steel bar skeleton is removed by using a depth filtering algorithm, so as to retain the outer layer steel bar point cloud data after removing the target tie bar, and obtain ROI point cloud data; Downsampling the ROI point cloud data to obtain downsampled point cloud data, and determining cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the downsampled point cloud data to obtain a first surface equation and a second surface equation; wherein the first straight steel bar and the second straight steel bar are straight steel bars that carry the target tie bars; The tie bar template library is used to perform template matching on the tie bars in the downsampled point cloud data, and the intersection point of the target tie bar and the first straight steel bar, and / or the intersection point of the target tie bar and the second straight steel bar are determined according to the first surface equation and the second surface equation to obtain the target welding point.

2. A visually guided welding method for a steel frame according to claim 1, characterized in that: The process of identifying the target tie bar in the global image of the target steel bar skeleton, determining the position coordinates of the target tie bar in the world coordinate system, and obtaining the target position coordinates includes: Inputting the global image of the target steel bar skeleton into a target recognition network; The target tie bar is identified, and the position coordinates of the target tie bar in the world coordinate system are determined by hand-eye calibration to obtain the target position coordinates.

3. A visually guided welding method for a steel frame according to claim 1, characterized in that: The process of downsampling the ROI point cloud data to obtain downsampled point cloud data, and determining cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the downsampled point cloud data to obtain the first surface equation and the second surface equation includes: Projecting the ROI point cloud data onto the X-axis of the camera coordinate system, and establishing a point cloud density spectrum with the Y-axis of the camera coordinate system as an independent variable, to obtain a first point cloud density spectrum; Selecting point cloud data that meets a first value range in the first point cloud density spectrum to obtain first point cloud data; Projecting the first point cloud data onto the X-axis of the camera coordinate system, and establishing a point cloud density spectrum with the Z-axis of the camera coordinate system as an independent variable, to obtain a second point cloud density spectrum; Selecting point cloud data that conforms to a second value range in the second point cloud density spectrum to obtain second point cloud data; Projecting the ROI point cloud data onto the Y axis of the camera coordinate system, and establishing a point cloud density spectrum with the X axis of the camera coordinate system as an independent variable, to obtain a third point cloud density spectrum; Selecting point cloud data that conforms to a third value range in the third point cloud density spectrum to obtain third point cloud data; Projecting the third point cloud data on the Y axis of the camera coordinate system, and establishing a point cloud density spectrum with the Z axis of the camera coordinate system as an independent variable, to obtain a fourth point cloud density spectrum; Selecting point cloud data that conforms to a fourth value range in the fourth point cloud density spectrum to obtain fourth point cloud data; The cylindrical surface equations corresponding to the first linear steel bar and the second linear steel bar are respectively fitted according to the second point cloud data and the fourth point cloud data to obtain the first surface equation and the second surface equation.

4. A visually guided welding method for a steel frame according to claim 1, characterized in that: The process of performing template matching on the tie bars in the downsampled point cloud data by using the tie bar template library, and determining the intersection of the target tie bar and the first straight steel bar, and / or the intersection of the target tie bar and the second straight steel bar according to the first surface equation and the second surface equation, to obtain the target welding point includes: Reading point cloud data in the tie bar template library to obtain template point cloud data; Processing the template point cloud data and the downsampled point cloud data respectively to obtain FPFH features, thereby obtaining first processed point cloud data and second processed point cloud data; Using a Ransac algorithm to roughly match the first processed point cloud data and the second processed point cloud data, and using an ICP algorithm to precisely match the first processed point cloud data and the second processed point cloud data to obtain matching point cloud data; The intersection point of the target tie bar and the first straight steel bar, and / or the intersection point of the target tie bar and the second straight steel bar are determined according to the matching point cloud data, the first surface equation and the second surface equation to obtain the target welding point.

5. A visually guided welding method for a steel bar skeleton according to claim 4, characterized in that: The process of controlling the robot arm to weld the target welding point includes: Obtaining the position coordinates of the target welding point in the world coordinate system to obtain the six-degree-of-freedom coordinates of the welding point position; Based on the six-degree-of-freedom coordinates of the welding point position, and using a robot arm inverse solution algorithm, each joint angle of the robot arm is calculated to obtain a joint angle set; The robot arm is controlled to weld the target welding point according to the joint angle set.

6. A visually guided welding device for a steel bar skeleton, characterized in that: include: A tie bar identification module is used to identify the target tie bar in the global image of the target steel bar skeleton, and determine the position coordinates of the target tie bar in the world coordinate system to obtain the target position coordinates; wherein the tie bars in the target steel bar skeleton all have bending degrees; the global image includes RGB information and depth information of the target steel bar skeleton; the target tie bar is any tie bar in the target steel bar skeleton; A robot arm guiding module is used to guide the movement of a three-dimensional camera carried on the robot arm using the target position coordinates, and to determine point cloud data of the target tie bar location area in a world coordinate system using the image taken by the three-dimensional camera of the target tie bar, thereby obtaining target point cloud data; A welding point welding module is used to process the target point cloud data to determine the intersection between the target tie bar and the straight steel bar carrying the target tie bar, obtain the target welding point, and control the robot arm to weld the target welding point; The robot arm guiding module comprises: A robot arm guiding submodule, used to guide the three-dimensional camera carried by the end of the robot arm to move using the target position coordinates, and to determine the RGB image and depth map of the target tie bar location area using the image taken by the three-dimensional camera, and to convert them into original point cloud data; A coordinate conversion submodule is used to perform coordinate conversion on the original point cloud data to obtain point cloud data of the target tie bar location area in the world coordinate system, thereby obtaining the target point cloud data; The welding point welding module comprises: The ROI selection submodule is used to identify and segment the target tie bars in the RGB image and depth map of the target tie bar location area using a target recognition and segmentation algorithm, and remove the depth information of the target tie bars, and remove the background depth information of the target steel bar skeleton using a depth filtering algorithm, thereby retaining the outer layer steel bar point cloud data after removing the target tie bars, and obtaining ROI point cloud data; A downsampling submodule is used to downsample the ROI point cloud data to obtain downsampled point cloud data, and determine the cylindrical surface equations corresponding to the first straight steel bar and the second straight steel bar according to the downsampled point cloud data to obtain the first surface equation and the second surface equation; wherein the first straight steel bar and the second straight steel bar are straight steel bars that carry the target tie bars; The template matching submodule is used to perform template matching on the tie bars in the downsampled point cloud data using the tie bar template library, and determine the intersection of the target tie bar and the first straight steel bar, and / or the intersection of the target tie bar and the second straight steel bar according to the first surface equation and the second surface equation, to obtain the target welding point.

7. A robotic arm, characterized in that: include: Memory for storing computer programs; A processor is used to implement the steps of a visually guided welding method for a steel skeleton as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the visually guided welding method for a steel bar skeleton as described in any one of claims 1 to 5 are implemented.

Images