A welding robot pose calculation method, system, device and medium

Abstract

The application discloses a welding robot pose calculation method, system, device and medium, the method comprises adopting binocular camera to grab the image of the workpiece to be welded and positioning, the first local image and the second local image grabbed are respectively subjected to welding surface segmentation, and the first local welding surface and the second local welding surface are obtained; the first local welding surface and the second local welding surface are respectively subjected to feature point detection and feature point matching; the sparse three-dimensional point cloud of the local welding surface is subjected to plane fitting, and the fitted welding plane and the normal vector of the fitted welding plane are obtained; the spatial position of the weld starting point in the camera coordinate system is converted to the robot coordinate system for subsequent attitude control. The application positions the weld starting point of the workpiece to be welded by combining binocular stereo vision, obtains the normal vector of the local welding surface according to the positioning, obtains the robot attitude, and effectively realizes the accurate positioning of the weld starting point of the workpiece to be welded.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot control, and in particular relates to a welding robot pose calculation method, system, device and medium. BACKGROUND

[0002] With the rapid development of manufacturing industry, the requirements for the autonomy and adaptability of welding robots are also increasing. To improve the autonomy and adaptability of welding robots, the first problem to be solved is the accurate identification and precise positioning of the weld. The weld starting point as the starting part of the weld, the positioning accuracy of the robot will directly affect the quality of subsequent welding.

[0003] Most of the traditional methods for positioning the weld starting point are offline programming and teaching reproduction, which are achieved by fixing the position of a single type of welding workpiece, and it is difficult to make adjustments according to real-time information, and lack of robustness. With the development of machine learning and machine vision, visual sensors have been gradually widely used in robot automated welding, and how to use visual sensors and design recognition and positioning visual algorithms has become the key and inevitable trend to realize autonomous intelligent welding of robots.

[0004] Neural networks have strong representation capabilities in extracting image features and semantic information. Based on traditional image processing to realize the positioning of the weld starting point has been gradually replaced by neural networks. Compared with traditional methods, the method using neural networks has higher accuracy, robustness and universality in the task of positioning the weld starting point. SUMMARY

[0005] The present application aims to at least solve one of the technical problems existing in the prior art, and provides a welding robot pose calculation method, system, device and medium, which can improve the welding accuracy, robustness and universality of the welding robot.

[0006] According to the first aspect of the present application, a welding robot pose calculation method is provided, comprising the following steps:

[0007] The binocular camera captures the image of the workpiece to be welded and performs positioning to obtain the first image position, the second image position of the weld starting point of the workpiece to be welded and the spatial position of the weld starting point in the camera coordinate system;

[0008] The first local image is obtained with the first image position as the center, the second local image is obtained with the second image position as the center, and the first local weld surface and the second local weld surface are obtained by performing weld surface segmentation on the first local image and the second local image, respectively.

[0009] perform feature point detection and feature point matching on the first local welding surface and the second local welding surface respectively to obtain a sparse three-dimensional point cloud of the local welding surface;

[0010] perform plane fitting on the sparse three-dimensional point cloud of the local welding surface to obtain a fitted welding plane and a normal vector of the fitted welding plane, and obtain the robot pose according to the normal vector of the fitted welding plane;

[0011] convert the spatial position of the weld start point in the camera coordinate system to the robot coordinate system, and control the pose of the robot according to the obtained spatial coordinates of the weld start point in the robot coordinate system, so that the robot aims the welding torch at the weld start point of the workpiece to be welded.

[0012] According to the first aspect of the present application, further, the acquisition of the first image position, the second image position of the weld start point and the spatial position of the weld start point in the camera coordinate system specifically includes the following steps:

[0013] acquire the first image and the second image containing the weld start point of the workpiece to be welded by using a binocular camera;

[0014] perform target recognition according to the first image to obtain a first detection result box;

[0015] perform target recognition according to the second image to obtain a second detection result box;

[0016] take the center point of the first detection result box and the center point of the second detection result box as the first image position and the second image position of the weld start point;

[0017] acquire the three-dimensional coordinates of the weld start point in the camera coordinate system according to the center point of the first detection result box and the center point of the second detection result box as the spatial position of the weld start point in the camera coordinate system.

[0018] According to the first aspect of the present application, further, the acquisition of the three-dimensional coordinates of the weld start point in the camera coordinate system specifically includes the following steps:

[0019] acquire the parallax of the center point of the first detection result box and the center point of the second detection result box;

[0020] acquire the three-dimensional coordinates of the weld start point in the camera coordinate system according to the parallax.

[0021] According to the first aspect of the present application, further, the acquisition of the first local welding surface and the second local welding surface specifically includes the following steps:

[0022] cutting a certain size image from the first image grabbed by the binocular camera centering on the first image position, to obtain the first local image;

[0023] cutting a certain size image from the second image grabbed by the binocular camera centering on the second image position, to obtain the second local image;

[0024] segmenting the welding surface in the first local image to obtain the first local welding surface;

[0025] segmenting the welding surface in the second local image to obtain the second local welding surface.

[0026] According to the first aspect of the present application, further, the acquisition of the sparse three-dimensional point cloud of the local welding surface specifically comprises the following steps:

[0027] detecting feature points of the first local welding surface to obtain a feature point group of the first local welding surface;

[0028] detecting feature points of the second local welding surface to obtain a feature point group of the second local welding surface;

[0029] matching the feature point group of the first local welding surface and the feature point group of the second local welding surface to obtain a feature point pair set of the local welding surface;

[0030] acquiring the sparse three-dimensional point cloud of the local welding surface in the camera coordinate system according to the feature point pair set;

[0031] converting the coordinates of the sparse three-dimensional point cloud from the camera coordinate system to the robot coordinate system, as the sparse three-dimensional point cloud of the local welding surface.

[0032] According to the first aspect of the present application, further, plane fitting is performed on the sparse three-dimensional point cloud of the local welding surface to obtain a fitted welding plane and a normal vector of the fitted welding plane, in the case of one or two welding surfaces:

[0033] if there is only one welding surface, performing plane fitting on the sparse three-dimensional point cloud of the first plane of the local welding surface to obtain a first fitted welding plane; if there are two welding surfaces, performing plane fitting on the sparse three-dimensional point clouds of the first plane and the second plane of the local welding surface to obtain a first fitted welding plane and a second fitted welding plane;

[0034] If there is only one welding surface, a normal vector passing through the weld seam starting point is obtained according to the spatial coordinates of the first fitting welding plane and the weld seam starting point in the robot coordinate system as a first normal vector; if there are two welding surfaces, normal vectors passing through the weld seam starting point are obtained according to the spatial coordinates of the first fitting welding plane, the second fitting welding plane and the weld seam starting point in the robot coordinate system as a first normal vector and a second normal vector;

[0035] If there is only one normal vector, the Euler angle of the robot is obtained according to the normal vector as the robot pose; if there are two normal vectors, a perpendicular vector of the normal vector is obtained as a weld seam initial vector, and the Euler angle of the robot is obtained according to the normal vector and the weld seam initial vector as the robot pose.

[0036] According to the first aspect of the present application, further, the spatial position of the weld seam starting point in the camera coordinate system is converted to the robot coordinate system, and the robot is controlled in pose according to the obtained spatial coordinates of the weld seam starting point in the robot coordinate system, and the robot aims the welding torch at the weld seam starting point of the workpiece to be welded, which specifically includes the following steps:

[0037] According to the spatial position coordinate value of the weld seam starting point in the camera coordinate system, the camera coordinate system and the robot coordinate system, the spatial coordinate value of the weld seam starting point in the robot coordinate system is obtained as the spatial coordinates of the weld seam starting point in the robot coordinate system;

[0038] According to the spatial coordinates of the weld seam starting point in the robot coordinate system and the robot pose, the 6D pose of the robot is obtained as the pose of the robot aiming the welding torch at the weld seam starting point of the workpiece to be welded.

[0039] According to the second aspect of the present application, a welding robot pose calculation system is provided, which is based on the above welding robot pose calculation method and includes:

[0040] A weld seam starting point positioning module is configured to position the weld seam starting point of the workpiece to be welded by using a binocular camera, and obtain the first image position, the second image position and the spatial position of the weld seam starting point in the camera coordinate system.

[0041] a robot pose estimation module configured to: obtain a first local image and a second local image by taking images of a certain size centered on the first image position and the second image position; perform weld face segmentation on the first local image and the second local image to obtain a first local weld face and a second local weld face; perform feature point detection and feature point matching on the first local weld face and the second local weld face to obtain a sparse three-dimensional point cloud of the local weld face; perform plane fitting on the sparse three-dimensional point cloud of the local weld face to obtain a fitted weld plane and a normal vector of the fitted weld plane; and obtain the robot pose according to the normal vector of the fitted weld plane;

[0042] a coordinate conversion module configured to convert a space coordinate in a camera coordinate system into a space coordinate in a robot base coordinate system;

[0043] a pose estimation module configured to: obtain a pose of the robot aiming the welding torch at the weld seam starting point according to the obtained space coordinate and the robot pose, and control the robot to move to the weld seam starting point according to the pose.

[0044] According to a third aspect of the present application, a welding robot pose calculation device is provided, comprising:

[0045] at least one processor;

[0046] at least one memory configured to store at least one program;

[0047] When the at least one program is executed by the at least one processor, the at least one processor can implement the welding robot pose calculation method described above.

[0048] According to a fourth aspect of the present application, a computer readable storage medium is provided, wherein the computer readable storage medium stores a program executable by a processor, and the program executable by the processor is configured to execute the welding robot pose calculation method described above when executed by the processor.

[0049] The beneficial effects of the embodiments of the present application at least include: the present application positions the weld seam starting point of the workpiece to be welded by combining binocular stereo vision, obtains the normal vector of the local weld face according to the positioning, and obtains the robot pose, thereby effectively realizing accurate positioning of the weld seam starting point of the workpiece to be welded. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly described below. Obviously, the described drawings are only some of the embodiments of the present application, not all embodiments, and those skilled in the art can obtain other design solutions and drawings according to these drawings without creative labor.

[0051] Figure 1 is a structural block diagram of a weld starting point positioning platform of the present application;

[0052] Figure 2 is a step flow chart of a welding robot pose calculation method of an embodiment of the first aspect of the present application;

[0053] Figure 3 is a working schematic diagram of a binocular camera in the present application;

[0054] Figure 4 is a segmentation schematic diagram of a welding surface in the present application;

[0055] Figure 5 is a segmentation schematic diagram of two welding surfaces in the present application;

[0056] Figure 6 is a local welding surface feature point detection schematic diagram in the present application;

[0057] Figure 7 is a schematic diagram of estimating a robot pose based on a fitted welding plane normal vector in the present application. DETAILED DESCRIPTION

[0058] This part will describe the specific embodiments of the present application in detail, and the preferred embodiments of the present application are shown in the accompanying drawings, which serve to supplement the description in the text part and enable people to intuitively and visually understand each technical feature and the overall technical scheme of the present application, but it cannot be understood as a limitation on the protection scope of the present application.

[0059] In the description of the present application, it should be understood that, in relation to the description of the orientation, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application.

[0060] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, etc. is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the sequence of technical features indicated.

[0061] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be understood in a broad sense, and the person skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical scheme.

[0062] Most of the traditional methods of positioning the starting point of the weld are offline programming, teaching and reappearing, which are achieved by fixing the position of a single type of welding workpiece, and it is difficult to make adjustments according to real-time information, and it lacks robustness. With the development of machine learning and machine vision, visual sensors are gradually widely used in robot automatic welding, and how to use visual sensors and design recognition and positioning visual algorithms has become the key and inevitable trend to realize the autonomous intelligent welding of robots.

[0063] Neural networks have shown strong representation capabilities in extracting image features and semantic information. Based on traditional image processing to realize the positioning of the starting point of the weld has been gradually replaced by neural networks. Compared with traditional methods, the method using neural networks has higher accuracy, robustness and universality in the task of positioning the starting point of the weld.

[0064] To this end, the application provides a welding robot pose calculation method, system, device and medium, which positions the starting point of the weld of the workpiece to be welded by combining binocular stereo vision, and obtains the robot pose according to the normal vector of the local welding surface obtained by positioning, thereby effectively realizing accurate positioning of the starting point of the weld of the workpiece to be welded.

[0065] As shown in Figure 1 The application relates to a welding robot pose calculation platform, which comprises a control system 101, a collection system 102, a processing system 103 and a robot system 104.

[0066] The control system 101 is responsible for controlling the collection system 102, the processing system 103 and the robot system 104, and communicating and exchanging data with the collection system 102, the processing system 103 and the robot system 104.

[0067] The collection system 102 collects left and right camera raw image data after receiving the collection instruction of the control system 101, and sends the raw image to the control system 101. The collection system 102 is composed of two cameras of the same type.

[0068] The processing system 103 comprises an image data preprocessing module, a weld starting point positioning module, a robot pose estimation module and a final positioning pose calculation module. The processing system 103 calls the corresponding modules according to the command sent by the control system 101.

[0069] The robot system 104 controls the robot to accurately move to the starting point of the weld after receiving the positioning instruction and positioning pose data of the control system 101. The robot system 104 comprises a six-axis mechanical arm, a controller of the six-axis mechanical arm and a welding gun.

[0070] The principle that the processing system 103 calls the corresponding module according to the command sent by the control system 101 is that, after the processing system 103 receives the positioning instruction sent by the control system 101, the image data preprocessing module, the positioning module and the robot pose estimation module are executed; after the processing system 103 receives the positioning pose calculation instruction sent by the control system 101, the final positioning pose calculation module is executed.

[0071] The principle that the control system 101 and the processing system 103 communicate and exchange data is that, the control system 101 sends the original image data to the processing system 103 at the same time of sending the positioning instruction to the processing system 103, and the processing system 103 sends the position data of the weld starting point to the control system 101 after processing; the control system 101 sends the local welding image to the processing system 103 at the same time of sending the robot pose estimation instruction to the processing system 103, and the processing system 103 sends the pose of the robot at the weld starting point to the control system after processing; the control system 101 sends the spatial position of the weld starting point and the pose of the robot to the processing system 103 at the same time of sending the positioning pose calculation instruction to the processing system 103, and the processing system 103 sends the pose information required by the robot to position the weld starting point to the control system 101 after processing.

[0072] As an optional implementation, in the processing system 103, the image data preprocessing module performs stereo correction; when the robot pose estimation module is used, the image is also cropped according to the target detection frame, and the local image of the weld starting point is cropped separately; wherein the stereo correction is realized by using opencv, and the required parameters are obtained by double target positioning.

[0073] The weld starting point positioning module includes weld type detection, weld starting point positioning, left and right image target positioning result matching, and weld starting point spatial position calculation. Among them, the weld type detection and the weld starting point positioning adopt a deep learning target detection network, including but not limited to the FCOS target detection network.

[0074] The left and right image target positioning result matching method is that, first, the target detection result frames of the left and right images are sorted in order from left to right according to the pixel coordinates of the center points of the result frames; then, taking the target detection result frame of the left image as a reference, a matching frame is searched from left to right in the target detection result frame of the right image, and the matching principle is that the x and y coordinate differences of the center points of the two frames to be matched do not exceed 210 pixel values and 20 pixel values respectively, and the weld detection types of the two frames are the same; all matching frames of the left image target detection result frame are searched and saved according to this method.

[0075] The method for calculating the spatial position of the weld starting point is as follows: first, the disparity of the center points of the two matching boxes is calculated, a homogeneous matrix M is formed by the 2-pixel coordinate value of the center point of the left matching box, the disparity and a numerical value 1, the homogeneous matrix M is multiplied by the Q matrix to obtain a homogeneous matrix Y, the 4-dimensional homogeneous matrix is changed into a 3-dimensional matrix A, and the 3-dimensional matrix A is the coordinate of the center point of the matching box in the camera coordinate system, and the coordinate of the center point in the camera coordinate system is the spatial position of the weld starting point in the camera coordinate system.

[0076] The robot pose estimation calculation module includes welding surface segmentation, feature point detection and matching, welding plane fitting and normal vector calculation, and robot pose calculation. The welding surface segmentation adopts a deep learning instance segmentation network, including but not limited to a YOLACT segmentation network. The feature point detection includes but is not limited to ORB feature point detection, and the feature point matching includes but is not limited to brute force matching. The welding plane fitting and normal vector calculation includes but is not limited to the least squares method.

[0077] The method for calculating the pose of the welding robot is as follows: first, the pose vector of the welding torch of the robot is obtained according to the fitted plane normal vector. If there is only one normal vector, the normal vector is directly taken as the pose vector of the robot. If there are two normal vectors, the normal vector is first obtained according to the normal vector to obtain a perpendicular vector of the normal vector as an initial weld vector, and then the synthetic vector of the normal vector and the initial weld vector is taken as the pose vector. Then the pose vector is converted into a rotation matrix by the formula R = cos a I + (1-cos a)nn Τ + n' sin a, and the Euler angle of the robot, i.e. the robot pose, is obtained. In the formula, I is a unit vector, a is the included angle between the welding torch pose vector and the unit vector of the Z-axis of the robot base coordinate system, n is the cross product of the welding torch pose vector and the unit vector of the Z-axis of the robot base coordinate system, and the result is normalized to obtain a unit vector, and n' is the skew-symmetric matrix of n.

[0078] The final positioning pose calculation module includes spatial coordinate system transformation and pose generation.

[0079] The method for calculating the spatial coordinate system transformation is as follows: the spatial coordinate value in the camera coordinate system is (x C , y C , z C ), and the spatial coordinate value (x B , y B , z B ) in the robot coordinate system can be calculated by the formula . is the conversion matrix of the robot base coordinate system and the welding torch coordinate system, is the conversion matrix of the camera coordinate system and the welding torch coordinate system. Wherein, the robot end flange coordinate system is established at the center of the robot end flange, and the tool coordinate system is established at the tip of the welding torch. The transformation relationship between the end flange coordinate system and the robot base coordinate system can be read from the robot controller, that is, The transformation relationship between the tool coordinate system and the end flange coordinate system is obtained by tool calibration, that is, Therefore, The camera coordinate system and the end flange coordinate system are obtained by hand-eye calibration, that is, Therefore,

[0080] The pose generation calculation method is: combining the spatial coordinate value (x B , y B , z B ) of the welding torch aiming at the starting point of the weld seam of the workpiece in the robot base coordinate system and the pose (A, B, C) of the robot, to obtain a 6D pose (X, Y, Z, A, B, C).

[0081] As an optional implementation, the control system 101 uses the Windows system as the operating system of the host computer, and realizes module interaction in the client / server mode.

[0082] The above welding robot pose calculation method comprises the following steps:

[0083] S201, after all preparations are completed, the control system sends a collection instruction to the collection system.

[0084] S202, after receiving the collection instruction, the collection system collects the left and right camera raw image data, and sends the raw image data to the control system.

[0085] S203, after receiving the raw image data, the control system sends the raw image data to the processing system and sends a weld seam starting point positioning instruction to the processing system.

[0086] S204, after receiving the weld seam starting point positioning instruction and the raw image data, the processing system first calls the image data preprocessing module to perform stereo correction on the image, then calls the weld seam starting point positioning module to obtain the weld seam starting point target detection frame and the spatial position of the weld seam starting point; then calls the robot pose estimation module to obtain the pose of the robot at the weld seam starting point; finally, calls the final positioning pose calculation module to obtain the positioning pose of the robot to the weld seam starting point, and then sends the positioning pose to the control module.

[0087] ​​​​S205, the control module sends the acquired positioning pose and positioning instruction to the robot system; the robot system controls the robot to complete the positioning task.

[0088] The welding robot pose calculation method provided in the embodiment realizes identification, positioning of the weld starting point and robot pose estimation; the precise positioning of the weld starting point is realized by using a target detection method in deep learning, and the method of deep learning is also used in robot pose estimation, which well solves the problem that the weld starting point pose cannot be detected in the case where the workpiece pose changes greatly; the method can be applied to the positioning task of the weld starting point of the lap joint, butt joint and corner joint welding types.

[0089] Based on the welding seam starting point positioning platform, Figure 1 the embodiment also provides a welding robot pose calculation method, which needs to be noted that the welding robot pose calculation method can be realized by the welding seam starting point positioning platform, but is not limited to the welding seam starting point positioning platform, and can also be realized by other systems. The method specifically includes the following steps:

[0090] S1, a binocular camera is used to capture the image of the workpiece to be welded and to position, so as to obtain the first image position, the second image position and the spatial position of the weld starting point of the workpiece to be welded in the camera coordinate system.

[0091] An image containing the weld starting point of the workpiece to be welded is acquired, and an object recognition network is used to recognize and process the image to obtain a detection frame of the weld starting point; wherein the object recognition network can be trained by using an existing network model. Since the binocular camera has the function of measuring distance (depth), the weld starting point in the image can be positioned based on this function to obtain the spatial position information of the weld starting point.

[0092] Step S1 can further include:

[0093] S11, a binocular camera is used to acquire a first image and a second image containing the weld starting point of the workpiece to be welded;

[0094] S12, target recognition is performed according to the first image to obtain a first detection result frame;

[0095] S13, target recognition is performed according to the second image to obtain a second detection result frame;

[0096] S14, the center points of the first detection result frame and the second detection result frame are taken as the first image position and the second image position of the weld starting point. Binocular triangulation is performed on the center points of the first detection result frame and the second detection result frame to obtain the three-dimensional coordinates of the weld starting point in the camera coordinate system as the spatial position of the weld starting point in the camera coordinate system.

[0097] The binocular camera includes a first camera and a second camera, the first camera collects a first image, and the second camera collects a second image. In some embodiments, referring to Figure 3 , the image contains several weld start points, so the weld start points identified in the first image and the second image need to be matched; in this embodiment, the workpiece to be welded includes three types of welding workpieces, i.e., butt joint, lap joint and corner joint, and the weld start point refers to the position or endpoint where the weld starts.

[0098] Specifically, referring to Figure 3 , the matching method of the target detection result boxes of the left image (the first image) and the right image (the second image) is as follows: first, the target detection result boxes of the left and right images are sorted in order from left to right according to the pixel coordinates of the center points of the result boxes; then, taking the left image target detection result box as a reference, a matching box is found in the right image target detection result box from left to right, and the matching principle is that the x and y coordinate differences of the center points of the two boxes to be matched do not exceed 210 pixel values and 20 pixel values respectively, and the weld detection types of the two boxes are the same; all matching boxes of the left image target detection result boxes are found according to this method, and the result boxes of the successful matching are saved.

[0099] The space position calculation method of the weld start point is as follows: first, the disparity of the center points of the two matching boxes is calculated, a homogeneous matrix M is composed of the pixel coordinate value of the left matching box center point, the disparity and the value 1, the homogeneous matrix M is left multiplied by the Q matrix to obtain a homogeneous matrix Y, the 4-dimensional homogeneous matrix is changed into a 3-dimensional matrix A, and the 3-dimensional matrix A is the coordinate of the matching box center point in the camera coordinate system, which is the space position of the weld start point in the camera coordinate system.

[0100] S2, a certain size of image is intercepted respectively with the first image position and the second image position as the center to obtain a first local image and a second local image, the first local image and the second local image are subjected to weld surface segmentation to obtain a first local weld surface and a second local weld surface.

[0101] The detection box position of the weld start point is obtained through the identification and positioning of step S1. In this embodiment, the detection box is a square box. According to the center position of the detection box, a certain size of image of the original image is intercepted. The size of the intercepted image is determined according to the size of the workpiece to be welded.

[0102] Each pixel belonging to the weld surface is obtained by identifying and segmenting the intercepted image through the object segmentation network. The object segmentation network can be trained using an existing network model.

[0103] Specifically, step S2 includes S21-S24:

[0104] S21, cropping a certain size image from the first image with the first image position as the center to obtain a first local image;

[0105] S22, cropping a certain size image from the second image with the second image position as the center to obtain a second local image;

[0106] S23, segmenting the welding surface in the first local image according to the object segmentation network to obtain a first local welding surface;

[0107] S24, segmenting the welding surface in the second local image according to the object segmentation network to obtain a second local welding surface;

[0108] Specifically, referring to Figure 4 , the butt joint type welding workpiece, since the two welding surfaces are in the same plane, only one welding surface is obtained after the local image is segmented by the object segmentation network; referring to Figure 5 , the lap joint and corner joint type welding workpiece, since the two welding surfaces intersect to form a certain angle, two welding surfaces are obtained after the local image is segmented by the object segmentation network.

[0109] S3, feature point detection and feature point matching are performed on the first local welding surface and the second local welding surface to obtain a sparse three-dimensional point cloud of the local welding surface.

[0110] After the segmentation of S2, referring to Figure 4 and Figure 5 , a mask image of the local image welding surface is obtained. Feature point detection is performed on the original local image, and whether the feature point belongs to the welding surface is determined by the RGB value of the pixel at the corresponding coordinate of the mask image.

[0111] The step S3 specifically includes S31-S34:

[0112] S31, feature point detection is performed on the first local welding surface to obtain a feature point group of the first local welding surface;

[0113] S32, feature point detection is performed on the second local welding surface to obtain a feature point group of the second local welding surface;

[0114] S33, matching is performed on the feature point group of the first local welding surface and the feature point group of the second local welding surface to obtain a feature point pair group of the local welding surface;

[0115] S34, according to the feature point pair group, a sparse three-dimensional point cloud of the local welding surface in the camera coordinate system is obtained, and the coordinates of the sparse three-dimensional point cloud are converted from the camera coordinate system to the robot coordinate system.

[0116] Specifically, referring to Figure 6The feature point matching manner of the left image and the right image is as follows: for each feature point in a certain welding surface of the left image, the similarity of the feature point with all feature points in the corresponding welding surface of the right image is calculated, wherein the most similar feature point is the matching point of the feature point. In this embodiment, the distance between the feature point descriptors is used as the similarity measure.

[0117] Since the feature point matching may have false matching or multiple matching, matching screening is needed to improve the accuracy of matching. In this embodiment, screening is performed based on a threshold: a threshold is set, and only the feature point of the right image closest to the feature point of the left image is accepted as the matching, and the matching with a distance exceeding the threshold is excluded.

[0118] After the feature point matching, a feature point pair set is obtained. Binocular triangulation is performed on each point pair in the set to obtain the three-dimensional coordinates of each point pair in the camera coordinate system, that is, a sparse three-dimensional point cloud of the local welding surface is obtained. The calculation manner can refer to the calculation manner of the spatial position of the weld start point in step S1, which is not described herein.

[0119] According to the spatial position coordinate values of the sparse three-dimensional point cloud in the camera coordinate system, the camera coordinate system and the robot coordinate system, the spatial coordinate values of the sparse three-dimensional point cloud in the robot coordinate system are obtained.

[0120] S4, plane fitting is performed on the sparse three-dimensional point cloud of the local welding surface to obtain a fitted welding plane and a normal vector of the fitted welding plane, and a robot pose is obtained according to the normal vector of the fitted welding plane.

[0121] Specifically, step S4 includes S41-S5.

[0122] S41, plane fitting is performed on the sparse three-dimensional point cloud of the first plane of the local welding surface to obtain a first fitted welding plane;

[0123] S42, if there are two welding surfaces, plane fitting is performed on the sparse three-dimensional point cloud of the second plane of the local welding surface to obtain a second fitted welding plane;

[0124] S43, a normal vector passing through the weld start point is obtained as a first normal vector according to the first fitted welding plane and the spatial coordinates of the weld start point in the robot coordinate system.

[0125] S44, if there are two welding surfaces, a normal vector passing through the weld start point is obtained as a second normal vector according to the second fitted welding plane and the spatial coordinates of the weld start point in the robot coordinate system.

[0126] S45. If there is only one normal vector: obtain the robot's Euler angles based on the normal vector, which will be used as the robot's pose; if there are two normal vectors: obtain the perpendicular vector of the normal vector as the initial vector of the weld; obtain the robot's Euler angles based on the normal vector and the initial vector of the weld, which will be used as the robot's pose.

[0127] Specifically, see Figure 7 (a) The unit normal vector v1 of the fitted welding plane is obtained from the weld start point, which is the attitude vector of the robot welding torch at the weld start point; see [reference]. Figure 7 (b) The unit normal vectors v1 and v2 of the fitted welding plane are obtained from the weld start point. Then, the unit weld start vector v3 is obtained based on v1 and v2. The final welding torch attitude vector is obtained by vector synthesis of v1, v2, and v3, determined by the formula v = αv1 + βv2 + γv3, where α, β, and γ can be determined according to welding process requirements. In this embodiment, α, γ, and β are all equal to 1. The attitude vector is obtained by the formula R = cosαI + (1 - cosα)nn Τ +n′sinα is converted into a rotation matrix, which yields the Euler angles of the robot, i.e., the robot's pose. In the formula, I is a unit vector, α is the angle between the welding torch pose vector and the unit vector of the robot's base coordinate system Z-axis, n is the cross product of the welding torch pose vector and the unit vector of the robot's base coordinate system Z-axis, and the result is normalized to obtain a unit vector, and n' is the antisymmetric matrix of n.

[0128] S5. Transform the spatial position of the weld start point in the camera coordinate system to the robot coordinate system. Based on the spatial coordinates of the weld start point in the robot coordinate system and the robot's posture, control the robot to align the welding torch with the weld start point of the workpiece to be welded.

[0129] Specifically, step S5 includes S51 to S52:

[0130] S51. Based on the spatial coordinates of the weld start point in the camera coordinate system, the camera coordinate system, and the robot coordinate system, obtain the spatial coordinates of the weld start point in the robot coordinate system.

[0131] S52. Obtain the robot's 6D pose based on the spatial coordinates of the weld start point in the robot coordinate system and the robot's posture, and use it as the robot's pose for aligning the welding torch with the weld start point of the workpiece to be welded.

[0132] Specifically, the spatial coordinates of the weld start point in the camera coordinate system are (x... C y C , z C ), can be derived from the formula The spatial coordinates (x, y) of the seam initiation point in the robot coordinate system were calculated. B yB , z B ). is a transformation matrix from the robot base coordinate system to the welding torch coordinate system, is a transformation matrix from the camera coordinate system to the welding torch coordinate system.

[0133] The robot end flange coordinate system is established at the center of the robot end flange, and the tool coordinate system is established at the tip of the welding torch. The transformation relationship from the end flange coordinate system to the robot base coordinate system can be read from the robot controller, that is, The transformation relationship between the tool coordinate system and the end flange coordinate system is obtained by tool calibration, that is, and , the welding torch coordinate system can be obtained The camera coordinate system and the end flange coordinate system are obtained by hand-eye calibration, that is, and , the welding torch coordinate system can be obtained

[0134] The robot aligns the welding torch to the position of the starting point of the weld of the workpiece to be welded, that is, the spatial coordinate values (x B , y B , z B ) and the pose (A, B, C) of the robot, and combines them to obtain a 6D pose (X, Y, Z, A, B, C).

[0135] The embodiment also provides a welding robot pose calculation system, comprising:

[0136] A weld starting point positioning module is configured to position the starting point of the weld of the workpiece to be welded by using a binocular camera, and obtain the left image position, the right image position, and the spatial position of the starting point of the weld in the camera coordinate system.

[0137] A robot pose estimation module is configured to obtain a left local image and a right local image by cutting a certain size of image centered on the left image position and the right image position; obtain a left local weld surface and a right local weld surface by performing weld surface segmentation on the left local image and the right local image; obtain a sparse three-dimensional point cloud of the local weld surface by performing feature point detection and feature point matching on the left local weld surface and the right local weld surface; obtain a fitted weld plane and a normal vector of the fitted weld plane by performing plane fitting on the sparse three-dimensional point cloud of the local weld surface; and obtain the pose of the robot according to the normal vector of the fitted weld plane.

[0138] A coordinate conversion module is configured to convert the spatial coordinates in the camera coordinate system into spatial coordinates in the robot base coordinate system.

[0139] ​​The pose estimation module is configured to obtain a pose of the robot aiming the welding torch at the starting point of the weld according to the obtained spatial coordinates and the robot pose, and control the robot to move to the starting point of the weld according to the pose.

[0140] The welding robot pose calculation system provided in the embodiment can execute the welding robot pose calculation method provided in the method embodiment of the present application, can execute the implementation steps of any combination of the method embodiments, and has the corresponding functions and beneficial effects of the method.

[0141] The welding robot pose calculation system provided in the embodiment can execute the welding robot pose calculation method provided in the method embodiment of the present application, can execute the implementation steps of any combination of the method embodiments, and has the corresponding functions and beneficial effects of the method.

[0142] at least one processor;

[0143] at least one memory for storing at least one program;

[0144] When the at least one program is executed by the at least one processor, the at least one processor implements the method. Figure 2 Method.

[0145] The welding robot pose calculation system provided in the embodiment can execute the welding robot pose calculation method provided in the method embodiment of the present application, can execute the implementation steps of any combination of the method embodiments, and has the corresponding functions and beneficial effects of the method.

[0146] The welding robot pose calculation system provided in the embodiment can execute the welding robot pose calculation method provided in the method embodiment of the present application, can execute the implementation steps of any combination of the method embodiments, and has the corresponding functions and beneficial effects of the method.

[0147] In some alternative embodiments, the functions / operations mentioned in the block diagrams can not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially concurrently or the blocks can sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flow diagrams of the present application are provided by way of example only. There is no intention, therefore, to limit the disclosed methods to the precise operations and logic flow presented in this specification. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations of a larger operation are performed in parallel.

[0148] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the functions and / or features of the'and / or other features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be understood that detailed discussion of the actual implementation of each module is unnecessary to an understanding of the present application. Rather, the actual implementation of the modules, in light of the description of the properties, functions and interrelationships of the various functional modules disclosed herein, will be apparent to a skilled artisan in engineering, in view of the present disclosure. Accordingly, the present application is not limited to the details of the illustrated implementation. Rather, the present application is limited only by the scope of the claims appended hereto and equivalents thereof.

[0149] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer- readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0150] The logic and / or steps represented in the flowcharts and / or otherwise described herein, for example, can be embodied in non-transitory computer-readable media, which can be executed by, used with, or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0151] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via an optical scanner, then compiled, interpreted, or otherwise processed, using an appropriate medium, into a computer program in a suitable language.

[0152] It should be understood that aspects of the application can be implemented in hardware, software, firmware, or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example in software or firmware, which are stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0153] In the above description of the present specification, the description referring to the terms "one embodiment", "another embodiment", or "certain embodiments" or the like means that a specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present application. Descriptive expressions of the above terms in the present specification do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner.

[0154] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are included in the scope defined by the claims of the present application.

Claims

1. A method for calculating the pose of a welding robot, characterized in that, Includes the following steps: A binocular camera is used to capture images of the workpiece to be welded and to locate it, thereby obtaining a first image and a second image containing the starting point of the weld seam of the workpiece to be welded. Target recognition is performed on the first image to obtain a first detection result box; Target recognition is performed based on the second image to obtain a second detection result box; The center point of the first detection result box and the center point of the second detection result box are used as the first image position and the second image position of the weld start point; The three-dimensional coordinates of the weld start point in the camera coordinate system are obtained based on the center point of the first detection result box and the center point of the second detection result box, and are used as the spatial position of the weld start point in the camera coordinate system. A first local image is obtained with the first image position as the center, and a second local image is obtained with the second image position as the center. The first local image and the second local image are respectively segmented into welding surfaces to obtain a first local welding surface and a second local welding surface. Feature point detection and feature point matching are performed on the first local welding surface and the second local welding surface respectively to obtain a sparse three-dimensional point cloud of the local welding surface; Plane fitting is performed on the sparse three-dimensional point cloud of the local welding surface to obtain the fitted welding plane and the normal vector of the fitted welding plane. The robot posture is obtained based on the normal vector of the fitted welding plane. The spatial position of the weld start point in the camera coordinate system is transformed to the robot coordinate system. Based on the obtained spatial coordinates of the weld start point in the robot coordinate system, the robot is subjected to attitude control, and the robot aligns the welding torch with the weld start point of the workpiece to be welded. Specifically, a plane fitting is performed on the sparse 3D point cloud of the local welding surface to obtain the fitted welding plane and the normal vector of the fitted welding plane. The number of welding surfaces can be one or two. If there is only one welding surface, perform plane fitting on the sparse three-dimensional point cloud of the first plane of the local welding surface to obtain the first fitted welding plane; if there are two welding surfaces, perform plane fitting on the sparse three-dimensional point clouds of the first plane and the second plane of the local welding surface to obtain the first fitted welding plane and the second fitted welding plane. If there is only one welding surface, the normal vector passing through the weld start point is obtained based on the first fitted welding plane and the spatial coordinates of the weld start point in the robot coordinate system, and is used as the first normal vector; if there are two welding surfaces, the normal vector passing through the weld start point is obtained based on the first fitted welding plane, the second fitted welding plane and the spatial coordinates of the weld start point in the robot coordinate system, and is used as the first normal vector and the second normal vector. If there is only one normal vector: obtain the Euler angles of the robot based on the normal vector, and use them as the robot's posture; if there are two normal vectors: obtain the perpendicular vector of the normal vector as the initial vector of the weld, and then obtain the Euler angles of the robot based on the normal vector and the initial vector of the weld, and use them as the robot's posture.

2. The welding robot pose calculation method according to claim 1, characterized in that: The acquisition of the three-dimensional coordinates of the weld start point in the camera coordinate system includes the following steps: Obtain the disparity between the center point of the first detection result box and the center point of the second detection result box; The three-dimensional coordinates of the weld start point in the camera coordinate system are obtained based on the parallax.

3. The welding robot pose calculation method according to claim 1, characterized in that: Obtaining the first partial weld surface and the second partial weld surface specifically includes the following steps: A certain size image is cropped from the first image captured by the binocular camera with the first image position as the center to obtain the first partial image; A certain size image is cropped from the second image captured by the binocular camera with the second image position as the center to obtain the second partial image; The first local welding surface is obtained by segmenting the welding surface in the first local image; The second local welding surface is obtained by segmenting the welding surface in the second local image.

4. The welding robot pose calculation method according to claim 3, characterized in that: The acquisition of the sparse three-dimensional point cloud of the local welded surface specifically includes the following steps: Feature point detection is performed on the first local weld surface to obtain a feature point group of the first local weld surface; Feature point detection is performed on the second local weld surface to obtain a feature point group of the second local weld surface; The feature point group of the first local welding surface and the feature point group of the second local welding surface are matched to obtain a set of feature point pairs of the local welding surface; Based on the aforementioned set of feature point pairs, obtain the sparse 3D point cloud of the local welding surface in the camera coordinate system; The coordinates of the sparse 3D point cloud are transformed from the camera coordinate system to the robot coordinate system, and used as the sparse 3D point cloud of the local welding surface.

5. The welding robot pose calculation method according to claim 1, characterized in that: The spatial position of the weld start point in the camera coordinate system is transformed to the robot coordinate system. Based on the obtained spatial coordinates of the weld start point in the robot coordinate system, the robot's attitude is controlled, and the robot aligns the welding torch with the weld start point of the workpiece to be welded. This specifically includes the following steps: Based on the spatial position coordinates of the weld start point in the camera coordinate system, the camera coordinate system, and the robot coordinate system, the spatial coordinates of the weld start point in the robot coordinate system are obtained and used as the spatial coordinates of the weld start point in the robot coordinate system. The 6D pose of the robot is obtained based on the spatial coordinates of the weld start point in the robot coordinate system and the robot posture, and is used as the pose for the robot to align the welding torch with the weld start point of the workpiece to be welded.

6. A welding robot pose calculation system, based on the welding robot pose calculation method according to any one of claims 1 to 5, characterized in that, include: The weld start point positioning module is used to locate the weld start point of the workpiece to be welded using a binocular camera, and to obtain the first image position, the second image position, and the spatial position of the weld start point in the camera coordinate system. The robot pose estimation module is used to crop images of a certain size centered on the positions of the first and second images to obtain a first local image and a second local image; to segment the first and second local images into welding surfaces to obtain a first local welding surface and a second local welding surface; to perform feature point detection and feature point matching on the first and second local welding surfaces to obtain a sparse 3D point cloud of the local welding surface; to perform plane fitting on the sparse 3D point cloud of the local welding surface to obtain a fitted welding plane and a normal vector of the fitted welding plane; and to obtain the robot pose based on the normal vector of the fitted welding plane. The coordinate transformation module is used to convert spatial coordinates in the camera coordinate system into spatial coordinates in the robot base coordinate system; Pose estimation module: used to obtain the pose of the robot aligning the welding torch with the weld start point based on the obtained spatial coordinates and the robot posture, and control the robot to move to the weld start point based on the pose.

7. A welding robot pose calculation device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor is able to implement the welding robot pose calculation method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to perform the welding robot pose calculation method as described in any one of claims 1 to 5.

Images