Workpiece welding control method and device, storage medium and electronic equipment

Abstract

The application discloses a workpiece welding control method and device, a storage medium and electronic equipment. The method comprises the following steps: controlling a welding robot to move to a predetermined photographing position, and controlling a two-dimensional camera installed on the welding robot to take a photograph of a target workpiece to obtain a target two-dimensional image, wherein a target mark is arranged on a predetermined position of the target workpiece; segmenting the target two-dimensional image to obtain a mark image comprising the target mark; identifying the mark image to determine a first relative pose between the two-dimensional camera and the target mark; converting the first relative pose to determine a second relative pose between the welding robot and the target workpiece; and in the case that the second relative pose matches a predetermined welding pose, controlling the welding robot to perform welding processing on the target workpiece. The application solves the technical problem of high limitation of related art positioning depending on a three-dimensional image acquisition device and setting of the robot.

Description

Technical Field

[0001] This invention relates to the field of robot positioning technology, and more specifically, to a workpiece welding control method, apparatus, storage medium, and electronic device. Background Technology

[0002] In related technologies, automated welding equipment mainly achieves positioning through mechanical positioning or 3D (three-dimensional) vision camera positioning. However, the mechanical devices used for positioning require specialized design and manufacturing, resulting in low efficiency. When the automated production line changes or is replaced, the original mechanical positioning devices also need to be disassembled and reassembled, which is time-consuming and labor-intensive. Furthermore, positioning with 3D structured light cameras or 3D laser cameras suffers from problems such as obscure algorithms, cumbersome deployment, and long debugging cycles. Relying on 3D vision cameras for workpiece positioning also presents significant limitations in the setup of welding robots.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides a workpiece welding control method, apparatus, storage medium, and electronic device to at least solve the technical problems in related technologies that rely on three-dimensional image acquisition equipment for positioning and have high limitations on robot settings.

[0005] According to one aspect of the present invention, a workpiece welding control method is provided, comprising: controlling a welding robot to move to a predetermined photographing position, and controlling a two-dimensional camera mounted on the welding robot to photograph a target workpiece to obtain a target two-dimensional image, wherein a target mark is provided at a predetermined position on the target workpiece; segmenting the target two-dimensional image to obtain a mark image including the target mark; recognizing the mark image to determine a first relative pose between the two-dimensional camera and the target mark; converting the first relative pose to determine a second relative pose between the welding robot and the target workpiece; and controlling the welding robot to perform welding processing on the target workpiece when the second relative pose matches a predetermined welding pose.

[0006] Optionally, segmenting the target two-dimensional image to obtain a labeled image including the target marker includes: converting the target two-dimensional image to grayscale to obtain a grayscale image; determining the target pixel value corresponding to the pixel in the grayscale image, and the neighboring pixel value corresponding to the neighboring pixel adjacent to the pixel; determining an adaptive threshold for the grayscale image based on the target pixel value and the neighboring pixel value; and segmenting the target two-dimensional image using the adaptive threshold to obtain the labeled image.

[0007] Optionally, the target marker includes multiple corner points, and the step of identifying the marker image to determine the first relative pose between the two-dimensional camera and the target marker includes: identifying the marker image to determine the two-dimensional corner point images corresponding to the multiple corner points respectively; determining the corner point pixel coordinates corresponding to the multiple corner points based on the two-dimensional corner point images corresponding to the multiple corner points respectively; and determining the first relative pose based on the corner point pixel coordinates corresponding to the multiple corner points respectively.

[0008] Optionally, determining the first relative pose based on the pixel coordinates of the corner points respectively includes: determining the focal length, principal point coordinates, and distortion coefficients of the two-dimensional camera, wherein the principal point coordinates are the position of the center of the image generated by the two-dimensional camera in the pixel coordinate system; determining an intrinsic parameter matrix representing the optical characteristics of the two-dimensional camera based on the focal length, the principal point coordinates, and the distortion coefficients; and calibrating the pixel coordinates of the corner points respectively based on the intrinsic parameter matrix to obtain the first relative pose.

[0009] Optionally, the welding robot includes a movable robotic arm, which is mounted on a predetermined base. The two-dimensional camera is mounted on the robotic arm. The second relative pose is the relative pose between the base and the target workpiece. The step of converting the first relative pose to determine the second relative pose between the welding robot and the target workpiece includes: acquiring a third relative pose between the two-dimensional camera and the base, and acquiring a fourth relative pose between the target workpiece and the target mark; and determining the second relative pose based on the first relative pose, the third relative pose, and the fourth relative pose.

[0010] Optionally, obtaining the third relative pose between the two-dimensional camera and the base includes: determining the rotation matrix and translation vector of the two-dimensional camera; determining the extrinsic parameter matrix representing the pose of the two-dimensional camera in the world coordinate system based on the rotation matrix and the translation vector; and calibrating the relative pose between the two-dimensional camera and the base based on the extrinsic parameter matrix to obtain the third relative pose.

[0011] Optionally, the method further includes: determining a pose deviation between the second relative pose and the predetermined welding pose when the second relative pose does not match the predetermined welding pose; adjusting the second relative pose based on the pose deviation to obtain an updated pose; and controlling the welding robot to perform welding processing on the target workpiece using the updated pose.

[0012] According to another aspect of the present invention, a workpiece welding control device is provided, comprising: a photographing module, configured to control a welding robot to move to a predetermined photographing position and control a two-dimensional camera mounted on the welding robot to photograph a target workpiece to obtain a target two-dimensional image, wherein a target mark is provided at a predetermined position on the target workpiece; a mark acquisition module, configured to segment the target two-dimensional image to obtain a mark image including the target mark; an image recognition module, configured to recognize the mark image and determine a first relative pose between the two-dimensional camera and the target mark; a conversion module, configured to convert the first relative pose to determine a second relative pose between the welding robot and the target workpiece; and a welding control module, configured to control the welding robot to perform welding processing on the target workpiece when the second relative pose matches a predetermined welding pose.

[0013] According to another aspect of the present invention, a non-volatile storage medium is provided, the non-volatile storage medium storing a plurality of instructions adapted for loading by a processor and executing any one of the workpiece welding control methods described herein.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising: one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the workpiece welding control methods described herein.

[0015] In this embodiment of the invention, a welding robot is controlled to move to a predetermined photographing position, and a two-dimensional camera mounted on the welding robot is controlled to photograph the target workpiece to obtain a target two-dimensional image. A target mark is set at the predetermined position on the target workpiece. The target two-dimensional image is segmented to obtain a mark image including the target mark. The mark image is identified to determine a first relative pose between the two-dimensional camera and the target mark. The first relative pose is transformed to determine a second relative pose between the welding robot and the target workpiece. When the second relative pose matches the predetermined welding pose, the welding robot is controlled to perform welding processing on the target workpiece. This achieves the goal of reducing the limitations of welding robot setup, realizes the technical effect of using a two-dimensional image acquisition device to achieve three-dimensional pose positioning, and thus solves the technical problem in related technologies that rely on three-dimensional image acquisition devices for positioning, resulting in high limitations on robot setup. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0017] Figure 1 This is a flowchart of an optional workpiece welding control method provided according to an embodiment of the present invention;

[0018] Figure 2 This is a schematic diagram of an optional workpiece welding control method provided according to an embodiment of the present invention;

[0019] Figure 3 This is a schematic flowchart of an optional workpiece welding control method provided according to an embodiment of the present invention;

[0020] Figure 4 This is a schematic diagram of an optional workpiece welding control device provided according to an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Currently, in applications where robots are used for workpiece welding, due to poor consistency of incoming workpieces, tooling and fixture errors, and interference during the operation, if the welding robot continues to execute according to a predetermined trajectory during operation, it cannot be guaranteed that the welding robot trajectory is consistent with the actual workpiece path. In other words, the welding robot's operation process cannot be adjusted online according to the actual working conditions, thus failing to meet the requirements of high-precision processing.

[0024] In related technologies, automated welding equipment mainly achieves positioning through mechanical positioning or 3D (three-dimensional) vision camera positioning. These two approaches have at least the following drawbacks: the mechanical devices used for positioning require specialized design and manufacturing, resulting in low efficiency; this method also has certain requirements for workpiece consistency; mechanical positioning is sensitive to the size and shape information of the workpiece, and the mechanical positioning device often fails after a model change, requiring redesign; mechanical positioning methods have poor flexibility. When the automated production line changes or is replaced, the original mechanical positioning device also needs to be disassembled and reassembled accordingly, which is time-consuming and labor-intensive. Furthermore, 3D structured light cameras or 3D laser cameras for positioning suffer from problems such as obscure algorithms, cumbersome deployment, and long debugging cycles, resulting in very high overall costs. Therefore, related technologies relying on 3D vision cameras for workpiece positioning have significant limitations in the setup of welding robots.

[0025] To address the aforementioned problems, this invention provides a method embodiment for workpiece welding control. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] Figure 1 This is a flowchart of a workpiece welding control method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0027] Step S102: Control the welding robot to move to the predetermined photo-taking position, and control the two-dimensional camera installed on the welding robot to take a photo of the target workpiece to obtain a two-dimensional image of the target, wherein a target mark is set at the predetermined position of the target workpiece.

[0028] It is understandable that the welding robot possesses mobility, allowing it to move to a predetermined photographing position according to a set route, and control the 2D camera mounted on the welding robot to photograph the target workpiece, obtaining a target 2D image including the entire workpiece. To achieve 3D pose localization of the target workpiece using 2D hardware, target markers are pre-set at predetermined positions on the target workpiece. It should be noted that the relative position between the target workpiece and the target markers is fixed.

[0029] Optionally, the target markers can be of various types, with ArUco markers being preferred. ArUco markers are image marking systems based on two-dimensional barcodes, commonly used in computer vision and augmented reality to increase the amount of information when mapping the two-dimensional world to the three-dimensional world. ArUco markers have unique codes; each marker has a unique identifier, allowing the location and orientation in the three-dimensional world to be determined by detecting and recognizing the markers. They also typically have specific dimensions and scales, which can be used to estimate the scale and size of objects, facilitating a more accurate mapping of the two-dimensional world to the three-dimensional world. Other options include, for example, RFID tags. RFID tags can be used for wireless positioning, while barcode markers can be used to determine the location and related information of a workpiece by scanning the barcode.

[0030] Step S104: Segment the target two-dimensional image to obtain a labeled image including the target marker;

[0031] It is understandable that after the 2D camera acquires the scene where the workpiece is located, the background needs to be removed and the target image, including the target identifier, needs to be segmented from the target 2D image.

[0032] In one optional embodiment, segmenting a target two-dimensional image to obtain a labeled image including the target marker includes: converting the target two-dimensional image to grayscale to obtain a grayscale image; determining the target pixel value corresponding to a pixel in the grayscale image, and the neighboring pixel value corresponding to a neighboring pixel adjacent to the pixel; determining an adaptive threshold for the grayscale image based on the target pixel value and the neighboring pixel value; and segmenting the target two-dimensional image using the adaptive threshold to obtain the labeled image.

[0033] Adaptive thresholding is an image processing technique used to automatically determine the threshold for each pixel, thereby achieving image segmentation. When segmenting the target marker image from a target 2D image, the target 2D image can be converted to grayscale, as grayscale images have only one channel, facilitating image processing. The target pixel value within the grayscale image can be determined, as can the values ​​of its neighboring pixels. Adaptive thresholding dynamically adjusts the threshold based on the target pixel value and its neighboring pixel values, thus determining an adaptive threshold for the grayscale image. Using this adaptive threshold, the target 2D image can be segmented, extracting the marker image from the background.

[0034] Optionally, the grayscale image can also be smoothed to remove noise. Filtering methods used for smoothing include Gaussian filtering and median filtering. Gaussian filtering is a method for smoothing images by weighted averaging of the pixel values ​​surrounding each pixel to reduce noise. Gaussian filtering blurs the image, making it smoother. Median filtering is a non-linear filtering algorithm that sorts the pixel values ​​surrounding each pixel and takes the median value as that pixel's value. Median filtering reduces noise by removing outliers. Using Gaussian or median filtering algorithms removes noise from the image, making it smoother. An adaptive threshold segmentation algorithm determines the threshold for each pixel based on local image features, dividing the image into target markers and background. This allows for better extraction of target marker information from the two-dimensional image, which is beneficial for subsequent image processing tasks.

[0035] Optionally, the aforementioned adaptive thresholding method can be a local adaptive thresholding method. This method divides the target two-dimensional image into several local regions and then calculates a threshold in each local region. Methods that can be used include the local mean method and the local median method. The local mean method sets the threshold for each pixel to the average gray value of its surrounding neighborhood, while the local median method sets the threshold for each pixel to the median gray value of its surrounding neighborhood.

[0036] Optionally, the aforementioned adaptive thresholding method can be a global adaptive thresholding method. This method calculates the threshold based on the statistical characteristics of the entire target 2D image. Methods that can be used include Otsu's Thresholding Method and the maximum entropy method. Otsu's method determines the threshold by maximizing the inter-class variance to maximize the difference between the segmented target and the background. The maximum entropy method maximizes the entropy between the segmented target and the background to make the segmentation result more accurate. Otsu's method is used to automatically determine the optimal threshold for a binary image by maximizing the inter-class variance to maximize the difference between the target's labeled image, the target 2D image, and the background. By calculating the histogram of the image pixels' gray levels, Otsu's method can automatically determine a threshold that divides the image into two parts: a labeled image and the background.

[0037] Optionally, the target two-dimensional image can be segmented to obtain an initial labeled image. The initial labeled image is then post-processed to remove small noise and fill holes to obtain the labeled image.

[0038] Optionally, the marker image can be extracted from the target two-dimensional image, and the contour recognition of the marker can also be performed. The contours are extracted and those that are not convex or not close to a square are discarded. The above-mentioned selection of the square contour is because the target marker is selected as a square. When the target marker is other predetermined shapes, the selected contour recognition method is adjusted accordingly. Some contour filtering is also applied (removing contours that are too small or too large, removing contours that are too close to each other, etc.).

[0039] Optionally, the contour recognition algorithms described above can be of various types. For example, the Sobel operator, a gradient-based edge detection algorithm, finds the edge contours of the target marker by calculating the horizontal and vertical gradients of the target 2D image. The Laplacian operator, a second-derivative-based edge detection algorithm, can detect edge contours in the target 2D image. The Roberts operator, a difference-based edge detection algorithm, finds the edge contours of the target marker by calculating the diagonal difference of the target 2D image. Edge tracking algorithms are based on the continuity between pixels, identifying the contours of the marker image by tracing the connection relationships between the edge pixels of the target marker.

[0040] Step S106: Recognize the marked image to determine the first relative pose between the 2D camera and the target mark;

[0041] It is understandable that by recognizing the marked image, the first relative pose between the two-dimensional camera and the target mark can be determined using the two-dimensional image.

[0042] In one optional embodiment, the target marker includes multiple corner points. Identifying the marker image and determining the first relative pose between the 2D camera and the target marker includes: identifying the marker image and determining the 2D corner point images corresponding to the multiple corner points respectively; determining the corner point pixel coordinates corresponding to the multiple corner points based on the 2D corner point images corresponding to the multiple corner points respectively; and determining the first relative pose based on the corner point pixel coordinates corresponding to the multiple corner points respectively.

[0043] It can be understood that the 2D camera, target workpiece, and target marker are all actual objects existing in the world coordinate system. To determine the 3D pose based on the 2D image, it is necessary to determine the first relative pose in the camera coordinate system. The marker image is identified to determine the 2D corner images corresponding to multiple corner points. Based on the transformation of the 2D corner images corresponding to multiple corner points in the camera coordinate system, the world coordinate system for the corner pixel coordinates of each corner point can be obtained. Based on the corner pixel coordinates, the first relative pose is determined. Through the above processing, the first relative pose between the 2D camera and the target marker can be determined based on the acquired 2D image, which is used to subsequently obtain the second relative pose between the welding robot and the target workpiece.

[0044] Optionally, when the target is marked as an ArUco mark or a QR code, there are four obvious corner points and internal binary encoding, which can increase the amount of information when mapping from two-dimensional coordinates to three-dimensional coordinates, thus realizing attitude projection.

[0045] In one optional embodiment, determining a first relative pose based on the pixel coordinates of multiple corner points includes: determining the focal length, principal point coordinates, and distortion coefficients of the 2D camera, wherein the principal point coordinates are the position of the center of the image generated by the 2D camera in the pixel coordinate system; determining an intrinsic parameter matrix to represent the optical characteristics of the 2D camera based on the focal length, principal point coordinates, and distortion coefficients; and calibrating the pixel coordinates of the multiple corner points based on the intrinsic parameter matrix to obtain the first relative pose.

[0046] It is understandable that, in order to determine the first relative pose in the camera coordinate system, it is necessary to calibrate the corner pixel coordinates of multiple corner points based on the intrinsic parameters of the 2D camera. The camera's focal length determines the camera's angle of view and magnification, and the distortion coefficients reflect the degree of distortion of the camera lens. These coefficients are used to correct distortion in the image. The principal point coordinates of the camera represent the origin position on the image plane. The corner pixel coordinates are then converted into the first relative pose in the camera coordinate system. Through the above processing, the first relative pose in the camera coordinate system can be obtained after the transformation.

[0047] Optionally, the aforementioned distortion coefficients may include radial distortion and tangential distortion. By subtracting the principal point coordinates from the corner pixel coordinates, multiplying by the focal length, and then correcting the image distortion using the distortion coefficients, the first relative pose in the camera coordinate system can be obtained.

[0048] Step S108: Convert the first relative pose to determine the second relative pose between the welding robot and the target workpiece;

[0049] It is understandable that by performing coordinate transformation on the first relative pose, the second relative pose between the welding robot and the target workpiece in the world coordinate system can be obtained.

[0050] In one optional embodiment, the welding robot includes a movable robotic arm mounted on a predetermined base. A two-dimensional camera is mounted on the robotic arm. The second relative pose is the relative pose between the base and the target workpiece. The process of converting the first relative pose to determine the second relative pose between the welding robot and the target workpiece includes: acquiring a third relative pose between the two-dimensional camera and the base, and acquiring a fourth relative pose between the target workpiece and a target mark; and determining the second relative pose based on the first, third, and fourth relative poses.

[0051] It is understood that a welding robot includes a movable robotic arm for performing welding. The robotic arm is mounted on a predetermined base and is equipped with a 2D camera for positioning. The robot acquires a third relative pose between the 2D camera and the base in the world coordinate system, and a fourth relative pose between the target workpiece and a target marker. This fourth relative pose is a fixed value that does not change after the target marker is fixed. Based on the first, third, and fourth relative poses obtained above, a second relative pose can be determined.

[0052] Optionally, such as Figure 2 As shown, the robot robot consists of a robotic arm 201, a target workpiece 202, a 2D camera 203, a target marker 204, and a trolley platform 205. During actual operation, the target workpiece 202 is placed at a fixed position near the robotic arm 201, and the target marker 204, used for positioning, is fixed at a predetermined position inside the target workpiece 202. The 2D camera 203 can acquire a 2D image of the target workpiece 202, which is then used for image processing to ultimately output the accurate pose of the target workpiece 202 relative to the robotic arm 201.

[0053] Optionally, Let be the second relative pose between the welding robot and the target workpiece in the world coordinate system, which is also the transformation matrix that needs to be obtained in the end. It can be expressed as: in, For a two-dimensional camera relative to the base

[0054] The third relative pose between the seats in the world coordinate system. It is the first relative pose of the target marker with respect to the 2D camera. This is the fourth relative pose of the target workpiece relative to the target mark. Once the target mark is fixed, it will not change and is a fixed value. Optionally, the above-mentioned fourth relative pose can be obtained by manual teaching.

[0055] In one optional embodiment, obtaining the third relative pose between the 2D camera and the base includes: determining the rotation matrix and translation vector of the 2D camera; determining the extrinsic parameter matrix representing the pose of the 2D camera in the world coordinate system based on the rotation matrix and translation vector; and calibrating the relative pose between the 2D camera and the base based on the extrinsic parameter matrix to obtain the third relative pose.

[0056] It is understandable that calibrating the relative pose between the 2D camera and the base is necessary to facilitate the transformation of camera coordinates to the world coordinate system. This requires considering the camera's external parameters, including rotation and translation, which are implemented using the 2D camera's rotation matrix and translation vector. Through the transformation of external parameters, the camera coordinates can be converted into 3D point coordinates in the world coordinate system, thus obtaining the third relative pose between the 2D camera and the base in the world coordinate system.

[0057] Step S110: When the second relative pose matches the predetermined welding pose, control the welding robot to perform welding on the target workpiece.

[0058] It is understandable that when the second relative pose matches the predetermined welding pose, it is considered that the welding robot has positioned the target workpiece for welding, and the welding robot is controlled to perform welding processing on the target workpiece.

[0059] In an optional embodiment, the method further includes: determining a pose deviation between the second relative pose and the predetermined welding pose when the second relative pose does not match the predetermined welding pose; adjusting the second relative pose based on the pose deviation to obtain an updated pose; and controlling the welding robot to perform welding processing on the target workpiece using the updated pose.

[0060] It is understandable that if the second relative pose does not match the predetermined welding pose, it is considered that there is a positioning deviation between the welding robot and the target workpiece. In order to correct the deviation, it is necessary to determine the pose deviation between the second relative pose and the predetermined welding pose. Based on the pose deviation, the second relative pose is adjusted to obtain an updated pose. Using the corrected updated pose to control the welding robot to perform the welding process helps to improve the welding accuracy.

[0061] Through the above steps S102, the welding robot is controlled to move to the predetermined imaging position, and the two-dimensional camera installed on the welding robot is controlled to take a picture of the target workpiece to obtain a target two-dimensional image, wherein a target mark is set at the predetermined position of the target workpiece; in step S104, the target two-dimensional image is segmented to obtain a mark image including the target mark; in step S106, the mark image is identified to determine the first relative pose between the two-dimensional camera and the target mark; in step S108, the first relative pose is converted to determine the second relative pose between the welding robot and the target workpiece; in step S110, if the second relative pose matches the predetermined welding pose, the welding robot is controlled to perform welding processing on the target workpiece. This can reduce the limitations of welding robot setup, realize the technical effect of three-dimensional pose positioning using two-dimensional image acquisition equipment, and thus solve the technical problem of high limitation of robot setup due to reliance on three-dimensional image acquisition equipment for positioning in related technologies.

[0062] Based on the above embodiments and optional embodiments, the present invention proposes an optional implementation method, such as... Figure 3 As shown, the welding robot has a mobile function and includes a movable robotic arm for performing welding. The robotic arm is mounted on a predetermined base and is equipped with a 2D camera for positioning. Following a predetermined route, the robot moves to a predetermined photographing position and controls the 2D camera mounted on the welding robot to photograph the target workpiece. If the photographing fails, it is retaken. If the photographing is successful, a target 2D image including the entire target workpiece is obtained. Target markers, preferably ArUco markers, are already set on the target workpiece. The target 2D image is segmented, and an adaptive thresholding method is used to extract the marker image including the target markers.

[0063] The marked image is identified to determine the two-dimensional corner images corresponding to multiple corner points. Based on the transformation of these two-dimensional corner images in the camera coordinate system, the world coordinate system for the pixel coordinates of each corner point is obtained. Based on these pixel coordinates, the first relative pose is determined and denoted as...

[0064] Calibrate using the intrinsic parameters of the 2D camera, subtract the principal point coordinates from the corner pixel coordinates, multiply by the focal length, and then correct image distortion using distortion coefficients. This yields the first relative pose in the camera coordinate system, which is used to subsequently obtain the second relative pose between the welding robot and the target workpiece, denoted as [missing information].

[0065] The second relative pose between the welding robot and the target workpiece in the world coordinate system can be obtained in the following way.

[0066]

[0067] in, This represents the third relative pose of the 2D camera relative to the base in the world coordinate system. The third relative pose is obtained using rotation matrices and translation vectors. It is the first relative pose of the target marker with respect to the 2D camera. It is the fourth relative pose of the target workpiece with respect to the target mark.

[0068] When the second relative pose matches the predetermined welding pose, it is considered that the welding robot has positioned the target workpiece for welding, and the welding robot is controlled to perform welding processing on the target workpiece.

[0069] The above-described optional implementation methods achieve at least the following effects: No specially customized mechanical positioning mechanism is required in the application scenario, nor is it necessary to provide additional external cooperating devices and mechanisms; it can adapt to workpieces of various shapes and sizes. Adjustments to automated production lines will not negatively impact the operation of this positioning algorithm. Compared to the use of 3D cameras for positioning in related technologies, the specific implementation method provided by this invention offers higher accuracy, lower overall cost, and increased robot setup flexibility. It has advantages such as ease of use, good generalization performance, high flexibility, high intelligence, and low overall cost. The method of using a 2D camera to achieve 3D pose positioning can concisely, efficiently, and stably calculate the posture of the workpiece being welded in the workspace.

[0070] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0071] This embodiment also provides a workpiece welding control device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the terms "module" and "device" can refer to a combination of software and / or hardware that performs a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0072] According to embodiments of the present invention, an embodiment of an apparatus for implementing a workpiece welding control method is also provided. Figure 4 This is a schematic diagram of a workpiece welding control device according to an embodiment of the present invention, such as... Figure 4 As shown, the above-mentioned workpiece welding control device includes: a photo-taking module 402, a mark acquisition module 404, an image recognition module 406, a conversion module 408, and a welding control module 410. The device will be described below.

[0073] The photo-taking module 402 is used to control the welding robot to move to the predetermined photo-taking position and control the two-dimensional camera installed on the welding robot to take a photo of the target workpiece to obtain a two-dimensional image of the target, wherein a target mark is set at the predetermined position of the target workpiece.

[0074] The marker acquisition module 404 is connected to the imaging module 402 and is used to segment the target two-dimensional image to obtain a marker image including the target marker;

[0075] The image recognition module 406, connected to the marker acquisition module 404, is used to recognize the marker image and determine the first relative pose between the two-dimensional camera and the target marker;

[0076] The conversion module 408, connected to the image recognition module 406, is used to convert the first relative pose and determine the second relative pose between the welding robot and the target workpiece.

[0077] The welding control module 410, connected to the conversion module 408, is used to control the welding robot to perform welding processing on the target workpiece when the second relative posture matches the predetermined welding posture.

[0078] In a workpiece welding control device provided in this embodiment of the invention, a photographing module 402 is used to control a welding robot to move to a predetermined photographing position and control a two-dimensional camera mounted on the welding robot to photograph the target workpiece to obtain a target two-dimensional image, wherein a target mark is set at the predetermined position of the target workpiece; a mark acquisition module 404, connected to the photographing module 402, is used to segment the target two-dimensional image to obtain a mark image including the target mark; an image recognition module 406, connected to the mark acquisition module 404, is used to recognize the mark image and determine a first relative pose between the two-dimensional camera and the target mark; a conversion module 408, connected to the image recognition module 406, is used to convert the first relative pose to determine a second relative pose between the welding robot and the target workpiece; and a welding control module 410, connected to the conversion module 408, is used to control the welding robot to perform welding processing on the target workpiece when the second relative pose matches the predetermined welding pose. This achieves the goal of reducing the limitations of welding robot setup, realizes the technical effect of using two-dimensional image acquisition equipment to achieve three-dimensional pose positioning, and thus solves the technical problem of relying on three-dimensional image acquisition equipment for positioning, which has high limitations on robot setup.

[0079] It should be noted that the above modules can be implemented by software or hardware. For example, for the latter, it can be implemented in the following ways: the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0080] It should be noted that the aforementioned photographing module 402, marking acquisition module 404, image recognition module 406, conversion module 408, and welding control module 410 correspond to steps S102 to S110 in the embodiments. The instances and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above embodiments. It should be noted that the aforementioned modules, as part of the device, can run on a computer terminal.

[0081] It should be noted that the optional or preferred implementation methods of this embodiment can be found in the relevant descriptions in the embodiments, and will not be repeated here.

[0082] The aforementioned workpiece welding control device may also include a processor and a memory. The photographing module 402, the marking acquisition module 404, the image recognition module 406, the conversion module 408, the welding control module 410, etc., are all stored in the memory as program units. The processor executes the aforementioned program units stored in the memory to realize the corresponding functions.

[0083] The processor contains a core that retrieves the corresponding program unit from memory. One or more cores may be configured. Memory may include non-persistent memory in computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory includes at least one memory chip.

[0084] This invention provides a non-volatile storage medium storing a program that, when executed by a processor, implements a workpiece welding control method.

[0085] This invention provides an electronic device including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs the following steps: controlling a welding robot to move to a predetermined photographing position, and controlling a two-dimensional camera mounted on the welding robot to photograph a target workpiece to obtain a target two-dimensional image, wherein a target mark is set at the predetermined position on the target workpiece; segmenting the target two-dimensional image to obtain a mark image including the target mark; recognizing the mark image to determine a first relative pose between the two-dimensional camera and the target mark; converting the first relative pose to determine a second relative pose between the welding robot and the target workpiece; and, if the second relative pose matches a predetermined welding pose, controlling the welding robot to perform welding processing on the target workpiece. The device described herein may be a server, PC, etc.

[0086] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program comprising the following steps: controlling a welding robot to move to a predetermined photographing position, and controlling a two-dimensional camera mounted on the welding robot to photograph a target workpiece to obtain a target two-dimensional image, wherein a target mark is set at a predetermined position on the target workpiece; segmenting the target two-dimensional image to obtain a mark image including the target mark; recognizing the mark image to determine a first relative pose between the two-dimensional camera and the target mark; converting the first relative pose to determine a second relative pose between the welding robot and the target workpiece; and, if the second relative pose matches a predetermined welding pose, controlling the welding robot to perform welding processing on the target workpiece.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0092] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0093] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0094] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0095] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for controlling workpiece welding, characterized in that, include: The welding robot is controlled to move to a predetermined photographing position, and the two-dimensional camera installed on the welding robot is controlled to photograph the target workpiece to obtain a two-dimensional image of the target. The predetermined position of the target workpiece is marked with a target mark, which includes multiple corner points. The target 2D image is segmented to obtain a labeled image including the target marker; The marked image is identified to determine the two-dimensional corner images corresponding to the plurality of corner points respectively; based on the two-dimensional corner images corresponding to the plurality of corner points respectively, the corner pixel coordinates corresponding to the plurality of corner points are obtained; based on the corner pixel coordinates corresponding to the plurality of corner points respectively, the first relative pose between the two-dimensional camera and the target mark is determined. The first relative pose is transformed to determine the second relative pose between the welding robot and the target workpiece; When the second relative pose matches the predetermined welding pose, the welding robot is controlled to perform welding on the target workpiece.

2. The method according to claim 1, characterized in that, The step of segmenting the target two-dimensional image to obtain a labeled image including the target label includes: The target two-dimensional image is converted to grayscale to obtain a grayscale image; Determine the target pixel value corresponding to the pixel point included in the grayscale image, and the neighboring pixel value corresponding to the neighboring pixel point adjacent to the pixel point; Based on the target pixel value and the neighboring pixel values, an adaptive threshold for the grayscale image is determined; The target two-dimensional image is segmented using the adaptive threshold to obtain the labeled image.

3. The method according to claim 1, characterized in that, The step of determining the first relative pose based on the pixel coordinates of the corner points respectively includes: Determine the focal length, principal point coordinates, and distortion coefficients of the two-dimensional camera, wherein the principal point coordinates are the position of the center of the image generated by the two-dimensional camera in the pixel coordinate system; Based on the focal length, the principal point coordinates, and the distortion coefficient, an intrinsic parameter matrix is ​​determined to represent the optical characteristics of the two-dimensional camera; Based on the intrinsic parameter matrix, the pixel coordinates of the corner points corresponding to the multiple corner points are calibrated to obtain the first relative pose.

4. The method according to claim 1, characterized in that, The welding robot includes a movable robotic arm mounted on a predetermined base. A two-dimensional camera is mounted on the robotic arm. The second relative pose is the relative pose between the base and the target workpiece. The step of converting the first relative pose to determine the second relative pose between the welding robot and the target workpiece includes: Obtain the third relative pose between the 2D camera and the base, and obtain the fourth relative pose between the target workpiece and the target mark; The second relative pose is determined based on the first relative pose, the third relative pose, and the fourth relative pose.

5. The method according to claim 4, characterized in that, The step of obtaining the third relative pose between the 2D camera and the base includes: Determine the rotation matrix and translation vector of the two-dimensional camera; Based on the rotation matrix and the translation vector, determine the extrinsic parameter matrix used to represent the pose of the two-dimensional camera in the world coordinate system; Based on the extrinsic parameter matrix, the relative pose between the two-dimensional camera and the base is calibrated to obtain the third relative pose.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: If the second relative pose does not match the predetermined welding pose, determine the pose deviation between the second relative pose and the predetermined welding pose; Based on the pose deviation, the second relative pose is adjusted to obtain the updated pose; The welding robot is controlled to perform welding on the target workpiece using the updated pose.

7. A workpiece welding control device, characterized in that, include: The camera module is used to control the welding robot to move to a predetermined camera position and control the two-dimensional camera installed on the welding robot to take a picture of the target workpiece to obtain a two-dimensional image of the target. The predetermined position of the target workpiece is marked with a target mark, which includes multiple corner points. The marker acquisition module is used to segment the target two-dimensional image to obtain a marker image including the target marker; An image recognition module is used to recognize the marked image and determine the two-dimensional corner images corresponding to the plurality of corner points respectively; based on the two-dimensional corner images corresponding to the plurality of corner points respectively, obtain the corner pixel coordinates corresponding to the plurality of corner points respectively; and based on the corner pixel coordinates corresponding to the plurality of corner points respectively, determine the first relative pose between the two-dimensional camera and the target mark. A conversion module is used to convert the first relative pose to determine a second relative pose between the welding robot and the target workpiece. The welding control module is used to control the welding robot to perform welding processing on the target workpiece when the second relative pose matches the predetermined welding pose.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores multiple instructions adapted for loading and execution by a processor of the workpiece welding control method according to any one of claims 1 to 6.

9. An electronic device, characterized in that, include: One or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the workpiece welding control method according to any one of claims 1 to 6.

Images