A laser axis-pair pose solving method and system based on visual geometry constraint

Abstract

The application discloses a laser axis alignment pose solving method based on visual geometric constraints, comprising: establishing rigid transformation relationships among a mechanical arm base coordinate system, an end tool coordinate system, a camera coordinate system and a laser coordinate system; obtaining a three-dimensional position of a target point in the camera coordinate system; determining a laser optical axis direction and a laser coordinate system origin position through spatial straight line fitting; uniformly converting the target point and laser installation parameters to the mechanical arm base coordinate system; jointly establishing an alignment constraint of the laser optical axis passing through the target point and a camera depth constraint, solving an end tool desired pose; and outputting the desired pose and driving the mechanical arm to perform laser axis alignment movement. The application solves the technical problem that systematic axis alignment errors caused by installation deviation and optical axis deflection cannot be compensated in the solving stage, and achieves the technical effect that the end laser axis alignment desired pose simultaneously satisfies two types of geometric constraints that the laser optical axis passes through the target point and the target point maintains a preset depth relative to the camera.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of robot vision guidance and control technology, and in particular to a laser-based axis pose solving method and system based on visual geometric constraints. Background Technology

[0002] With the continuous development of industrial robots and visual perception technology, laser operations at the end effector of vision-guided robotic arms are widely used in scenarios such as laser welding, laser cutting, laser marking, and precision measurement. In these operations, the end-effector camera is responsible for acquiring the spatial position information of the target point. Based on this information, the robotic arm drives the end-effector laser to move to the vicinity of the target and perform laser irradiation or action, which places high demands on the alignment accuracy of the end-effector laser.

[0003] In existing vision-guided methods, hand-eye calibration is typically used to obtain the extrinsic parameter relationship between the camera and the end effector, and the image observation results of the target point are converted into robotic arm motion commands. Some methods simplify the end effector laser actuator to a point actuator located at the origin of the tool coordinate system, or only compensate for the target position by a fixed offset, thereby approximating the spatial position of the laser emission point.

[0004] The inventors of this disclosure have noted that current vision guidance methods do not take into account the simultaneous existence of two types of installation parameters: translational offset of the laser emission point relative to the origin of the tool coordinate system and installation angle deviation of the laser optical axis relative to the tool coordinate axis. They have not established a unified geometric constraint relationship between the target point, the camera depth direction, the end-effector coordinate system and the laser optical axis. As a result, although the robotic arm can move the end-effector to the vicinity of the target, there is still a lateral deviation between the actual laser irradiation point and the target point, resulting in insufficient axis accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a laser alignment axis pose solution method and system based on visual geometric constraints, which solves the technical problem in related technologies that lacks the ability to solve the laser optical axis alignment constraint and the camera depth constraint simultaneously, resulting in the end-effector laser being unable to accurately align with the target point in the robotic arm operation scenario.

[0006] This invention provides a method for solving the laser axis pose based on visual geometric constraints, comprising the following steps: Obtain the rigid body transformation relationships between the end-effector coordinate system and the camera coordinate system and the laser coordinate system, respectively; The three-dimensional position of the target point in the camera coordinate system is obtained based on the visual perception unit; Based on the coordinates of multiple laser irradiation points in the end-tool coordinate system, a straight line fit is performed to generate the unit direction vector of the laser optical axis in the end-tool coordinate system and the position of the origin of the laser coordinate system. Transform the target point position and laser installation parameters to the robot arm's base coordinate system; The alignment constraint of the laser optical axis passing through the target point and the depth constraint of the target point maintaining a preset depth relative to the camera are jointly established to solve the desired laser-axis pose; and the desired laser-axis pose is output to the motion control unit to drive the end effector of the robotic arm to move to the desired pose. The process of solving the desired laser alignment axis pose includes: determining the desired pose and calculating the desired directions of the laser optical axis and the camera depth axis in the robot arm base coordinate system; solving the directed distance from the laser emission point to the target point by combining the alignment constraints and depth constraints; and calculating the desired position of the end effector based on the directed distance and combining them into the desired pose.

[0007] Further, the linear fitting based on the coordinates of multiple laser irradiation points in the end-tool coordinate system includes: controlling the end-tool laser to sequentially irradiate the calibration target surface located at different spatial positions; acquiring images under each irradiation state by the end-tool camera and extracting the pixel coordinates and corresponding depth values ​​of the laser spot center; obtaining the three-dimensional position of each laser irradiation point in the camera coordinate system through back projection using the camera pinhole imaging model; transforming the three-dimensional position of each laser irradiation point to the end-tool coordinate system using the rotation matrix and position vector of the camera coordinate system relative to the end-tool coordinate system; performing spatial linear fitting on the transformed spatial point set based on the criterion of minimizing the sum of the vertical distances from each sampling point to the fitted line; the output of the spatial linear fitting is the unit direction vector of the laser optical axis in the end-tool coordinate system; determining the foot of the perpendicular from the origin of the end-tool coordinate system to the fitted optical axis as the origin position of the laser coordinate system, wherein the foot of the perpendicular is obtained by subtracting the projection component of the centroid in the optical axis direction from the centroid of all laser irradiation points in the end-tool coordinate system.

[0008] Furthermore, the step of obtaining the three-dimensional position of the target point in the camera coordinate system based on the visual perception unit includes: obtaining the pixel coordinates and corresponding depth values ​​of the target point in the image plane; and back-projecting the pixel coordinates and depth values ​​of the target point into the camera coordinate system according to the camera intrinsic parameter matrix and the camera pinhole imaging model to obtain the coordinate components of the target point along each axis of the camera coordinate system. The coordinate components of each axis are obtained by multiplying the difference between the pixel coordinates and the principal point coordinates by the depth value and then dividing by the equivalent focal length in the corresponding direction. The depth direction coordinate component is the depth value itself.

[0009] Further, the method of solving the directed distance from the laser emission point to the target point by simultaneously combining the alignment constraints and depth constraints includes: the alignment constraint is expressed as the position of the target point in the robot arm base coordinate system equal to the desired position of the end-effector plus the offset of the origin of the laser coordinate system in the robot arm base coordinate system under the desired attitude plus the product of the directed distance and the desired direction of the laser optical axis; the depth constraint is expressed as the position of the target point in the robot arm base coordinate system equal to the desired position of the end-effector plus the offset of the origin of the camera coordinate system in the robot arm base coordinate system under the desired attitude plus the product of the preset camera depth and the desired direction of the camera depth axis; by simultaneously combining the alignment constraints and depth constraints to eliminate the desired position of the end-effector, the directed distance is obtained as the preset camera depth minus the inner product of the desired direction of the camera depth axis and the position difference between the origin of the laser coordinate system and the origin of the camera coordinate system in the end-effector coordinate system after rotation in the desired attitude, and then divided by the inner product of the desired direction of the camera depth axis and the desired direction of the laser optical axis.

[0010] Furthermore, the method further includes: before solving the directed distance, determining whether the absolute value of the inner product of the desired direction of the camera depth axis and the desired direction of the laser optical axis is less than a preset direction projection threshold; if it is less than the direction projection threshold, terminating the solution and reporting an anomaly; after obtaining the target point depth value, determining whether the depth value is within the range between a preset minimum effective depth value and a maximum effective depth value; if it does not meet the range or any component of the three-dimensional coordinates of the target point is not a finite real number, then the target point is determined to be invalid and will not proceed to the subsequent pose solving process.

[0011] Furthermore, after obtaining the three-dimensional position of the target point in the camera coordinate system and before performing line fitting, the method further includes: transforming the three-dimensional position of the target point to the robot arm base coordinate system through the relationship between the camera and tool extrinsic parameters and the current pose of the end-effector; solving for the expected position of the end-effector during the camera open-loop approach phase based on the preset camera observation distance, the unit direction of the camera depth axis in the camera coordinate system, and the geometric requirement that the target point is located in front of the camera central axis and maintains the preset observation distance, the expected position of the end-effector during the camera open-loop approach phase is calculated. The expected position of the end-effector is equal to the position of the target point in the robot arm base coordinate system minus the component of the preset observation distance along the desired direction of the camera depth axis minus the component of the offset of the camera coordinate system origin in the end-effector coordinate system after rotation by the desired pose; and using the expected pose of the camera open-loop approach phase as the target input for the robot arm motion, so that the end-effector camera moves to the vicinity of the target.

[0012] Furthermore, after the camera's open-loop approach is completed, visual closed-loop correction is also performed: obtaining the pixel deviation between the real-time pixel coordinates of the target point and the preset desired pixel coordinates, and the depth deviation between the real-time depth of the target point and the desired depth; controlling the end effector of the robotic arm to generate a corrective motion based on the pixel deviation and depth deviation; calculating the planar pixel error of the target point relative to the desired pixel position, and if the planar pixel error is not greater than the allowable pixel error threshold and the absolute value of the depth deviation is not greater than the allowable depth error threshold within a consecutive preset stable period, then the closed-loop correction is determined to be complete.

[0013] Furthermore, after driving the end effector of the robotic arm to move to the desired pose, the method further includes: calculating the desired translation of the end effector; if the norm of the desired translation exceeds a preset maximum single movement distance, scaling the desired translation to the maximum single movement distance; after the robotic arm completes the movement, re-acquiring the actual pose of the end effector, calculating the lateral error from the target point to the laser optical axis after the movement, wherein the lateral error is obtained by subtracting the projection component of the vector in the direction of the laser optical axis after the movement from the vector between the target point and the laser emission point after the movement; if the norm of the lateral error is greater than the allowable laser axis error threshold, re-performing the coordinate transformation and desired pose solution based on the current observation results until the lateral error meets the accuracy requirements or reaches the preset maximum number of iterations.

[0014] Furthermore, an electronic device is also provided, comprising: at least one memory storing computer-executable instructions non-transiently; and at least one processor configured to run the computer-executable instructions, wherein the computer-executable instructions, when run by the processor, implement the laser-axis pose solving method based on visual geometric constraints disclosed in this invention.

[0015] Furthermore, a laser alignment pose control system based on visual geometric constraints is provided. This system includes a calibration unit, a visual perception unit, a pose solving unit, and a motion control unit. The calibration unit is configured to acquire the rigid body transformation relationships between the end-effector coordinate system and the camera coordinate system and the laser coordinate system, respectively, and to perform linear fitting based on the coordinates of multiple laser illumination points in the end-effector coordinate system to generate the laser optical axis direction and the origin position of the laser coordinate system. The visual perception unit is configured to acquire the three-dimensional position of the target point in the camera coordinate system. The pose solving unit is configured to convert the target point position and laser installation parameters to the robot arm base coordinate system, and jointly establish alignment constraints where the laser optical axis passes through the target point and depth constraints where the target point maintains a preset depth relative to the camera to solve for the desired laser alignment pose. The motion control unit is configured to output the desired laser alignment pose as the robot arm motion control target, driving the robot arm end-effector to move to the desired pose.

[0016] This invention incorporates the camera coordinate system, end-effector coordinate system, laser coordinate system, and robotic arm base coordinate system into a unified geometric transformation framework. By separately acquiring the extrinsic parameters of the camera and tool, as well as the laser optical axis direction and the position of the origin of the laser coordinate system, the offset of the laser emission point and the skewness of the optical axis direction can be accurately expressed in the pose solving stage. This solves the technical problem that the systematic axis-alignment errors caused by installation deviations and optical axis skewness cannot be compensated in the solving stage. It achieves the technical effect of ensuring that the desired pose of the end-effector laser simultaneously satisfies two types of geometric constraints: the laser optical axis passing through the target point and the target point maintaining a preset depth relative to the camera. Furthermore, through lateral error verification and iterative correction, it achieves the technical effect of gradually approximating the target when the accuracy requirement is not met in a single axis-alignment movement, thereby improving the geometric consistency of vision-guided laser operations. Attached Figure Description

[0017] Figure 1 This is a flowchart of the end-laser pair axis pose solving method based on camera and laser joint geometric constraints provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall flowchart of the robot end-effector laser alignment control system provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the multi-coordinate system transformation relationship provided in an embodiment of the present invention; Figure 4 This is a schematic diagram showing the coordinate system extrinsic parameter calibration results provided in this embodiment of the invention: a comparison of the position components of the camera and the laser origin. Figure 5 This is a schematic diagram of the laser irradiation point sampling coordinate distribution (end-effector coordinate system T) provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the summarization of the three-axis components of the key point position in the base coordinate system provided in the embodiment of the present invention; Figure 7 This is a schematic diagram comparing the end-effector positions before and after solving the axis pose, provided in an embodiment of the present invention. Figure 8 This is a schematic diagram of the components and norm of the expected translational amount about the axis provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the laser-axis desired pose joint constraint solution process provided in an embodiment of the present invention. Detailed Implementation

[0018] Reference Figure 1-9 This invention provides a method for solving the axis pose of a laser based on visual geometric constraints, comprising the following steps: Step S100: Establish the coordinate system and external parameter relationships of the robot operation system.

[0019] In step S100, the coordinate system definitions and rigid body transformation relationships between the rigid body components in the robot operation system are obtained, and a set of coordinate system parameters for subsequent pose calculation is generated. The coordinate system includes at least the robot arm base coordinate system. End tool coordinate system Camera coordinate system and laser coordinate system Among them, the robot arm base coordinate system The end-effector coordinate system serves as the global reference coordinate system for robot motion planning and pose representation. Fixed to the end flange or tool mounting end of the robotic arm, camera coordinate system Fixed to the end camera, laser coordinate system Fixed to the end laser and its positive The axial direction is defined as the direction of the laser emission axis.

[0020] Rigid body transformations between any two coordinate systems are represented by homogeneous transformation matrices. Let the coordinate systems be... Relative to coordinate system The rigid body transformation is as follows:

[0021] In the formula, Representing the coordinate system To coordinate system The homogeneous transformation matrix; Representing the coordinate system Relative to coordinate system rotation matrix; Representing the coordinate system Origin in coordinate system The position vector below.

[0022] Based on the above transformation representation, the end-effector coordinate system is obtained. To the camera coordinate system External parameter relationships:

[0023] In the formula, Representing the camera coordinate system Relative to the end tool coordinate system The rigid body transformation matrix; Representing the camera coordinate system Relative to the end tool coordinate system The rotation matrix; Representing the camera coordinate system The origin is in the end tool coordinate system The position vector below. End tool coordinate system. To the camera coordinate system The external parameter relationships are obtained through hand-eye calibration.

[0024] Furthermore, hand-eye calibration involves controlling the robotic arm to move to multiple different poses and acquiring images of the calibration board. Then, by using the end-effector pose in each pose and the camera's observation of the calibration board, a system of equations is established to solve for the camera coordinate system. Relative to the end tool coordinate system rotation matrix and position vector The solution algorithms employed include classic hand-eye calibration algorithms such as the Tsai method or the Park-Martin method. The inputs are the end-effector pose sequence and the corresponding camera-to-calibration-board pose sequence for each calibration pose, and the output is... .

[0025] Obtain the coordinate system of the end tool To the laser coordinate system External parameter relationships:

[0026] In the formula, Representing the laser coordinate system Relative to the end tool coordinate system The rigid body transformation matrix; Representing the laser coordinate system Relative to the end tool coordinate system The rotation matrix; Representing the laser coordinate system The origin is in the end tool coordinate system The position vector below.

[0027] laser coordinate system of righteousness The axis is defined as the direction of the laser emission axis. Laser coordinate system Given the unit optical axis vector, we have ,in Indicates transpose. According to the laser coordinate system. Relative to the end tool coordinate system The rotational relationship of the laser optical axis in the end-effector coordinate system Downward direction for:

[0028] Laser optical axis in end-effector coordinate system Downward direction Used to characterize the actual laser emission direction in the end-effector coordinate system.

[0029] Further obtain the end-effector coordinate system Relative to the robot arm's base coordinate system Current pose:

[0030] In the formula, Indicates the coordinate system of the end tool Relative to the robot arm's base coordinate system The current rotation matrix; Indicates the coordinate system of the end tool The origin is in the robot arm's base coordinate system. The current position vector below. End tool coordinate system. Relative to the robot arm's base coordinate system The current pose is obtained from the state feedback of the robotic arm.

[0031] It should be noted that the above coordinate system definition may also include the camera optical coordinate system. This is used to describe the direction of the camera's imaging center axis. When the three-dimensional position of the target point output by the visual perception unit is expressed in the camera's optical coordinate system... When descending, the camera's optical coordinate system needs to be used. To the camera coordinate system The rigid body transformation relationship transforms the three-dimensional position of the target point to the camera coordinate system. This transformation is based on the camera's optical coordinate system. Relative to camera coordinate system rotation matrix and position vector Sure.

[0032] In March 20XX, a precision electronic component laser welding production line was equipped with a six-degree-of-freedom robotic arm laser welding workstation. The end effector integrates a depth camera and a semiconductor pulsed laser for performing precision laser soldering operations on soldering targets on printed circuit boards. Due to a structural mounting offset between the laser output port and the end effector flange, and the laser optical axis relative to the end effector tool… The shaft has an installation tilt of approximately 2°. If this deviation is not systematically modeled, the laser irradiation point will continuously deviate from the center of the welding target. Therefore, a complete coordinate system establishment and external parameter calibration must be completed before production.

[0033] The system employs the Park-Martin method for hand-eye calibration, controlling the robotic arm to move sequentially to 15 different poses and acquiring images of the chessboard calibration board, thus solving for the camera coordinate system. Relative to the end tool coordinate system The external parameter relationship. After calibration, the camera origin is in the end-effector coordinate system. The position vector below The origin of the laser coordinate system is in the end-tool coordinate system. The position vector below All were determined, and the rotation matrix was obtained simultaneously.

[0034] Table 1. Coordinate system external parameter calibration results

[0035] Laser optical axis direction in the table above of The component 0.0349 corresponds to an installation tilt angle of approximately 2°, which is consistent with the actual mechanical installation measurement results, verifying the effectiveness of the calibration process.

[0036] Step S200: Obtain the three-dimensional position of the target point in the camera coordinate system.

[0037] In step S200, the target point is obtained based on the output data of the visual perception unit. In the camera coordinate system The three-dimensional position vector below .

[0038] For example, when the visual perception unit outputs the target point Pixel coordinates in the image plane and corresponding depth value At that time, among them Indicates the target point Pixel coordinates in the horizontal direction of the image. Indicates the target point Pixel coordinates in the vertical direction of the image. Let the distance from the target point along the camera depth direction to the camera imaging reference plane be denoted by the camera intrinsic parameter matrix. for:

[0039] In the formula, This indicates the camera's equivalent focal length in the horizontal direction of the image (in pixels). This represents the equivalent focal length of the camera in the vertical direction of the image (in pixels). This represents the pixel coordinates of the camera's principal point in the horizontal direction of the image; This represents the pixel coordinates of the camera principal point in the vertical direction of the image.

[0040] Based on the camera pinhole imaging model, the target point Backprojection of pixel coordinates and depth values ​​into the camera coordinate system Below, we obtain:

[0041] In the formula, , , Representing the target points respectively In the camera coordinate system lower edge axis, shaft and The coordinate components along the axial direction. In the above formula, the molecule... The unit of measurement is pixels·meter, and the denominator is... The dimension of is pixels, and the result after dividing the two is in meters. The dimensions are consistent, and each coordinate component has a length dimension, thus the dimensional relationship holds. Therefore, the target point... In the camera coordinate system The three-dimensional position vector below Represented as:

[0042] It should be noted that the above method of obtaining the 3D position of the target point is a specific form of back projection of pixel coordinates and depth values. If the visual perception unit directly outputs the target point... In the camera coordinate system If the target's three-dimensional coordinates are directly output by a visual perception unit such as a depth camera, structured light camera, or binocular camera, then these three-dimensional coordinates can be directly marked as... ,in This indicates transpose, and the pixel backprojection process will no longer be performed.

[0043] It should be noted that if the target point The initial three-dimensional position is expressed in the target observation coordinate system. Below, recorded as Then, using the target observation coordinate system To the camera coordinate system The rigid body transformation relationship is used to transform the target point to the camera coordinate system. :

[0044] In the formula, Represents the target observation coordinate system Relative to camera coordinate system The rotation matrix; Represents the target observation coordinate system Origin in camera coordinate system The position vector below. When the target observation coordinate system... For the camera optical coordinate system When, the above formula is specifically written as ,in Indicates the target point In the camera optical coordinate system The three-dimensional position vector below.

[0045] In this embodiment of the disclosure, to avoid the impact of depth holes, false detections, or coordinate anomalies on subsequent pose solving, the following step is included after obtaining the 3D position of the target point: performing a validity judgment on the depth value of the target point, that is, judging whether the depth value meets the requirements. In the formula This indicates the minimum allowed effective depth value. This represents the maximum allowed effective depth value; both are positive real numbers pre-set based on the actual operational scenario, satisfying... If any component of the target point's three-dimensional coordinates is not a finite real number, or if the depth value does not meet the above range, the target point is deemed invalid and will not proceed to the subsequent pose calculation process.

[0046] The visual perception unit performs detection on the pad target point numbered PAD-037 on the circuit board, in the camera intrinsic parameter matrix. Pixels Pixels Pixels Pixels. The effective depth range is set to... mm, mm. The visual perception unit outputs the target pixel coordinates. Depth value If the depth value falls within the valid range (mm), the target point is considered valid.

[0047] Substituting the above values ​​into the back projection formula:

[0048]

[0049]

[0050] Table 2 Calculation results of target point back projection

[0051] Step S300: Determine the laser optical axis direction and tool and laser extrinsic parameters based on the collected laser irradiation point.

[0052] In step S300, the end-effector laser is controlled to illuminate the calibration area, and the end-effector camera acquires multiple laser illumination points in the camera coordinate system. The position below will be used to transform each laser irradiation point to the end-tool coordinate system. Afterwards, spatial straight line fitting is performed to generate the laser optical axis in the end-effector coordinate system. The unit direction vector below and laser coordinate system The origin is in the end tool coordinate system The lower position .

[0053] For example, let the first A laser illumination point in the camera coordinate system The position below is ,in , This indicates the number of valid sampling points. Using the extrinsic relationship between the camera and the end effector, each laser illumination point is transformed to the end effector coordinate system. Down:

[0054] In the formula, Indicates the first The laser irradiation point is in the end-tool coordinate system. The position vector below; Representing the camera coordinate system Relative to the end tool coordinate system The rotation matrix; Representing the camera coordinate system The origin is in the end tool coordinate system The position vector below.

[0055] Furthermore, the aforementioned laser illumination points are located in the camera coordinate system. The lower position The following method is used to obtain the laser: The end-effector laser sequentially illuminates a calibration target at different depths or spatial locations. An end-effector camera captures images under each illumination state. Image processing methods are used to extract the pixel coordinates of the laser spot center. These coordinates are then combined with depth images or structured light ranging to obtain the corresponding depth values. Finally, the laser illumination points are back-projected onto the camera's pinhole imaging model to obtain the coordinates of each laser illumination point within the camera coordinate system. The three-dimensional position below The back projection method is the same as the method for obtaining the three-dimensional position of the target point in step S200.

[0056] For the aforementioned set of spatial points, a straight line is fitted, and the unit direction vector is obtained by minimizing the sum of the perpendicular distances from each sampling point to the fitted line. Assume all laser illumination points are in the end-effector coordinate system. The lower center of mass for:

[0057] In the formula, Indicates the number of valid sampling points. The traversal index for summation, starting from... Received The straight line fitting is achieved using the least squares method or singular value decomposition, and its input is as described above. coordinates of a spatial point The output is a spatial straight-line direction vector that best represents the direction of laser propagation. .

[0058] Furthermore, the end-tool coordinate system The foot of the perpendicular from the origin to the fitted optical axis is defined as the laser coordinate system. Origin position :

[0059] In the formula, Indicates transpose; This represents the projected length of the centroid position vector along the optical axis. This definition ensures that the origin of the laser coordinate system lies on the fitted optical axis and guarantees that the origin of the laser coordinate system has a definite geometric meaning relative to the origin of the tool coordinate system.

[0060] During the laser optical axis calibration stage, three ceramic calibration target plates of different thicknesses are placed sequentially in front of the end laser. The target plate depths are 120mm, 160mm, and 200mm respectively (in the camera coordinate system). of (Using the axis as a reference), the end laser illuminates each target plate in low-power continuous wave mode. The end depth camera acquires images of the light spots and extracts the coordinates of the center pixel of the light spots. After back projection, the coordinates of each illumination point in the camera coordinate system are obtained. The three-dimensional position is then transformed to the end-tool coordinate system via extrinsic parameter transformation. A total of One valid sampling point.

[0061] Table 3. Sampling and conversion data of laser irradiation points

[0062] For the end tool coordinate system in the table above Singular value decomposition and line fitting are performed on the three points below, and the centroid is... mm, the fitted unit direction vector is The calibration results are consistent with those of step S100, verifying the internal consistency of the two-stage calibration.

[0063] Step S400: Transform the target point to the robot arm's base coordinate system and calculate the current laser emission point position and laser optical axis direction.

[0064] In step S400, based on the end-effector coordinate system Relative to the robot arm's base coordinate system Current pose The three-dimensional position of the target point and the laser installation parameters are uniformly converted to the robot arm's base coordinate system. The target point is generated in the robot arm's base coordinate system. The lower position Current laser emission point location and the direction of the laser optical axis .

[0065] Obtain the coordinate system of the end tool Relative to the robot arm's base coordinate system Current pose The target point is transformed to the end-tool coordinate system by utilizing the extrinsic relationship between the camera and the tool. Down:

[0066] In the formula, Indicates the target point In the end tool coordinate system The three-dimensional position vector below. Further, the target point is moved from the end-tool coordinate system. Transform to the robot arm base coordinate system The following steps are performed to obtain the current position of the target point in the global motion space. :

[0067] Based on the established tool and laser extrinsic parameters, the laser coordinate system The origin is in the end tool coordinate system The position below is Calculate the current laser emission point in the robot arm's base coordinate system. The lower position :

[0068] At the same time, according to the laser optical axis direction Calculate the direction of the laser optical axis In the robot arm's base coordinate system Current direction :

[0069] The laser optical axis direction obtained is determined by two rotational relationships: one is the installation posture of the laser relative to the end effector, and the other is the current posture of the end effector relative to the robot arm's base coordinate system.

[0070] It should be noted that after completing the above calculations, the relative positional relationship between the target point and the current laser optical axis can be obtained. Let the vector from the current laser emission point to the target point be... Then the perpendicular error vector from the target point to the current laser optical axis is:

[0071] In the formula, This represents the lateral deviation vector of the target point relative to the current laser optical axis; Indicates transpose; This represents the projection length of the target point's relative vector along the laser optical axis. If... and If they are collinear, it means that the current laser axis has passed through the target point; if they are not collinear, it means that there is still a lateral deviation between the laser axis and the target point, and the end effector's axis-corrected pose needs to be solved in subsequent steps.

[0072] The current state feedback of the robotic arm provides the coordinate system of the end effector. In the robot arm's base coordinate system The current position vector below mm, current orientation is approximately a unit rotation (end-of-care tool) (axis vertically downwards), that is (A third-order identity matrix) is used to simplify the explanation of the numerical transfer relationships between the variables.

[0073] The result obtained in step S200 mm and the result obtained in step S100 Substitute mm into the coordinate system of the end tool; the target point is in the coordinate system of the end tool. The following position is:

[0074] (Here, the rotation matrix is ​​taken as the identity matrix, and the vectors are directly added.) The target point is in the robot arm's base coordinate system The following position:

[0075] The current laser emission point is in the robot arm's base coordinate system. The following position:

[0076] Current laser optical axis direction (The direction remains unchanged when the rotation matrix is ​​an identity matrix).

[0077] Lateral deviation vector The calculations show that the target point is not on the current laser optical axis, with a lateral deviation of about 38.1 mm. Therefore, it is necessary to proceed to step S500 to solve for the axis pose.

[0078] Table 4 Summary of key point locations in the base coordinate system

[0079] Step S500: Solve for the desired pose of the end laser relative to the axis based on the joint geometric constraints of the camera and laser.

[0080] In step S500, based on the target point in the robot arm's base coordinate system The lower position Laser optical axis in end-effector coordinate system Downward direction Position of the origin of the laser coordinate system In addition to the relationship between the camera and the tool's extrinsic parameters, the alignment constraint of the laser optical axis passing through the target point and the depth constraint of the target point maintaining a preset depth relative to the camera are jointly established, and the coordinate system of the end-tool is solved. Laser-axis desired pose Specifically, it includes the following sub-steps: a) Determine the desired orientation and calculate the desired directions of the laser optical axis and the camera depth axis. Let the desired orientation of the laser-axis-aligned end-effector be... superscript This represents the desired amount of laser beam along the axis. In one example, Get the current end-effector pose ,Right now The end effector's orientation remains unchanged. Under this desired orientation, the laser optical axis lies in the robot arm's base coordinate system. The expected direction is:

[0081] In the formula, This indicates the position of the laser optical axis in the robot arm's base coordinate system under the desired laser-axis pose. The unit direction vector below. The camera depth axis in the end-tool coordinate system. Downward unit direction It is obtained from the rotation relationship between the camera and the tool's extrinsic parameters, i.e. ,in This indicates transpose, with the camera depth axis in the robot arm's base coordinate system. The expected direction is:

[0082] In the formula, This represents the unit direction vector of the camera depth axis in the robot arm's base coordinate system under the desired pose of the laser axis.

[0083] b) Establish alignment constraints where the laser optical axis passes through the target point. If the target point is located on the laser optical axis, then the target point... The desired laser emission point and laser optical axis direction should satisfy:

[0084] In the formula, It represents the directed distance from the laser emission point along the laser optical axis to the target point, with the dimension of length; This represents the desired position of the end effector. All terms in this formula are position vectors with length dimensions, and the dimensional relationships hold. This formula indicates that the target point should be located on a straight line formed by extending the laser emission point along the laser optical axis.

[0085] Furthermore, The physical meaning is: when At that time, the target point is located in front of the laser emission point along the laser emission direction, that is, on the positive extension side of the laser optical axis; when When the target point is located on the opposite side of the laser emission direction, this situation usually does not meet the physical feasibility in actual operation and should be verified in the subsequent solution results.

[0086] c) Establish camera depth constraints. Let the desired camera depth be... ,in , representing the distance from the target point along the camera depth axis to the camera origin under the desired laser-axis pose, and the following must be satisfied under the desired laser-axis pose:

[0087] d) Solve for the directed distance by simultaneously applying two types of constraints. Substitute the alignment constraint into the depth constraint to eliminate... The solution is as follows:

[0088] In the formula, Indicates transpose; in the molecule The desired camera depth is expressed in units of length. The denominator is the dot product of the unit direction vector and the length vector, with the dimension being length; The dot product of two unit direction vectors, denoted by a dimensionless scalar, represents the directional projection relationship between the camera depth axis and the laser optical axis; overall result. The dimension is length, consistent with the definition of directed distance, and the dimensional relationship holds. When the absolute value of this denominator is too small, it indicates that the camera's depth axis is approximately perpendicular to the laser's optical axis, and the depth constraint cannot be stably determined. If this condition is not met, the pose solution conditions should be determined. In at least one example of the embodiments of this disclosure, a directional projection threshold can be set. ,in ,when The solution will terminate and an exception will be reported.

[0089] Furthermore, The value of should be predetermined based on the angle range between the camera depth axis and the laser optical axis in the actual installation structure. Its physical meaning is: when the absolute value of the projection of the two axes is less than , At that time, camera depth constraints The constraint sensitivity approaches zero, and the solution is extremely sensitive to small perturbations in the input parameters. Therefore, it is used in... As a criterion for numerical stability.

[0090] e) Determine the desired position of the end effector based on the alignment constraints and combine them with the desired pose. After obtaining... Then, reverse the process to find the expected position of the end tool:

[0091] Therefore, the desired pose of the end laser axis. Represented as:

[0092] End-effector laser-oriented axis desired pose It simultaneously satisfies two geometric requirements: the laser optical axis passes through the target point, and the target point maintains a preset depth relationship with the camera.

[0093] Perform axis pose solving on the PAD-037 pad target point, with the desired camera depth set to... mm (consistent with the intermediate target plate depth during calibration in step S300, to ensure observation stability). The rotation matrix is ​​taken as the identity matrix, then... , (The camera depth axis is approximately parallel to the end tool) axis).

[0094] denominator Much greater than the threshold The solution conditions are satisfied.

[0095] In molecules of Components mm, In the rotation matrix, the identity matrix is ​​taken only. The component contributes to the inner product, and the result is: mm, substituting into the formula:

[0096] The target point is located in front of the laser emission direction, which is physically feasible. The desired position of the end effector is then calculated (using the identity matrix as the rotation matrix). ):

[0097]

[0098]

[0099]

[0100] Table 5 shows the key intermediate quantities for axis pose determination.

[0101] Step S600: Output the desired pose of the end effector and perform laser-axis motion.

[0102] In step S600, the end laser is aligned to the desired pose along the axis. As the target output for the motion control of the robotic arm, it drives the end effector of the robotic arm to move from its current pose to the desired pose of the laser-aligned axis. Align the laser optical axis with the target point Satisfy the preset alignment relationship.

[0103] For example, calculating the expected translation of the end tool. The desired translation amount With expected posture As the target input for the robotic arm's trajectory planning, the robotic arm executes the motion accordingly.

[0104] It should be noted that the desired translation amount can be limited before outputting it. If... Exceeding the preset maximum single movement distance Then scale the desired translation amount proportionally: The corresponding target position after limiting is ;when At that time, take directly This process can prevent the robotic arm from making unreasonable large movements due to sudden changes in the target point, depth anomalies, or excessively large solution results.

[0105] After the robotic arm completes the laser-axis alignment, it reacquires the actual pose of the end effector and calculates the position of the laser emission point after the movement, following the procedure in step S400. and the direction of the laser optical axis Calculate the lateral error from the target point to the laser optical axis. :

[0106] In the formula, Indicates transpose. If satisfied If so, it is determined that the laser optical axis has been aligned. This indicates the permissible laser alignment error threshold. After the axis error meets the requirements, the control end laser performs the target action according to the preset power and working mode.

[0107] In this embodiment of the disclosure, in order to further improve the alignment accuracy when the accuracy requirements are not met in a single axis movement, based on step S600, if the lateral error does not meet the requirements... Then, based on the current observation results, steps S400 to S600 are re-executed to perform one or more iterative corrections until the axis accuracy requirements are met or the preset maximum number of iterations is reached.

[0108] In this embodiment of the disclosure, to improve the observation stability when the camera approaches the target in an open-loop manner, the following steps are included after step S200 and before step S300: solving the camera's open-loop approach pose based on the geometric relationship between the camera and the tool, and performing the camera's open-loop motion. Specifically, this is achieved by utilizing the extrinsic parameters of the camera and the tool, and the target point in the camera coordinate system. The three-dimensional position below and end-tool coordinate system Relative to the robot arm's base coordinate system Current pose Transform the target point to the robot arm's base coordinate system Down: Let the preset camera observation distance be... ,in Let the camera's central axis lie in the camera coordinate system. The unit direction vector below is ,in Indicates transpose. Let the desired rotation matrix of the end-effector after the camera's open-loop approach be... superscript This represents the expected value during the open-loop approach phase, and the corresponding camera coordinate system. Relative to the robot arm's base coordinate system The expected rotation matrix is Based on the geometric requirement that the target point is located in front of the camera's central axis while maintaining a preset observation distance, the desired position of the camera origin after open-loop approach is... Should meet Therefore, the open-loop end effector is close to the desired position: The camera's open-loop approach pose is Camera open-loop approach pose As the target input for the open-loop motion of the robotic arm, the end-effector camera is first moved to the vicinity of the target and the target point is kept observable within the camera's field of view and depth range.

[0109] In this embodiment of the disclosure, in order to further reduce the pixel deviation and depth deviation of the target point after open-loop motion, and to ensure that the subsequent laser axis-to-axis solution is based on stable target observation, the following step is further included after the camera's open-loop approach is completed: performing visual closed-loop correction. Let the target point during the closed-loop correction process be... The expected pixel coordinates are The real-time pixel coordinates of the target point are The real-time depth of the target point is Expected depth is Define the visual error vector for:

[0110] in, This represents the pixel deviation of the target point in the horizontal direction of the image. This represents the pixel deviation of the target point in the vertical direction of the image. This represents the depth deviation of the target point relative to the desired observation depth. Visual error vector. The first two terms are in pixels, and the third term is in length. Since these three terms have different dimensions, the visual error vector... It is only used to determine the convergence condition for independent component analysis; it does not perform cross-component addition, subtraction, or norm operations. Based on the visual error vector... Controlling the end effector of the robotic arm to generate corrective motion to reduce the visual error vector Gradually decrease. Let the planar pixel error of the target point relative to the desired pixel position be... If in consecutive Simultaneously satisfy within each control cycle and If the closed-loop calibration is completed, then the closed-loop calibration is considered complete. This indicates the preset number of consecutive stable periods. This indicates the allowable pixel error threshold. This indicates the allowable depth error threshold. , .

[0111] The robot end-effector laser alignment control system provided in at least one embodiment of this disclosure includes at least one processor, a vision perception unit, a robotic arm, an end-effector camera, an end-effector laser, a calibration unit, a pose solving unit, and a motion control unit. For a detailed description of the vision perception unit, please refer to the relevant description of step S200 in the embodiments of the end-effector laser alignment pose solving method based on camera and laser joint geometric constraints described above; it will not be repeated here. For a detailed description of the calibration unit, please refer to the relevant descriptions of steps S100 and S300; it will not be repeated here. For a detailed description of the pose solving unit, please refer to the relevant descriptions of steps S400 and S500; it will not be repeated here. For a detailed description of the motion control unit, please refer to the relevant description of step S600; it will not be repeated here. The technical effects of this robot end-effector laser alignment control system are the same as those of the aforementioned end-effector laser alignment pose solving method based on camera and laser joint geometric constraints; they will not be repeated here.

[0112] The expected translation is:

[0113]

[0114] Preset maximum single movement distance mm, mm No bandwidth limit required, directly using The robot arm motion control unit serves as the target input. After the robot arm performs the motion, the axis error threshold is set to... mm. After the actual motion is completed, the end effector pose is reacquired, and the laser emission point and optical axis direction are recalculated according to step S400 to obtain the lateral error. mm, satisfying mm, indicating alignment is complete, the laser then performs pulsed laser soldering on the PAD-037 pads according to the preset peak power and pulse width.

[0115] Table 6 Results of shaft motion execution and error verification

[0116] This implementation incorporates the camera coordinate system, end-effector coordinate system, laser coordinate system, and robot arm base coordinate system into a unified geometric transformation framework. Steps S100 to S300 respectively obtain the extrinsic parameters between the camera and the tool, as well as the laser optical axis direction and the position of the laser coordinate system origin. This ensures that the laser emission point offset and optical axis skew can be accurately expressed in subsequent solutions, rather than simplifying the laser actuator to the tool coordinate system origin or equating the laser optical axis to the tool coordinate axis. Therefore, the systematic alignment errors caused by installation deviations and optical axis skew can be compensated for during the pose calculation stage.

[0117] In step S500, an alignment constraint where the laser optical axis passes through the target point and a depth constraint where the target point maintains a preset depth relative to the camera are jointly established, ensuring that the obtained end-effector desired pose simultaneously satisfies the laser action direction requirement and the camera's effective observation distance requirement. Compared to end-effector position compensation based solely on the target point's spatial coordinates or a fixed offset, this joint constraint enables directional distance... The solution is constrained by the camera depth condition, thus avoiding unreasonable changes in the observation distance between the camera and the target when the optical axis passes through the target condition.

[0118] Through optional camera open-loop approach and visual closed-loop correction, the pixel and depth deviations of the target point are reduced before the laser axis solution, resulting in higher observation stability of the 3D position of the target point used for coordinate transformation in step S400. This, in turn, ensures that the joint constraint solution in step S500 is based on reliable input data. Through lateral error verification and iterative correction in step S600, gradual approximation can be performed when the accuracy requirements are not met in a single axis movement, thereby improving the geometric consistency and execution reliability of vision-guided laser operations.

[0119] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A method for solving the laser pair axis pose based on visual geometric constraints, characterized in that, Includes the following steps: Obtain the rigid body transformation relationships between the end-effector coordinate system and the camera coordinate system and the laser coordinate system, respectively; The three-dimensional position of the target point in the camera coordinate system is obtained based on the visual perception unit; Based on the coordinates of multiple laser irradiation points in the end-tool coordinate system, a straight line fit is performed to generate the unit direction vector of the laser optical axis in the end-tool coordinate system and the position of the origin of the laser coordinate system. Transform the target point position and laser installation parameters to the robot arm's base coordinate system; By jointly establishing the alignment constraint of the laser optical axis passing through the target point and the depth constraint of the target point maintaining a preset depth relative to the camera, the desired pose of the laser about the axis is solved. as well as The laser-aligned desired pose is output to the motion control unit, driving the robotic arm end effector to move to the desired pose. The process of solving the desired laser orientation includes: determining the desired orientation and calculating the desired directions of the laser optical axis and the camera depth axis in the robot arm base coordinate system; solving the directed distance from the laser emission point to the target point by combining the alignment constraints and depth constraints; and calculating the desired position of the end effector based on the directed distance and combining them into the desired orientation.

2. The laser axis pose solving method based on visual geometric constraints according to claim 1, characterized in that, The linear fitting based on the coordinates of multiple laser illumination points in the end-effector coordinate system includes: The end laser is controlled to sequentially irradiate the calibration target surface located at different spatial positions. The end camera acquires images under each irradiation state and extracts the pixel coordinates and corresponding depth values ​​of the laser spot center. The three-dimensional position of each laser irradiation point in the camera coordinate system is obtained by back projection through the camera pinhole imaging model. The three-dimensional positions of each laser irradiation point are transformed to the end-tool coordinate system using the rotation matrix and position vector of the camera coordinate system relative to the end-tool coordinate system; The transformed spatial point set is fitted with a spatial line based on the criterion of minimizing the sum of the vertical distances from each sampling point to the fitted line. The output of the spatial line fitting is the unit direction vector of the laser optical axis in the end-tool coordinate system. The foot of the perpendicular from the origin of the end-tool coordinate system to the fitted optical axis is determined as the origin of the laser coordinate system. The foot of the perpendicular is obtained by subtracting the projection component of the centroid in the optical axis direction from the centroid of all laser irradiation points in the end-tool coordinate system.

3. The laser axis pose solving method based on visual geometric constraints according to claim 1, characterized in that, The process of obtaining the three-dimensional position of the target point in the camera coordinate system based on the visual perception unit includes: Obtain the pixel coordinates and corresponding depth value of the target point in the image plane; Based on the camera intrinsic parameter matrix and the camera pinhole imaging model, the pixel coordinates and depth values ​​of the target point are back-projected onto the camera coordinate system to obtain the coordinate components of the target point along each axis of the camera coordinate system. Each axis coordinate component is obtained by multiplying the difference between the pixel coordinates and the principal point coordinates by the depth value and then dividing by the equivalent focal length in the corresponding direction. The depth direction coordinate component is the depth value itself.

4. The laser axis pose solving method based on visual geometric constraints according to claim 1, characterized in that, Solving the directed distance from the laser emission point to the target point by simultaneously applying the alignment constraints and depth constraints includes: The alignment constraint is expressed as the position of the target point in the robot arm base coordinate system being equal to the desired position of the end tool plus the offset of the origin of the laser coordinate system in the robot arm base coordinate system under the desired attitude plus the product of the directional distance and the desired direction of the laser optical axis. The depth constraint is expressed as the position of the target point in the robot arm base coordinate system being equal to the desired position of the end tool plus the offset of the camera coordinate system origin in the robot arm base coordinate system under the desired attitude plus the product of the preset camera depth and the desired direction of the camera depth axis. The alignment constraint and depth constraint are combined to eliminate the desired position of the end tool, and the directional distance is equal to the preset camera depth minus the inner product of the position difference between the desired direction of the camera depth axis and the laser coordinate system origin and the camera coordinate system origin in the end tool coordinate system after rotation of the desired attitude, and then divided by the inner product of the desired direction of the camera depth axis and the desired direction of the laser optical axis.

5. The laser axis pose solving method based on visual geometric constraints according to claim 4, characterized in that, The method further includes: Before solving the directed distance, it is determined whether the absolute value of the inner product of the desired direction of the camera depth axis and the desired direction of the laser optical axis is less than a preset direction projection threshold. If it is less than the direction projection threshold, the solution is terminated and an anomaly is reported. After obtaining the depth value of the target point, it is determined whether the depth value is within the range between the preset minimum effective depth value and the maximum effective depth value. If the range is not met or any component of the three-dimensional coordinates of the target point is not a finite real number, the target point is determined to be invalid and will not proceed to the subsequent pose solving process.

6. The laser pair axis pose solving method based on visual geometric constraints according to claim 1, characterized in that, After obtaining the 3D position of the target point in the camera coordinate system and before performing line fitting, the following steps are also included: The 3D position of the target point is transformed to the robot arm's base coordinate system using the relationship between the camera and tool extrinsic parameters and the current pose of the end-effector. Based on the preset camera observation distance, the unit direction of the camera depth axis in the camera coordinate system, and the geometric requirement that the target point is located in front of the camera center axis and maintains the preset observation distance, the expected position of the end tool during the camera open-loop approach phase is solved. The expected position of the end tool is equal to the position of the target point in the robot arm base coordinate system minus the component of the preset observation distance along the expected direction of the camera depth axis, and then minus the component of the offset of the camera coordinate system origin in the end tool coordinate system after rotation by the expected attitude. The desired pose of the camera during the open-loop approach phase is used as the target input for the robotic arm's motion, causing the end-effector camera to move to the vicinity of the target.

7. The laser axis pose solving method based on visual geometric constraints according to claim 6, characterized in that, After the camera's open-loop approach is complete, visual closed-loop correction is also performed. Obtain the pixel deviation between the real-time pixel coordinates of the target point and the preset expected pixel coordinates, as well as the depth deviation between the real-time depth of the target point and the expected depth; The robotic arm end effector is controlled to generate corrective motion based on the pixel deviation and depth deviation. Calculate the planar pixel error of the target point relative to the desired pixel position. If the planar pixel error is not greater than the allowable pixel error threshold and the absolute value of the depth deviation is not greater than the allowable depth error threshold within a consecutive preset number of stable periods, then the closed-loop correction is determined to be complete.

8. The laser pair axis pose solving method based on visual geometric constraints according to claim 1, characterized in that, After driving the robotic arm end effector to the desired pose, the process also includes: Calculate the expected translation amount of the end effector. If the norm of the expected translation amount exceeds the preset maximum single movement distance, then scale the expected translation amount to the maximum single movement distance proportionally. After the robotic arm completes its movement, the actual pose of the end effector is reacquired, and the lateral error from the target point to the laser optical axis after the movement is calculated. The lateral error is obtained by subtracting the projection component of the vector in the direction of the laser optical axis after the movement from the vector between the target point and the laser emission point after the movement. If the norm of the lateral error is greater than the allowable laser alignment error threshold, then coordinate transformation and desired pose solution are re-executed based on the current observation results until the lateral error meets the accuracy requirements or reaches the preset maximum number of iterations.

9. A laser-axis pose control system based on visual geometric constraints, used to execute the laser-axis pose solving method based on visual geometric constraints according to any one of claims 1 to 8, characterized in that, include: The calibration unit is configured to obtain the rigid body transformation relationship between the end-effector coordinate system and the camera coordinate system and the laser coordinate system, respectively, and to perform linear fitting based on the coordinates of multiple laser irradiation points in the end-effector coordinate system to generate the laser optical axis direction and the position of the origin of the laser coordinate system. The visual perception unit is configured to acquire the three-dimensional position of the target point in the camera coordinate system; The pose solving unit is configured to transform the target point position and laser installation parameters to the robot arm base coordinate system, and jointly establish the alignment constraint of the laser optical axis passing through the target point and the depth constraint of the target point maintaining a preset depth relative to the camera to solve the laser's desired pose about the axis; The motion control unit is configured to output the desired laser-axis pose as the target motion control for the robotic arm, and drive the end effector of the robotic arm to move to the desired pose.

Images