Intelligent crane pipe hand-eye calibration method, system and equipment based on conformal reference field and medium

Abstract

The invention provides an intelligent crane pipe hand-eye calibration method, system and equipment based on a conformal reference field and a medium, and belongs to the technical field of crane pipe intelligent calibration. A reference ring containing a multicolor concentric ring belt is designed and prepared and installed at the tail end of a crane pipe, and a kinematic model is constructed to describe the pose relation between the tail end and a base; building a conformal reference field, and driving a crane pipe to move to form a reference ring array covering a working space; detecting the position of the reference ring and synchronously recording pose information; calibrating external parameters of the camera, and solving a coordinate system transformation relation between the camera and the reference ring; calculating a transformation matrix of the reference ring and the base by using a kinematic model; finally, the matrixes are connected in series, and the pose mapping relation from the camera to the crane pipe base is determined. Visual identification precision is improved through a multi-color concentric ring belt, calibration robustness is enhanced by covering a working space through a conformal reference field, high-precision pose mapping is realized through fusion of a kinematic model and visual calibration, and automatic operation efficiency and reliability are improved.

Description

Technical Field

[0001] This invention belongs to the field of intelligent calibration technology for loading arms, specifically relating to an intelligent hand-eye calibration method, system, device, and medium for loading arms based on a conformal reference field. Background Technology

[0002] In the chemical liquid transportation industry, tank trucks are the core transportation tool, and the accuracy and efficiency of their filling operations are directly related to the safety and economy of chemical liquid logistics.

[0003] In related technologies, hand-eye calibration uses a black and white checkerboard pattern or a metal reference sphere. However, due to the large fluctuations in lighting in industrial settings, the reflectivity of metal, and color crosstalk, calibration accuracy is affected. For example, when the lighting changes, the edges of the black and white checkerboard become blurred, leading to overexposure under strong light and underexposure under weak light, resulting in a high error rate in feature point extraction. Multiple positioning operations are required when installing multiple reference spheres, resulting in large coaxiality deviations and causing 3D-2D point-to-point matching errors in extrinsic parameter calibration. Ultimately, the calibration error exceeds the allowable range, failing to meet the high-precision docking requirements of loading arms.

[0004] In related technologies, multi-degree-of-freedom models are used, requiring the identification of parameters such as link offsets, rotation angles, and joint clearances. However, to simplify this process, single-joint models are employed, neglecting the coupling effects of multi-joint coordinated motion. It can be seen that the multi-parameter models in these technologies have high Jacobian matrix dimensions, leading to an exponential increase in the number of iterations during parameter identification, making real-time calculations in industrial settings difficult. Single-joint models, by ignoring joint coupling, increase pose calculation errors, resulting in distortion of the reference loop pose transfer and affecting the accuracy of subsequent hand-eye mapping.

[0005] In related technologies, extrinsic parameter calibration using the poses of 3-5 reference points can easily lead to non-compliance with coplanarity and distance invariance. Accumulated errors from multi-image stitching result in a rotation matrix deviation ≥0.02 rad and a translation vector deviation ≥0.005 m between the camera coordinate system and the reference ring coordinate system. These errors are further amplified during subsequent hand-eye mapping, ultimately making it difficult to meet the requirements for loading arm docking deviation. Summary of the Invention

[0006] This invention proposes a hand-eye calibration method for intelligent loading arms based on a conformal reference field, aiming to greatly simplify the calibration process and significantly improve the absolute positioning accuracy of intelligent loading arm filling operations. This method calibrates the transformation matrix from the camera coordinate system to the loading arm base coordinate system by visually assisting in measuring the relative motion distance of the end-effector reference ring. This bypasses the reliance on high-precision measuring tools, allowing even non-professionals to easily perform the calibration work, thus improving the absolute positioning accuracy of the intelligent loading arm end effector.

[0007] The methods include: S101: Design and fabricate a reference ring for visual recognition, the reference ring comprising multiple concentric colored rings having different colors and sizes, and mounted on the end of a smart loading arm; S102: Construct the kinematic model of the intelligent loading arm, and describe the pose mapping relationship between the end effector of the robotic arm and the base coordinate system by defining the joint coordinate system and geometric parameters; S103: Construct a conformal reference field, fix multiple reference rings to the end of the intelligent loading arm, drive the loading arm to move to multiple typical poses in the joint space according to a preset trajectory, and form a reference ring array covering the work space. S104: Acquire image data and detect reference rings, control the ambient lighting and camera parameters, use target recognition methods to extract the position information of each reference ring from the image, and simultaneously record joint angles and loading arm pose data. S105: Calibrate camera extrinsic parameters. Based on the known world coordinates of the reference ring in the conformal reference field and the 2D position of the image detection, solve the rotation matrix and translation vector of the camera coordinate system relative to the reference ring coordinate system. S106: Obtain the pose relationship between the reference ring and the loading arm, and use the kinematic model of the intelligent loading arm to calculate the homogeneous transformation matrix of the reference ring coordinate system relative to the loading arm base coordinate system. S107: Determine the pose mapping from the camera to the loading arm base. By concatenating the transformation matrices between the camera and the reference loop, and between the reference loop and the loading arm base, obtain the pose transformation relationship from the camera coordinate system to the loading arm base coordinate system.

[0008] This application also provides an intelligent arm-mounted loading arm hand-eye calibration system based on a conformal reference field, the system comprising: A reference ring preparation module is used to design and prepare a reference ring for visual recognition. The reference ring includes multiple concentric colored rings with different colors and sizes, and is installed at the end of an intelligent loading arm. The kinematic modeling module is used to construct the kinematic model of the intelligent loading arm. By defining the joint coordinate system and geometric parameters, it describes the pose mapping relationship between the end effector of the robotic arm and the base coordinate system. The reference field construction module is used to construct a conformal reference field, fix multiple reference rings to the end of the intelligent loading arm, and drive the loading arm to move to multiple typical poses in the joint space according to a preset trajectory, forming a reference ring array covering the workspace. The image acquisition and detection module is used to acquire image data and detect reference rings, control the ambient lighting and camera parameters, extract the position information of each reference ring from the image using target recognition methods, and simultaneously record joint angle and loading arm pose data. The camera extrinsic parameter calibration module is used to calibrate the camera extrinsic parameters. Based on the known world coordinates of the reference ring in the conformal reference field and the 2D position of the image detection, it solves the rotation matrix and translation vector of the camera coordinate system relative to the reference ring coordinate system. The pose calculation module is used to obtain the pose relationship between the reference ring and the loading arm. Using the kinematic model of the intelligent loading arm, it calculates the homogeneous transformation matrix of the reference ring coordinate system relative to the loading arm base coordinate system. The hand-eye pose mapping module is used to determine the pose mapping from the camera to the loading arm base. By concatenating the transformation matrices between the camera and the reference loop, and between the reference loop and the loading arm base, the pose transformation relationship from the camera coordinate system to the loading arm base coordinate system is obtained.

[0009] According to another embodiment of this application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the intelligent loading arm hand-eye calibration method based on a conformal reference field.

[0010] According to another embodiment of this application, a storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the intelligent loading arm hand-eye calibration method based on a conformal reference field.

[0011] As can be seen from the above technical solutions, the present invention has the following advantages: This invention provides an intelligent loading arm hand-eye calibration method based on a conformal reference field. This method combines red, green, and blue concentric rings and isolation slots to suppress color crosstalk. After CIE LAB calibration, it maintains stable recognition under different lighting conditions. The DH theory simplifies the double rotary joint model, balancing model accuracy and computational complexity. It avoids parameter redundancy in multi-degree-of-freedom models and ensures reduced pose calculation errors through orthogonal coordinate system standardization. A single reference ring and loading arm motion replace the physical placement of multiple reference rings, avoiding coaxiality errors during installation. Nine typical poses are distributed in a square grid, forming strong spatial constraints and providing rich and reliable 3D-2D point pairs for extrinsic parameter calibration. Polarizing filters suppress metallic reflections, and uniform diffuse light sources eliminate highlights / shadows, ensuring stable image quality. An improved YOLOv5s model enables rapid and accurate detection of the reference ring, avoiding point pair matching errors caused by image blurring or missed detections.

[0012] The IPPE algorithm of this invention quickly acquires initial extrinsic parameters, and the LM algorithm optimizes reprojection errors, ensuring a reliable rigid transformation relationship between the camera and the reference ring coordinate system. A coordinate measuring machine is used to confirm the offset deviation between the reference ring and the end effector. Combined with a kinematic model calculation, a direct pose mapping between the reference ring and the loading arm base is established, avoiding error propagation caused by relative displacement. Direct docking of the visual coordinate system and the mechanical coordinate system is achieved through matrix concatenation. Consistency verification reduces random errors, ultimately achieving accurate conversion of image features to 3D coordinates in the base coordinate system. Attached Figure Description

[0013] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 The flowchart shows the intelligent loading arm hand-eye calibration method based on a conformal reference field. Figure 2 A schematic diagram of the three-dimensional model of the reference ring; Figure 3 This is a schematic diagram of the link coordinate system; Figure 4 This is a schematic diagram of the image data; Figure 5 A schematic diagram of the target recognition results; Figure 6 A schematic diagram of the conformal reference field; Figure 7 This is a schematic diagram of an electronic device. Detailed Implementation

[0015] This invention provides a hand-eye calibration method for intelligent loading arms based on a conformal reference field, addressing the problem of insufficient relative pose accuracy between the camera coordinate system and the loading arm coordinate system in chemical liquid transportation. It constructs a conformal reference field by using a reference ring at the loading arm's end according to specified rules, acquires images of this field using a vision system, and achieves rapid target identification using a deep learning model. Finally, it calculates the transformation matrix between the camera coordinate system and the loading arm coordinate system through camera extrinsic parameter calibration techniques. This method bypasses the complex homogeneous matrix equation solving in traditional hand-eye calibration, reducing calibration complexity and enabling calibration of both the vision and mechanical systems.

[0016] This invention establishes the transformation matrix from the end of the loading arm to its base through articulated kinematic chains. A conformal reference field is constructed by repeatedly and regularly moving the position of the reference ring at the end of the loading arm; the feature points of this field form the corner points of a circular checkerboard. The transformation matrix between the camera coordinate system and the conformal reference field coordinate system is calculated using camera extrinsic parameter calibration technology. Then, the transformation matrix between the camera coordinate system and the loading arm coordinate system is calculated using the rigid body transformation principle of the hand-eye system. Application results show that this method can significantly improve the relative pose accuracy between the camera coordinate system and the loading arm coordinate system, shorten calibration time and reduce safety risks, providing an efficient and low-cost calibration solution for liquid loading and unloading systems in the chemical industry.

[0017] The following describes in detail the intelligent loading arm hand-eye calibration method based on a conformal reference field. Specific details, such as particular system structures and techniques, are presented for illustrative purposes and not for limitation, to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application can also be implemented in other embodiments without these specific details.

[0018] It should be understood that, when used in this specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0019] The terms "one embodiment" or "some embodiments" used in this application mean that one or more embodiments of this application include the specific features, structures, or characteristics described in that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this application do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] Please see Figure 1 The diagram shows a flowchart of a smart loading arm hand-eye calibration method based on a conformal reference field in a specific embodiment. The method includes: S101: Design and fabricate a reference ring for visual recognition, the reference ring comprising multiple concentric colored bands having different colors and sizes, and mounted on the end of a smart loading arm.

[0022] In some embodiments, the design of the loading arm reference ring incorporates, for example... Figure 2 The three-dimensional model shown serves as the core calibration carrier of the visual recognition system. It adopts a high-precision concentric ring topology configuration, with three high-saturation spectral feature rings arranged from the inside out: red (R: wavelength 620-750nm), green (G: wavelength 495-570nm), and blue (B: wavelength 450-495nm).

[0023] The dimensions of the reference rings in this embodiment are shown in Table 1. The diameters of each color ring strictly follow a geometric series ratio design to achieve multi-scale feature decoupling. The substrate is made of 304 austenitic stainless steel (standard: GB / T 20878-2007) and precision CNC machined to ensure a surface roughness Ra≤0.8μm to guarantee optical reflection consistency. A 0.5mm wide mirror-polished isolation groove is provided between the rings to effectively suppress color crosstalk. The three-dimensional structure adopts a hollow and lightweight design, with an integrated M6 standard threaded mounting interface (tolerance grade 6H) on the back side to meet the rigid assembly requirements of industrial scenarios. This multi-color concentric structure is quantized and calibrated using the CIE LAB color space, with a color difference ΔE*ab<1.5 (measured under D65 light source), ensuring robust color recognition under different lighting conditions.

[0024] Table 1: Dimensions of the reference ring band

[0025] The reference ring serves as the spatial reference for the visual servoing system. A scale-invariant feature transform (SIFT) descriptor is constructed using the diameter ratios of the multi-color rings, achieving an attitude angle measurement accuracy of 0.05° at VGA resolution. In dynamic tracking scenarios, the frequency domain features of the three color rings (low-frequency red / high-frequency blue) support Kalman filtering for multi-target tracking, with a position update rate of up to 200Hz. Through a built-in diameter-color encoding mechanism, the system can automatically distinguish installation postures and compensate for eccentricity errors. This design has been successfully applied to the guidance system of intelligent loading arms, improving anti-interference capability by 20% compared to traditional calibration boards.

[0026] As can be seen, this embodiment utilizes the spectral differences and geometric series design of the three-color rings to provide scale-invariant features for the vision system; high-saturation colors and isolation grooves suppress color crosstalk, and the stainless steel substrate ensures mechanical rigidity and reflectivity consistency, so that the reference ring can still be stably identified under different lighting and poses.

[0027] S102: Construct the kinematic model of the intelligent loading arm, and describe the pose mapping relationship between the end effector of the robotic arm and the base coordinate system by defining the joint coordinate system and geometric parameters.

[0028] In some embodiments, a smart loading arm is a device used for loading and unloading liquids or gases, typically installed in tank farms, docks, or loading / unloading stations. The main structural components of a smart loading arm include: a rotary joint, a column, an inner arm, an outer arm, a drop arm, a cylinder, a sealing device, a flange, a reference ring, and other auxiliary components.

[0029] Constructing a kinematic model of the intelligent loading arm is the initial step in system calibration. Essentially, it involves establishing a mathematical representation by analyzing the geometric structure and motion characteristic parameters of the mechanical system. The core function of this model is to accurately describe the spatial transformation relationship between the robotic arm's end effector and the base coordinate system.

[0030] Differences in modeling methods directly affect two key indicators: first, the number of parameters required for identification varies significantly across different theoretical frameworks; second, the model's fitting accuracy to real motion exhibits a positive correlation. It is worth noting that while increasing the parameter dimension can improve the model's ability to capture the nonlinear characteristics of mechanical systems, it significantly increases the computational complexity of the parameter identification process, specifically manifested in an exponential increase in the number of iterations due to the expansion of the Jacobian matrix dimension. Therefore, in engineering practice, a Pareto optimal solution needs to be sought between model accuracy and calibration feasibility.

[0031] A schematic diagram of the link coordinate system is shown below. Figure 3 As shown, the main steps of kinematic modeling are: (1) When setting up the joint coordinate system, it is necessary to first clarify the coordinate system. Z i The orientation of the axis. Based on the motion characteristics of each joint, all joints are classified as rotational, and can be defined according to the line containing their rotation axis. Z i The spatial position of the axis. The positive direction of this coordinate axis is determined by the right-hand screw rule; that is, when matching the direction of rotation with the direction of the four fingers' bend, the direction the thumb points is the positive direction. Z i The positive direction of the axis.

[0032] (2) Constructing the connecting rod i With adjacent links i The common perpendicular line between +1 and , this geometric characteristic line is... X i The spatial positioning reference of the axis. Its pointing rule is defined as: from the current link i Starting from the centroid or characteristic point, pointing along the common perpendicular towards the subsequent member. i The position corresponding to +1 is used to determine the direction of the coordinate system.

[0033] (3) When constructing the joint coordinate system, the rule for determining the origin position is: take the joint Z i shaft and X i The spatial intersection of the axes serves as the reference origin of the coordinate system.

[0034] (4) During the construction of the joint coordinate system, when X i shaft and Z i After the axis is defined, Y i The direction of the axis can be derived using the right-hand coordinate system rule. The specific method is as follows: [The text abruptly ends here, likely due to an incomplete sentence or missing information.] Z i The axis can be considered as the direction axis of the right-hand screw's forward movement. X i The axis is considered as the common perpendicular pointing towards the axis, and is determined by the direction of the right four fingers bending from... Z i Axle Steering X i When the thumb is pointing vertically, that is... Y i The positive direction of the axis.

[0035] The orthogonal coordinate system construction method in this embodiment ensures the standardization of joint kinematic modeling and the consistency of mathematical calculations. The poses of each link are represented by four geometric parameters: (1) Link length a For along X i Axis, from Z i Move to Z i+1 The distance; (2) Linkage angle α To bypass X i Axis, from Z i Rotate to Z i+1 Angle; (3) Linkage offset d For along Z i Axis, from X i-1 Move to X i The distance; (4) Joint angle θ To bypass Z i Axis, from Xi-1 Rotate to X i The angle.

[0036] The intelligent loading arm employs a dual-rotation joint series design, achieving planar spatial trajectory control through the coordinated motion of the two axes. Compared to multi-degree-of-freedom systems, this simplified architecture demonstrates greater economic and reliability advantages in scenarios such as fluid transport and docking. The kinematic model is shown in Table 2. Based on the Denavit-Hartenberg (DH) modeling theory, the system's kinematic parameter set can be simplified to joint angles and link lengths.

[0037] Based on the Denavit-Hartenberg (DH) theory modeling, when defining the joint coordinate system, the Zi axis is along the rotation axis (the positive direction is determined by the right-hand screw rule), the Xi axis is the common perpendicular of adjacent links (pointing to the subsequent link), the origin is the intersection of Xi and Zi, and the Yi axis is derived by the right-hand rule; the intelligent loading arm is a double rotary joint series structure, and the DH parameters are: α1=α2=0°, a1=1.300m, a2=1.655m, d1=d2=0m, and the joint angles θ1 (joint A) and θ2 (joint B) are variables; the motion of each joint is connected by a homogeneous transformation matrix to derive the pose of the end effector relative to the base.

[0038] Table 2 Kinematic Model

[0039] It can be seen that the mechanical structure of the loading arm is decomposed into links and joints. The pose relationship between adjacent links is described by the link length a, rotation angle α, offset d, and joint angle θ. The transformation of each joint is connected in series by homogeneous matrix multiplication to establish a mathematical mapping between the end pose and the joint angle, thereby realizing the conversion from joint angle to Cartesian space pose.

[0040] As one implementation based on step S102, S102 specifically includes the following steps: Step S1021: Determine the rotation axis direction of each joint of the intelligent loading arm, and define the Zi axis of each joint coordinate system according to the right-hand screw rule, wherein the Zi axis coincides with the joint rotation axis.

[0041] In some embodiments, the mechanical structure of the intelligent loading arm is analyzed to identify the physical rotation axis of each joint. For example, for rotary joints, the Zi axis is along the joint axis. The right-hand screw rule is applied as follows: the direction of the four fingers' bend indicates the direction of joint rotation, and the thumb's direction is the positive direction of the Zi axis. This ensures the uniformity of the Zi axis definition across all joint coordinate systems.

[0042] Step S1022: Construct a common perpendicular line between adjacent links, using this common perpendicular line as the Xi axis, with the direction pointing from the current link to the next link.

[0043] In some embodiments, the shortest distance line between adjacent links i and i+1, also known as the common perpendicular, is calculated. In three-dimensional space, the common perpendicular can be solved using the cross product and dot product of vectors, ensuring that it is perpendicular to the Zi and Zi+1 axes. The Xi axis runs along the direction of the common perpendicular, pointing from the origin of the current link to the origin of the next link.

[0044] Step S1023: Set the origin of each joint coordinate system, and take the spatial intersection point of the Zi axis and Xi axis as the reference origin of the coordinate system.

[0045] In some embodiments, the intersection of the Zi and Xi axes is determined geometrically and used as the origin of the coordinate system. For example, if the Zi and Xi axes do not intersect directly, an approximate solution is obtained using projection or the least squares method. Anchoring the abstract coordinate system to the physical structure allows the model to be directly mapped to the actual mechanical system.

[0046] Step S1024: Based on the defined Zi and Xi axes, derive the direction of the Yi axis using the right-hand coordinate system rule to complete the establishment of the joint coordinate system.

[0047] In some embodiments, after the Zi and Xi axes are defined, the Yi axis is automatically generated through the cross product, and its direction is perpendicular to the Zi-Xi plane. Mathematically, the cross product result needs to be normalized to ensure a unit vector. Utilizing the orthogonality principle in three-dimensional space, a complete right-handed coordinate system is constructed, thereby ensuring mathematical consistency during coordinate system transformations.

[0048] Step S1025: Assign four geometric parameters to each link, including link length a, link rotation angle α, link offset d, and joint angle θ, and calculate the homogeneous transformation matrix between adjacent coordinate systems based on these parameters.

[0049] This embodiment assigns four DH parameters a, α, d, θ to each link, parameterizing the geometric relationships of the mechanical system and describing the kinematic chain through linear algebraic transformations. This allows the model to calculate pose in real time through parameter updates. This improves the model's scalability and computational efficiency, makes the system easy to calibrate and optimize, and is suitable for real-time control applications.

[0050] S103: Construct a conformal reference field, fix multiple reference rings to the end of the intelligent loading arm, drive the loading arm to move to multiple typical poses in the joint space according to a preset trajectory, and form a reference ring array covering the work space.

[0051] In some embodiments, the conformal reference field is constructed based on the pose modeling method of the robotic arm end effector, which is based on its kinematic model. The kinematic model uses the chain principle to transfer and synthesize the motion components generated by each joint coordinate system step by step, and finally derives the mapping relationship between the pose of the robotic arm end effector and the kinematic parameters of each joint.

[0052] After constructing the joint coordinate system of the robotic arm, the spatial transformation relationship between adjacent joint coordinate systems can be decomposed into four ordered motion steps: Along the joint i -1 X i-1 Translation along the axis a i-1 Distance; then, around the same X i-1 Axis rotation α i-1 Angle; then, along the joint i of Z i Translation along the axis d i Distance; finally, around Z i Axis rotation θ i Angle. Based on the above kinematic chain derivation, an angle can be established from the coordinate system { i} to coordinate system { i The homogeneous transformation matrix of {-1} : Formula (1) Given a clearly defined coordinate system for each link and corresponding parameter definitions, the kinematic model of the robotic arm can be directly constructed. By analytically calculating the homogeneous transformation matrix of each joint, the coordinate system of the end effector can be solved by matrix multiplication in the order of the kinematic chain. n} relative to the base coordinate system { 0 The pose transformation matrix of}: Formula (2) In the formula, For about n Functions of joint variables Let be the position vector of the robotic arm's end effector. This represents the rotation matrix of the robotic arm's end effector. After acquiring real-time measurement data from the robotic arm's joint position sensors, the precise pose of the end effector in the Cartesian coordinate system can be determined through forward kinematics calculations. .

[0053] When constructing the conformal reference field, a reference ring is fixed to the end of the intelligent loading arm, driving it to move along a preset trajectory within the joint space, covering multiple typical poses within its workspace. Under each motion pose, the stereo vision system synchronously acquires images of the reference ring and records the corresponding joint angle data. To ensure the conformal reference field meets the calibration requirements of camera extrinsic parameters, a combination of translational and rotational motions of the intelligent loading arm can be used to make the reference ring exhibit regular angle and distance changes within the camera's field of view. This simplifies the process of calculating the transformation matrix between the camera coordinate system and the conformal reference field coordinate system using camera extrinsic parameter calibration techniques. The joint angles are defined in Table 3, which defines nine typical poses, each controlling the intelligent loading arm to move to a predetermined position.

[0054] Table 3: Definition of Joint Angles

[0055] The intelligent loading arm deploys multiple high-precision reference ring arrays within its working space to construct a conformal reference field. The reference rings are arranged in a square grid topology. Arranged along the X and Y axes. The positions of the conformal reference field are shown in Table 4, which defines the positions of the conformal reference field consisting of 9 identical reference rings, where the position of the conformal reference field is referenced to the position of the first reference ring.

[0056] Table 4 Location of the conformal reference field

[0057] As can be seen, fixing the reference ring to the end of the loading arm via threads ensures no relative displacement during installation. Preset joint angles are shown in Table 3, with multiple typical poses covering the X (0-0.2m) and Y (0-0.2m) ranges of the workspace. The loading arm is driven to move along trajectories along the center, X-axis edge, and Y-axis edge, pausing for 3 seconds in each pose to ensure stability. The reference rings are arranged in a square grid, with world coordinates shown in Table 4 (Z=0, coplanar), forming a conformal reference field covering the workspace. The coplanar grid topology provides rigid distances and angles that remain unchanged, reducing ambiguity in extrinsic parameter calibration.

[0058] As one implementation of this embodiment, S103 specifically includes the following steps: Step S1031: Fix the reference ring to the intelligent loading arm end effector, ensure that the mounting surface of the reference ring is perpendicular to the axis of the end effector through the mechanical interface, and calibrate the center of the reference ring to coincide with the origin of the end coordinate system.

[0059] In this embodiment, the reference ring is installed at the end of the intelligent loading arm via a threaded interface. A level or laser calibration tool is used to ensure that the plane of the reference ring is perpendicular to the axis of the end effector. Through mechanical constraints and geometric measurements, the physical position of the reference ring is correlated with the end coordinate system in the kinematic model, reducing installation errors.

[0060] Step S1032: Plan the motion trajectory of the intelligent loading arm, and preset multiple typical poses based on the joint angles to make the reference ring form a regular spatial distribution within the camera's field of view.

[0061] In this embodiment, based on the working space range of the intelligent loading arm, nine typical joint angles are selected as shown in Table 3, covering the grid points in the XY plane. Using inverse kinematics principles, the grid points in Cartesian space are mapped to joint angles, ensuring that the reference loop presents a regular square topology within the camera's field of view. Joint constraints and motion smoothness must be considered during planning to avoid abrupt changes.

[0062] Step S1033: Drive the intelligent loading arm to move along the planned trajectory and pause at each preset pose, controlling the movement speed to ensure that the reference ring is stationary during image acquisition.

[0063] In some embodiments, the intelligent loading arm moves along a planned trajectory, pausing and remaining stationary at each preset pose, with the pause time set according to the camera exposure time. Closed-loop feedback control of the joint angles ensures pose accuracy; the pause mechanism allows the camera to capture clear images, avoiding motion blur. An encoder integrated into the control system monitors the joint angles in real time, ensuring pose repeatability.

[0064] Step S1034: Under each preset pose, use a camera to synchronously acquire images of the reference ring and record the corresponding joint angle data, and realize data synchronization through hardware triggering.

[0065] In some embodiments, an external camera trigger mode is used to send a trigger signal to the camera and data recording system when the intelligent loading arm reaches a preset pose. This ensures a one-to-one correspondence between the image and pose data, avoiding asynchronous errors.

[0066] Step S1035: Based on the recorded joint angle data and the kinematic model of the intelligent loading arm, calculate the world coordinates of the reference ring in the conformal reference field under each pose to form the spatial topology of the reference ring array.

[0067] In some embodiments, the coordinates of the reference ring in the base coordinate system for each pose are calculated based on the recorded joint angles and DH parameters. Joint motion is transferred to the end effector via chained matrix multiplication, and the precise position of the reference ring is obtained by combining this with the installation offset. The numerical stability of the matrix multiplication must be addressed during the calculation to ensure coordinate accuracy.

[0068] S104: Acquire image data and detect reference rings, control the ambient lighting and camera parameters, use target recognition methods to extract the position information of each reference ring from the image, and simultaneously record joint angles and loading arm pose data.

[0069] In some embodiments, to ensure the measurement accuracy of the vision system, the data acquisition environment and camera parameters are controlled. First, a stable optical experimental platform is built to avoid external vibration interference, and a uniformly diffused light source is arranged in the experimental area to eliminate the influence of specular highlights and shadows on image quality. When installing the camera, the optical axis of the vision system is perpendicular to the experimental platform. The camera resolution is selected as 3840×2160 pixels, and the frame rate is selected as 30FPS.

[0070] Before camera calibration, intrinsic parameter calibration is required. The Zhang Zhengyou calibration method is used to obtain parameters such as focal length, principal point coordinates, and distortion coefficients for subsequent image correction and depth calculation. During the experiment, it is necessary to control the ambient light intensity and use polarizing filters to suppress reflective interference if necessary.

[0071] To address the limitation of ordinary RGB cameras lacking depth information, a multi-frame fusion strategy is employed: 10-20 frames are continuously acquired for each robotic arm in a static pose, random noise is suppressed through temporal averaging, and feature point tracking algorithms (such as LK optical flow) are used to verify the static state of the reference loop. Furthermore, to compensate for changes in ambient light, a reference brightness board is placed in the scene to dynamically adjust camera exposure parameters. All image data includes timestamps, joint angles, and robotic arm pose information for subsequent offline processing.

[0072] The original images require rigorous preprocessing to improve calibration reliability. First, an intrinsic parameter-based distortion correction model is applied; second, histogram equalization is used to enhance image contrast and highlight the characteristic edges of the reference ring. Image data such as... Figure 4 As shown, the image is arranged in a square grid topology. Arranged along the X and Y axes.

[0073] The target recognition method is based on an improved YOLOv5 deep learning model, focusing on efficiently detecting pre-defined reference loops from images from a vision system. Considering the imaging characteristics of ordinary cameras (such as light sensitivity), a lightweight YOLOv5s model is selected as the baseline architecture, achieving a balance between speed and accuracy, and can be adapted to small-sample scenarios through transfer learning. The model input resolution is set to 640×640 pixels to ensure that reference loop features can be effectively extracted at different distances, while avoiding excessive consumption of computational resources.

[0074] Image data was divided into training, validation, and test sets in an 8:1:1 ratio to ensure the model's generalization ability. The LabelImg tool was used for annotation, accurately annotating target bounding boxes in PASCAL VOC format with an error controlled within 2 pixels. To improve model robustness, multi-dimensional data augmentation was implemented: geometric transformation, photometric adjustment, random occlusion, and Gaussian noise injection to simulate the impact of complex industrial environments on imaging.

[0075] The model training employs a transfer learning strategy, loading COCO data to train weights and accelerate convergence. Hyperparameters include a batch size of 16, an initial learning rate of 0.01 (adjusted by cosine annealing), and a weighted CIoU Loss loss function (weight 0.8) to optimize boundary regression accuracy. To address the characteristics of the baseline ring, K-means clustering is used to analyze the actual baseline ring size, generating customized anchor boxes to improve small target detection capabilities. A hard sample mining mechanism is introduced during training, specifically enhancing missed samples in the validation set before retraining.

[0076] The model achieved 98.2% precision and 96.7% recall on the independent test set, with an mAP of 97.5% at 0.5 and a single-frame inference time of 8.3 milliseconds (NVIDIA RTX 3080 Ti GPU). In actual deployment, the model is seamlessly integrated with the intelligent loading arm control system, and the output target bounding box information is directly used for subsequent calibration equation construction, improving the overall system efficiency by 40%. This method has been successfully applied to high-precision equipment such as intelligent loading arms, providing a reliable visual perception foundation for industrial automation scenarios. The target recognition results are as follows: Figure 5 As shown, the yellow border represents the result of target recognition.

[0077] As can be seen, this embodiment controls illumination and camera parameters to ensure stable image quality, and intrinsic parameter calibration eliminates lens distortion; multi-frame fusion suppresses noise, and preprocessing enhances features; the YOLOv5s model achieves rapid and accurate detection of baseline loops through transfer learning and data augmentation.

[0078] As one implementation of step S104, the following methods are also included: Step S1041: Build a stable optical experimental platform, arrange uniform diffuse reflection light sources, control the ambient light intensity, and use polarizing filters to suppress reflection interference to ensure the consistency of the image acquisition environment.

[0079] In some embodiments, the optical platform isolates external vibrations through a shock-absorbing device, and the diffuse reflection light source uses an LED array to uniformly illuminate the experimental area, avoiding specular highlights and shadows. A polarizing filter is installed in front of the camera lens to filter reflected light at a specific angle. Step S1042: Configure camera parameters, including a resolution of 3840×2160 pixels and a frame rate of 30FPS, and use the Zhang Zhengyou calibration method to obtain the camera intrinsic parameter matrix and distortion coefficients for image correction.

[0080] Step S1043: Continuously acquire 10-20 frames of images in each robotic arm static pose, synchronously record joint angle and pose data through hardware triggering, and verify the static state of the reference loop using time-series averaging and LK optical flow algorithms.

[0081] In some embodiments, in each stationary pose of the robotic arm, the camera continuously acquires multiple frames of images via hardware triggering, simultaneously recording joint angle sensor data. A time-series averaging algorithm performs pixel-level averaging on the multiple frames to suppress random noise. The LK optical flow algorithm tracks reference loop feature points to verify whether they are stationary during acquisition. Time-series analysis reduces the impact of noise, and motion consistency ensures data validity. A synchronization mechanism aligns image and pose data using timestamps to avoid asynchronous errors.

[0082] Step S1044: Apply an intrinsic parameter-based distortion correction model to correct the original image, and use histogram equalization to enhance image contrast and highlight the reference ring feature edges.

[0083] In some embodiments, distortion correction first maps the image coordinates to the camera coordinate system based on intrinsic parameters, applies distortion formulas for correction, and then reprojects the image onto the image plane. Histogram equalization redistributes the image gray levels, expanding the dynamic range. Optical distortion is corrected and image quality is improved through mathematical transformations, resulting in sharper reference ring edges. Preprocessing steps are performed in parallel on the GPU for improved efficiency.

[0084] Step S1045: Use the improved YOLOv5 deep learning model for target recognition, including data augmentation, transfer learning and anchor box clustering, to extract the bounding box location information of the reference ring from the image.

[0085] In some embodiments, the model input resolution is set to 640×640 pixels, the training set is partitioned in an 8:1:1 ratio, and data augmentation techniques such as geometric transformation, photometric adjustment, random occlusion, and Gaussian noise are applied. Transfer learning loads pre-trained COCO weights, with hyperparameters including a batch size of 16 and a learning rate of 0.01. This achieves high-precision and fast benchmark loop detection, adapts to complex industrial environments, and improves the overall system efficiency and calibration reliability.

[0086] S105: Calibrate camera extrinsic parameters. Based on the known world coordinates of the reference ring in the conformal reference field and the 2D position of the image detection, solve for the rotation matrix and translation vector of the camera coordinate system relative to the reference ring coordinate system.

[0087] In some embodiments, such as Figure 6 As shown, the camera extrinsic calibration integrates nine reference ring images within the workspace into a single conformal reference field image to construct a strongly spatially constrained extrinsic calibration environment. By capturing the spatial distribution relationship of the reference rings through multiple imaging operations, the rigid geometric constraints between the points (distance invariance, coplanarity) significantly improve the accuracy of the camera extrinsic calibration. This technique effectively avoids the problem of error accumulation from multi-image stitching.

[0088] World coordinates of 9 reference loops on a preloaded conformal reference field ,in The world coordinate system is defined as follows: the origin is referenced to the position of the lower right corner reference ring; the X-axis reference is the vector formed by the red reference rings; the Y-axis reference is the vector formed by the green reference rings; and the Z-axis reference is the vector defined by the right-hand rule. Coordinate data is stored in array format and can be directly read by subsequent calculation modules.

[0089] Load the camera intrinsic parameter matrix: Formula (3) In the formula, f x , f y These are the equivalent focal lengths along the X and Y axes, respectively. c x , c y The coordinates of the main point.

[0090] Loading distortion coefficients: Formula (4) In the formula, k 1, k 2, k 3 represents the radial distortion coefficient. p 1, p 2 represents the tangential distortion coefficients. These parameters are calculated and stored through pre-performed camera intrinsic calibration (Zhang Zhengyou calibration method).

[0091] On the image of the conformal reference field, the Shi-Tomasi corner detection method is used to identify the corner positions. A one-to-one correspondence is established between the detected image corners and the known world coordinates of the reference ring on the conformal reference field according to the mesh topology.

[0092] The camera coordinate system is defined as follows: the origin is referenced to the optical center of the camera (i.e., the optical center of the lens); the Z-axis is referenced to the scene in front of the camera, coinciding with the camera's optical axis; and the X and Y axes are referenced to the row and column directions parallel to the camera's imaging sensor, respectively. Assume a point on the world coordinate system... The coordinates in the camera coordinate system are The camera coordinates are then calculated using the following rigid body transformation formula: Formula (5) In the formula, To handle rotation matrices in the coordinate system orientation, To handle the translation vector of the coordinate system position.

[0093] The rotation matrix of the projection equation is solved based on 9 sets of 3D-2D point pairs in the conformal reference field, as well as the camera's intrinsic parameter matrix and distortion coefficients. R Translation vector TThe efficient and robust IPPE (Infinitesimal Plane-Based Pose Estimation) algorithm is selected to calculate the initial estimates of the camera extrinsic parameters. The LM (Levenberg-Marquardt) algorithm is used to optimize the initial estimates of the camera extrinsic parameters. Its core idea is to achieve stable convergence of parameter corrections by dynamically balancing the characteristics of gradient descent and the Gauss-Newton method. In the kinematic calibration of the intelligent loading arm, this algorithm optimizes geometric parameters to minimize the error between the actual distance value and the theoretical distance value predicted by the kinematic model. Formula (6) The parameter update equation combines the advantages of gradient descent and the Gauss-Newton method, and its form is: Formula (7) In the formula, Let be the Jacobian matrix of the error function with respect to the parameters. This is the damping factor, used to adjust the direction of the algorithm. This is the parameter correction amount.

[0094] To improve the robustness of the algorithm, damping factor Dynamic adjustment is required: the initial value is set to... If the current iteration reduces the error, it is reduced to accelerate convergence; if the error increases, it is increased to enhance stability. Simultaneously, parameter sensitivity analysis is introduced to eliminate redundant parameters, and the global sensitivity index of each parameter is calculated. Formula (8) In the formula, For the first i Global sensitivity indicators of joints Let be the error of the k-th iteration. If If so, it is removed from the optimization variables to reduce the computational dimensionality. The results of the camera extrinsic parameter calibration are as follows: , Formula (9) This embodiment is based on the perspective projection model. By using the correspondence between 3D world points and 2D image points, the rigid transformation (R, T) between the camera coordinate system and the reference ring coordinate system is solved. The IPPE algorithm quickly obtains the initial solution, and the LM algorithm optimizes the reprojection error through dynamic damping factor optimization, thereby improving the accuracy of the extrinsic parameters.

[0095] As part of S105 of the present invention, the following steps are also involved: Step S1051: Load the world coordinates of the reference loop in the conformal reference field, the camera intrinsic parameter matrix, and the distortion coefficients to establish the correspondence between the world coordinate system and the image coordinate system.

[0096] In some embodiments, the world coordinates of nine reference rings on the conformal reference field are preloaded. The world coordinate system is defined with the lower right reference ring as the origin, the red reference rings forming a vector to define the X-axis, the green reference rings forming a vector to define the Y-axis, and the Z-axis defined by the right-hand rule. Simultaneously, the camera intrinsic parameter matrix and distortion coefficients, pre-obtained using the Zhang Zhengyou calibration method, are loaded.

[0097] Step S1052: Use the Shi-Tomasi corner detection method on the conformal reference field image to identify the corner positions of the reference ring and establish a one-to-one correspondence mapping with the known world coordinates.

[0098] In some embodiments, a Shi-Tomasi corner detector is applied to the image of the conformal reference field to identify the characteristic corners of each reference ring. Based on the square grid topology of the reference rings, a correspondence is established between the detected image corners and known world coordinates.

[0099] S1053: Based on the 3D-2D point pair correspondence, the IPPE algorithm is used to calculate the initial estimates of the rotation matrix and translation vector of the camera extrinsic parameters.

[0100] In some embodiments, the IPPE algorithm is specifically designed for coplanar point sets. It calculates two possible pose solutions and selects the one with the smaller reprojection error as the initial estimate. This algorithm leverages the coplanarity of the reference rings to quickly obtain initial extrinsic parameters through analytical solutions.

[0101] Step S1054: Use the Levenberg-Marquardt algorithm to perform nonlinear optimization on the initial external parameter estimates, and achieve stable convergence by dynamically adjusting the damping factor.

[0102] In some embodiments, the output of the IPPE algorithm is used as the initial value, and the reprojection error is minimized using the LM algorithm. By combining the advantages of gradient descent and the Gauss-Newton method, fast optimization is achieved while ensuring convergence stability.

[0103] Step S1055: Perform parameter sensitivity analysis, eliminate redundant parameters in the optimization process, and verify the accuracy and stability of the external parameter calibration results.

[0104] By analyzing the sensitivity of parameters to the objective function, insensitive redundant parameters can be identified and eliminated, simplifying the optimization problem. This reduces computational complexity, improves calibration efficiency, and ensures the reliability of extrinsic parameter results.

[0105] S106: Obtain the pose relationship between the reference ring and the loading arm, and use the kinematic model of the intelligent loading arm to calculate the homogeneous transformation matrix of the reference ring coordinate system relative to the loading arm base coordinate system.

[0106] In some embodiments, a coordinate measuring machine is used to confirm that the coordinate system of the reference ring coincides with that of the end effector of the loading arm, and the transformation matrix is ​​fixed as the identity matrix; 9 sets of joint angles are extracted from S103, and the average value over 3 seconds is calculated; the transformation matrix from the end effector to the base is calculated based on the DH model, and the fixed transformation of the reference ring and the end effector is concatenated to obtain the matrix from the reference ring to the base; the translation vector of the matrix from the reference ring to the base is verified to have a deviation of ≤0.005m from the known world coordinates, and stored in XML format.

[0107] It can be seen that by using the kinematic model of the loading arm, the joint angle is converted into the pose of the end effector relative to the base; since the reference ring is rigidly connected to the end effector, the pose of the end effector is transferred to the reference ring through a fixed transformation matrix, thus establishing the pose mapping between the reference ring and the base.

[0108] Step S106 of the present invention further includes the following: Step S1061: Determine the installation relationship between the reference ring and the intelligent loading arm end effector. By measuring the spatial offset between the mounting surface of the reference ring and the end coordinate system, establish a fixed transformation matrix from the reference ring coordinate system to the end coordinate system.

[0109] In some embodiments, a coordinate measuring machine or laser tracker is used to accurately measure the spatial relationship between the mounting surface of the reference ring and the coordinate system of the end of the loading arm. By measuring the positional deviation between the center point of the reference ring and the origin of the end coordinate system, as well as the angular relationship between the plane of the reference ring and each axis of the end coordinate system, a fixed transformation matrix is ​​calculated.

[0110] Step S1062: Obtain real-time angle data of each joint of the intelligent loading arm by acquiring joint angle sensor signals through an encoder or potentiometer installed at the rotating joint.

[0111] In some embodiments, high-precision absolute encoders are installed at the two rotary joints of the intelligent loading arm to acquire joint angle signals in real time. The analog signals are converted into digital signals by a data acquisition card, and a filtering algorithm is used to eliminate noise interference.

[0112] Step S1063: Based on the DH kinematic parameters and real-time joint angles of the intelligent loading arm, calculate the homogeneous transformation matrix of the end effector coordinate system relative to the base coordinate system.

[0113] In some embodiments, the transformation matrix is ​​calculated step by step according to the DH parameters in Table 2 and the real-time acquired joint angles θ1 and θ2, following the DH modeling theory. The pose of the end effector in the base coordinate system is obtained by matrix multiplication.

[0114] Step S1064: Combine the fixed transformation matrix from the reference ring to the end point and the transformation matrix from the end point to the base to calculate the complete pose transformation relationship between the reference ring coordinate system and the loading arm base coordinate system.

[0115] In some embodiments, the installation transformation matrix obtained in step S1061 is multiplied by the end pose matrix calculated in step S1063 to obtain the complete transformation relationship between the reference ring coordinate system and the loading arm base coordinate system. The order and dimensions of the matrix multiplication must be kept consistent during the calculation process. The reference ring pose is unified into the base coordinate system through a coordinate transformation chain.

[0116] Step S1065: Verify the accuracy of the reference ring pose calculation. Ensure the reliability of the pose transformation relationship through repeated measurements and comparative analysis.

[0117] In some embodiments, under multiple known poses, the actual pose of the reference ring is directly measured using a laser tracker or a vision measurement system, and compared with the theoretical pose calculated by the kinematic model. Position and attitude errors are calculated to verify the accuracy of the kinematic model. A truth reference is provided by an external measurement device to evaluate the accuracy of the pose calculation.

[0118] Step S1066: Record all pose data and establish a mapping table to provide a complete pose correspondence between the reference ring and the loading arm base coordinate system for subsequent hand-eye calibration.

[0119] In some embodiments, all calculated reference ring pose data, corresponding joint angles, and time information are organized into a structured mapping table. A database management system is used to store and manage the pose data, and an index is created for fast querying. Complete historical pose data records are provided to support data backtracking and analysis in subsequent calibration processes.

[0120] S107: Determine the pose mapping from the camera to the loading arm base. By concatenating the transformation matrices between the camera and the reference loop, and between the reference loop and the loading arm base, obtain the pose transformation relationship from the camera coordinate system to the loading arm base coordinate system.

[0121] In some embodiments, the arm-mounted loading arm hand-eye calibration method involves a fixed extrinsic parameter matrix, pre-determined using camera extrinsic parameter calibration technology, relative to the conformal reference field coordinate system at the end of the loading arm. This matrix fully describes the rigid transformation relationship of the camera relative to the conformal reference field. Additionally, a homogeneous transformation matrix is ​​calculated from the conformal reference field coordinate system at the end of the loading arm to the loading arm base coordinate system using the forward kinematics of the intelligent loading arm. This matrix fully describes the position and orientation of the conformal reference field in the base coordinate system. , Formula (10) To establish a direct geometric mapping from the camera coordinate system to the loading arm base coordinate system, the two known spatial transformation matrices mentioned above need to be concatenated. Specifically, the pose of the camera coordinate system relative to the loading arm base coordinate system can be achieved by matrix composition of the transformation relationship from the conformal reference field coordinate system to the camera coordinate system and the transformation relationship from the conformal reference field coordinate system to the loading arm base coordinate system. Essentially, this process transforms points or vectors in the camera coordinate system, through the intermediate link of the conformal reference field coordinate system, to ultimately express them in the loading arm base coordinate system.

[0122] Formula (11) Mathematically, this solution process involves the multiplication of two homogeneous transformation matrices. This matrix multiplication operation strictly follows the chain rule for rigid body transformations in three-dimensional space, and its simplified calculation result is as follows: Formula (12) It plays a fundamental role in the vision system of intelligent loading arms. It establishes a geometric bridge between visual perception information and the base coordinate system upon which the motion control of the intelligent loading arm depends. Based on this matrix, combined with camera intrinsic parameters, it is possible to back-project the coordinates of the target feature points identified in the image to their three-dimensional spatial positions in the base coordinate system of the intelligent loading arm, or directly use them to calculate the motion commands required to guide the end effector of the intelligent loading arm to the desired observation pose, thereby significantly improving the accuracy and intelligence level of automated operations.

[0123] This embodiment uses a reference ring coordinate system as an intermediate link to decompose the pose relationship between the camera and the base into two transformations: camera to reference ring and reference ring to base. A direct mapping from the camera to the base is obtained by concatenating these transformations using matrix multiplication, conforming to the rigid body transformation transitivity. This concatenated transformation establishes a direct connection between the vision system and the mechanical system, allowing image features to be directly converted into 3D coordinates in the base coordinate system. It also reduces the error from multiple averaging and verification processes, thus meeting the high-precision docking requirements of the loading arm.

[0124] As one implementation of step S107, the following methods are also included: Step S1071: Unify the spatial reference datum of each coordinate system and determine the mathematical expression relationship between the camera coordinate system, the reference ring coordinate system and the loading arm base coordinate system.

[0125] In some embodiments, the reference datum for each coordinate system is explicitly defined: the camera coordinate system has the optical center as its origin and the optical axis as the Z-axis; the reference ring coordinate system has the lower right reference ring as its origin and the axis is determined according to the color ring distribution; the loading arm base coordinate system takes the fixed mounting base as its reference. By establishing a unified right-handed coordinate system rule, the mathematical consistency of all transformation matrices is ensured.

[0126] Step S1072: Based on the camera extrinsic calibration results, obtain the rigid transformation matrix from the camera coordinate system to the reference ring coordinate system.

[0127] In some embodiments, the camera extrinsic parameters obtained through step S105 calibration include a rotation matrix and a translation vector. The validity of the extrinsic parameter matrix is ​​verified, ensuring that the rotation matrix satisfies orthogonality and has a determinant of 1. The pose relationship is represented as a 4×4 transformation matrix in homogeneous coordinates. Using the pre-calibrated camera extrinsic parameters, a rigid transformation relationship between the camera observation space and the reference ring physical space is directly established.

[0128] Step S1073: Based on the kinematic model, calculate and obtain the pose transformation matrix from the reference ring coordinate system to the loading arm base coordinate system.

[0129] In some embodiments, the pose matrix of the reference ring in the loading arm base coordinate system calculated in step S106 is used. The pose matrix integrates the kinematic model parameters of the intelligent loading arm, real-time joint angles, and reference ring installation parameters. Forward kinematic calculations ensure the accuracy and real-time nature of the pose information. Through kinematic modeling of the mechanical system, a mathematical relationship is established between the physical position of the reference ring and the loading arm base coordinate system.

[0130] Step S1074: Solve the complete transformation relationship from the camera coordinate system to the loading arm base coordinate system through chain matrix multiplication.

[0131] In some embodiments, operations are performed according to a strict matrix multiplication order, first ensuring that the matrix dimensions match. During the calculation, the rotation matrix part undergoes matrix multiplication, and the translation vector part undergoes a composite operation of rotation and translation. The validity of the resulting matrix is ​​verified to ensure that it satisfies the properties of rigid body transformation. Through the chain rule of coordinate transformation, the two independent transformation relationships are combined into a complete direct mapping from the camera to the base.

[0132] Step S1075: Verify the accuracy and reliability of the hand-eye calibration results and evaluate the accuracy performance of the transformation matrix in practical applications.

[0133] In some embodiments, under multiple verification poses, feature points in the known base coordinate system are transformed to the camera coordinate system using the calibrated TCBTCB transformation, then projected onto the image plane, and compared with the actual image observation position. Simultaneously, the consistency of the transformation matrix under different poses is checked. Using data independent of the calibration process, the accuracy and stability of the hand-eye calibration results in practical applications are verified.

[0134] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0135] The following are embodiments of the intelligent loading arm hand-eye calibration system based on a conformal reference field provided in this disclosure. This system and the intelligent loading arm hand-eye calibration methods based on conformal reference fields in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the intelligent loading arm hand-eye calibration system based on a conformal reference field, please refer to the embodiments of the intelligent loading arm hand-eye calibration methods based on conformal reference fields described above.

[0136] The system includes: a reference ring preparation module for designing and preparing a reference ring for visual recognition, wherein the reference ring comprises multiple concentric colored rings with different colors and sizes, and is installed at the end of an intelligent loading arm; The kinematic modeling module is used to construct the kinematic model of the intelligent loading arm. By defining the joint coordinate system and geometric parameters, it describes the pose mapping relationship between the end effector of the robotic arm and the base coordinate system. The reference field construction module is used to construct a conformal reference field, fix multiple reference rings to the end of the intelligent loading arm, and drive the loading arm to move to multiple typical poses in the joint space according to a preset trajectory, forming a reference ring array covering the workspace. The image acquisition and detection module is used to acquire image data and detect reference rings, control the ambient lighting and camera parameters, extract the position information of each reference ring from the image using target recognition methods, and simultaneously record joint angle and loading arm pose data. The camera extrinsic parameter calibration module is used to calibrate the camera extrinsic parameters. Based on the known world coordinates of the reference ring in the conformal reference field and the 2D position of the image detection, it solves the rotation matrix and translation vector of the camera coordinate system relative to the reference ring coordinate system. The pose calculation module is used to obtain the pose relationship between the reference ring and the loading arm. Using the kinematic model of the intelligent loading arm, it calculates the homogeneous transformation matrix of the reference ring coordinate system relative to the loading arm base coordinate system. The hand-eye pose mapping module is used to determine the pose mapping from the camera to the loading arm base. By concatenating the transformation matrices between the camera and the reference loop, and between the reference loop and the loading arm base, the pose transformation relationship from the camera coordinate system to the loading arm base coordinate system is obtained.

[0137] like Figure 7 As shown, this application also provides an electronic device, including a display module 103, a memory 102, a processor 101, a communication module 104, and a computer program stored in the memory and executable on the processor 101. When the processor 101 executes the program, it implements the steps of a smart loading arm hand-eye calibration method based on a conformal reference field.

[0138] In embodiments of the present invention, electronic devices include, but are not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed herein.

[0139] In this embodiment, processor 101 may be implemented using at least one of an application-specific integrated circuit, a programmable logic device, a field-programmable gate array, a processor, a controller, a microcontroller, a microprocessor, or an electronic unit designed to perform the functions described herein. In some cases, such an implementation may be implemented within a controller. For software implementation, implementations such as processes or functions may be implemented with separate software modules that allow the performance of at least one function or operation. Software code may be implemented by a software application (or program) written in any suitable programming language, and the software code may be stored in memory and executed by the controller.

[0140] The display module 103 is used to display information input by the user or information provided to the user. The display module 103 may include a display panel, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like.

[0141] The memory 102 can be used to store software programs and various data. The memory 102 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0142] The communication module 104 transmits radio signals to and / or receives radio signals from at least one of a base station, an external terminal, and a server. Such radio signals may include voice call signals, video call signals, or various types of data sent and / or received according to text and / or multimedia messages.

[0143] This application also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of an intelligent loading arm hand-eye calibration method based on a conformal reference field.

[0144] The storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0145] The storage medium stores a program product capable of implementing the methods described above in this specification. In some possible implementations, various aspects of this disclosure may also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0146] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A conformal reference field based intelligent hose hand-eye calibration method, characterized in that, The method comprises: S101: designing and preparing a reference ring for visual recognition, the reference ring comprising a plurality of concentric color bands, the concentric color bands having different colors and sizes, and being installed on the end of the smart gooseneck; S102: constructing a kinematic model of the smart gooseneck, describing the pose mapping relationship between the end effector of the manipulator and the base coordinate system by defining the joint coordinate system and the geometric parameters; S103: constructing a conformal reference field, fixing a plurality of the reference rings on the end of the smart gooseneck, driving the gooseneck to move in the joint space according to a preset trajectory to a plurality of typical poses, and forming an array of reference rings covering the working space; S104: collecting image data and detecting the reference rings, controlling the experimental environment illumination and camera parameters, using a target recognition method to extract the position information of each reference ring from the image, and synchronously recording the joint angle and gooseneck pose data; S105: calibrating the camera extrinsic parameters, based on the known world coordinates of the reference rings in the conformal reference field and the 2D positions detected from the image, solving the rotation matrix and translation vector of the camera coordinate system relative to the reference ring coordinate system; S106: obtaining the pose relationship between the reference ring and the gooseneck, using the kinematic model of the smart gooseneck to calculate the homogeneous transformation matrix of the reference ring coordinate system relative to the gooseneck base coordinate system; S107: determining the pose mapping from the camera to the gooseneck base, obtaining the pose transformation relationship from the camera coordinate system to the gooseneck base coordinate system by concatenating the transformation matrices of the camera and the reference ring, and the reference ring and the gooseneck base.

2. The conformal reference field based smart hose hand-eye calibration method of claim 1, wherein, S102 specifically comprises the following steps: determining the rotation axis direction of each joint of the smart gooseneck, defining the Zi axis of each joint coordinate system according to the right-hand screw rule, wherein the Zi axis coincides with the joint rotation axis; constructing a common perpendicular line between adjacent links, taking the common perpendicular line as the Xi axis, and the direction pointing from the current link to the next link; setting the origin of each joint coordinate system, taking the spatial intersection point of the Zi axis and the Xi axis as the coordinate system reference origin; based on the defined Zi axis and Xi axis, the Yi axis direction is derived through the right-hand coordinate system rule to complete the establishment of the joint coordinate system; assigning four geometric parameters to each link, including link length a, link rotation angle a, link offset d and joint angle θ, and calculating the homogeneous transformation matrix between adjacent coordinate systems based on these parameters.

3. The conformal reference field based smart hosed hand-eye calibration method of claim 1, wherein, S103 specifically comprises the following steps: fixing the reference ring on the end effector of the smart gooseneck, ensuring that the mounting surface of the reference ring is perpendicular to the axis of the end effector through the mechanical interface, and calibrating the coincidence of the center of the reference ring and the origin of the end coordinate system; planning the motion trajectory of the smart gooseneck, presetting a plurality of typical poses based on the joint angle, so that the reference rings form a regular spatial distribution in the camera field of view; driving the smart gooseneck to move according to the planned trajectory, and pausing at each preset pose, controlling the motion speed to ensure that the reference rings are in a static state during image acquisition; at each preset pose, synchronously collecting the image of the reference ring using the camera, and recording the corresponding joint angle data, and realizing data synchronization through hardware triggering; based on the recorded joint angle data and the kinematic model of the smart gooseneck, calculating the world coordinates of the reference rings in the conformal reference field at each pose, and forming the spatial topology of the array of reference rings.

4. The conformal reference field based smart hosed hand-eye calibration method of claim 1, wherein, S104 specifically comprises the following steps: Build a stable optical experiment platform, arrange a uniform diffuse reflection light source, control the intensity of environmental light, and use a polarizing filter to suppress reflected light interference to ensure the consistency of the image acquisition environment; Configure camera parameters, including resolution of 3840x2160 pixels and frame rate of 30FPS, and use Zhang Zhengyou calibration method to obtain camera intrinsic matrix and distortion coefficients for image correction; Collect 10-20 frames of images continuously at each static pose of the mechanical arm, record joint angle and pose data synchronously through hardware triggering, and verify the reference ring static state using time sequence average and LK optical flow algorithm; Apply the distortion correction model based on intrinsic parameters to correct the original image, and use histogram equalization to enhance the image contrast and highlight the feature edges of the reference ring; Use the improved YOLOv5 deep learning model for target recognition, including data enhancement, transfer learning and anchor box clustering, to extract the boundary box position information of the reference ring from the image.

5. The conformal reference field based smart hosed hand-eye calibration method of claim 1, wherein, S105 specifically comprises the following steps: Step S1051: Load the world coordinates of the reference ring in the conformal reference field, camera intrinsic matrix and distortion coefficients, and establish the correspondence between the world coordinate system and the image coordinate system; Use the Shi-Tomasi corner detection method to identify the corner positions of the reference ring on the conformal reference field image, and establish a one-to-one mapping with the known world coordinates; Based on the correspondence between 3D-2D point pairs, use IPPE algorithm to calculate the initial estimate of the rotation matrix and translation vector of the camera extrinsic parameters; Use Levenberg-Marquardt algorithm to nonlinearly optimize the initial extrinsic parameter estimate, and realize stable convergence through dynamic adjustment of the damping factor; Perform parameter sensitivity analysis, eliminate redundant parameters in the optimization process, and verify the accuracy and stability of the extrinsic parameter calibration result.

6. The conformal reference field based smart hosed hand-eye calibration method of claim 1, wherein, S106 specifically comprises the following steps: Determine the installation relationship between the reference ring and the intelligent crane pipe end effector, measure the spatial offset between the reference ring mounting surface and the end coordinate system, and establish the fixed transformation matrix from the reference ring coordinate system to the end coordinate system; Obtain real-time angle data of each joint of the intelligent crane pipe, and collect joint angle sensor signals through encoders or potentiometers installed at rotating joints; Based on the DH kinematics parameters and real-time joint angles of the intelligent crane pipe, calculate the homogeneous transformation matrix of the end effector coordinate system relative to the base coordinate system; Combine the fixed transformation matrix from the reference ring to the end and the transformation matrix from the end to the base to calculate the complete pose transformation relationship of the reference ring coordinate system relative to the crane pipe base coordinate system; Verify the accuracy of the reference ring pose calculation, and ensure the reliability of the pose transformation relationship through repeated measurement and comparative analysis; Record all pose data and establish a mapping relationship table to provide complete pose correspondence between the reference ring and the crane pipe base coordinate system for subsequent hand-eye calibration.

7. The conformal reference field based smart hosed hand-eye calibration method of claim 1, wherein, S107 specifically comprises the following steps: Unify the spatial reference of each coordinate system to determine the mathematical expression relationship between the camera coordinate system, the reference ring coordinate system and the crane pipe base coordinate system; Based on the camera extrinsic calibration result, obtain the rigid transformation matrix from the camera coordinate system to the reference ring coordinate system; Based on the kinematics model calculation, the pose transformation matrix of the reference ring coordinate system to the riser base coordinate system is obtained; Through chain matrix multiplication operation, the complete transformation relationship of the camera coordinate system to the riser base coordinate system is solved; The accuracy and reliability of the hand-eye calibration result are verified, and the precision performance of the transformation matrix in actual application is evaluated.

8. A conformal reference field based intelligent hose hand-eye calibration system, comprising: The system is used to realize the intelligent riser hand-eye calibration method based on the conformal reference field as claimed in any one of claims 1 to 7. The system comprises: A reference ring preparation module for designing and preparing a reference ring for visual identification, the reference ring comprising a plurality of concentric color bands with different colors and sizes, and being installed at the end of the intelligent riser; A kinematics modeling module for building a kinematics model of the intelligent riser, describing the pose mapping relationship between the end effector of the manipulator and the base coordinate system by defining the joint coordinate system and geometric parameters; A reference field construction module for constructing a conformal reference field, fixing a plurality of the reference rings at the end of the intelligent riser, driving the riser to move to a plurality of typical poses according to a preset trajectory in the joint space, and forming an array of reference rings covering the working space; An image acquisition and detection module for acquiring image data and detecting the reference rings, controlling the experimental environment lighting and camera parameters, using a target recognition method to extract the position information of each reference ring from the image, and synchronously recording the joint angle and riser pose data; A camera extrinsic calibration module for calibrating the camera extrinsic parameters, based on the known world coordinates of the reference rings in the conformal reference field and the 2D positions detected from the image, solving the rotation matrix and translation vector of the camera coordinate system relative to the reference ring coordinate system; A pose calculation module for obtaining the pose relationship between the reference ring and the riser, using the kinematics model of the intelligent riser to calculate the homogeneous transformation matrix of the reference ring coordinate system relative to the riser base coordinate system; A hand-eye pose mapping module for determining the pose mapping of the camera to the riser base, obtaining the pose transformation relationship of the camera coordinate system to the riser base coordinate system by concatenating the transformation matrices of the camera and the reference ring, and the reference ring and the riser base.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to realize the steps of the intelligent riser hand-eye calibration method based on the conformal reference field as claimed in any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to realize the steps of the intelligent riser hand-eye calibration method based on the conformal reference field as claimed in any one of claims 1 to 7.

Images