Mechanical arm teaching method and system based on multi-infrared camera positioning

Abstract

The invention relates to a mechanical arm teaching method and system based on multi-infrared camera positioning, and the method comprises the steps: arranging reflective markers at the tail end of a mechanical arm welding gun and the tail end of a teaching gun, building a coordinate system, and determining the poses of a welding gun tip corresponding point M and a teaching gun tip N; a reflective marker coordinate system positioning system of multiple infrared cameras is established, camera parameter calibration and coordinate system construction and association are performed, a welding gun tip M is moved to a plurality of point positions, a transformation matrix from a camera coordinate system to a mechanical arm coordinate system is solved, a teaching gun related coordinate system transformation matrix is solved, and a multi-infrared-camera reflective marker coordinate system positioning system is obtained. Conversion from the pose of the teaching gun in a camera coordinate system to the pose of the teaching gun in a mechanical arm coordinate system is achieved, and flexible and high-precision teaching is achieved. Compared with the prior art, the multi-infrared camera and the reflective marker are combined, the shielding problem is effectively solved, the depth precision is improved, system integration is simplified by establishing a universal hand-eye calibration method, the teaching gun design is visual and easy to use, and the positioning accuracy, operation convenience and working efficiency of mechanical arm teaching are improved.

Description

Technical Field

[0001] The present invention relates to a teaching method for a robotic arm, and more particularly to a teaching method and system for a robotic arm based on multi-infrared camera positioning. Background Art

[0002] In the field of modern robotic arm technology, teaching is a key link in achieving precise motion control of robotic arms. Although traditional collaborative robotic arms provide a manual dragging teaching method, this method is extremely inconvenient in actual operation, with low teaching efficiency and difficult to meet the requirements of high-efficiency production. In the scenario of industrial robotic arms, the teaching process often relies on a teach pendant for complex operations, which not only takes time but also requires operators to have high professional skills and proficiency, greatly increasing the labor cost and time cost. For the teaching of the robotic arm of a humanoid robot, it also faces the dilemmas of cumbersome operation and inconvenient teaching.

[0003] In addition, with the continuous improvement of the accuracy requirements in industrial production, a teaching method based on two-dimensional barcodes and a single camera has been proposed. However, this method has obvious defects. In actual applications, a single camera is easily affected by occlusion. Once the barcode or the target object is partially occluded, the information obtained by the camera will be incomplete, resulting in a significant reduction in positioning accuracy, and its calibration range is very limited, unable to adapt to complex and changing working environments and diverse task requirements.

[0004] In summary, the existing robotic arm teaching methods have many deficiencies in terms of convenience, efficiency, and accuracy. There is an urgent need for a portable, wide calibration range, occlusion-free, and intuitive and efficient teaching positioning solution to promote the wide application and development of robotic arm technology in various fields. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above-mentioned defects existing in the prior art and provide a teaching method and system for a robotic arm based on multi-infrared camera positioning. By combining multiple infrared cameras with reflective markers, the occlusion problem is effectively solved and the depth accuracy is improved. The system integration is simplified by establishing a general hand-eye calibration method. The teaching gun is designed to be intuitive and easy to use, significantly improving the positioning accuracy, operation convenience, and working efficiency of robotic arm teaching, and strongly promoting the wide application of robotic arm technology in multiple fields and reducing the application cost and technical threshold.

[0006] The purpose of the present invention can be achieved by the following technical solutions:

[0007] The first aspect of the present invention provides a teaching method for a robotic arm based on multi-infrared camera positioning, including the following steps:

[0008] S1: Set 3 reflective markers at the end of the robotic arm welding torch and establish a coordinate system to determine the pose of the corresponding point M at the tip of the welding torch;

[0009] S2: Establish a positioning system for the reflective marker coordinate system based on multiple infrared cameras;

[0010] S3: Establish a rigid body of the reflective marker coordinate system in the positioning system for the reflective marker coordinate system, establish the corresponding point M of the welding torch, move the tip M of the welding torch to multiple positions, record the positions of the moved tip M of the welding torch in the camera and the robot arm base coordinate systems, and solve the transformation matrix from the camera coordinate system to the robot arm coordinate system;

[0011] S4: Set three reflective markers A, B, and C at the end of the teaching torch as the origin of the teaching torch coordinate system, with AB perpendicular to BC, and determine the pose of the teaching torch tip N;

[0012] S5: Obtain the coordinates of points A, B, and C on the teaching torch in the camera coordinate system through the positioning system for the reflective marker coordinate system based on multiple infrared cameras, track and obtain the 6D pose of the teaching torch rigid body, and calculate the pose of the teaching torch tip N;

[0013] S6: Solve the transformation matrix from the teaching torch tip coordinate system to the camera coordinate system using the pose of the teaching torch tip N;

[0014] S7: Solve the transformation matrix from the teaching torch coordinate system to the robot coordinate system, and repeat steps S6 to S7 until the teaching of each stage is completed.

[0015] Furthermore, in S1, the three reflective markers are distributed in different orientations at the end of the welding torch, and the three reflective markers are used to provide reflected light for the infrared cameras.

[0016] Furthermore, in S1, the pose of M in the coordinate system is calculated according to the CAD model.

[0017] Furthermore, in S2, it specifically includes the following steps:

[0018] Install infrared cameras according to the working range of the robot arm, the teaching area, and the occlusion situation, and calibrate the position and angle;

[0019] Configure the camera parameters, and perform internal parameter and external parameter calibration;

[0020] Construct a coordinate system with the reflective marker as the object, determine the origin and the axis directions, and associate the position information of the reflective marker in the camera coordinate system with the constructed reflective marker coordinate system, thereby establishing a complete positioning system.

[0021] Furthermore, in S3, it specifically includes the following steps:

[0022] Establish a rigid body of the reflective marker coordinate system in the positioning system for the reflective marker coordinate system based on multiple infrared cameras, establish the corresponding point M of the welding torch, track and obtain the rigid body coordinate system, and the 6D pose of the tip M of the welding torch in the camera coordinate system;

[0023] Move the welding torch tip M of the mobile robotic arm to point O, and then move it from point O to points X, Y, and Z along the XYZ axes respectively. A total of four positions, O, X, Y, and Z, are obtained. Construct the OXYZ coordinate system and record the positions in the camera coordinate system as The positions in the robotic arm base coordinate system are Thus, the transformation matrix T from the camera coordinate system to the robotic arm coordinate system is solved. cam2 b ase 。

[0024] Further, in S3, the transformation matrix T cam2base In it, taking as column vectors to form matrix R base2cnm From this, the rotation matrix in the transformation matrix T acm2base and the offset vector are obtained. Where is obtained in the following way:

[0025]

[0026] Further, in S4, it specifically includes the following steps:

[0027] Set three reflective markers A, B, and C at the end of the teaching gun, use them as the origin of the teaching gun coordinate system, and ensure that AB is perpendicular to BC. Specify the tip of the gun as point N;

[0028] Obtain the relevant structural dimension information of the teaching gun from the CAD model, and further calculate and determine the specific pose of point N in the set teaching gun coordinate system.

[0029] Further, in S5, it specifically includes the following steps:

[0030] With the help of a reflective marker coordinate system positioning system based on multiple infrared cameras, capture and obtain the coordinates of the three reflective markers A, B, and C set on the teaching gun in the camera coordinate system. Then, based on the rigid body formed by points A, B, and C, use the tracking function of the reflective marker coordinate system based on multiple infrared cameras to obtain the 6-degree-of-freedom pose of the teaching gun rigid body as a whole in the camera coordinate system. Based on the 6-degree-of-freedom pose and the correlation relationship between points, calculate and obtain the pose of the teaching gun tip N in the camera coordinate system.

[0031] Further, in S6, it specifically includes the following steps:

[0032] Taking the 6D pose of the tip of the teaching gun rigid body obtained by tracking through the multi-infrared camera positioning system as the basic data, based on the corresponding coordinate transformation principle and mathematical calculation method, the position and attitude correspondence relationship between the coordinate system of the teaching gun tip and the camera coordinate system is deduced, and the transformation matrix T that can accurately reflect the mutual conversion relationship between the two is solved. marker2cam 。

[0033] Further, in S7, it specifically includes the following steps:

[0034] Based on the obtained transformation matrix T from the coordinate system of the teaching gun tip to the camera coordinate system marker2cam , further solve the transformation matrix T from the teaching gun coordinate system to the robot coordinate system marker2base :

[0035] T marker2base = T cam2base * T marker2cam

[0036] After that, continuously repeat the steps of S6 to S7, and sequentially advance the teaching tasks at each stage until all are completed. Finally, convert the key points reflecting the pose in each stage into the form of straight lines and arc trajectories according to the rules, and export them as a trajectory annotation file in the json format, providing a basis for the robotic arm to move precisely along the established trajectory.

[0037] The second aspect of the present invention provides a robotic arm teaching system based on multi-infrared camera positioning, including:

[0038] A plurality of reflective markers provided at the end of the welding torch of the robotic arm assembly, used to cooperate with the infrared camera to achieve positioning;

[0039] A teaching gun device, with a plurality of reflective markers with specific positional relationships for constructing the coordinate system of the teaching gun provided at its end, and the coordinate system of the teaching gun is associated with the teaching operation of the robotic arm;

[0040] A multi-infrared camera group, distributed around the working area of the robotic arm, obtains image information by monitoring the reflective markers. The multi-infrared camera group has its own camera coordinate systems, and the camera coordinate systems can work together through calibration and association;

[0041] A control system, which is communicatively connected to a robotic arm assembly and a multi-infrared camera group. The control system is used to receive the image data collected by the cameras and process it based on a preset algorithm. The preset algorithm includes calculating and determining the transformation matrix between different coordinate systems according to the position information of the reflective markers in different coordinate systems. Specifically, by tracking and collecting data on the reflective markers of the robotic arm welding torch and the teaching gun in the reflective marker coordinate system positioning system, the transformation matrices from the camera coordinate system to the robotic arm coordinate system, from the teaching gun coordinate system to the camera coordinate system, and from the teaching gun coordinate system to the robot coordinate system are solved. The control system is also used to convert the teaching action information of the teaching gun into a motion instruction that the robotic arm can recognize and execute according to the above transformation matrix, so as to guide the robotic arm to move along a predetermined trajectory.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1) In the present invention, the multi-infrared camera positioning module effectively solves the occlusion problem. Multiple infrared cameras monitor the robotic arm and the teaching gun from different perspectives. Compared with a single-camera system, the risk of positioning failure caused by object occlusion is greatly reduced, ensuring the stability and continuity of positioning. This enables the robotic arm to accurately position even in a complex working environment, especially suitable for high-precision application scenarios such as welding where there are many possibilities of occlusion.

[0044] 2) The depth accuracy is significantly improved in the present invention. The multi-infrared camera positioning module uses multi-view information fusion technology and, through geometric correction and view compensation means, comprehensively calculates and analyzes the information captured by multiple cameras. Compared with the limited depth accuracy of a single-camera system, the present invention can more accurately determine the target pose in three-dimensional space, greatly improving the positioning accuracy, providing a reliable guarantee for the robotic arm operation that requires high-precision positioning, and effectively improving the product processing quality and production efficiency.

[0045] 3) A general hand-eye calibration method is proposed in the present invention. The present invention realizes the high-precision calibration of the robotic arm base coordinate system and the multi-camera system, and establishes an accurate conversion relationship from the camera coordinate system to the robotic arm coordinate system. This general method provides a solid technical support for the application of the multi-infrared camera positioning system in robotic arm teaching, simplifies the system integration and debugging process, reduces the technical threshold and application cost, and is conducive to the wide promotion of this technology in the industrial field.

[0046] 4) The teaching method in the present invention is intuitive and easy to use. The design of the teaching gun fully considers the operating habits of workers. Its shape and operation method are similar to those of common tools used by workers (such as welding torches and paintbrushes). Workers can easily start operating without complex additional learning. This user-friendly design improves the efficiency and accuracy of the teaching process, reduces errors and time waste caused by unskilled operations, and also reduces the requirements for the professional skills of operators, facilitating more ordinary workers to participate in the production operations related to robotic arms, further enhancing the flexibility and adaptability of production. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic structural diagram of the robotic arm teaching system based on the teaching gun in the present invention.

[0048] In the figure: 1, infrared camera; 2, robotic arm; 3, teaching gun; 4, reflective marker. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The present invention will be described in detail below with reference to the drawings and specific embodiments. Features such as component models, material names, connection structures, control methods, algorithms, etc. that are not clearly described in this technical solution are regarded as common technical features disclosed in the prior art.

[0050] Embodiment 1

[0051] The robotic arm teaching method based on multi-infrared camera positioning in this embodiment includes the following steps:

[0052] S1: Set 3 reflective markers at the end of the robotic arm welding torch and establish a coordinate system to determine the pose of the corresponding point M at the tip of the welding torch;

[0053] In S1, the 3 reflective markers are distributed in different orientations at the end of the welding torch, and the 3 reflective markers are used to provide reflected light to the infrared camera. Figure 1 It is a schematic structural diagram of the robotic arm teaching system based on the teaching gun in the present invention. Figure 1 It includes: infrared camera 1, robotic arm 2, teaching gun 3, reflective marker 4.

[0054] In S1, the pose of M in the coordinate system is calculated according to the CAD model.

[0055] The following is the principle explanation of the above steps:

[0056] Principle of setting 3 reflective markers

[0057] Three reflective markers are set at different orientations at the end of the robotic arm welding torch to provide reflected light to the infrared camera from multiple angles. Since the posture and position of the robotic arm change continuously during operation, the reflective markers at multiple different orientations can ensure that, regardless of the angle and position of the welding torch, there is a high probability that the reflected light signal can be captured by the infrared camera from different perspectives. This can avoid the situation where the position information of the welding torch cannot be accurately obtained due to a single reflective marker being blocked or in the blind spot of the camera monitoring, increasing the comprehensiveness and reliability of the positioning information acquisition, and thus laying a foundation for accurately determining the position and posture of the welding torch in space subsequently.

[0058] The purpose of multiple infrared cameras working together is to accurately position the robotic arm. As objects that can be clearly recognized by the camera and reflect light, the reflective markers become the key reference targets for positioning. Through the reflected light, the camera can clearly capture the positions of these markers, and then, based on their relative relationships with the entire welding torch and the tip of the welding torch, calculate the relevant pose information of the welding torch, which is an important prerequisite for the multi-infrared camera positioning system to function.

[0059] Principle of establishing a coordinate system

[0060] Establishing a coordinate system is to construct a unified and accurate positioning reference framework in space. In the positioning and motion control scenarios related to the robotic arm, it is necessary to clarify the specific positions and postures of each component in the three-dimensional space. The coordinate system provides such a standard quantification system, which can accurately describe the position of the welding torch and the key points on it (such as the corresponding point M of the welding torch tip) in space in the form of coordinate values, facilitating subsequent operations such as calculating various position relationships, converting between different coordinate systems, and planning the motion trajectory of the robotic arm.

[0061] By incorporating the three reflective markers and the corresponding point M of the welding torch tip into the established coordinate system, their relative position relationships can be clearly determined. For example, knowing the coordinate positions of the reflective markers in the coordinate system and combining their fixed relative positions with the welding torch tip (since they are installed at the established positions at the end of the welding torch), the coordinate position of the corresponding point M of the welding torch tip can be calculated through the corresponding geometric relationships and coordinate operations, and then its pose can be determined. At the same time, this coordinate system can also establish connections with other coordinate systems (such as the camera coordinate system, the robotic arm base coordinate system, etc.) in the future to achieve the conversion and unification of position information under different perspectives and different references.

[0062] Principle of calculating the pose of M in the coordinate system according to the CAD model

[0063] The CAD (Computer-Aided Design) model contains information such as the detailed structural design dimensions, shape, and precise positional relationships between components of the robotic arm welding torch. During the actual manufacturing process of the robotic arm, the welding torch is produced according to the specifications determined by the CAD model, so the relative positions of its various parts are known and determined. For the corresponding point M at the tip of the welding torch, in the CAD model, its exact positional relationship relative to the end of the welding torch, that is, relative to those 3 reflective markers, can be clearly defined, such as geometric parameters like relative distance and angle.

[0064] Based on the accurate structural and positional relationship information provided by the CAD model, through corresponding geometric coordinate calculation methods, the corresponding point M at the tip of the welding torch can be accurately positioned in this established coordinate system, and its specific coordinate values and attitude information (such as its orientation in three-dimensional space, etc.) can be calculated.

[0065] S2: Establish a positioning system for the reflective marker coordinate system based on multiple infrared cameras;

[0066] In S2, it specifically includes the following steps:

[0067] Install infrared cameras according to the working range, teaching area, and occlusion situation of the robotic arm, and calibrate the position and angle;

[0068] Configure the camera parameters and perform internal and external parameter calibration;

[0069] Construct a coordinate system with the reflective marker as the object, determine the origin and the direction of the coordinate axes, and associate the position information of the reflective marker in the camera coordinate system with the constructed reflective marker coordinate system, thereby establishing a complete positioning system.

[0070] The following is a detailed description of the principles of the above steps:

[0071] The teaching area is a specific spatial range where the operator performs the teaching operation of the robotic arm. Within this area, the robotic arm needs to move accurately according to the teaching requirements and be precisely positioned. Therefore, the installation position of the camera should be adjusted according to the position and size of the teaching area, and the camera should be mainly arranged in places where the key actions and position changes of the robotic arm during the teaching process can be clearly captured, ensuring that sufficient accurate positioning data can be obtained during the teaching session and providing a good observation basis for subsequent precise teaching positioning.

[0072] In the actual working environment, various obstacles may exist around the robotic arm. These obstacles are likely to block the observation of the camera and affect the positioning effect. By analyzing the occlusion situation in advance, such as identifying the directions and areas where there may be occlusion risks, the installation position and angle of the camera can be adjusted accordingly. Multiple cameras with different perspectives are used for complementarity, so that the monitoring fields of each camera cooperate with each other, minimizing the occurrence of situations where the reflected light of the reflective marker cannot be obtained due to occlusion as much as possible, ensuring the stability and continuity of the positioning system. Even if some perspectives are blocked, other cameras can still normally capture the reflective marker to complete the positioning.

[0073] Calibrating the position and angle of the camera is to ensure the accuracy and effectiveness of its monitoring field. An accurate position can ensure that the camera can accurately align with the area where the reflective marker is located at the end of the robotic arm and its welding torch. A suitable angle can minimize problems such as image distortion and reflection interference to the greatest extent, making the image of the reflective marker obtained by the camera clearer and more accurate. Furthermore, it can improve the accuracy of subsequent positioning calculations based on the image, enabling the captured image by the camera to truly reflect the actual position and posture of the reflective marker in space, providing reliable original image data for the entire positioning system.

[0074] Principle of configuring basic camera parameters

[0075] The camera itself has many adjustable parameters, such as exposure time, gain, resolution, etc. Reasonably configuring these parameters is carried out according to the specific working environment and positioning requirements. For example, in a working environment with relatively dim light, appropriately increasing the exposure time or gain can improve the brightness of the image, making the reflective marker more clearly distinguishable in the image. And according to the required positioning accuracy and data processing ability, selecting an appropriate resolution can not only ensure obtaining clear enough image details for positioning calculations, but also avoid excessive data volume caused by too high a resolution, increasing the processing burden. Thus, it optimizes the data acquisition link of the entire positioning system, ensuring that good-quality image data is obtained for subsequent analysis.

[0076] Principle of internal parameter calibration

[0077] The internal parameters of the camera include focal length, principal point coordinates, lens distortion coefficients, etc. These parameters reflect the optical characteristics of the camera lens itself and the inherent properties related to the imaging principle. Under different usage environments, individual differences of the camera, and changes caused by long-term use, etc., these internal parameters may deviate. Through internal parameter calibration, using specific calibration algorithms and tools such as calibration boards, accurately measuring and correcting these parameters can make the camera imaging conform to the ideal geometric model, reduce image distortion, and ensure the accuracy of the conversion from image coordinates to actual space coordinates. In this way, when subsequently calculating the real position of the reflective marker in space based on the image of the reflective marker obtained by the camera, calculations can be performed based on accurate internal parameters, improving the positioning accuracy.

[0078] Principle of External Parameter Calibration

[0079] External parameters mainly involve the position and attitude relationship of the camera relative to a known coordinate system (such as the world coordinate system or the base coordinate system of the robotic arm, etc.), that is, the rotation and translation relationships. Accurate external parameters are the key to realizing the conversion between different coordinate systems. Through external parameter calibration, using multiple control points with known positions (such as calibration objects placed at specific positions or reflective markers with known coordinates, etc.), combined with the image coordinates of these control points captured by the camera, and applying spatial geometric transformation algorithms, the position and attitude of the camera in the coordinate system of the entire positioning system can be accurately determined. This enables the subsequent accurate conversion of the position information of the reflective marker obtained in the camera coordinate system to other relevant coordinate systems, such as associating with the motion control coordinate system of the robotic arm to achieve precise control of the robotic arm based on camera positioning.

[0080] Principle of Constructing a Coordinate System with Reflective Markers

[0081] Reflective markers are the key targets observed by the camera on the robotic arm. Constructing a coordinate system with them as the object has clear positioning guidance. Since the core purpose of the entire positioning system is to determine the position and attitude of the robotic arm's welding torch, and there is a fixed installation position relationship between the reflective markers and the welding torch, establishing a coordinate system around the reflective markers is equivalent to establishing a local coordinate system closely connected to the key parts of the robotic arm. By determining the origin of this coordinate system (for example, a specific reflective marker or its geometric center, etc. can be selected as the origin) and the direction of the coordinate axes (reasonably set according to the actual situation such as the motion direction of the robotic arm and the working plane, such as taking the axis direction of the welding torch as the direction of a certain coordinate axis, etc.), it can provide a relatively intuitive and practical reference system for subsequent description of the position and attitude of the welding torch and the entire robotic arm, facilitating various quantitative representations and calculations of positions and attitudes.

[0082] Principle of Associating Position Information

[0083] Associating the position information of the reflective marker in the camera coordinate system with the constructed reflective marker coordinate system is a crucial step in realizing the integration of the entire positioning system. The camera coordinate system is established based on the camera imaging principle and reflects the position of the objects captured by the camera in its own imaging space. The reflective marker coordinate system is a local coordinate system established based on the actual targets on the robotic arm. Through specific coordinate transformation algorithms and calibration processes, the corresponding relationship between the two can be found. For example, by using the coordinates of multiple reflective markers in the camera coordinate system and their known relative positions in the reflective marker coordinate system, mathematical models such as coordinate transformation matrices are established, enabling the conversion of the position information of the reflective markers observed in the camera coordinate system to the reflective marker coordinate system, and further deriving the position and pose information of the robotic arm welding torch closely connected to the reflective marker.

[0084] S3: Establish a rigid body of the reflective marker coordinate system in the reflective marker coordinate system positioning system, establish the corresponding point M of the welding torch, move the tip M of the welding torch to multiple positions, record the positions of the moved tip of the welding torch in the camera and the robotic arm base coordinate systems, and solve the transformation matrix from the camera coordinate system to the robotic arm coordinate system;

[0085] In S3, it specifically includes the following steps:

[0086] Establish a rigid body of the reflective marker coordinate system in the multi-infrared-camera-based reflective marker coordinate system positioning system, establish the corresponding point M of the welding torch, track and obtain the rigid body coordinate system, and the 6D pose of the tip M of the welding torch in the camera coordinate system;

[0087] Move the tip M of the robotic arm welding torch to point O, and then move from point O to points X, Y, and Z along the XYZ axes respectively, obtaining a total of four positions O, X, Y, and Z. Construct the OXYZ coordinate system and record the positions in the camera coordinate system as The positions in the robotic arm base coordinate system are Thus, solve the transformation matrix T from the camera coordinate system to the robotic arm coordinate system cam2 b ase 。

[0088] In S3, the transformation matrix T cam2base Among them, taking as column vectors to form matrix R base2cam , thereby obtaining the transformation matrix T cam2base The rotation matrix in and the offset vector Among them The acquisition method of

[0089]

[0090] The following is a detailed explanation of the principle of the above steps:

[0091] First, establish a rigid body of the reflective marker coordinate system in the reflective marker coordinate system positioning system and establish the corresponding point M of the welding torch. The principle of this step is to associate the actual object of the robotic arm welding torch with the abstract coordinate system. The establishment of the rigid body of the reflective marker coordinate system can construct a relatively stable reference frame based on the relative fixed position and attitude of the reflective markers in space. As the point describing the key position of the welding torch, relying on this rigid body reference, through the monitoring of the reflective markers by multiple infrared cameras and the corresponding tracking algorithms, the rigid body coordinate system containing the tip M of the welding torch and its 6D pose information in the camera coordinate system can be obtained. This is like "positioning" the welding torch in the space observed by the camera and clarifying its attitude, providing basic data for further analyzing its position relationship in different coordinate systems. Then, move the tip M of the robotic arm welding torch to specific multiple points. The reason for doing this is to create multiple sets of sample data in different position states. Since only one position cannot accurately reflect the transformation relationship between the camera coordinate system and the robotic arm base coordinate system, by moving the tip of the welding torch to multiple representative points in different directions (moving along the XYZ axes) and constructing the corresponding OXYZ coordinate system, the position of the welding torch in the camera coordinate system and the robotic arm base coordinate system can be observed and recorded from multiple angles, thereby preparing sufficient and necessary information basis for solving the transformation matrix between the two coordinate systems.

[0092] The core principle of solving the transformation matrix from the camera coordinate system to the robotic arm coordinate system is to achieve coordinate transformation by establishing a mathematical correspondence between the two. After obtaining the position of the tip M of the welding torch in the camera coordinate system and its position in the robotic arm base coordinate system, using the corresponding position data in these different coordinate systems, in a specific calculation method, they are sorted into corresponding column vectors to form a matrix. Through such matrix construction and established mathematical operation rules, such as the given acquisition method, the rotation matrix and the offset vector are extracted from it, and finally the transformation matrix from the camera coordinate system to the robotic arm coordinate system is accurately obtained. This transformation matrix is like a "bridge", enabling the subsequent accurate conversion of the position information observed by the camera to the coordinate system of the robotic arm itself, thereby realizing precise control of the robotic arm movement based on camera positioning and laying a key mathematical foundation for accurately knowing the actual position of the robotic arm in the entire robotic arm teaching and subsequent work processes.

[0093] S4: Set three reflective markers A, B, and C at the end of the teaching gun as the origin of the teaching gun coordinate system, with AB perpendicular to BC, and determine the pose of the teaching gun tip N;

[0094] In S4, it specifically includes the following steps:

[0095] Set three reflective markers A, B, and C at the end of the teaching gun, use them as the origin of the teaching gun coordinate system, and ensure that AB and BC are perpendicular to each other. Define the tip of the gun as point N;

[0096] Obtain the relevant structural dimension information of the teaching gun from the CAD model, and further calculate and determine the specific pose of point N in the pre-set teaching gun coordinate system.

[0097] Set three reflective markers A, B, and C at the end of the teaching gun and use them as the origin of the teaching gun coordinate system. At the same time, require AB and BC to be perpendicular. These three reflective markers are set as the origin of the coordinate system because they have relatively fixed positions on the teaching gun and the reflected light can be clearly captured by multiple infrared cameras. Using them as a reference starting point to construct the coordinate system is like establishing an "anchor point" for the teaching gun in space, which is convenient for subsequent description of the positions and postures of various parts of the teaching gun in space. Making AB and BC perpendicular to each other is to construct a reference system that conforms to the rules of the space rectangular coordinate system. The rectangular coordinate system has the advantages of intuitiveness and convenience in operation in mathematical calculations and spatial positioning descriptions, and can more accurately and conveniently reflect the spatial relationships of different positions of the teaching gun through coordinate values.

[0098] Use the CAD model to obtain the relevant structural dimension information of the teaching gun to further calculate and determine the specific pose of point N in the pre-set teaching gun coordinate system. The principle is to utilize the precise design data contained in the CAD model. The CAD model details the key parameters such as the dimensions and positional relationships between the overall shape of the teaching gun and its internal components, which are accurate and reliable information determined during the design stage. Since there are fixed relative geometric relationships between point N (the tip of the teaching gun) and the three reflective markers A, B, and C set at the end, through the structural dimension information such as the distances and angles between them in the CAD model, corresponding geometric calculation methods can be used, such as vector operations and spatial geometric theorems. In the teaching gun coordinate system with A, B, and C as the origin that has been constructed, the coordinate values of point N and its pose situation (such as its orientation in space, etc.) can be accurately calculated, providing an accurate initial pose data basis based on the design source for subsequent accurate teaching based on the teaching gun and operations such as coordinate transformation with other coordinate systems.

[0099] S5: Obtain the coordinates of points A, B, and C on the teaching gun in the camera coordinate system through the reflective marker coordinate system positioning system based on multiple infrared cameras, track and obtain the 6D pose of the teaching gun rigid body, and calculate the pose of the teaching gun tip N;

[0100] In S5, it specifically includes the following steps:

[0101] With the help of a positioning system based on a multi-infrared camera for the reflective marker coordinate system, capture and obtain the coordinates of three reflective markers A, B, and C set on the teaching gun in the camera coordinate system. Subsequently, based on the rigid body formed by points A, B, and C, use the tracking function of the reflective marker coordinate system based on the multi-infrared camera to obtain the 6-degree-of-freedom pose of the entire teaching gun rigid body in the camera coordinate system. Then, based on the 6-degree-of-freedom pose and the correlation relationships between points, calculate and obtain the pose of the teaching gun tip N in the camera coordinate system.

[0102] With the help of a positioning system based on a multi-infrared camera for the reflective marker coordinate system to capture and obtain the coordinates of points A, B, and C on the teaching gun in the camera coordinate system, which is the basic operation of the entire positioning process. The multi-infrared camera can observe the reflective markers at the end of the teaching gun from different perspectives. Since these reflective markers can be clearly identified by the reflected light in the camera's field of view, the camera can accurately record their corresponding coordinate positions. And the subsequent operations are carried out based on the rigid body formed by points A, B, and C because the rigid body has a relatively fixed shape and position relationship in space, and its movement can be described by unified coordinate transformation and attitude changes. Using the tracking function of this positioning system to monitor this rigid body can obtain the 6-degree-of-freedom pose of the entire teaching gun rigid body in the camera coordinate system. The 6 degrees of freedom cover the translation and rotation information of the rigid body in three-dimensional space, comprehensively reflecting the position and attitude of the teaching gun in the space observed by the camera, providing an overall framework data basis for further accurately analyzing the pose of its key parts.

[0103] On this basis, further calculate and obtain the pose of the teaching gun tip N in the camera coordinate system based on the 6-degree-of-freedom pose and the correlation relationships between points. Since there are clear geometric relationships between the teaching gun tip N and the rigid body formed by the three reflective markers A, B, and C, such as relative distances and angles, these relationships are fixed in the structural design of the teaching gun. Then, when the 6-degree-of-freedom pose of the entire rigid body is known, spatial geometry calculation methods can be used, such as vector operations. According to the relative position vectors of points A, B, C and point N, combined with the translation and rotation of the rigid body, integrate this information for calculation, so as to accurately deduce the specific coordinate position and corresponding attitude of the teaching gun tip N in the camera coordinate system, and further provide key and refined positioning data for subsequent operations such as coordinate transformation between different coordinate systems and precise teaching of the robotic arm according to the teaching gun.

[0104] S6: Use the pose of the teaching gun tip N to solve the transformation matrix from the teaching gun tip coordinate system to the camera coordinate system;

[0105] In S6, it specifically includes the following steps:

[0106] Taking the 6D pose of the tip of the teaching gun rigid body obtained by tracking through the multi-infrared camera positioning system as the basic data, based on the corresponding coordinate transformation principle and mathematical calculation method, the position and attitude correspondence between the coordinate system of the teaching gun tip and the camera coordinate system is deduced, and the transformation matrix T that can accurately reflect the mutual conversion relationship between the two is solved. marker2cam 。

[0107] The reason for taking the 6D pose of the tip of the teaching gun rigid body obtained by tracking through the multi-infrared camera positioning system as the basic data is that this 6D pose contains the comprehensive position and attitude information of the teaching gun tip N in space. It details the translation of the teaching gun tip N along the three coordinate axes and the rotation around these three coordinate axes within the space observed by the camera, which is an accurate quantitative expression of the state of the teaching gun tip N. And our purpose is to find the conversion relationship between the coordinate system of the teaching gun tip and the camera coordinate system, so this accurate 6D pose data becomes the key starting basis for subsequent derivation and calculation.

[0108] Deriving and solving the transformation matrix according to the corresponding coordinate transformation principle and mathematical calculation method, the principle is that there are specific mathematical correlations between different coordinate systems. The coordinate system of the teaching gun tip focuses on the reference system for describing the teaching gun tip itself and its surrounding relative positions, while the camera coordinate system is a spatial positioning framework established based on camera imaging and observation. By applying knowledge such as the coordinate transformation theory in spatial geometry, like rotation matrices and translation vectors, using the known 6D pose of the teaching gun tip N, the coordinate representation form of the teaching gun tip in its own coordinate system is made to correspond to its coordinate situation in the camera coordinate system through a series of mathematical calculations. After rigorous derivation and operation, the transformation matrix that can accurately reflect the mutual conversion relationship between the two is finally obtained.

[0109] S7: Solve the transformation matrix from the teaching gun coordinate system to the robot coordinate system, and repeat S6 to S7 until the teaching of each stage is completed.

[0110] In S7, it specifically includes the following steps:

[0111] Based on the transformation matrix T from the coordinate system of the teaching gun tip to the camera coordinate system that has been obtained marker2cam , further solve the transformation matrix T from the teaching gun coordinate system to the robot coordinate system marker2base :

[0112] T marker2base =T cam2base *T marker2cam

[0113] Subsequently, steps S6 to S7 are continuously repeated to sequentially advance the teaching tasks of each stage until all are completed. Finally, the key points reflecting the pose in each stage are converted into straight line and circular arc trajectory forms according to the rules and exported as a trajectory annotation file in json format, providing a basis for the robotic arm to move precisely along the established trajectory.

[0114] Based on the transformation matrix from the teaching gun tip coordinate system to the camera coordinate system that has been obtained, further solve the transformation matrix from the teaching gun coordinate system to the robot coordinate system. The principle lies in constructing the association chain between different coordinate systems to achieve the precise control of the robotic arm. The teaching gun coordinate system is a reference system built around the teaching gun to describe its own position and pose, while the robot coordinate system is the key framework for positioning the overall movement of the entire robotic arm. The obtained transformation matrix from the teaching gun tip coordinate system to the camera coordinate system reflects the position and pose correspondence between the teaching gun tip and the camera observation. Utilizing this intermediate conversion relationship, combined with the fixed geometric relationships between the various parts of the teaching gun and the gun tip, as well as the structural and motion associations between the robotic arm and the teaching gun, through corresponding coordinate transformations, geometric derivations, and other mathematical means, the transformation matrix from the teaching gun coordinate system to the robot coordinate system can be derived, thereby accurately converting the position and pose information of the teaching gun to the coordinate system of the entire robotic arm, enabling the robotic arm to know how to perform corresponding actions according to the guidance of the teaching gun.

[0115] Continuously repeating steps S6 to S7 to advance the teaching tasks of each stage until all are completed is because the teaching of the robotic arm is often a complex and multi-stage process. Different stages may involve different actions, position changes, etc. Each time these two steps are repeated, the transformation matrix between different coordinate systems can be continuously updated and calibrated to ensure that the position and pose information of each link can be accurately converted between different coordinate systems during the entire teaching process, achieving precise positioning. And finally, the key points reflecting the pose in each stage are converted into straight line and circular arc trajectory forms according to the rules and exported as a trajectory annotation file in json format. This is to provide recognizable and easily processable data basis for the robotic arm to move precisely along the established trajectory. Converting the pose key points into specific geometric trajectory forms is an operation that conforms to the actual motion mode of the robotic arm. The json format is convenient for storage, transmission, and subsequent reading and parsing of these trajectory data by the robotic arm control system, enabling the robotic arm to strictly perform precise motion operations according to the pre-set trajectory and achieving efficient and accurate conversion of the teaching results.

[0116] Embodiment 2

[0117] In this embodiment, the robotic arm teaching system based on multi-infrared camera positioning includes: a plurality of reflective markers disposed at the end of the welding torch of the robotic arm assembly for cooperating with the infrared cameras to achieve positioning; a teaching gun device, at the end of which there are a plurality of reflective markers with specific positional relationships for constructing the teaching gun coordinate system, and the teaching gun coordinate system is associated with the teaching operation of the robotic arm; a multi-infrared camera group, distributed around the working area of the robotic arm, which obtains image information by monitoring the reflective markers. The multi-infrared camera group has its own camera coordinate systems, and the camera coordinate systems can work together through calibration and association; a control system, which is communicatively connected to the robotic arm assembly and the multi-infrared camera group. The control system is used to receive the image data collected by the cameras and process it based on a preset algorithm. The preset algorithm includes calculating and determining the transformation matrix between different coordinate systems according to the position information of the reflective markers in different coordinate systems. Specifically, by tracking and collecting data on the reflective markers of the robotic arm welding torch and the teaching gun in the reflective marker coordinate system, the transformation matrices from the camera coordinate system to the robotic arm coordinate system, from the teaching gun coordinate system to the camera coordinate system, and from the teaching gun coordinate system to the robot coordinate system are solved. The control system is also used to convert the teaching action information of the teaching gun into a motion instruction that the robotic arm can recognize and execute according to the above transformation matrices, so as to guide the robotic arm to move along a predetermined trajectory. The controller is one of a single-chip microcomputer or a processor with an x86 architecture, an ARM architecture, or a RISC-V architecture.

[0118] The number of reflective markers at the end of the welding torch of the robotic arm assembly is at least three, and their distribution method can ensure that reflected light is provided to the infrared cameras from multiple angles to avoid the situation where the position information of the welding torch cannot be accurately obtained due to a single reflective marker being possibly blocked or in the camera monitoring blind area. The relative positional relationship of the reflective markers at the end of the welding torch is known and fixed, and their positional relationship with the tip of the welding torch can be determined through a CAD model or other preset precise models, so as to deduce the pose of the tip of the welding torch based on the position information of the reflective markers. At the same time, the establishment method and parameters of the robotic arm coordinate system can be flexibly adjusted and optimized according to the actual structure and kinematic characteristics of the robotic arm to adapt to different types and specifications of robotic arms, and the conversion relationship between this coordinate system and other coordinate systems can be accurately calculated and updated in real time by the control system based on the collected data, not limited to specific coordinate system establishment methods and conversion algorithms, covering any effective technical means that can achieve accurate coordinate conversion and robotic arm motion control.

[0119] The number of end reflective markers of the teaching gun device is at least three and satisfies a specific geometric positional relationship. For example, some of the markers are perpendicular to each other or have other positional relationships that can clearly determine the direction and origin of the coordinate system. The pose of the teaching gun tip can be accurately calculated and determined by combining the position information of the reflective markers with a CAD model or other precise design data. The establishment and parameter setting of the teaching gun coordinate system can be flexibly adjusted according to the actual design and usage requirements of the teaching gun to facilitate the conversion and association with the manipulator coordinate system and the camera coordinate system. Moreover, the teaching gun can closely cooperate with the teaching operation of the manipulator in terms of structure and function. It can simulate various actions and poses of the manipulator and, at the same time, accurately transmit its own pose information to the control system to achieve precise teaching guidance for the manipulator, and is not limited to a specific teaching gun structure and teaching method, covering any reasonable design and operation method that can achieve an effective teaching function.

[0120] The number, position, and angle of the multi-infrared camera group can be automatically adjusted and optimized according to the working range of the manipulator, the teaching area, and the occlusion situation of the surrounding environment. The parameter configuration of the camera, including internal parameters such as exposure time, gain, resolution, focal length, principal point coordinates, and lens distortion coefficient, and external parameters of the camera relative to other coordinate systems, can be automatically calibrated and updated in real time by the control system based on the real-time collected data and preset calibration algorithms to ensure that the image data obtained by the camera is clear and accurate and can accurately reflect the actual position and posture of the reflective markers in space, thereby providing reliable original data for the entire positioning and teaching system.

[0121] The above description of the embodiments is to enable those of ordinary skill in the art to understand and use the invention. It is obvious that those skilled in the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without creative effort. Therefore, the present invention is not limited to the above embodiments, and the improvements and modifications made by those skilled in the art without departing from the scope of the present invention should be within the protection scope of the present invention.

Claims

1. A robot arm teaching method based on multi-infrared camera positioning, characterized in that: The following steps are involved: S1: Set three reflective markers at the end of the robot arm welding gun and establish a coordinate system to determine the position and posture of the corresponding point M at the welding gun tip; S2: Establish a reflective marker coordinate system positioning system based on multiple infrared cameras; S3: Establish a reflective marker coordinate system rigid body in the reflective marker coordinate system positioning system, establish a welding gun corresponding point M, move the welding gun tip M to multiple points, record the position of the moving welding gun tip in the camera and robot arm base coordinate system, and solve the camera coordinate system to the robot arm coordinate system transformation matrix; S4: Set three reflective markers A, B, and C at the end of the teaching gun as the origin of the teaching gun coordinate system, and AB is perpendicular to BC to determine the teaching gun tip N posture; S5: The coordinates of the three points ABC on the teaching gun in the camera coordinate system are obtained through the reflective marker coordinate system positioning system based on multiple infrared cameras, the 6d position of the teaching gun rigid body is tracked and obtained, and the position N of the teaching gun tip is calculated; S6: Using the teaching gun tip N poses to solve the transformation matrix from the teaching gun tip coordinate system to the camera coordinate system; S7: Solve the transformation matrix from the teaching gun coordinate system to the robot coordinate system, and repeat S6 to S7 until all stages of teaching are completed.

2. A robot arm teaching method based on multi-infrared camera positioning according to claim 1, characterized in that: In S1, three reflective markers are distributed at different positions at the end of the welding gun, and the three reflective markers are used to provide reflected light to the infrared camera; In S1, the pose of M in the coordinate system is calculated based on the CAD model.

3. The method for teaching a robotic arm based on multi-infrared camera positioning according to claim 1, characterized in that: S2 specifically includes the following steps: Install infrared cameras according to the robot's working range, teaching area, and shielding conditions, and calibrate the position and angle; Configure camera parameters and calibrate internal and external parameters; A coordinate system is constructed with the reflective marker as the object, the origin and the direction of the coordinate axis are determined, and the position information of the reflective marker in the camera coordinate system is associated with the constructed reflective marker coordinate system, so as to establish a complete positioning system.

4. The method for teaching a robotic arm based on multi-infrared camera positioning according to claim 1, characterized in that: S3 specifically includes the following steps: In the reflective marker coordinate system positioning system based on multiple infrared cameras, a reflective marker coordinate system rigid body is established, and a welding gun corresponding point M is established, and the rigid body coordinate system and the 6d pose of the welding gun tip M in the camera coordinate system are tracked and obtained; Move the tip of the robot welding gun M to point O, and then move from point O to points X, Y, and Z along the XYZ axis, obtaining four positions O, X, Y, and Z. Construct the OXYZ coordinate system and record the position in the camera coordinate system as The position in the robot base coordinate system is Thus, the transformation matrix T from the camera coordinate system to the robotic arm coordinate system is solved cam2 b ase .

5. A robot arm teaching method based on multi-infrared camera positioning according to claim 4, characterized in that: In S3, the transformation matrix T cam2base In The column vectors form the matrix R base2cam , thus we get the transformation matrix T cam2base The rotation matrix in and the offset vector in The way to obtain is:

6. The method for teaching a robotic arm based on multi-infrared camera positioning according to claim 1, characterized in that: S4 specifically includes the following steps: Set three reflective markers A, B, and C at the end of the teaching gun, and use them as the origin of the teaching gun coordinate system. Ensure that AB and BC are perpendicular to each other, and clearly define the gun tip as point N; The relevant structural dimension information of the teaching gun is obtained from the CAD model, and the specific position and posture of point N in the set teaching gun coordinate system is further calculated and determined.

7. A robot arm teaching method based on multi-infrared camera positioning according to claim 6, characterized in that: S5 specifically includes the following steps: With the help of the reflective marker coordinate system positioning system based on multiple infrared cameras, the coordinates of the three reflective markers A, B, and C set on the teaching gun in the camera coordinate system are captured and obtained. Then, according to the rigid body formed by the three points A, B, and C, the tracking function of the reflective marker coordinate system based on multiple infrared cameras is used to obtain the 6-degree-of-freedom posture of the entire teaching gun rigid body in the camera coordinate system. Based on the 6-degree-of-freedom posture and the relationship between each point, the posture of the teaching gun tip N in the camera coordinate system is calculated.

8. The method for teaching a robotic arm based on multi-infrared camera positioning according to claim 7, characterized in that: S6 specifically includes the following steps: Taking the 6D position and posture of the gun tip in the teaching gun rigid body obtained by tracking through the multi-infrared camera positioning system as the basic data, according to the corresponding coordinate transformation principle and mathematical calculation method, the position and posture correspondence between the teaching gun tip coordinate system and the camera coordinate system is derived, and the transformation matrix T that can accurately reflect the mutual conversion relationship between the two is solved. marker2cam .

9. A robot arm teaching method based on multi-infrared camera positioning according to claim 8, characterized in that: S7 specifically includes the following steps: Based on the obtained transformation matrix T from the teaching gun tip coordinate system to the camera coordinate system marker2cam , and further solve the transformation matrix T from the teaching gun coordinate system to the robot coordinate system marker2base : T marker2base =T cam2base *T marker2cam Then, steps S6 to S7 are repeated continuously, and the teaching tasks of each stage are carried out in sequence until all are completed. Finally, the key points reflecting the posture in each stage are converted into straight line and circular arc trajectory forms according to the rules, and exported as trajectory annotation files in json format, providing a basis for the robot arm to move accurately along the predetermined trajectory.

10. A robotic arm teaching system based on multi-infrared camera positioning, characterized in that: include: Multiple reflective markers at the end of the welding gun of the robotic arm assembly are used to cooperate with the infrared camera to achieve positioning; A teaching gun device, the end of which is provided with a reflective marker for constructing a plurality of specific positional relationships of a teaching gun coordinate system, the teaching gun coordinate system being associated with the teaching operation of the robot arm; A multi-infrared camera group is distributed around the working area of ​​the robot arm, and image information is obtained by monitoring reflective markers. The multi-infrared camera group has its own camera coordinate system, and each camera coordinate system can work together through calibration and association; A control system is communicatively connected with a robotic arm assembly and a multi-infrared camera group. The control system is used to receive image data collected by the camera and process the data based on a preset algorithm. The preset algorithm includes calculating and determining a transformation matrix between different coordinate systems according to position information of reflective markers in different coordinate systems. Specifically, the reflective markers of the robotic arm welding gun and the teaching gun are tracked and data is collected in a reflective marker coordinate system positioning system to solve the transformation matrix from the camera coordinate system to the robotic arm coordinate system, from the teaching gun coordinate system to the camera coordinate system, and from the teaching gun coordinate system to the robot coordinate system. The control system is also used to convert the teaching action information of the teaching gun into motion instructions that can be recognized and executed by the robotic arm according to the transformation matrix, thereby guiding the robotic arm to move according to a predetermined trajectory.

Images