A method and system for target tracking in motion

Abstract

The application discloses a target tracking method and system in a motion state, and the method comprises the following steps: carrying out binocular target positioning in an offline state, establishing a binocular 3D imaging system, and carrying out hand-eye calibration and tool coordinate system calibration; acquiring 3D coordinates of a target point in the binocular 3D imaging system in real time; converting the 3D coordinates into coordinates of a laser coordinate system; calculating rotation angles of each axis of a manipulator and a rotating disc through the relationship between the laser coordinate system and the manipulator coordinate system; and controlling the rotation of the manipulator and the rotating disc according to the rotation angles. The 3D imaging system is adopted to position the 3D coordinates of the target point, the positioning speed is higher, the positioning precision is higher, and meanwhile, the rotating disc is introduced, so that 3D full-space dead-angle-free positioning can be realized.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and more specifically to a method and system for tracking targets in motion. Background Technology

[0002] Currently, the most common devices for tracking and locating target objects are human or dynamic target tracking in the security field. In industrial production, devices and methods for target tracking using visual positioning are not common. The commonly used devices for target tracking using visual positioning have simple functions and single performance parameters. Many do not pursue positioning accuracy or only have 2D positioning information. Moreover, they perform poorly in terms of real-time performance and tracking range, and their practicality is generally limited. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a method and system for tracking targets in motion with higher positioning accuracy and 3D coordinate information.

[0004] This invention provides a target tracking method in motion, the method comprising: performing dual-target calibration offline, establishing a binocular 3D imaging system, and performing hand-eye calibration and tool coordinate system calibration; acquiring the 3D coordinates of the target point in the binocular 3D imaging system in real time; converting the 3D coordinates into coordinates in the laser coordinate system; calculating the rotation angles of each axis of the robot and the turntable based on the relationship between the laser coordinate system and the robot coordinate system; and controlling the rotation of the robot and the turntable according to the rotation angles.

[0005] Optionally, the steps of performing hand-eye calibration and tool coordinate system calibration include:

[0006] Hand-eye calibration of the 3D coordinate system and laser coordinate system of the binocular 3D imaging system;

[0007] Perform coordinate system calibration between the laser and the robotic arm.

[0008] Optionally, the steps of performing binocular targeting and establishing a binocular 3D imaging system in offline mode include:

[0009] In offline mode, the intrinsic parameters of the left and right cameras and the positional relationship between the left and right cameras are calibrated, and a binocular 3D imaging system is established based on the intrinsic parameters and positional relationship.

[0010] Optionally, the step of obtaining the 3D coordinates of the target point in the binocular 3D imaging system includes:

[0011] Images containing the target point are captured by taking pictures with the left and right cameras. The row and column coordinates of the target point are located in the images, and the 3D coordinates of the target point are obtained according to the binocular 3D imaging system.

[0012] Optionally, the rotation angles of each axis of the computational robot and the turntable include:

[0013] Calculate the deflection and pitch angles of each axis of the robot arm, as well as the rotation angle of the turntable.

[0014] Optionally, the left and right cameras are mounted on the left and right sides of the robotic arm and move with the robotic arm.

[0015] Optionally, the robotic arm uses an axis-driven motion mode.

[0016] Optionally, the left and right cameras are high frame rate cameras.

[0017] Optionally, the intrinsic parameters include the camera's focal length, and the positional relationship includes the distance between the left and right cameras.

[0018] Another embodiment of the present invention provides a target tracking system in motion, the system including a control device and a six-degree-of-freedom manipulator, the six-degree-of-freedom manipulator being equipped with a left camera and a right camera, the control device being used to execute the target tracking method in motion described in any of the above claims.

[0019] In the technical solution provided by this invention, a binocular 3D imaging system is established offline through binocular positioning, and hand-eye calibration and tool coordinate system calibration are performed. The 3D coordinates of the target point in the binocular 3D imaging system are obtained in real time, and the 3D coordinates are converted into laser coordinate system coordinates. The rotation angles of each axis of the robot and the turntable are calculated through the relationship between the laser coordinate system and the robot coordinate system, thereby controlling the rotation of the robot and the turntable. Compared with the prior art, this invention uses a binocular positioning method to calculate the 3D coordinate information of the target point, which has higher positioning accuracy and faster speed. At the same time, the robot adopts a transparent transmission mode, which allows for unobstructed transmission. Moreover, the robot does not have any starting, acceleration, or deceleration actions during dynamic real-time tracking, thus eliminating dynamic stuttering and time consumption. In addition, the introduction of the turntable can achieve 3D full-space positioning without blind spots. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a target tracking method under motion conditions according to the present invention.

[0021] Figure 2 This is the pinhole imaging perspective projection model of the present invention.

[0022] Figure 3 This is a schematic diagram of the dual-target calibration method of the present invention.

[0023] Figure 4 This is a schematic diagram illustrating the relationship between the 3D coordinate system of the camera and the coordinate system of the laser in this invention.

[0024] Figure 5This is a schematic diagram of the target point rotating in the XOY plane in this invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention provides a target tracking system in motion. The system includes a rotary table and a control device. A six-degree-of-freedom (DOF) manipulator is mounted on the rotary table, which drives the manipulator to rotate. A left camera and a right camera are mounted on the manipulator. The manipulator controls the movement of the left and right cameras, thereby enabling 3D tracking. The control device employs the target tracking method in motion described in the following embodiment.

[0027] Please see Figure 1 As shown, the present invention provides a target tracking method in motion, the method comprising the following steps:

[0028] Step S10: Perform binocular calibration offline, establish a binocular 3D imaging system, and perform hand-eye calibration of the 3D coordinate system of the binocular 3D imaging system and the laser coordinate system, as well as coordinate system calibration between the laser and the robotic arm.

[0029] The steps for performing dual-target localization and establishing a binocular 3D imaging system in offline mode include:

[0030] In offline mode, the intrinsic parameters of the left and right cameras and the positional relationship between the left and right cameras are calibrated, and a binocular 3D imaging system is established based on the intrinsic parameters and positional relationship.

[0031] Specifically, the following steps are performed offline: establishing a binocular 3D imaging system, calibrating the hand-eye coordinate system and the laser coordinate system of the binocular 3D imaging system, and calibrating the coordinate system between the laser and the robotic arm.

[0032] Step S20: Real-time acquisition of the 3D coordinates of the target point in the binocular 3D imaging system.

[0033] Specifically, step S20 includes taking pictures with the left and right cameras to obtain images including the target point, locating the row and column coordinates of the target point in the images, and obtaining the 3D coordinates of the target point according to the binocular 3D imaging system.

[0034] Step S30: Convert the 3D coordinates into coordinates in the laser coordinate system.

[0035] Step S40: Calculate the rotation angles of each axis of the robot and the turntable by using the relationship between the laser coordinate system and the robot coordinate system.

[0036] The rotation angles of each axis of the computational robot and the turntable include:

[0037] Calculate the deflection and pitch angles of each axis of the robot arm, as well as the rotation angle of the turntable.

[0038] Step S50: Control the robot arm and turntable to rotate according to the rotation angle.

[0039] Please refer to Figure 2 The image shown is a pinhole imaging perspective projection model, a single-camera imaging model. Feature points are imaged through a pinhole, exposed and projected onto the camera's CCD chip, then converted into an image through photoelectric conversion. Its imaging mathematical model is as follows:

[0040]

[0041] Where f is the camera focal length, r is the distortion factor, and S x S y Given the pixel size, u0 as the pixel offset, and v0 as the camera's intrinsic parameters, these parameters can be obtained through camera calibration. A point in space (X...) w ,Y w Z w The pixel coordinates (u,v) are obtained through the above transformation.

[0042] Please refer to Figure 3 The diagram shows a schematic of the binocular calibration method. This method allows for the calibration of the intrinsic parameters of the two cameras, including their focal length *f*, and also allows for the calibration of their positional relationship (after correction), including the parameter *b* (distance between the two cameras). This leads to the creation of a binocular model.

[0043]

[0044] This invention obtains the row and column coordinates of a target point through positioning using a left and right camera. Positioning with the left and right cameras primarily involves determining the row and column coordinates of the target point within their respective images. Then, using the intrinsic parameters obtained from the aforementioned dual-camera positioning method, the 3D coordinates of the target point in the binocular 3D imaging system are calculated. Therefore, the row and column coordinates of the target point (i.e., the feature point) are located in the left and right camera images respectively. Then, combining the intrinsic parameters from the dual-camera positioning, the 3D coordinates of the feature point in the binocular 3D imaging system are calculated. During real-time positioning, the two cameras capture and position the target point. Through the camera imaging mathematical model, the position (xl, yl) and (xr, yr) in their respective camera coordinate systems can be calculated, thus yielding:

[0045]

[0046] x = xl * z / f

[0047] y = yl * z / f

[0048] The coordinates (x, y, z) of the target point in the binocular 3D imaging system can then be obtained.

[0049] This invention uses a dual-camera calibration method to calibrate the intrinsic parameters and positional relationship of two cameras. Once calibrated, the two cameras can calculate the 3D coordinates of a point in the binocular 3D imaging system (camera coordinate system), that is, the coordinates of the target point in the camera coordinate system.

[0050] Please refer to Figure 4 As shown, the following transformation matrix is ​​obtained through the hand-eye calibration method.

[0051] Among them, R 3×3 Let T be the rotation matrix. 3×1 It is a translation matrix.

[0052]

[0053] In this step, N feature points are selected, and the first coordinates of each feature point in the laser coordinate system and the second coordinates in the camera coordinate system are obtained. The first coordinates and the second coordinates are substituted into the mathematical model of the coordinate system transformation matrix to obtain 3N equations. All the equations constitute an overdetermined system of equations. The overdetermined system of equations is solved by the generalized inverse method of the least squares method of nonlinear equations to obtain the transformation matrix.

[0054] Specifically, the coordinates of N target points in the laser coordinate system and the camera coordinate system are obtained. When selecting a target point, three equations can be generated based on the mathematical model described above. Since the rotation matrix has 12 unknowns, 12 equations generated from 4 sets of known target points are needed to solve the transformation matrix. Finally, an overdetermined system of equations is constructed using the mathematical model, the coordinates in the laser coordinate system, and the coordinates in the camera coordinate system. This overdetermined system has multiple solutions, and to improve calibration accuracy, the optimal solution must be found. The 3D hand-eye calibration method for solving the transformation matrix between the 3D camera coordinate system and the laser coordinate system employs a generalized inverse method of the least squares method for iterative solving of the nonlinear equation system, and finds the optimal solution. This improves the accuracy of the transformation matrix and precisely controls the transformation relationship between the camera coordinate system and the laser coordinate system.

[0055] In this invention, the rotation angles of each axis of the robot and the turntable are calculated by utilizing the relationship between the laser coordinate system and the robot's coordinate system. Specifically, the coordinates of N target points in the robot's coordinate system and the camera coordinate system are obtained. When selecting a target point, three equations can be generated based on the aforementioned mathematical model. However, the rotation matrix has 12 unknowns, so 12 equations generated from 4 sets of known target points are needed to solve the transformation matrix. Finally, an overdetermined system of equations is constructed using the mathematical model, the coordinates in the robot's coordinate system, and the coordinates in the camera coordinate system. This overdetermined system of equations has multiple solutions, and to improve calibration accuracy, the optimal solution must be found.

[0056] The rotation angle includes the yaw angle and the pitch angle. The pitch angle is the angle of rotation around the X-axis in the laser coordinate system, and the yaw angle is the angle of rotation around the Y-axis. Please refer to [reference needed]. Figure 5 As shown, the target point is on the XOY plane. Taking the laser point on the plane where the laser line hits as the origin, and the direction of the laser line as the Z direction, the projection onto the ZOX plane has a deflection angle with the Z-axis, and the projection onto the YOZ plane has a pitch angle with the Z-axis.

[0057] In the technical solution provided by this invention, a binocular 3D imaging system is established offline using a binocular positioning method, and hand-eye calibration and tool coordinate system calibration are performed. The 3D coordinates of the target point in the binocular 3D imaging system are acquired in real time and converted into laser coordinate system coordinates. The rotation angles of each axis of the robot and the turntable are calculated based on the relationship between the laser coordinate system and the robot's coordinate system, thereby controlling the rotation of the robot and the turntable. Compared to existing technologies, this invention uses a binocular positioning method to calculate the 3D coordinate information of the target point, resulting in higher positioning accuracy and faster speed. Simultaneously, the robot uses a transparent transmission mode, ensuring unobstructed transmission. Furthermore, the robot does not undergo start-up, acceleration, or deceleration during dynamic real-time tracking, thus eliminating dynamic stuttering and time consumption. Additionally, it introduces… The turntable enables 3D full-space positioning without blind spots. Utilizing a high-frame-rate camera, it achieves fast positioning speed. Simultaneously, the robotic arm employs a pass-through mode, ensuring unobstructed transmission. Furthermore, the robotic arm operates without start-up, acceleration, or deceleration during dynamic real-time tracking, eliminating dynamic stuttering and time consumption. The dynamic positioning and tracking frequency can reach over 30Hz. The motion structure of this invention employs a robotic arm and turntable configuration, with the positioning camera mounted on the robotic arm and moving alongside it. This combined approach significantly increases the tracking range, theoretically enabling positioning and tracking within a 360-degree space. Additionally, the robotic arm uses an axial motion mode, effectively resolving the singularity problem commonly encountered in robotic arm movement. The tracking target of this invention is not fixed and can be set according to requirements, offering flexibility and a wide range of applications.

[0058] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A target tracking method in motion, characterized in that, The method includes: Perform binocular calibration offline, establish a binocular 3D imaging system, and perform hand-eye calibration and tool coordinate system calibration; Real-time acquisition of the 3D coordinates of the target point in the binocular 3D imaging system; Transform 3D coordinates into laser coordinates; By using the relationship between the laser coordinate system and the robot coordinate system, the rotation angles of each axis of the robot and the turntable are calculated. The rotation angle is used to control the rotation of the robotic arm and the turntable; The robotic arm is mounted on the turntable, and the rotation of the turntable drives the robotic arm to rotate. The steps for performing hand-eye calibration and tool coordinate system calibration include: Hand-eye calibration of the 3D coordinate system and laser coordinate system of the binocular 3D imaging system; Perform coordinate system calibration between the laser and the robotic arm; The steps for performing dual-target localization and establishing a binocular 3D imaging system in offline mode include: In offline mode, the intrinsic parameters of the left and right cameras and the positional relationship between the left and right cameras are calibrated, and a binocular 3D imaging system is established based on the intrinsic parameters and positional relationship. The rotation angles of each axis and the turntable of the computational robot include: Calculate the deflection and pitch angles of each axis of the robot arm, as well as the rotation angle of the turntable; The left and right cameras are mounted on the left and right sides of the robotic arm and move with the robotic arm; The robotic arm moves using an axis-driven motion mode.

2. The target tracking method under motion conditions according to claim 1, characterized in that, The step of acquiring the 3D coordinates of the target point in the binocular 3D imaging system in real time includes: Images containing the target point are captured by taking pictures with the left and right cameras. The row and column coordinates of the target point are located in the images, and the 3D coordinates of the target point are obtained according to the binocular 3D imaging system.

3. The target tracking method under motion conditions according to claim 1 or 2, characterized in that, The left and right cameras are high frame rate cameras.

4. The target tracking method under motion conditions according to claim 1, characterized in that, The intrinsic parameters include the camera's focal length, and the positional relationship includes the distance between the left and right cameras.

5. A target tracking system in motion, characterized in that, The system includes a control device and a six-degree-of-freedom manipulator, the six-degree-of-freedom manipulator being equipped with a left camera and a right camera, and the control device being used to execute the target tracking method under motion conditions as described in any one of claims 1-4.

Images