Billiard hitting system based on visual guidance

Abstract

The invention discloses a billiard hitting system based on visual guidance, and belongs to the field of industrial vision, robot motion control and intelligent control. Target detection is carried out through an industrial camera, pixel coordinates of billiard images are obtained, and the pixel coordinates are converted into physical coordinates (relative to physical coordinates of the upper left corner of a billiard table) through external parameter calibration. And then, the physical coordinates are further converted into a six-axis mechanical arm base coordinate system, and therefore the three-dimensional coordinates (x, y and z, and z is a fixed value) of the billiard ball relative to the mechanical arm are obtained. According to the system, the z-axis direction of a tool coordinate system is calibrated to be in the direction of a billiard rod of a mechanical arm, and the spatial postures (Rx, Ry and Rz) of the tool coordinate system are reversely deduced. When the position of the mechanical arm needs to be adjusted, the whole mechanical arm is moved through the translation table to meet the ball hitting requirement. According to the method, target detection, visual calibration and mechanical arm motion planning are combined, efficient control over the billiard hitting process is achieved, and the method has the advantages of being accurate in positioning, stable in ball hitting and suitable for scene complexity.

Description

Technical Field

[0001] Based on computer vision technology and robot control technology, the present invention studies a vision-guided billiard hitting system. First, the yolov8 object detection algorithm is used to obtain the position information (pixel coordinates) of the billiard balls, and the detection speed is improved by means of lightweight and parallelization; then, the internal and external parameters of the camera are calibrated to obtain the physical coordinates of the billiard balls on the billiard table based on the robotic arm; then the physical coordinates are converted into the base coordinate system of the robotic arm and the tool coordinate system is calibrated to obtain the relative relationship between the billiard balls and the robotic arm; then the angles and forces given by other strategy systems (this system focuses on hitting control, and the hitting strategy is given by other methods) are converted into the 6D coordinates of the robotic arm and the force of the hitting device, thus completing the precise hitting of the robotic arm. The present invention belongs to the fields of industrial vision, robot motion control and intelligent control. Background Art

[0002] With the continuous development of robot technology and computer vision, vision-guided robot control systems have been widely used in fields such as automated operations, intelligent manufacturing and entertainment. Especially in the entertainment industry, the research on automated billiard hitting systems has attracted increasing attention. As a highly skilled and strategic sport, the operation process of billiard games usually requires high precision and complex path planning. Therefore, automated systems can play an important role in such tasks.

[0003] In the billiard hitting task, it is first necessary to accurately locate the position and attitude of the balls. Although traditional image processing methods can detect billiard balls, in the case of complex lighting, occlusion and multiple balls on the billiard table, the positioning accuracy often fails to meet the requirements of high-precision hitting. Existing vision systems often have pixel-level positioning errors and cannot effectively handle these complex scenarios.

[0004] The conversion from the camera coordinate system to the physical world coordinate system usually involves complex mathematical calculations. Especially in practical applications, the relative position and attitude between the camera and the billiard table are not fixed, resulting in large errors in the coordinate conversion process. In addition, the robotic arm control system needs to perform motion planning based on the coordinates of the billiard balls and the hitting angles to ensure that the robotic arm can accurately move the cue to the appropriate position for hitting. The error accumulation in the existing systems during the conversion process often leads to the failure or deviation of hitting.

[0005] To solve the above problems, through visual detection and physical coordinate transformation: By performing high-precision extrinsic calibration, pixel coordinates are converted into the physical coordinates of the billiard table, and the z coordinate is fixed to simplify the calculation; Further, the conversion result is optimized through error correction technology to ensure the stability and accuracy of the physical coordinates. Combining the TCP (Tool Center Point) and TCF (Tool Coordinate Frame) calibration methods, the end working point and direction of the cue are determined, and the spherical coordinate system method is used to dynamically generate the x, y, and z direction vectors. By continuously trying to avoid singularities, the effectiveness of the tool pose is ensured. The inclination angle and azimuth angle of the shot given by the strategy method are used, and the inverse kinematics algorithm is used to map the direction of the cue to the six-degree-of-freedom parameters of the robotic arm; Combining with the path planning algorithm, the motion trajectory of the robotic arm is optimized to ensure accurate hitting of the ball. Summary of the Invention

[0006] The system is a combination of software and hardware, and the overall structure is as Figure 1 shown: The purpose of this system is to realize the billiard hitting process based on visual guidance, integrating a visual detection module; a coordinate system conversion module; and a robotic arm motion module, the three core modules, to form an end-to-end closed-loop system. The environmental information on the billiard table is collected by an industrial camera, and finally, the robotic arm is controlled according to the hitting strategy to complete the precise hitting operation. Among them, a standard billiard table is used to achieve a practical billiard hitting system. A high-resolution industrial camera is used to obtain environmental information to ensure accuracy. At the same time, the most common general-purpose six-axis robotic arm is used for hitting by the robotic arm, and a translation stage is used for the free movement of the robotic arm to meet the hitting requirements in various situations. The specific modules are as follows:

[0007] (1) Visual Detection Module

[0008] The visual system is used to detect the position of the balls on the billiard table in real time, and the position and category of all balls are identified and located through object detection technology. Traditional image processing methods (such as edge detection and color segmentation) have low accuracy in complex lighting and multi-ball occlusion scenarios, while deep learning-based object detection models have powerful feature extraction capabilities, which can significantly improve the accuracy and robustness of billiard detection. By using a deep learning object detection model (such as the YOLO series), the positions of multiple billiard balls in a complex scene can be quickly and accurately located, and the center point of the bounding box output by the detection model is used as the pixel coordinates of the billiard ball.

[0009] (2) Coordinate System Conversion Module

[0010] Convert the pixel coordinates of the billiard balls captured by the industrial camera into physical coordinates (relative to the coordinate system of the billiard table), and further convert them into the coordinates in the base coordinate system of the robotic arm for the motion control of the robotic arm. For the external parameter calibration, the multi-point calibration method is used to obtain the transformation matrix (such as the homography matrix) between the camera coordinate system and the physical coordinate system of the billiard table. Using the calibration results, through the perspective transformation formula, map the pixel coordinates to the physical coordinates. Through rotation and translation transformations, convert the points in the physical coordinate system into the points in the base coordinate system of the robotic arm.

[0011] (3) Robotic arm motion control module

[0012] First, calibrate the coordinate system of the end tool (billiard cue) of the robotic arm to determine the tool center point (TCP) and the tool direction (x, y, z axes). Then, according to the target position and attitude parameters, control the end effector of the robotic arm to accurately move to the target position and perform the hitting operation. Adopt six-degree-of-freedom motion planning to calculate the target position x, y, z and attitude R of the end of the robotic arm x ,R y ,R z , and ensure that the motion path of the robotic arm avoids the collision points by dynamically generating the direction vector through the spherical coordinate system. Then, select an appropriate force through the electric cylinder hitting device to perform the hitting, and complete the entire billiard hitting process.

[0013] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0014] 1) Through the optimized external parameter calibration and error correction technology, the pixel coordinates can be accurately converted into physical coordinates, significantly improving the accuracy of position measurement. Especially in complex multi-ball and occlusion scenarios, the system can still provide high-precision position information, avoiding the problem of error accumulation in traditional methods. 2) The calibration problem of the tool coordinate system has always been a difficult point in robotic arm control. Traditional methods often rely on static calibration or calibration at a single angle, which is prone to calibration error accumulation and cannot ensure accuracy in complex operating environments. However, the present invention combines the TCP and TCF calibration methods, and dynamically generates the x, y, z axis directions of the tool coordinate system through the spherical coordinate system, avoiding the influence of control such as collisions. Through multiple attempts and verifications, the system can adaptively adjust the accuracy of the tool coordinate system to ensure precise control of the tool attitude and position. 3) Since the present invention adopts multi-module collaborative control based on deep learning and optimization algorithms, the system can handle various changing factors in complex scenarios (such as light changes, changes in the number and position of balls, etc.). By real-time updating the coordinate transformation parameters and dynamically adjusting the robotic arm path planning, the system can maintain high stability and robustness in an uncertain environment, significantly improving the overall operation success rate. Brief Description of the Drawings

[0015] Figure 1This is the overall structural diagram of the system of the present invention.

[0016] Figure 2 This is a schematic diagram of the direction vector of the spherical surface. Detailed implementation manners

[0017] According to the above description, the following is a specific implementation process, but the scope protected by this patent is not limited to this implementation process.

[0018] A billiard hitting system based on visual guidance, characterized by including the following steps:

[0019] Step 1: Target detection

[0020] Step 1.1: Construction of the scene data set

[0021] First, collect the data set in the hitting scene. Arrange the hitting scene on a standard billiard table to ensure stable lighting conditions. Fix an industrial camera above the billiard table to cover the visible range of the entire tabletop and ensure that the shooting angle meets the requirements of target detection. Adjust the placement position and angle of the billiards, and collect diverse hitting scene data. Collect data under different light intensities and directions to simulate the possible lighting changes in the actual use scenario.

[0022] Step 1.2: Data preprocessing

[0023] a) Adjust the size of the image to 1365x1000; b) Perform random data augmentation on the data set; specifically including flipping in the horizontal and vertical directions, 90° rotation in the clockwise, counterclockwise, and upside-down manners, cropping with a minimum scale of 0% and a maximum scale of 17%, rotation between -9° and +9°, and brightness adjustment between -5% and +5%.

[0024] Step 1.3: Setting of model parameters

[0025] Considering both accuracy and real-time performance, adopt the yolov8 target detection algorithm, and use Fasternet as the backbone network to improve the model speed. Yolov8 has three detection heads. Since the size of the billiards is unique and the requirement for multi-scale is reduced, choose to remove two large detection heads and only retain the small target detection head to further improve the detection speed. At the same time, combine with tensorrt and enable FP16 (half-precision) quantization to reduce the model size on supported hardware; the size of the training batch parameter is 4; the learning rate is set to 0.01; the label smoothing is set to 0.1; the total number of training batches is set to 200;

[0026] Step 1.4: Parallel processing of image display and target detection

[0027] A multi-threaded architecture is designed to separate the image display and object detection tasks into independent threads. The image display thread extracts frame data from the captured images in real-time, preprocesses and displays it. The object detection thread runs in parallel, uses YOLOv8 to perform object detection on the captured images, and outputs the object position and class information. The two threads interact with data through a shared thread-safe queue to ensure the decoupling of the display and detection processes, avoiding the impact of detection time-consuming on the real-time performance of image display. Through the parallel processing of image display and object detection, the response speed and processing efficiency of the billiard hitting system are effectively improved, and the FPS is increased from 14 frames per second to 24 frames per second. Through the above steps, the pixel coordinate information of the billiard ball can be obtained in real-time.

[0028] Step 2: Camera calibration and coordinate system transformation

[0029] Step 2.1: Calibrate the camera internal parameters

[0030] Use a checkerboard pattern as the calibration board, with a checkerboard size of 9×6, and prepare multiple checkerboard images taken at different angles and positions to ensure coverage of different areas of the camera's field of view. Detect the checkerboard corner points through the built-in functions of OpenCV to obtain the corner point coordinates (pixel coordinates) in the image, and optimize the corner point positions at the sub-pixel level. Then calculate the camera internal parameters, including the internal parameter matrix, distortion coefficients, rotation vector, and translation vector, by inputting the object points and image points, and store the above results for loading and use during subsequent image processing, thus completing the calibration of the camera internal parameters.

[0031] Step 2.2: Calibrate the camera external parameters

[0032] The pixel coordinates of the billiard ball have been obtained in Step 1. In this step, by calibrating the camera external parameters, the conversion relationship between the camera coordinate system and the physical coordinate system of the billiard table is established. Since the area of the billiard table is large, to solve the problem of insufficient overall perspective transformation accuracy, the present invention adopts a method of regional perspective transformation, divides the billiard table into several regions, where point 0 is the center position of the first pocket in the upper left corner, and the maximum point is the center position of the pocket in the lower right corner. These two points determine the entire tabletop. Here, it is divided into several regions (here are 4 quadrants), and each region is calibrated separately to ensure the accuracy of perspective transformation within the local region. In each region, several calibration points (such as checkerboards or predefined physical points) are selected as benchmarks, and the pixel coordinates and corresponding physical coordinates of the calibration points are recorded. The pixel coordinates are recorded as (u, v), and the physical coordinates are recorded as (x, y). For the set of calibration points in each region, use the homography matrix calculation method (such as perspective transformation based on the least squares method) to solve the perspective transformation matrix:

[0033]

[0034] H describes the linear mapping relationship from pixel coordinates to physical coordinates, and each element h in H ij is obtained by calculating the homography matrix of two sets of points, namely the known pixel coordinates and the corresponding physical coordinates in the image through a checkerboard:

[0035]

[0036] Calculate H for each region separately 1 , H 2 , H 3 , H 4 and other matrices are used for perspective transformation of the corresponding regions. According to the pixel coordinates (u, v) of the input point, determine the region it belongs to (for example, determine the corresponding H by the pixel coordinate range i ). For points falling within a single region, directly use the corresponding perspective matrix for transformation:

[0037]

[0038] For points falling on the boundary of adjacent regions, use the weighted average of the perspective transformation results of multiple regions to reduce the conversion error: where represents the position of the point in region i, represents the position of the point in region j, and regions i and j are adjacent regions. After the above steps, the pixel coordinates are successfully converted into the physical coordinates (x world , y world ) in the billiard coordinate system, and the origin of the physical coordinates is the center position of the upper left pocket of the billiard table, so as to provide accurate position information for the strategy.

[0039] Step 2.3: Conversion from the physical coordinate system to the robotic arm base coordinate system

[0040] The desktop physical coordinate system provides a unified reference benchmark for planning the relative position relationships between objects, such as the relative position relationships between balls and between balls and pockets. However, to control the robotic arm, the position of the object in the robotic arm coordinate system still needs to be obtained. In this step, the position of the object in the desktop physical coordinate system is converted into the position in the robotic arm base coordinate system to ensure that the robotic arm can perform precise operations based on the object positions detected by the vision system. This conversion process includes two steps: rotation and translation. The specific method is to convert the physical coordinates of the object into the robotic arm base coordinate system through a rotation matrix and a translation vector based on the known installation position and pose of the robotic arm. However, due to measurement errors and the accuracy problem of coordinate alignment, error correction needs to be performed in actual tests to ensure the accuracy of the conversion results.

[0041] The transformation from the physical coordinate system to the robotic arm base coordinate system first involves the rotation between coordinate systems. Set a rotation matrix R, which describes the rotation relationship between the physical coordinate system and the robotic arm base coordinate system:

[0042]

[0043] where (x arm , y arm ) is the position of the object in the robotic arm base coordinate system, and (x world , y world ) is the position of the object in the physical coordinate system.

[0044] Immediately followed by a translation operation, the origin of the physical coordinate system is translated to the position of the robotic arm base. Set a translation vector T, which represents the displacement from the origin of the physical coordinate system to the origin of the robotic arm base coordinate system:

[0045]

[0046] where T = [t x , t y T is the translation vector, representing the displacement of the origin of the physical coordinate system relative to the origin of the robotic arm base coordinate system. The z-axis value (height) of the desktop physical coordinate system is not considered, and the height of the object remains consistent throughout the process. Therefore, the z arm in the robotic arm base coordinate system is directly obtained by measuring the height of the billiard ball relative to the robotic arm base and setting it as a fixed value. At this time, the x arm , y arm , z arm values of the billiard ball relative to the robotic arm are obtained.

[0047] Due to the limitations of calibration and installation accuracy, there may be certain errors in the rotation matrix R and the translation vector T, resulting in the coordinate transformation result not being exactly the same as the actual position. In actual tests, correction amounts △x, △y, △z are added to the transformed x arm , y arm , z arm respectively to compensate for the system error, where the correction amounts are obtained by calculating the differences between the x arm , y arm , z arm and the actual x, y, z in several regions:

[0048] x = x arm + △x

[0049] y = y arm + △y

[0050] z = z arm + △z ​

[0051] Through the above conversion, the present invention accurately maps the position of an object in the desktop physical coordinate system to the coordinate system of the robotic arm base, ensuring that the robotic arm can precisely operate the target object at a fixed height z. In addition, by adding error correction to the final conversion result, the influence of calibration errors and measurement deviations on the overall system accuracy is significantly reduced, providing reliable data support for subsequent motion planning of the robotic arm.

[0052] Step 2.4: Tool coordinate system calibration

[0053] Using the TCP and TCF calibration methods, first, fix a tool (such as a billiard cue) to the end of the robotic arm, ensuring a firm installation and a known pose. Move the end of the robotic arm to a standard pose (usually the initial reference point) and record the current pose as the benchmark. Control the robotic arm to align the tool tip (TCP) with a fixed point, and record the coordinates of the end of the robotic arm at multiple different poses respectively. Use the least squares method to calculate the intersection position of these points to determine the tool center point. Based on the TCP calibration result, determine the z-axis direction of the tool coordinate system by measuring the direction of the billiard cue. And use the right-hand rule to construct the x and y-axis directions, thereby completing the calibration of the tool coordinate system and determining the position and direction of the tool coordinate system relative to the end coordinate system of the robotic arm.

[0054] Step 3: Vision-guided robotic arm reference hitting

[0055] In this step, we achieve precise control of the motion of the robotic arm in the robotic arm coordinate system based on the target position calculated by the vision system and the strategy module. The strategy module can give the tilt angle α, azimuth angle β, and hitting force of the hit. To ensure that the end effector of the robotic arm can accurately reach the target position, we not only need to consider the position coordinates x, y, z, but also need to calculate the rotation angles Rx, Ry, Rz of the end effector of the robotic arm to ensure the correct pose of the robotic arm, and Rx, Ry, Rz and the end direction vector [x, y, z] of the robotic arm are a corresponding relationship. Since the angles given by the strategy system are based on the spherical coordinate system and can only give the angle of the direction vector z. Therefore, reverse calculation is required to obtain the rotation angles required by the robotic arm. The angles provided by the strategy module are based on the spherical coordinate system, which has a certain difference from the rotation angles of the end effector of the robotic arm. Therefore, reverse calculation is required. The specific inverse solution process is as follows:

[0056] Step 3.1: Calculate the direction vector of the sphere

[0057] First, according to the direction of the cue axis (i.e., the cue stick axis direction) given by the strategy system, we need to generate the directions of the x and y axes through the spherical coordinate system. These axis vectors are generated through the right - hand rule and sampling calculations on the sphere. According to the angles given by the strategy, we have calculated the direction of the z - axis using the longitude and latitude angles in the spherical coordinate system, and generated multiple candidate direction vectors by gradually increasing the angles (such as θ and φ). These candidate direction vectors represent possible x and y axis directions. Each time a direction vector is generated, a small step size (such as θ step represents the latitude angle and represents the longitude angle, both defaulting to 15°) Each time a direction vector is generated, it advances in these two step sizes to ensure that the entire sphere can be covered, as Figure 2 shown.

[0058] Step 3.2: Verify orthogonality and reachability

[0059] For each generated candidate direction vector x a First, check whether it is parallel to the given z - axis direction (i.e., the billiard cue direction):

[0060] |x·z|<0.99

[0061] This condition ensures that the angle between the candidate direction vector x and the z - axis is greater than 8°, avoiding their parallelism. If the candidate direction vector is not parallel to the z - axis, calculate the y - axis using the cross - product:

[0062] y = z×x

[0063] Then normalize the obtained vector to ensure it is a unit vector. At this time, the three direction vectors of the end - effector coordinate system are obtained for subsequent inverse kinematics use

[0064] Step 3.3: Direction vector calculation

[0065] Given two angles in the spherical coordinate system (the tilt angle α and the azimuth angle β), calculate the direction vector [x, y, z] of the end - effector of the robotic arm. The tilt angle α represents the tilt angle of the end - effector relative to the horizontal plane. The azimuth angle β represents the horizontal rotation angle of the end - effector around the target point. Through these two angles, the direction vector of the end - effector in three - dimensional space can be calculated, and the formula is:

[0066] x = cos(α)·sin(β)

[0067] y = cos(α)·cos(β)

[0068]

[0069] where both α and β are values after converting the angles to radians.

[0070] Step 3.4: Inverse solution of the rotation matrix

[0071] Using the calculated direction vector [x, y, z], the rotation angles R about the three axes can be calculated using the inverse calculation method. x , R y , R z :

[0072] The rotation angle R about the Z axis z is calculated from the x and y components of the direction vector:

[0073] R z = atan2(y, x)

[0074] The rotation angle R about the Y axis y is calculated from the z and x, y components of the direction vector:

[0075]

[0076] The rotation angle Rx about the X axis is calculated from the Y and Z components of the direction vector obtained from the previous two calculations:

[0077] R x = atan2(Y z , Z z )

[0078] Finally, the three rotation angles R x , R y , R z are obtained, in degrees.

[0079] Step 3.4: Calculate the rotation direction of the end effector

[0080] Using the calculated rotation angles R x , R y , R z , the rotation matrix R is further constructed, which represents the rotation of the end effector in three-dimensional space: R = R z ·R y ·R x . By combining these three rotation matrices, the rotation state of the end effector is obtained, thereby determining the attitude of the end effector of the robotic arm.

[0081] Step 4: Control the robotic arm to strike

[0082] Using the given (x, y, z, R x , R y , R z)The 6D coordinates can be completed through the instructions of the robotic arm. If a collision occurs or the target position cannot be reached during operation, the robotic arm can return to the position before the instruction execution through the return system and attempt the next set of feasible coordinates, thus achieving precise control of the robotic arm. The ball is mainly hit by an electric cylinder hitting device. By setting the rotation speed of the hitting motor, different magnitudes of force can be generated. After the strategy system gives the required hitting force, the magnitude of the force can be adjusted to complete the hitting.

Claims

1. A billiard hitting system based on vision guidance, characterized in that: The following steps are involved: Step 1: Object Detection Step 1.1: Scene dataset construction First, we collect data sets in the hitting scene. We set up the hitting scene on a standard billiard table and use an industrial camera fixed above the billiard table to cover the entire visible range of the tabletop to ensure that the shooting angle meets the requirements of target detection. Adjust the position and angle of billiard balls to collect data for various hitting scenarios; collect data under different light intensities and directions to simulate the lighting changes that may occur in actual usage scenarios; Step 1.2: Data preprocessing a) resize the image to 1365x1000; b) perform random data augmentation on the dataset, including horizontal and vertical flipping, 90° rotation in clockwise, counterclockwise, and upside-down mode, cropping with a minimum zoom of 0% and a maximum zoom of 17%, rotation between -9° and +9°, and brightness adjustment between -5% and +5%. Step 1.3: Model parameter setting Considering the comprehensive accuracy and real-time performance, the Yolov8 target detection algorithm is adopted, and Fasternet is used as the backbone network to improve the model speed; Yolov8 has three detection heads. Since the size of billiard balls is unique, the requirements for multi-scale are reduced. We choose to remove two large detection heads and only keep the small target detection head to further improve the detection speed. At the same time, it is combined with TensorRT and FP16 (half-precision) quantization is enabled to reduce the model size on supported hardware; the size of the training batch processing parameter is 4; the learning rate is set to 0.01; the label smoothing is set to 0.1; The total training batch size is set to 200; Step 1.4: Parallel processing of image display and object detection Designed a multi-threaded architecture to separate image display and object detection tasks into independent threads; The image display thread extracts frame data from the acquired image in real time, and pre-processes and displays it; The target detection threads run in parallel, using yolov8 to detect targets on the captured images and output target location and category information; the two threads interact with each other through a shared thread-safe queue to ensure the decoupling of the display and detection processes, and avoid the real-time performance of image display affected by time-consuming detection; through the parallel processing of image display and target detection, the FPS is increased from 14 frames / second to 24 frames / second; after the above steps, the pixel coordinate information of the billiard ball is obtained in real time; Step 2: Camera calibration and coordinate system conversion Step 2.1: Calibrate camera intrinsic parameters A checkerboard pattern with a size of 9×6 is used as the calibration board. Multiple checkerboard images taken at different angles and positions are prepared to ensure that different areas of the camera's field of view are covered. The checkerboard corners are detected through the built-in functions of OpenCV to obtain the corner coordinates in the image, i.e., pixel coordinates, and perform sub-pixel corner position optimization. The camera intrinsic parameters are then calculated by inputting object points and image points, including the intrinsic parameter matrix, distortion coefficient, rotation vector, and translation vector. The above results are stored for loading and use in subsequent image processing, thus completing the camera's intrinsic parameter calibration. Step 2.2: Calibrate camera extrinsics The pixel coordinates of the billiard ball have been obtained in step 1. In this step, the conversion relationship between the camera coordinate system and the physical coordinate system of the billiard table is established by calibrating the external parameters of the camera. The billiard table is divided into several areas by using the regional perspective transformation method, where the 0 point is the center position of the first pocket in the upper left corner, and the maximum point is the center position of the pocket in the lower right corner. The two points determine the entire table surface. It is divided into 4 quadrants here, and each area is calibrated separately to ensure the accuracy of perspective transformation in the local area. In each area, several calibration points are selected as the reference, and the pixel coordinates and corresponding physical coordinates of the calibration points are recorded. The pixel coordinates are recorded as (u, v), and the physical coordinates are recorded as (x, y). For the calibration point set in each area, the perspective transformation matrix is ​​solved using the homography matrix calculation method: H describes the linear mapping relationship from pixel coordinates to physical coordinates. Each element h in H ij The homography matrix of two sets of points, the known pixel coordinates in the image and the corresponding physical coordinates, is obtained by calculating the checkerboard: Calculate the H1, H2, H3, H4 matrices for each region for perspective transformation of the corresponding region; according to the pixel coordinates (u, v) of the input point, determine the region to which it belongs, that is, determine the H matrix to which it belongs through the pixel coordinate range. i , for points that fall within a single area, directly use the corresponding perspective matrix for transformation: For points that fall on the boundaries of adjacent regions, the weighted average of the perspective transformation results of multiple regions is used to reduce the transformation error: Where P Hi represents the position of the midpoint of region i, represents the position of the midpoint of region j, and regions i and j are adjacent regions; after the above steps, the pixel coordinates are converted into the physical coordinates of the billiard coordinate system (x world ,y world ), the physical coordinate origin is the center of the hole in the upper left corner of the billiard table; Step 2.3: Convert the physical coordinate system to the robot base coordinate system The transformation from the physical coordinate system to the robot base coordinate system first involves the rotation between the coordinate systems; a rotation matrix R is set, which describes the rotation relationship between the physical coordinate system and the robot base coordinate system: Among them, (x arm ,y arm ) is the object position in the robot base coordinate system, (x world ,y world ) is the position of the object in the physical coordinate system; Then perform a translation operation to translate the origin of the physical coordinate system to the position of the robot base; set a translation vector T to represent the displacement from the origin of the physical coordinate system to the origin of the robot base coordinate system: Where T = [t x ,t y ] T is a translation vector, which indicates the displacement of the origin of the physical coordinate system relative to the origin of the robot base coordinate system. The desktop physical coordinate system does not consider the value of the z-axis direction, and the height of the object remains consistent throughout the process. Therefore, z in the robot base coordinate system arm Directly measure the height of the billiard ball relative to the base of the robotic arm and set it as a fixed value; at this time, the x value of the billiard ball relative to the robotic arm is obtained. arm ,y arm ,z arm The value of In the actual test, the converted x arm ,y arm ,z arm Corrections △x, △y, △z are added to compensate for system errors, where the corrections are calculated in several areas x arm ,y arm ,z arm The difference between the actual x, y, and z is obtained: x=x arm +△x y=y arm +△y z=z arm +△z Step 2.4: Tool coordinate system calibration Using the TCP and TCF calibration methods, first fix the tool, i.e. the billiard cue, on the end of the robot arm, move the end of the robot arm to a standard posture, and record the current posture as a reference; control the robot arm to align the end of the tool with the fixed point, and record the coordinates of the end of the robot arm in multiple different postures; use the least squares method to calculate the intersection of these points and determine the center point of the tool; based on the TCP calibration results, determine the z-axis direction of the tool coordinate system by measuring the direction of the billiard cue; and use the right-hand rule to construct the x- and y-axis directions, thereby completing the calibration of the tool coordinate system and determining the position and direction of the tool coordinate system relative to the coordinate system of the end of the robot arm; Step 3: Vision-guided Robotic Arm Fiducial Hitting Given the inclination angle α, azimuth angle β and striking force of the impact; in order to ensure that the end effector of the robot arm can accurately reach the target position, it is necessary not only to consider the position coordinates x, y, z, but also to calculate the rotation angles Rx, Ry, Rz of the end effector of the robot arm to ensure the correct posture of the robot arm. Rx, Ry, Rz and the end direction vector [x, y, z] of the robot arm are a set of corresponding relationships; Since the angle given by the strategy system is based on the spherical coordinate system and can only give the angle of the direction vector z, it is necessary to perform a reverse calculation to obtain the required rotation angle of the robot arm. The specific process of the reverse calculation is as follows: Step 3.1: Calculate the direction vector of the sphere First, according to the z-axis direction of the club given by the strategy system, the directions of the x and y axes need to be generated through the spherical coordinate system; these axis vectors are generated by the right-hand rule and sampling calculations on the sphere; according to the angle given by the strategy, the direction of the z axis has been calculated. Using the longitude and latitude angles in the spherical coordinate system, multiple candidate direction vectors are generated by gradually increasing the angles, namely θ and φ; these candidate direction vectors represent possible x and y axis directions; each time a direction vector is generated, a small step size, namely θ, is used step Represents latitude angle and Indicates the longitude angle, the default is 15°; each time a direction vector is generated, it advances in these two steps to ensure that the entire sphere is covered; Step 3.2: Verify orthogonality and reachability For each generated candidate direction vector x a First check if it is parallel to the given z-axis direction, i.e. the direction of the pool cue: |x·z|<0.99 This condition ensures that the angle between the candidate direction vector x and the z axis is greater than 8° to avoid the two being parallel; if the candidate direction vector is not parallel to the z axis, the cross product is used to calculate the y axis: y=z×x Then the obtained vector is normalized to ensure that it is a unit vector; at this time, the three direction vectors of the terminal coordinate system are obtained; Step 3.3: Direction vector calculation Given two angles in the spherical coordinate system, namely the tilt angle α and the azimuth angle β, the direction vector [x, y, z] of the end effector of the robot arm is calculated; the tilt angle α represents the tilt angle of the end effector relative to the horizontal plane; the azimuth angle β represents the horizontal rotation angle of the end effector around the target point; through these two angles, the direction vector of the end effector in three-dimensional space is calculated, and the formula is: x=cos(α)·sin(β) y=cos(α)·cos(β) Among them, α and β are the values ​​after the angle is converted into radians; Step 3.4: Inverse rotation matrix The calculated direction vector [x, y, z] is used to calculate the rotation angle R around the three axes using the reverse solution method. x , R y , R z : Rotation angle R around the Z axis z Calculated from the direction vector x and y components: R z =atan2(y,x) The rotation angle R around the Y axis y Calculated by the z and x,y components of the direction vector: The rotation angle Rx around the X axis is calculated by the Y and Z components of the direction vector obtained in the first two steps: R x =atan2(Y z ,Z z ) Finally, we get three rotation angles R x , R y , R z , in degrees; Step 3.4: Calculate the end effector rotation direction By calculating the rotation angle R x , R y , R z , further construct the rotation matrix R, which represents the rotation of the end effector in three-dimensional space: R = R z ·R y ·R x ; By combining these three rotation matrices, the rotation state of the end effector is obtained, and then the posture of the end effector of the robot arm is determined; Step 4: Robotic Arm Controls the Hit By giving (x, y, z, R x ,R y ,R z )6D coordinates can be completed through the instructions of the robot arm; if a collision occurs or the robot cannot reach the target during operation, the robot arm returns to the position before the instruction is executed through the return system and tries the next set of feasible coordinates, thereby completing the precise control of the robot arm; the ball is struck by an electric cylinder striking device, which converts the speed of the striking motor into different forces by setting it. After the strategy system gives the required striking force, the force can be adjusted and the strike can be completed.

Images