Table tennis ball rotation estimation method and rotation estimation system based on visual measurement

Abstract

This application provides a table tennis rotation estimation method and system based on visual measurement, comprising: placing three solid circular markers of different colors on a table tennis ball; symmetrically placing a left camera and a right camera on the left and right sides of the net in the longitudinal direction; identifying the circular area of ​​the table tennis ball and the pixel coordinates of the circle center in the image captured by the cameras, and obtaining the three-dimensional coordinates of the center of the table tennis ball; identifying the solid circular marker on the table tennis ball within the circular area of ​​the table tennis ball, and obtaining the pixel coordinates of the marked area and the center of the marked area; calculating the posture of the table tennis ball based on the three-dimensional coordinates of the center of the table tennis ball and the pixel coordinates of the center of the marked area; calculating the rotation speed and direction of the table tennis ball based on the table tennis ball posture unit vector at a preset time; and comprehensively analyzing the results of the three markers to obtain the final rotation speed and direction of the table tennis ball. This application can effectively reduce costs and improve practicality.

Description

Technical Field

[0001] The present application belongs to the field of table tennis rotation estimation, and specifically relates to a table tennis rotation estimation method and a rotation estimation system based on visual measurement. Background Art

[0002] Table tennis is a beloved sport in China, with a broad mass base and widespread market application. Spin is a crucial technical factor in table tennis, and quantitatively measuring the spin speed of a flying ping-pong ball is crucial. This not only facilitates analysis of table tennis matches and training and improvement of table tennis skills and tactics, but also improves the return success rate of table tennis robots.

[0003] Although the rotation of a table tennis ball is crucial to the sport, accurately estimating its speed and direction has always been a major challenge in this field. Existing literature discloses methods and devices for estimating the rotation of a flying table tennis ball, but these methods suffer from the following practical limitations:

[0004] Methods for estimating rotational velocity based on a ping-pong ball's trajectory suffer from significant errors. This is because the effect of rotational velocity on the ball's trajectory is affected by other factors, such as the ball's speed. Furthermore, errors in trajectory measurement are amplified by the rotational velocity estimate. Visual tracking of the ball's markers can lead to method failure due to persistent occlusion. The ultra-high frame rate cameras and gimbal systems used in visual measurement suffer from limited fields of view, complex control, and high costs. Summary of the Invention

[0005] In order to at least overcome the problems existing in the related art to a certain extent, the present application provides a table tennis rotation estimation method and a rotation estimation system based on visual measurement.

[0006] According to a first aspect of the embodiments of the present application, the present application provides a table tennis ball rotation estimation method based on visual measurement, which includes the following steps:

[0007] Marking the ping-pong ball and arranging the vision system cameras involves the following steps: placing three solid circular marks of different colors on the ping-pong ball, with the line connecting the center of the first circular mark and the center of the second circular mark passing through the center of the ping-pong ball, and the line connecting the center of the third circular mark and the center of the ping-pong ball being perpendicular to the line connecting the center of the first circular mark and the center of the second circular mark; the vision system cameras include a left camera and a right camera, symmetrically arranged on the left and right sides of the length of the net, with the centerline of the net height as the axis of symmetry. The left and right cameras are both located in the same plane as the net.

[0008] Identify the circular area and center pixel coordinates of the ping-pong ball in the image captured by the vision system camera, and obtain the three-dimensional coordinates of the center of the ping-pong ball based on the center pixel coordinates;

[0009] Recognize the solid circular mark on the ping-pong ball in the circular area of ​​the ping-pong ball to obtain the marked area on the ping-pong ball and the center pixel coordinates of the marked area;

[0010] The posture of the table tennis ball is calculated based on the three-dimensional coordinates of the center of the table tennis ball and the center pixel coordinates of the marked area on the table tennis ball;

[0011] Calculating the rotation speed and rotation direction of the table tennis ball according to the unit vector of the table tennis ball posture at a preset time;

[0012] After estimating the rotation speed and direction of the table tennis ball according to the three markers of different colors, the results of the three markers are comprehensively analyzed to obtain the final rotation speed and direction of the table tennis ball.

[0013] In the above-mentioned table tennis ball rotation estimation method based on visual measurement, the specific process of identifying the circular area and the pixel coordinates of the center of the table tennis ball in the image captured by the visual system camera, and obtaining the three-dimensional coordinates of the center of the table tennis ball based on the pixel coordinates of the center of the circle is as follows:

[0014] Using a left camera and a right camera to synchronously capture an image of a table tennis ball with a solid circular mark, a left image and a right image are obtained;

[0015] Use the HSV color recognition algorithm to perform image recognition on the left and right images to obtain the circular area and center pixel coordinates of the ping-pong ball in the left and right images;

[0016] The three-dimensional reconstruction of the center of the table tennis ball is completed using the parallax algorithm based on the pixel coordinates of the center of the circular area in the left image and the pixel coordinates of the center of the circular area in the right image.

[0017] Furthermore, the specific process of identifying the solid circular mark on the table tennis ball in the circular area of ​​the table tennis ball and obtaining the marked area on the table tennis ball and the center pixel coordinates of the marked area is as follows:

[0018] The marked area on the ping pong ball in the image is identified by using the frame difference method between the marked ping pong ball and the unmarked ping pong ball to obtain the marked area;

[0019] Perform HSV color recognition on the marked area to determine the color of the circular mark on the ping-pong ball in the left and right images;

[0020] The center pixel coordinates of the marked areas in the left and right images are obtained using an ellipse fitting algorithm.

[0021] Furthermore, the specific process of calculating the posture of the table tennis ball based on the three-dimensional coordinates of the center of the table tennis ball and the central pixel coordinates of the marked area on the table tennis ball includes:

[0022] Assume that the center pixel coordinates of the marked area identified in the left image are (u bl ,v bl ), the three-dimensional coordinates of the center of the table tennis ball in the world coordinate system are (x cw ,y cw , z cw );

[0023] Based on the pinhole imaging principle of the camera, the constraint relationship between the center pixel coordinates of the marked area on the ping-pong ball and the marked area on the ping-pong ball in the world coordinate system is obtained as follows:

[0024]

[0025] Where z bl Indicates the depth distance of the center of the marked area relative to the left camera, u bl Indicates the row where the center of the marked region is in the left image, v bl Indicates the column where the center of the marked region is in the left image, M l represents the intrinsic parameter matrix of the left camera; l R w Represents the extrinsic parameter matrix of the left camera, that is, the rotation matrix of the world coordinate system relative to the left camera coordinate system; l T w Represents the translation vector of the left camera's extrinsic matrix world coordinate system relative to the left camera coordinate system; the camera's intrinsic and extrinsic matrix are obtained through camera calibration. is the column vector form of the three-dimensional coordinates of the center of the marked area in the world coordinate system, and the parameters in the column vector are the parameters to be solved;

[0026] According to the constraint relationship between the center pixel coordinates of the marked area on the ping-pong ball and the marked area on the ping-pong ball in the world coordinate system, the column vector form of the three-dimensional coordinates of the center of the marked area in the world coordinate system is obtained. for:

[0027]

[0028] set up According to The parametric equation of the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball is obtained from the expression:

[0029]

[0030] Convert the parametric equation of the space line into a point-wise form:

[0031]

[0032] Assume that the direction vector of the space line represented by the point-to-point formula is The point through which the straight line in the point-to-point formula passes is make The distance between the center of the ping-pong ball and the center of the marked area on the ping-pong ball is:

[0033]

[0034] Theoretically, the distance between the center of the ping-pong ball and the center of the marked area on the ping-pong ball is the radius r of the ping-pong ball. If |dr|>ε, where ε represents the preset allowable error, the three-dimensional coordinate solution of the center of the marked area on the ping-pong ball fails. Otherwise, the foot of the perpendicular line from the center of the ping-pong ball to the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball is calculated. The coordinates of the foot of the perpendicular line are the three-dimensional coordinates of the center of the marked area on the ping-pong ball in the world coordinate system (x bw ,y bw , z bw );

[0035] make Represents the direction vector of the perpendicular line from the center of the ping-pong ball to the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball. According to this perpendicular relationship get:

[0036] A1(x cw -x bw )+A2(y cw -y bw )+A3(z cw -z bw )=0;

[0037] According to the parametric equation of the space line, we get:

[0038] A1(x cw -A1z bl -B1)+A2(y cw -A2z bl -B2)+A3(z cw -A3z bl -B3)=0;

[0039] Then we get the parameter z bl for:

[0040]

[0041] According to z blThe expression of and the parametric equation of the space line are used to obtain the three-dimensional coordinates of the center of the marked area on the ping-pong ball in the world coordinate system;

[0042] According to the three-dimensional coordinates (x bw ,y bw , z bw ) and the three-dimensional coordinates of the center of the table tennis ball in the world coordinate system (x cw ,y cw , z cw ), the unit vector of the table tennis posture at each moment is obtained as:

[0043]

[0044] Furthermore, the process of calculating the rotation speed and rotation direction of the table tennis ball according to the table tennis ball posture unit vector at the preset time is:

[0045] Use the current position and time of the ping-pong ball to determine whether the current state is a new ping-pong ball trajectory. If so, record the data, reset the parameters, and continue solving the ping-pong ball posture. Otherwise, determine whether the time of the current trajectory posture acquisition is greater than the preset time. If so, analyze the rules of ping-pong ball mark recognition; otherwise, record the data and continue solving the ping-pong ball posture.

[0046] The number of rotations of a ping-pong ball within a preset time period is estimated by using the appearance and disappearance patterns of circular markers of the same color, thereby analyzing the patterns of ping-pong ball marker recognition. The patterns of marker appearance and disappearance include continuous marker appearance, continuous marker disappearance, marker appearance with intervals between adjacent frames, and marker appearance without intervals between adjacent frames.

[0047] Analyze the table tennis ball posture unit vectors corresponding to the three colored circular marks;

[0048] For the interval appearance of the mark with adjacent frames, the final rotation direction of the rotation speed is determined by the adjacent frames;

[0049] For intervals where no adjacent frames are marked, the final rotation direction is determined based on the adjacent marked frames;

[0050] Calculate the rotation speed of the table tennis ball, which includes the number of rotations and the start and end frame angles.

[0051] After estimating the rotation speed and direction of the table tennis ball according to the three markers of different colors, the results of the three markers are comprehensively analyzed to obtain the final rotation speed and direction of the table tennis ball.

[0052] Furthermore, the process of analyzing the table tennis ball posture unit vectors corresponding to the three colored circular marks is as follows:

[0053] The rotation axis of the table tennis ball is obtained by plane fitting the posture unit vectors marked with circles of the same color;

[0054] Using the table tennis posture unit vector at each moment Get the posture unit vectors of each color mark at different times, and assume that one of the posture unit vectors is Where, subscript i represents the color mark index, and j represents the vector index;

[0055] Solve the average coordinates of all posture unit vectors marked with the same color Construct the error matrix:

[0056]

[0057] Where n represents the number of posture vectors of the same color circle mark solved within the preset time T;

[0058] For the error matrix M ie Perform SVD decomposition, M ie =UDV T , where U represents an n×n left singular vector matrix; D represents an n×3 non-negative real diagonal matrix, whose diagonal elements are the error matrix M ie The singular values ​​of ; V represents a 3×3 orthogonal matrix, The axis of rotation of the table tennis ball is

[0059] Furthermore, for the markers that appear at intervals of adjacent frames, the specific process of determining the final rotation direction of the rotation speed through the adjacent frames is as follows:

[0060] The table tennis ball's rotation axis has two rotation directions, positive and negative, for the rotation speed;

[0061] In the pose unit vector sequence Select adjacent frames and make the following analysis and judgment on the adjacent frames:

[0062] Set the rotation direction judgment parameter r d : in, represents the pose unit vector of the previous frame, Represents the unit vector of the posture of the next frame; the rotation direction of the ping-pong ball is:

[0063] The final rotation direction is determined by majority voting based on the rotation direction calculated from multiple sets of adjacent frames. The judgment of topspin, bottomspin, leftspin, rightspin, clockwise and counterspin is completed based on the rotation direction vector.

[0064] Furthermore, for the intervals where no adjacent frames are marked, the specific process of determining the final rotation direction based on the adjacent marked frames is as follows:

[0065] Closely labeled frames are two labeled frames with only unlabeled frames in between;

[0066] In the pose unit vector sequence Select similar marked frames and make the following analysis and judgment on the similar marked frames:

[0067] Set the rotation direction judgment parameter r d : in, represents the pose unit vector of the previous frame of the similarly marked frame, Indicates the pose unit vector of the frame after the marker frame appears again; the rotation direction of the ping-pong ball is:

[0068] The rotation direction is calculated based on multiple sets of marked similar frames, and the final rotation direction is determined by majority voting. The judgment of topspin, backspin, left spin, right spin, clockwise spin and counterspin is completed based on the rotation direction vector.

[0069] Furthermore, the process of calculating the rotation speed of the table tennis ball is:

[0070] Contains the number of rotations and the start and end frame angles;

[0071] The number of rotations is determined by the appearance and disappearance of the mark. If the appearance, disappearance, and reappearance are approximately one circle, the number of circles n within the time T can be obtained. r ;

[0072] The angle between the start and end frames is:

[0073] The analysis of the start and end frame angle direction is as follows:

[0074] Set the rotation angle judgment parameter r a : in, Represents the pose unit vector calculated in the first frame of the trajectory, Represents the pose unit vector calculated from the last frame of the trajectory within the preset time T, is the rotation direction, and the angle of rotation of the ping-pong ball in the first and last frames is:

[0075] The angular velocity of the ping-pong ball is: Among them, t s Indicates the start time of the mark, t e Indicates the stop time of the marker appearance.

[0076] Furthermore, the specific process of comprehensively analyzing the results of the three different color marks to obtain the final rotation speed and rotation direction of the table tennis ball is as follows:

[0077] Determine the number of marks among the three marks that meet the conditions and appear at intervals with adjacent frames. If the number of marks that meet the conditions is greater than 1, the result of the mark with more adjacent frames is used as the final result.

[0078] If the number of marks that meet the conditions is 1, the result that meets the conditions is taken as the final result;

[0079] If none of the three marks meet the condition that the marks have interval appearance with adjacent frames and have adjacent frames, then the number of marks that meet the interval appearance condition in the marks is determined. Such interval appearance marks have no adjacent frames;

[0080] If the number of markers appearing in the interval is greater than 1, the marker result with more markers appearing is taken as the final result;

[0081] If the number of marks appearing in the interval is 1, the result that meets the conditions is taken as the final result;

[0082] If all three marks appear or disappear continuously, the ball will be deemed as no spin.

[0083] According to the second aspect of the embodiment of the present application, the present application also provides a table tennis rotation estimation system based on visual measurement, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor processes the computer program, it implements any of the above-mentioned table tennis rotation estimation methods based on visual measurement.

[0084] According to a third aspect of the embodiments of the present application, the present application further provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, it implements any of the above-mentioned table tennis rotation estimation methods based on visual measurement.

[0085] According to the above-mentioned specific implementation methods of the present application, it can be seen that there are at least the following beneficial effects: the table tennis rotation estimation method based on visual measurement provided by the present application can effectively ensure the success rate of mark recognition in the camera image by designing reasonable marks on the surface of the table tennis ball and the layout of the visual system camera, thereby reducing system costs and improving practicality.

[0086] This application uses HSV to first identify the table tennis ball, and then uses the frame difference method combined with HSV to perform mark identification on the identified table tennis ball area, which can effectively filter out environmental interference and improve recognition speed and accuracy.

[0087] This application uses the camera pinhole imaging principle and the distance constraint relationship between the mark and the center of the ball to complete the three-dimensional reconstruction of the table tennis mark, thereby determining the posture of the table tennis ball.

[0088] This application uses plane fitting to determine the rotation axis of the table tennis ball, uses adjacent frames and similar marked frames to complete the determination of the rotation direction, and combines the number of circles and the start and end frame angle compensation to effectively reduce the estimation error and improve the estimation accuracy.

[0089] This application can effectively eliminate interference and improve estimation accuracy through comprehensive analysis of the three circular marks on the ping-pong ball.

[0090] It should be understood that the above general description and the following detailed description are only exemplary and illustrative and do not limit the scope of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] The accompanying drawings below are part of the specification of the present application, which illustrate embodiments of the present application. The accompanying drawings, together with the description in the specification, are used to explain the principles of the present application.

[0092] Figure 1 A flowchart of a table tennis ball rotation estimation method based on visual measurement is provided in an embodiment of the present application.

[0093] Figure 2 A schematic diagram of the design of a circular mark on a table tennis ball in a table tennis ball rotation estimation method based on visual measurement provided in an embodiment of the present application.

[0094] Figure 3 A schematic diagram of the camera layout design in a table tennis rotation estimation method based on visual measurement provided in an embodiment of the present application.

[0095] Figure 4 A flowchart for calculating the rotation direction and speed in a table tennis rotation estimation method based on visual measurement provided in an embodiment of the present application.

[0096] Figure 5 This is a flowchart of the comprehensive analysis of the rotation speed results of each marker in a table tennis rotation estimation method based on visual measurement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0097] In order to make the objectives, technical solutions and advantages of the embodiments of the present application more clearly understood, the spirit of the contents disclosed in the present application will be clearly illustrated with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present application, any technician in the relevant technical field can change and modify the contents of the present application based on the techniques taught by the contents of the present application without departing from the spirit and scope of the contents of the present application.

[0098] The exemplary embodiments and descriptions of the present application are used to explain the present application, but are not intended to limit the present application. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0099] The terms “first,” “second,” etc. used herein do not specifically refer to an order or sequence, nor are they intended to limit this application. They are merely used to distinguish elements or operations described with the same technical terms.

[0100] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0101] As used herein, "and / or" includes any and all combinations of the items mentioned.

[0102] Regarding "plurality" herein, "plurality" includes "two" and "more than two"; regarding "plurality groups" herein, "plurality groups" includes "two groups" and "more than two groups".

[0103] Certain terms used to describe the present application are discussed below, or elsewhere in this specification, to provide additional guidance to those skilled in the art regarding the description of the present application.

[0104] like Figure 1 As shown, the table tennis ball rotation estimation method based on visual measurement provided in the embodiment of the present application includes the following steps:

[0105] S1. Mark the ping-pong balls and arrange the vision system cameras. The specific process is as follows:

[0106] like Figure 2 As shown, three solid circular marks of different colors are set on the ping-pong ball, wherein the line connecting the center of the first circular mark and the center of the second circular mark passes through the center of the ping-pong ball, and the line connecting the center of the third circular mark and the center of the ping-pong ball is perpendicular to the line connecting the center of the first circular mark and the center of the second circular mark.

[0107] Specifically, the table tennis ball used in this application is orange, and the three solid circular markings are in the distinctive colors of red, green, and white, with diameters ranging from 20 to 28 mm. This design ensures that the center of at least one of the circular markings is not lower than the horizontal equator of the table tennis ball, which is more conducive to visual observation.

[0108] like Figure 3 As shown, the vision system cameras used in this application include a left camera and a right camera.

[0109] The left and right cameras are symmetrically placed on the left and right sides of the net along its length, with the centerline of the net at its height as the axis of symmetry. Both cameras are located in the same plane as the net.

[0110] This large baseline layout can not only improve the accuracy of three-dimensional reconstruction of the center of the ping-pong ball, but also improve the success rate of observation of the circular mark on the ping-pong ball.

[0111] S2. Identify the circular area and pixel coordinates of the center of the ping-pong ball in the image captured by the visual system camera, and obtain the three-dimensional coordinates of the center of the ping-pong ball based on the pixel coordinates of the center of the circle. The specific process is as follows:

[0112] The left camera and the right camera are used to synchronously capture images of a table tennis ball with a solid circular mark, thereby obtaining a left image and a right image.

[0113] The HSV color recognition algorithm is used to perform image recognition on the left image and the right image to obtain the circular area and center pixel coordinates of the ping-pong ball in the left image and the right image.

[0114] Among them, the core of the HSV color recognition algorithm is to convert the image in RGB space into the image in HSV space, so as to achieve the purpose of color recognition.

[0115] The three-dimensional reconstruction of the center of the table tennis ball is completed using the parallax algorithm based on the pixel coordinates of the center of the circular area in the left image and the pixel coordinates of the center of the circular area in the right image.

[0116] It should be noted that the HSV color recognition algorithm and the disparity algorithm are both mature algorithms in the field of image processing and will not be described in detail here.

[0117] S3. Recognize the solid circular mark on the ping-pong ball in the circular area of ​​the ping-pong ball to obtain the marked area on the ping-pong ball and the center pixel coordinates of the marked area. The specific process is as follows:

[0118] The marked area on the ping pong ball in the image is identified by using the frame difference method between the marked ping pong ball and the unmarked ping pong ball to obtain the marked area.

[0119] HSV color recognition is performed on the marked area to determine the color of the circular mark on the ping-pong ball in the left and right images. At the same time, the ellipse fitting algorithm is used to obtain the central pixel coordinates of the marked area in the left and right images.

[0120] S4. Calculate the posture of the table tennis ball based on the three-dimensional coordinates of the center of the table tennis ball and the coordinates of the center pixel of the marked area on the table tennis ball. The specific process is as follows:

[0121] The following uses the marked area on the ping-pong ball identified in the left image as an example to calculate the posture of the ping-pong ball.

[0122] Assume that the center pixel coordinates of the marked area identified in the left image are (u bl ,v bl ), the three-dimensional coordinates of the center of the table tennis ball in the world coordinate system are (x cw ,y cw , z cw ).

[0123] Based on the pinhole imaging principle of the camera, the constraint relationship between the center pixel coordinates of the marked area on the ping-pong ball and the marked area on the ping-pong ball in the world coordinate system can be obtained as follows:

[0124]

[0125] In formula (1), z bl Indicates the depth distance of the center of the marked area relative to the left camera, u bl Indicates the row where the center of the marked region is in the left image, v bl Indicates the column where the center of the marked region is in the left image, M l represents the intrinsic parameter matrix of the left camera; l R w Represents the extrinsic parameter matrix of the left camera, that is, the rotation matrix of the world coordinate system relative to the left camera coordinate system; l T w Represents the translation vector of the left camera's extrinsic matrix relative to the left camera's coordinate system. The camera's intrinsic and extrinsic matrix are obtained through camera calibration. is the column vector form of the three-dimensional coordinates of the center of the marked area in the world coordinate system, and the parameters in the column vector are the parameters to be solved.

[0126] According to formula (1), the column vector form of the three-dimensional coordinates of the center of the marked area in the world coordinate system is obtained for:

[0127]

[0128] Let's set According to formula (2), the parametric equation of the spatial line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball can be obtained:

[0129]

[0130] Convert equation (3) into a point-wise equation:

[0131]

[0132] Let the direction vector of the space line represented by equation (4) be The point through which the straight line in the point-to-point formula passes is make The distance between the center of the ping-pong ball and the center of the marked area on the ping-pong ball is:

[0133]

[0134] Theoretically, the distance between the center of the ping-pong ball and the center of the marked area on the ping-pong ball is the radius r of the ping-pong ball. If |dr|>ε (ε is the preset allowable error, set according to the actual situation), the three-dimensional coordinate solution of the center of the marked area on the ping-pong ball fails; otherwise, calculate the foot of the perpendicular line from the center of the ping-pong ball to the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball. The coordinates of the foot of the perpendicular line are the three-dimensional coordinates of the center of the marked area on the ping-pong ball in the world coordinate system (x bw ,y bw , z bw ).

[0135] make Represents the direction vector of the perpendicular line from the center of the ping-pong ball to the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball. According to this perpendicular relationship You can get:

[0136] A1(x cw -x bw )+A2(y cw -y bw )+A3(z cw -z bw )=0 (6)

[0137] Substituting formula (3) into formula (6), we get:

[0138] A1(x cw -A1z bl -B1)+A2(y cw -A2z bl -B2)+A3(z cw -A3z bl -B3)=0 (7)

[0139] According to formula (7), the parameter z can be obtained bl for:

[0140]

[0141] Substituting equation (8) into equation (3), we can obtain the three-dimensional coordinates of the center of the marked area on the ping-pong ball in the world coordinate system.

[0142] According to the three-dimensional coordinates (x bw ,y bw , z bw) and the three-dimensional coordinates of the center of the table tennis ball in the world coordinate system (x cw ,y cw , z cw ), the unit vector of the table tennis posture at each moment is obtained as:

[0143]

[0144] S5, such as Figure 4 As shown, the rotation speed and rotation direction of the table tennis ball are calculated according to the table tennis ball posture unit vector at the preset time T. The specific process is:

[0145] S51. Use the current position and time of the table tennis ball to determine whether the current state is a new table tennis ball trajectory. If so, record the data, reset the parameters, and continue to solve the table tennis ball posture; otherwise, go to step S52.

[0146] S52: Determine whether the time for collecting the current trajectory posture is greater than the preset time T. If so, proceed to step S53; otherwise, record the data and return to step S51 to continue solving the table tennis posture.

[0147] Specifically, the threshold T for starting the calculation is set to 100 ms according to actual needs.

[0148] S53, analyzing the rules of table tennis mark recognition, the specific process is as follows:

[0149] The number of revolutions of the table tennis ball within the preset time T is estimated based on the appearance of circular marks of the same color. Specifically, the judgment is mainly made based on the appearance and disappearance patterns of the marks.

[0150] The appearance and disappearance patterns of the mark include the continuous appearance of the mark, the continuous disappearance of the mark, the interval appearance of the mark with adjacent frames, and the interval appearance of the mark without adjacent frames.

[0151] If the mark continues to appear or the mark continues to disappear, it can be determined that the ball is not spinning.

[0152] If the mark appears with an interval of adjacent frames or the mark appears without an interval of adjacent frames, the process proceeds to step S54.

[0153] S54: Analyze the table tennis ball posture unit vectors corresponding to the three colored circular marks. The specific process is as follows:

[0154] The rotation axis of the table tennis ball is obtained by plane fitting the posture unit vectors of the circular markers of the same color. The posture unit vectors of each color marker at different times can be obtained using formula (9) in step S4. Let one of the posture unit vectors be Among them, the subscript i represents the color mark index, j represents the vector index, and then the average coordinates of all posture unit vectors under the same color mark are solved Construct the error matrix:

[0155]

[0156] In formula (10), n represents the number of posture vectors of the same color circular mark solved within time T.

[0157] For the error matrix M ie Perform SVD decomposition, M ie =UDV T , where U represents an n×n left singular vector matrix; D represents an n×3 non-negative real diagonal matrix, whose diagonal elements are the error matrix M ie The singular values ​​of ; V represents a 3×3 orthogonal matrix, The axis of rotation of the table tennis ball is

[0158] S55 . For the mark that appears at intervals between adjacent frames, determine the final rotation direction of the rotation speed through the adjacent frames.

[0159] The table tennis ball's rotation axis has two rotation directions: positive and negative.

[0160] In the pose unit vector sequence Select adjacent frames and make the following analysis and judgment on the adjacent frames:

[0161] Set the rotation direction judgment parameter r d : in, represents the pose unit vector of the previous frame, Represents the unit vector of the posture of the next frame; the rotation direction of the ping-pong ball is:

[0162] The final rotation direction is determined by majority voting based on the rotation direction calculated from multiple sets of adjacent frames. The judgment of topspin, bottomspin, leftspin, rightspin, clockwise and counterspin is completed based on the rotation direction vector.

[0163] S56 , for intervals where no adjacent frames are marked, determine the final rotation direction based on adjacent marked frames.

[0164] Among them, the adjacent marked frames are two marked frames with only unmarked frames in between.

[0165] In the pose unit vector sequence Select similar marked frames and make the following analysis and judgment on the similar marked frames:

[0166] Set the rotation direction judgment parameter r d : in, represents the pose unit vector of the previous frame of the similarly marked frame, Indicates the pose unit vector of the frame after the marker frame appears again; the rotation direction of the ping-pong ball is:

[0167] The rotation direction is calculated based on multiple sets of marked similar frames, and the final rotation direction is determined by majority voting. The judgment of topspin, backspin, left spin, right spin, clockwise spin and counterspin is completed based on the rotation direction vector.

[0168] S57: Calculate the rotation speed of the table tennis ball, which includes determining the number of rotations and the start and end frame angles.

[0169] The number of rotations is determined by the appearance and disappearance of the mark. If the appearance, disappearance, and reappearance can be approximated as one circle, the number of circles n within the time T can be obtained. r .

[0170] The angle between the start and end frames is:

[0171] The analysis of the start and end frame angle direction is as follows:

[0172] Set the rotation angle judgment parameter r a : in, Represents the pose unit vector calculated in the first frame of the trajectory, Represents the pose unit vector calculated from the last frame of the trajectory within the preset time T, is the rotation direction obtained by step S55 or step S56, so that the rotation angle of the table tennis ball in the first frame and the last frame can be obtained as follows:

[0173] The angular velocity of the ping-pong ball is further obtained as: Among them, t s Indicates the start time of the mark, t e Indicates the stop time of the marker appearance.

[0174] S6, such as Figure 5 As shown in FIG, after estimating the rotation speed and rotation direction of the table tennis ball according to the three different colored marks, the results of the three marks are comprehensively analyzed to obtain the final rotation speed and rotation direction of the table tennis ball. The specific process is as follows:

[0175] like Figure 4As shown, first determine the number of tags among the three tags that meet the conditions and appear at intervals with adjacent frames. If the number of tags that meet the conditions is greater than 1, the result of the tag with more adjacent frames is used as the final result.

[0176] If the number of marks that meet the conditions is 1, the result that meets the conditions is taken as the final result;

[0177] If none of the three marks meet the condition that the marks have interval appearance with adjacent frames and have adjacent frames, then the number of marks that meet the interval appearance condition in the marks is determined. Such interval appearance marks have no adjacent frames;

[0178] If the number of markers appearing in the interval is greater than 1, the marker result with more markers appearing is taken as the final result;

[0179] If the number of marks appearing in the interval is 1, the result that meets the conditions is taken as the final result;

[0180] If the three marks do not meet the conditions for interval appearance, that is, the three marks appear continuously or disappear continuously, the ball without spin will be directly regarded as the final result.

[0181] This application utilizes a rational design of ping-pong ball markers and camera layout, then uses the HSV algorithm to identify the ball and reconstruct the ball's center in 3D. Frame difference and the HSV algorithm are then used to identify the markers within the ball's area. The ball's posture is then calculated using camera pinhole imaging and distance constraints. Finally, the ball's rotational speed and direction are accurately estimated using the ball's posture sequence. This application offers the ability to estimate ball rotation in real time over a large field of view, while being low-cost and highly practical.

[0182] This application uses only binocular stereo vision to estimate the rotation of a table tennis ball, which can effectively reduce costs and improve practicality; the innovative marker design can effectively reduce the problem of marker occlusion and improve the robustness of the system; the three-dimensional reconstruction of the table tennis ball markers is completed by solving the point-line distance constraint, and the accurate and effective solution of the table tennis ball posture is achieved; plane fitting and the use of the averaging principle of adjacent frames / similar marker frames can effectively improve the accuracy of the rotation direction; the angular step size of the start and end frames can effectively improve the accuracy of the rotation angular velocity; and the comprehensive analysis of multiple markers can improve the effectiveness and applicability of the entire system.

[0183] In an exemplary embodiment, based on the table tennis rotation estimation method based on visual measurement provided in the embodiments of the present application, the embodiments of the present application also provide a table tennis rotation estimation system based on visual measurement, which includes a memory and a processor coupled to the memory, and the processor is configured to execute the table tennis rotation estimation method based on visual measurement in any embodiment of the present application based on instructions stored in the memory.

[0184] The memory may be a system memory or a fixed non-volatile storage medium, etc. The system memory may store an operating system, application programs, a boot loader, a database, and other programs, etc.

[0185] It should be noted that the table tennis rotation estimation system based on visual measurement provided in the above embodiment and the table tennis rotation estimation method embodiment based on visual measurement belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0186] In an exemplary embodiment, the present application also provides a computer storage medium, which is a computer-readable storage medium, for example, a memory including a computer program, and the above-mentioned computer program can be executed by a processor to complete the table tennis rotation estimation method based on visual measurement in any embodiment of the present application.

[0187] The above-mentioned embodiments of the present application can be implemented in various hardware, software coding, or a combination of the two. For example, the embodiments of the present application can also represent program code for executing the above-mentioned method in a digital signal processor. The present application can also relate to various functions performed by a computer processor, a digital signal processor, a microprocessor, or a field programmable gate array. The above-mentioned processor can be configured according to the present application to perform specific tasks, which are completed by executing machine-readable software code or firmware code that defines the specific methods disclosed in the present application. The software code or firmware code can be developed to represent different programming languages ​​and different formats or forms. It can also represent different target platform compiled software codes. However, the different code styles, types, and languages ​​of the software code for performing tasks according to the present application and other types of configuration code do not depart from the spirit and scope of the present application.

[0188] The above description is only an illustrative embodiment of the present application. Without departing from the concept and principle of the present application, any equivalent changes and modifications made by any technician in this field should fall within the scope of protection of the present application.

Claims

1. A table tennis ball rotation estimation method based on visual measurement, characterized in that: The following steps are involved: Marking the ping-pong ball and arranging the vision system cameras involves the following steps: placing three solid circular marks of different colors on the ping-pong ball, with the line connecting the center of the first circular mark and the center of the second circular mark passing through the center of the ping-pong ball, and the line connecting the center of the third circular mark and the center of the ping-pong ball being perpendicular to the line connecting the center of the first circular mark and the center of the second circular mark; the vision system cameras include a left camera and a right camera, symmetrically arranged on the left and right sides of the length of the net, with the centerline of the net height as the axis of symmetry. The left and right cameras are both located in the same plane as the net. Identify the circular area and center pixel coordinates of the ping-pong ball in the image captured by the vision system camera, and obtain the three-dimensional coordinates of the center of the ping-pong ball based on the center pixel coordinates; Recognize the solid circular mark on the ping-pong ball in the circular area of ​​the ping-pong ball to obtain the marked area on the ping-pong ball and the center pixel coordinates of the marked area; The posture of the table tennis ball is calculated based on the three-dimensional coordinates of the center of the table tennis ball and the center pixel coordinates of the marked area on the table tennis ball; Calculating the rotation speed and rotation direction of the table tennis ball according to the unit vector of the table tennis ball posture at a preset time; After estimating the rotation speed and direction of the table tennis ball according to the three markers of different colors, the results of the three markers are comprehensively analyzed to obtain the final rotation speed and direction of the table tennis ball.

2. The table tennis rotation estimation method based on visual measurement according to claim 1, characterized in that: The specific process of identifying the circular area and the pixel coordinates of the center of the table tennis ball in the image captured by the visual system camera, and obtaining the three-dimensional coordinates of the center of the table tennis ball according to the pixel coordinates of the center of the circle is as follows: Using a left camera and a right camera to synchronously capture an image of a table tennis ball with a solid circular mark, a left image and a right image are obtained; Use the HSV color recognition algorithm to perform image recognition on the left and right images to obtain the circular area and center pixel coordinates of the ping-pong ball in the left and right images; The three-dimensional reconstruction of the center of the table tennis ball is completed using the parallax algorithm based on the pixel coordinates of the center of the circular area in the left image and the pixel coordinates of the center of the circular area in the right image.

3. The table tennis rotation estimation method based on visual measurement according to claim 2, characterized in that: The specific process of identifying the solid circular mark on the table tennis ball in the circular area of ​​the table tennis ball and obtaining the marked area on the table tennis ball and the center pixel coordinates of the marked area is as follows: The marked area on the ping pong ball in the image is identified by using the frame difference method between the marked ping pong ball and the unmarked ping pong ball to obtain the marked area; Perform HSV color recognition on the marked area to determine the color of the circular mark on the ping-pong ball in the left and right images; The center pixel coordinates of the marked areas in the left and right images are obtained using an ellipse fitting algorithm.

4. The table tennis rotation estimation method based on visual measurement according to claim 3, characterized in that: The specific process of calculating the posture of the table tennis ball according to the three-dimensional coordinates of the center of the table tennis ball and the central pixel coordinates of the marked area on the table tennis ball includes: The center pixel coordinates of the marked area identified in the left image are (u bl ,v bl ), the three-dimensional coordinates of the center of the table tennis ball in the world coordinate system are (x cw ,y cw , z cw ); Based on the pinhole imaging principle of the camera, the constraint relationship between the center pixel coordinates of the marked area on the ping-pong ball and the marked area on the ping-pong ball in the world coordinate system is obtained as follows: Where z bl Indicates the depth distance of the center of the marked area relative to the left camera, u bl Indicates the row where the center of the marked region is in the left image, v bl Indicates the column where the center of the marked region is in the left image, M l represents the intrinsic parameter matrix of the left camera; l R w Represents the extrinsic parameter matrix of the left camera, that is, the rotation matrix of the world coordinate system relative to the left camera coordinate system; l T w Represents the translation vector of the left camera's extrinsic matrix world coordinate system relative to the left camera coordinate system; the camera's intrinsic and extrinsic matrix are obtained through camera calibration. is the column vector form of the three-dimensional coordinates of the center of the marked area in the world coordinate system, and the parameters in the column vector are the parameters to be solved; According to the constraint relationship between the center pixel coordinates of the marked area on the ping-pong ball and the marked area on the ping-pong ball in the world coordinate system, the column vector form of the three-dimensional coordinates of the center of the marked area in the world coordinate system is obtained. for: set up According to The parametric equation of the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball is obtained from the expression: Convert the parametric equation of the space line into a point-wise form: Assume that the direction vector of the space line represented by the point-to-point formula is The point through which the straight line in the point-to-point formula passes is make The distance between the center of the ping-pong ball and the center of the marked area on the ping-pong ball is: Theoretically, the distance between the center of the ping-pong ball and the center of the marked area on the ping-pong ball is the radius r of the ping-pong ball. If |dr|>ε, where ε represents the preset allowable error, the three-dimensional coordinate solution of the center of the marked area on the ping-pong ball fails. Otherwise, the foot of the perpendicular line from the center of the ping-pong ball to the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball is calculated. The coordinates of the foot of the perpendicular line are the three-dimensional coordinates of the center of the marked area on the ping-pong ball in the world coordinate system (x bw ,y bw , z bw ); make Represents the direction vector of the perpendicular line drawn from the center of the ping-pong ball to the line connecting the origin of the camera coordinate system and the center of the marked area on the ping-pong ball. get: <h2 style=";text-align:left;direction:ltr">A1(x<h2 style=";text-align:left;direction:ltr"> cw <h2 style=";text-align:left;direction:ltr"> -x<h2 style=";text-align:left;direction:ltr"> bw <h2 style=";text-align:left;direction:ltr"> )+A2(y<h2 style=";text-align:left;direction:ltr"> cw <h2 style=";text-align:left;direction:ltr"> -y<h2 style=";text-align:left;direction:ltr"> bw <h2 style=";text-align:left;direction:ltr"> )+A3(z<h2 style=";text-align:left;direction:ltr"> cw <h2 style=";text-align:left;direction:ltr"> -z<h2 style=";text-align:left;direction:ltr"> bw <h2 style=";text-align:left;direction:ltr"> )=0; According to the parametric equation of the space line, we get: <h2 style=";text-align:left;direction:ltr">A1(x<h2 style=";text-align:left;direction:ltr"> cw <h2 style=";text-align:left;direction:ltr"> -A1z<h2 style=";text-align:left;direction:ltr"> bl <h2 style=";text-align:left;direction:ltr"> -B1)+A2(y<h2 style=";text-align:left;direction:ltr"> cw <h2 style=";text-align:left;direction:ltr"> -A2z<h2 style=";text-align:left;direction:ltr"> bl <h2 style=";text-align:left;direction:ltr"> -B2)+A3(z<h2 style=";text-align:left;direction:ltr"> cw <h2 style=";text-align:left;direction:ltr"> -A3z<h2 style=";text-align:left;direction:ltr"> bl <h2 style=";text-align:left;direction:ltr"> -B3)=0; Then we get the parameter z bl for: According to z bl The expression of and the parametric equation of the space line are used to obtain the three-dimensional coordinates of the center of the marked area on the ping-pong ball in the world coordinate system; According to the three-dimensional coordinates (x bw ,y bw , z bw ) and the three-dimensional coordinates of the center of the table tennis ball in the world coordinate system (x cw ,y cw , z cw ), the unit vector of the table tennis posture at each moment is obtained as:

5. The table tennis rotation estimation method based on visual measurement according to claim 4, characterized in that: The process of calculating the rotation speed and rotation direction of the table tennis ball according to the table tennis ball posture unit vector at the preset time is as follows: Use the current position and time of the ping-pong ball to determine whether the current state is a new ping-pong ball trajectory. If so, record the data, reset the parameters, and continue solving the ping-pong ball posture. Otherwise, determine whether the time of the current trajectory posture acquisition is greater than the preset time. If so, analyze the rules of ping-pong ball mark recognition; otherwise, record the data and continue solving the ping-pong ball posture. The number of rotations of a ping-pong ball within a preset time period is estimated by using the appearance and disappearance patterns of circular markers of the same color, thereby analyzing the patterns of ping-pong ball marker recognition. The patterns of marker appearance and disappearance include continuous marker appearance, continuous marker disappearance, marker appearance with intervals between adjacent frames, and marker appearance without intervals between adjacent frames. Analyze the table tennis ball posture unit vectors corresponding to the three colored circular marks; For the interval appearance of the mark with adjacent frames, the final rotation direction of the rotation speed is determined by the adjacent frames; For intervals where no adjacent frames are marked, the final rotation direction is determined based on the adjacent marked frames; The rotation speed of the table tennis ball is calculated, which includes the number of rotations and the start and end frame angles.

6. The table tennis rotation estimation method based on visual measurement according to claim 5, characterized in that: The process of analyzing the table tennis posture unit vectors corresponding to the three colored circular marks is as follows: The rotation axis of the table tennis ball is obtained by plane fitting the posture unit vectors marked with circles of the same color; Using the table tennis posture unit vector at each moment Get the posture unit vectors of each color mark at different times, and assume that one of the posture unit vectors is Where, subscript i represents the color mark index, and j represents the vector index; Solve the average coordinates of all posture unit vectors marked with the same color Construct the error matrix: Where n represents the number of posture vectors of the same color circle mark solved within the preset time T; For the error matrix M ie Perform SVD decomposition, M ie =UDV T , where U represents an n×n left singular vector matrix; D represents an n×3 non-negative real diagonal matrix, whose diagonal elements are the error matrix M ie The singular values ​​of ; V represents a 3×3 orthogonal matrix, The axis of rotation of the table tennis ball is 7. The table tennis rotation estimation method based on visual measurement according to claim 6, characterized in that: The specific process of determining the final rotation direction of the rotation speed through the adjacent frames when the mark appears at intervals of adjacent frames is as follows: The table tennis ball's rotation axis has two rotation directions, positive and negative, for the rotation speed; In the pose unit vector sequence Select adjacent frames and make the following analysis and judgment on the adjacent frames: Set the rotation direction judgment parameter r d : in, represents the pose unit vector of the previous frame, Represents the unit vector of the posture of the next frame; the rotation direction of the ping-pong ball is: The final rotation direction is determined by majority voting based on the rotation direction calculated from multiple sets of adjacent frames. The judgment of topspin, bottomspin, leftspin, rightspin, clockwise and counterspin is completed based on the rotation direction vector.

8. The table tennis rotation estimation method based on visual measurement according to claim 7, characterized in that: For the intervals where no adjacent frames are marked, the specific process of determining the final rotation direction based on the adjacent marked frames is as follows: Closely labeled frames are two labeled frames with only unlabeled frames in between; In the pose unit vector sequence Select similar marked frames and make the following analysis and judgment on the similar marked frames: Set the rotation direction judgment parameter r d : in, represents the pose unit vector of the previous frame of the similarly marked frame, Indicates the pose unit vector of the frame after the marker frame appears again; the rotation direction of the ping-pong ball is: The rotation direction is calculated based on multiple sets of marked similar frames, and the final rotation direction is determined by majority voting. The judgment of topspin, backspin, left spin, right spin, clockwise spin and counterspin is completed based on the rotation direction vector.

9. The table tennis rotation estimation method based on visual measurement according to claim 8, characterized in that: The process of calculating the rotation speed of the table tennis ball is as follows: Contains the number of rotations and the start and end frame angles; The number of rotations is determined by the appearance and disappearance of the mark. If one mark appears, disappears, and then appears again, it is considered a circle. The number of circles n within the time T is obtained. r ; The angle between the start and end frames is: The analysis of the start and end frame angle direction is as follows: Set the rotation angle judgment parameter r a : in, Represents the pose unit vector calculated in the first frame of the trajectory, Represents the pose unit vector calculated from the last frame of the trajectory within the preset time T, is the rotation direction, and the angle of rotation of the ping-pong ball in the first and last frames is: The angular velocity of the ping-pong ball is: Among them, t s Indicates the start time of the mark, t e Indicates the stop time of the marker appearance.

10. The table tennis rotation estimation method based on visual measurement according to claim 5, characterized in that: The specific process of comprehensively analyzing the results of the three different colored markers to obtain the final rotation speed and rotation direction of the table tennis ball is as follows: Determine the number of tags among the three tags that meet the conditions and appear at intervals with adjacent frames. If the number of tags that meet the conditions is greater than 1, the result of the tag with more adjacent frames is used as the final result. If the number of marks that meet the conditions is 1, the result that meets the conditions is taken as the final result; If none of the three marks meet the condition that the marks have interval appearance with adjacent frames and have adjacent frames, then the number of marks that meet the interval appearance condition in the marks is determined. Such interval appearance marks have no adjacent frames; If the number of markers appearing in the interval is greater than 1, the marker result with more markers appearing is taken as the final result; If the number of marks appearing in the interval is 1, the result that meets the conditions is taken as the final result; If all three marks appear or disappear continuously, the ball will be deemed as no spin.

11. A table tennis ball rotation estimation system based on visual measurement, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor processes the computer program, the method for estimating table tennis rotation based on visual measurement as described in any one of claims 1 to 10 is implemented.

Images