A tennis landing point determination method and system based on tennis court key point detection

Abstract

The application relates to the field of image processing and discloses a tennis landing point judgment method and system based on a tennis court key point detection, which generates a key point heat map and performs foreground enhancement, avoids a tendency to predict a fuzzy and expanded heat map, improves the accuracy of heat map regression, and further improves the generalization ability and accuracy, then performs fisheye camera parameter calibration, avoids serious imaging distortion caused by the super wide-angle field of view of the fisheye camera, accurately obtains the landing point of the tennis ball, further improves the accuracy of the judgment, and further performs distortion correction processing, avoids the distortion of the position information of the key point, improves the accuracy of the judgment of the landing point of the tennis ball, finally judges the landing point of the tennis ball according to the corrected key point, avoids the high cost caused by the use of a complex camera system, avoids distortion misjudgment caused by the super wide-angle lens, and improves the accuracy of the judgment of the landing point of the tennis ball.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method and system for determining the landing point of a tennis ball based on the detection of key points on a tennis court. Background Technology

[0002] With the rapid development of computer vision technology, computer vision recognition technology has been widely applied in various fields, especially in sports. In tennis, a racket-based, net-based competitive match, the ball speed is so fast that it is difficult for referees to judge whether the ball is in or out of bounds by sensory perception, resulting in many disputed balls. At this time, using computer vision technology to determine the landing point of the tennis ball has become an indispensable means.

[0003] In existing technologies, multiple high-speed cameras are typically used to build an Hawk-Eye system to monitor tennis courts in real time and obtain information on the tennis court and the movement of the tennis ball. However, building a Hawk-Eye system is expensive and difficult to deploy in training venues, failing to meet the growing practical needs of a wide range of enthusiasts. Ordinary cameras have a smaller angle of view, and if the view needs to cover the entire tennis court, multiple cameras must be deployed simultaneously to collect data in real time, thus increasing the complexity of the system. Furthermore, although wide-angle cameras have a large angle of view, they have greater distortion, making it difficult to accurately determine the landing point of the tennis ball.

[0004] Therefore, how to design a method for determining the landing point of a tennis ball while avoiding high construction costs and improving the accuracy of the determination has become an urgent problem to be solved. Summary of the Invention

[0005] Based on this, the present invention proposes a method and system for determining the landing point of a tennis ball based on the detection of key points on a tennis court. By generating a heatmap of key points and performing foreground enhancement, it avoids the tendency to predict blurry and expand the heatmap, improving the accuracy of heatmap regression, thereby improving generalization ability and accuracy. Then, fisheye camera parameter calibration is performed to avoid severe imaging distortion caused by the ultra-wide field of view of the fisheye camera, so as to accurately obtain the landing point of the tennis ball, further improving the accuracy of the determination. Distortion correction processing is also performed to avoid distortion of the position information of key points, thereby improving the accuracy of the determination of the landing point of the tennis ball. Finally, the landing point of the tennis ball is determined based on the corrected key points. This invention avoids the high cost caused by using complex camera systems and avoids the distortion misjudgment caused by ultra-wide-angle lenses. The present invention improves the accuracy of tennis ball landing point determination.

[0006] This invention proposes a method for determining the landing point of a tennis ball based on the detection of key points on a tennis court, comprising:

[0007] Acquire an image of the target tennis court and generate a key point heatmap. Perform foreground enhancement processing on the key point heatmap. The key point heatmap is generated based on a Gaussian kernel function. The key points include the intersection of court landmark lines.

[0008] Fisheye camera parameters are calibrated based on the key point heatmap after foreground enhancement processing. The fisheye camera parameter calibration is used to obtain the internal parameters and distortion coefficients of the fisheye camera.

[0009] Distortion correction is performed based on the calibrated fisheye camera parameters to obtain key correction points. The distortion correction is based on the normalized camera coordinate system.

[0010] The tennis ball landing point is determined based on the correction key points to obtain the final tennis ball landing point. The tennis ball landing point determination is based on homography matrix mapping.

[0011] In summary, the tennis landing point determination method based on key point detection on a tennis court, as described above, avoids predictive ambiguity and expands the heatmap by generating a key point heatmap and performing foreground enhancement, thus improving the accuracy of heatmap regression and consequently enhancing generalization ability and accuracy. Furthermore, fisheye camera parameter calibration avoids severe imaging distortion caused by the ultra-wide-angle field of view of the fisheye camera, accurately acquiring the tennis landing point and further improving the determination accuracy. Distortion correction processing further avoids distortion of the key point's positional information, improving the accuracy of tennis landing point determination. Finally, the tennis landing point is determined based on the corrected key points, avoiding the high costs associated with using complex camera systems and avoiding distortion misjudgments caused by ultra-wide-angle lenses. Therefore, this invention improves the accuracy of tennis landing point determination. Specifically, the process involves acquiring an image of the target tennis court and generating a keypoint heatmap. This heatmap is then enhanced with foreground enhancement. The heatmap is generated based on a Gaussian kernel function, and the keypoints include intersections of court landmarks. This avoids the tendency to predict blurry and expanded heatmaps, improving the accuracy of heatmap regression and thus enhancing generalization ability and accuracy. Based on the enhanced keypoint heatmap, fisheye camera parameters are calibrated. This calibration obtains the fisheye camera's internal parameters and distortion coefficients, preventing severe imaging distortion caused by the ultra-wide field of view of the fisheye camera, thus ensuring accurate... This invention accurately obtains the landing point of the tennis ball, further improving the accuracy of the determination. Distortion correction processing is performed based on the calibrated fisheye camera parameters to obtain key correction points. This distortion correction processing is based on a normalized camera coordinate system, avoiding distortion of the positional information of key points, thus improving the accuracy of tennis ball landing point determination. The final tennis ball landing point is determined based on the corrected key points. This tennis ball landing point determination is based on homography matrix mapping, avoiding the high costs associated with using complex camera systems and avoiding distortion misjudgments caused by ultra-wide-angle lenses. Therefore, this invention improves the accuracy of tennis ball landing point determination.

[0012] Furthermore, the step of acquiring the target tennis court image and generating a key point heatmap specifically includes:

[0013] Acquire images of the target tennis court and perform preprocessing, including data cleaning, normalization, and data augmentation.

[0014] Key points are set based on the image of the target tennis court, including the intersections of court markings. A heatmap of the key points is generated using a Gaussian kernel function. The specific algorithm for generating the heatmap of the key points is as follows:

[0015] =exp(- ),

[0016] ,

[0017] in, This represents the single-point thermal value of a key point heatmap. , These represent the x and y coordinates of a single point on the heatmap, respectively. , These represent the x and y coordinates of the key points, respectively. σ The standard deviation of the Gaussian kernel. The scaling factor representing the Gaussian kernel size. , These represent the width and height of the target tennis court image, respectively. This represents the maximum value function.

[0018] Furthermore, the step of performing foreground enhancement processing on the key point heatmap specifically includes:

[0019] Weights are generated for each pixel in the target tennis court image to perform foreground enhancement processing on the keypoint heatmap. The loss algorithm for the foreground enhancement processing is as follows:

[0020] ,

[0021] log(1+ + ε ),

[0022] in, This indicates the loss from foreground enhancement processing. Indicates the number of key points. Indicates the ordinal number of the key point. Indicates the first i A heatmap of key real-world labels. A heatmap representing the prediction. This represents the weight generation function. ε This represents a small perturbation. ε> 0.

[0023] Furthermore, the step of calibrating fisheye camera parameters based on the keypoint heatmap after foreground enhancement specifically includes:

[0024] A tennis court coordinate system is constructed based on the target tennis court image. The origin of this coordinate system is any base corner point, and the baseline corresponding to each base corner point is... x In the axial direction, the edge line corresponding to the bottom corner point is y Axial direction;

[0025] The tennis ball key points in the camera coordinate system are then regenerated. The specific algorithm for generating the tennis ball key points in the camera coordinate system is as follows:

[0026] ,

[0027] in, Indicates the camera coordinate system. Represents the coordinate system of the tennis court. Indicates the location of key points in the camera coordinate system. This indicates the spatial location of key points in the tennis court coordinate system. This represents the rotation matrix of the tennis court coordinate system relative to the camera coordinate system. This represents the position vector of the origin of the tennis court coordinate system relative to the origin of the camera coordinate system. Represents the spatial coordinates of key tennis points in the camera coordinate system;

[0028] Based on the equidistant projection model of a fisheye camera, a relationship function between the incident angle and the projection point is constructed. The specific relationship function between the incident angle and the projection point is as follows:

[0029] ,

[0030] ,

[0031] ,

[0032] in, This represents the distance from the projection point to the center of the image. f This indicates the focal length of the fisheye camera. Indicates the angle of incidence. , , The horizontal, vertical, and ordinate coordinates of the key points in the camera coordinate system represent their spatial positions.

[0033] A function relating the distorted angle of refraction to the angle of incidence is constructed based on higher-order distortion coefficients. The specific function relating the distorted angle of refraction to the angle of incidence is as follows:

[0034] ,

[0035] in, Indicates the angle of refraction. , , , Indicates the distortion coefficient;

[0036] The image pixels corresponding to key points are calculated based on the relationship function between the distorted refraction angle and the incident angle. The specific algorithm for calculating the image pixels corresponding to key points is as follows:

[0037] ,

[0038] in, Image pixels representing key points, This indicates the internal parameters of the fisheye camera;

[0039] The optimization is performed based on the tennis ball key points in the camera coordinate system, the relationship function between the incident angle and the projection point, the relationship function between the distorted refraction angle and the incident angle, and the image pixels corresponding to the key points.

[0040] Furthermore, the step of optimizing based on the tennis ball key points in the camera coordinate system, the relationship function between the incident angle and the projection point, the relationship function between the distorted refraction angle and the incident angle, and the image pixels corresponding to the key points specifically includes:

[0041] The objective optimization function is constructed, and the specific algorithm for the objective optimization function is as follows:

[0042] ,

[0043] in, Describe the objective optimization function. This represents the rotation matrix of the tennis court coordinate system relative to the camera coordinate system. This represents the position vector of the origin of the tennis court coordinate system relative to the origin of the camera coordinate system. , , , Indicates the distortion coefficient. This indicates the internal parameters of the fisheye camera. Indicates the number of key points. Indicates the ordinal number of the key point. , The x and y coordinates represent the theoretical pixel positions of keypoints. , The x and y coordinates represent the actual pixel positions of key points.

[0044] Furthermore, the step of performing distortion correction processing based on the calibrated fisheye camera parameters to obtain key correction points specifically includes:

[0045] Based on the calibrated fisheye camera parameters, a keypoint coordinate transformation is performed. This keypoint coordinate transformation is based on a normalized camera coordinate system, and the specific algorithm for the keypoint coordinate transformation is as follows:

[0046] ,

[0047] in, , Represents the x and y coordinates of key points in the normalized camera coordinate system. This represents the inverse of the intrinsic parameter matrix of the fisheye camera. , The x and y coordinates represent the actual pixel positions of the key points;

[0048] The refraction angle is calculated based on the transformed key point coordinates. The specific algorithm for calculating the refraction angle is as follows:

[0049] ,

[0050] in, Indicates the angle of refraction;

[0051] The incident angle is then optimized based on the refraction angle. The specific algorithm for optimizing the incident angle is as follows:

[0052] ,

[0053] in, Indicates the angle of incidence. This represents the optimization solution function. , , , Indicates the distortion coefficient;

[0054] The distance from the distortion-free projection point to the image center is calculated based on the incident angle. The specific algorithm for calculating the distance from the distortion-free projection point to the image center is as follows:

[0055] ,

[0056] in, This represents the distance from the undistorted projection point to the center of the image;

[0057] The distortion-free pixel coordinates are calculated to obtain the correction key points. The specific algorithm for calculating the distortion-free pixel coordinates is as follows:

[0058] ,

[0059] ,

[0060] in, , Represents the x and y coordinates of a pixel without distortion.

[0061] Furthermore, the step of determining the tennis ball's landing point based on the correction key points to obtain the final tennis ball landing point specifically includes:

[0062] The mapping relationship between the calibration key points and the key points of the actual standard tennis court is constructed. The specific algorithm for the mapping relationship between the calibration key points and the key points of the actual standard tennis court is as follows:

[0063] ,

[0064] ,

[0065] in, , This represents the x and y coordinates of key points on a standard tennis court. , Represents the x and y coordinates of a pixel without distortion. Represents the homography matrix. This represents the homography matrix optimization function. Indicates the number of key points. Indicates the ordinal number of the key point. Represents the function to be minimized. , Indicates the first The x and y coordinates of each key pixel without distortion. , Indicates the first The x and y coordinates of key points on a standard tennis court;

[0066] Based on the mapping relationship between the corrected key points and the key points of the actual standard tennis court, the final tennis ball landing point is obtained. The specific algorithm for the final tennis ball landing point is as follows:

[0067] ,

[0068] in, , The x and y coordinates represent the final landing point of the tennis ball. , The horizontal and vertical coordinates of the image pixels representing the final landing point of the tennis ball.

[0069] This invention proposes a tennis ball landing point determination system based on the detection of key points on a tennis court, comprising:

[0070] A key point generation module is used to acquire an image of a target tennis court and generate a key point heatmap, and to perform foreground enhancement processing on the key point heatmap. The key point heatmap is generated based on a Gaussian kernel function, and the key points include intersections of court landmarks.

[0071] The camera parameter calibration module is used to calibrate the fisheye camera parameters based on the key point heatmap after foreground enhancement processing. The fisheye camera parameter calibration is used to obtain the internal parameters and distortion coefficients of the fisheye camera.

[0072] The distortion correction module is used to perform distortion correction processing based on the calibrated fisheye camera parameters to obtain key correction points. The distortion correction processing is based on the normalized camera coordinate system.

[0073] The landing point determination module is used to determine the landing point of the tennis ball based on the correction key points to obtain the final landing point of the tennis ball. The tennis ball landing point determination is based on homography matrix mapping.

[0074] The present invention also provides a storage medium storing one or more programs, which, when executed by a processor, implement the tennis ball landing point determination method based on the detection of key points on a tennis court as described above.

[0075] The present invention also provides a computer device, the computer device including a memory and a processor, wherein:

[0076] The memory is used to store computer programs;

[0077] When the processor executes the computer program stored in the memory, it implements the tennis ball landing point determination method based on the detection of key points on the tennis court, as described above. Attached Figure Description

[0078] Figure 1 This is a flowchart of the tennis landing point determination method based on the detection of key points on a tennis court, as proposed in the first embodiment of the present invention.

[0079] Figure 2 This is a schematic diagram of the tennis landing point determination system based on the detection of key points on the tennis court, as proposed in the second embodiment of the present invention.

[0080] Figure 3 This is a schematic diagram of key points of a tennis court according to the first embodiment of the present invention;

[0081] Figure 4This is a schematic diagram of deep learning-based tennis court key point detection according to the first embodiment of the present invention.

[0082] Figure 5 This is a schematic diagram of the fisheye camera distortion according to the first embodiment of the present invention;

[0083] Figure 6 This is a schematic diagram of distortion correction according to the first embodiment of the present invention;

[0084] Figure 7 This is a schematic diagram illustrating the landing point determination of the first embodiment of the present invention.

[0085] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0086] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0087] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0088] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0089] Please see Figure 1 The diagram shows a flowchart of a tennis ball landing point determination method based on tennis court key point detection according to the first embodiment of the present invention. This tennis ball landing point determination method based on tennis court key point detection includes steps S01 to S04, wherein:

[0090] Step S01: Acquire the image of the target tennis court and generate a key point heatmap, and perform foreground enhancement processing on the key point heatmap;

[0091] It should be noted that in this embodiment, the image of the target tennis court is acquired using a fisheye camera. For information on fisheye camera distortion effects, please refer to [link / reference needed]. Figure 5 ,in , These represent the x and y coordinates of a single point on the heatmap, respectively. , These represent the x and y coordinates of the image pixels, respectively. This represents the deviation from the image center. Indicates the angle of refraction. f This indicates the focal length of the fisheye camera. Indicates the center of the image. , , Represents the spatial position of keypoints in the camera coordinate system using their x, y, and x coordinates. Indicates the origin of the camera coordinate system. Indicates the angle of incidence. Indicates the actual point captured by the fisheye camera. This represents the point after the incident ray enters the fisheye camera from the actual point. , This represents the abscissa and ordinate of the spatial location of keypoints in the tennis court coordinate system. In this embodiment, the generation of keypoint heatmaps and foreground enhancement processing are based on a deep learning network model. For the specific process of deep learning-based tennis court keypoint detection, please refer to [link to documentation]. Figure 4 The key point heatmap is generated based on a Gaussian kernel function. The key points include the intersections of court markings. For specific key points of a tennis court, please refer to [link / reference needed]. Figure 3 Where #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, and #19 are key points of the tennis court, indicating that the target tennis court image has been acquired and preprocessed. The preprocessing includes data cleaning, normalization, and data augmentation.

[0092] Key points are set based on the image of the target tennis court, including the intersections of court markings. A heatmap of the key points is generated using a Gaussian kernel function. The specific algorithm for generating the heatmap of the key points is as follows:

[0093] =exp(- ),

[0094] ,

[0095] in, This represents the single-point thermal value of a key point heatmap. , These represent the x and y coordinates of a single point on the heatmap, respectively. , These represent the x and y coordinates of the key points, respectively. σ The standard deviation of the Gaussian kernel. The scaling factor representing the Gaussian kernel size. , These represent the width and height of the target tennis court image, respectively. Represents the maximum value function;

[0096] Weights are generated for each pixel in the target tennis court image to perform foreground enhancement processing on the keypoint heatmap. The loss algorithm for the foreground enhancement processing is as follows:

[0097] ,

[0098] log(1+ + ε ),

[0099] in, This indicates the loss from foreground enhancement processing. Indicates the number of key points. Indicates the ordinal number of the key point. Indicates the first i A heatmap of key real-world labels. A heatmap representing the prediction. This represents the weight generation function. ε This represents a small perturbation. ε> 0.

[0100] Step S02: Calibrate the fisheye camera parameters based on the key point heatmap after foreground enhancement processing;

[0101] It should be noted that in this embodiment, the fisheye camera parameter calibration is used to obtain the fisheye camera's internal parameters and distortion coefficients, and to construct a tennis court coordinate system based on the target tennis court image. The origin of this coordinate system is any bottom corner point, and the baseline corresponding to each bottom corner point is... x In the axial direction, the edge line corresponding to the bottom corner point is y Axial direction;

[0102] The tennis ball key points in the camera coordinate system are then regenerated. The specific algorithm for generating the tennis ball key points in the camera coordinate system is as follows:

[0103] ,

[0104] in, Indicates the camera coordinate system. Represents the coordinate system of the tennis court. Indicates the location of key points in the camera coordinate system. This indicates the spatial location of key points in the tennis court coordinate system. This represents the rotation matrix of the tennis court coordinate system relative to the camera coordinate system. This represents the position vector of the origin of the tennis court coordinate system relative to the origin of the camera coordinate system. Represents the spatial coordinates of key tennis points in the camera coordinate system;

[0105] Based on the equidistant projection model of a fisheye camera, a relationship function between the incident angle and the projection point is constructed. The specific relationship function between the incident angle and the projection point is as follows:

[0106] ,

[0107] ,

[0108] ,

[0109] in, This represents the distance from the projection point to the center of the image. f This indicates the focal length of the fisheye camera. Indicates the angle of incidence. , , The horizontal, vertical, and ordinate coordinates of the key points in the camera coordinate system represent their spatial positions.

[0110] A function relating the distorted angle of refraction to the angle of incidence is constructed based on higher-order distortion coefficients. The specific function relating the distorted angle of refraction to the angle of incidence is as follows:

[0111] ,

[0112] in, Indicates the angle of refraction. , , , Indicates the distortion coefficient;

[0113] The image pixels corresponding to key points are calculated based on the relationship function between the distorted refraction angle and the incident angle. The specific algorithm for calculating the image pixels corresponding to key points is as follows:

[0114] ,

[0115] in, Image pixels representing key points, This indicates the internal parameters of the fisheye camera;

[0116] The optimization is performed based on the tennis ball key points in the camera coordinate system, the relationship function between the incident angle and the projection point, the relationship function between the distorted refraction angle and the incident angle, and the image pixels corresponding to the key points.

[0117] The objective optimization function is constructed, and the specific algorithm for the objective optimization function is as follows:

[0118] ,

[0119] in, Describe the objective optimization function. This represents the rotation matrix of the tennis court coordinate system relative to the camera coordinate system. This represents the position vector of the origin of the tennis court coordinate system relative to the origin of the camera coordinate system. , , , Indicates the distortion coefficient. This indicates the internal parameters of the fisheye camera. Indicates the number of key points. Indicates the ordinal number of the key point. , The x and y coordinates represent the theoretical pixel positions of keypoints. , The x and y coordinates represent the actual pixel positions of key points.

[0120] Step S03: Perform distortion correction processing based on the calibrated fisheye camera parameters to obtain key correction points;

[0121] It should be noted that, in this embodiment, the specific effect of distortion correction can be found in the following reference: Figure 6 Among them, #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, and #19 are key points on the tennis court. The distortion correction process is based on the normalized camera coordinate system. According to the calibrated fisheye camera parameters, the key point coordinates are transformed. The specific algorithm for the key point coordinate transformation is as follows:

[0122] ,

[0123] in, , Represents the x and y coordinates of key points in the normalized camera coordinate system. This represents the inverse of the intrinsic parameter matrix of the fisheye camera. , The x and y coordinates represent the actual pixel positions of the key points;

[0124] The refraction angle is calculated based on the transformed key point coordinates. The specific algorithm for calculating the refraction angle is as follows:

[0125] ,

[0126] in, Indicates the angle of refraction;

[0127] The incident angle is then optimized based on the refraction angle. The specific algorithm for optimizing the incident angle is as follows:

[0128] ,

[0129] in, Indicates the angle of incidence. This represents the optimization solution function. , , , Indicates the distortion coefficient;

[0130] The distance from the distortion-free projection point to the image center is calculated based on the incident angle. The specific algorithm for calculating the distance from the distortion-free projection point to the image center is as follows:

[0131] ,

[0132] in, This represents the distance from the undistorted projection point to the center of the image;

[0133] The distortion-free pixel coordinates are calculated to obtain the correction key points. The specific algorithm for calculating the distortion-free pixel coordinates is as follows:

[0134] ,

[0135] ,

[0136] in, , Represents the x and y coordinates of a pixel without distortion.

[0137] Step S04: Determine the tennis ball landing point based on the key points of the correction to obtain the final tennis ball landing point;

[0138] It should be noted that, in this embodiment, the specific effect of tennis ball landing point determination can be found in the following reference: Figure 7 Where #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12, #13, #14, #15, #16, #17, #18, and #19 are key points on the tennis court. The determination of the tennis ball's landing point is based on homography matrix mapping, constructing a mapping relationship between the corrected key points and the actual standard tennis court key points. The specific algorithm for the mapping relationship between the corrected key points and the actual standard tennis court key points is as follows:

[0139] ,

[0140] ,

[0141] in, , This represents the x and y coordinates of key points on a standard tennis court. , Represents the x and y coordinates of a pixel without distortion. Represents the homography matrix. This represents the homography matrix optimization function. Indicates the number of key points. Indicates the ordinal number of the key point. Represents the function to be minimized. , Indicates the first The x and y coordinates of each key pixel without distortion. , Indicates the first The x and y coordinates of key points on a standard tennis court;

[0142] Based on the mapping relationship between the corrected key points and the key points of the actual standard tennis court, the final tennis ball landing point is obtained. The specific algorithm for the final tennis ball landing point is as follows:

[0143] ,

[0144] in, , The x and y coordinates represent the final landing point of the tennis ball. , The horizontal and vertical coordinates of the image pixels representing the final landing point of the tennis ball.

[0145] In summary, the tennis landing point determination method based on key point detection on a tennis court, as described above, avoids predictive ambiguity and expands the heatmap by generating a key point heatmap and performing foreground enhancement, thus improving the accuracy of heatmap regression and consequently enhancing generalization ability and accuracy. Furthermore, fisheye camera parameter calibration avoids severe imaging distortion caused by the ultra-wide-angle field of view of the fisheye camera, accurately acquiring the tennis landing point and further improving the determination accuracy. Distortion correction processing further avoids distortion of the key point's positional information, improving the accuracy of tennis landing point determination. Finally, the tennis landing point is determined based on the corrected key points, avoiding the high costs associated with using complex camera systems and avoiding distortion misjudgments caused by ultra-wide-angle lenses. Therefore, this invention improves the accuracy of tennis landing point determination. Specifically, the process involves acquiring an image of the target tennis court and generating a keypoint heatmap. This heatmap is then enhanced with foreground enhancement. The heatmap is generated based on a Gaussian kernel function, and the keypoints include intersections of court landmarks. This avoids the tendency to predict blurry and expanded heatmaps, improving the accuracy of heatmap regression and thus enhancing generalization ability and accuracy. Based on the enhanced keypoint heatmap, fisheye camera parameters are calibrated. This calibration obtains the fisheye camera's internal parameters and distortion coefficients, preventing severe imaging distortion caused by the ultra-wide field of view of the fisheye camera, thus ensuring accurate... This invention accurately obtains the landing point of the tennis ball, further improving the accuracy of the determination. Distortion correction processing is performed based on the calibrated fisheye camera parameters to obtain key correction points. This distortion correction processing is based on a normalized camera coordinate system, avoiding distortion of the positional information of key points, thus improving the accuracy of tennis ball landing point determination. The final tennis ball landing point is determined based on the corrected key points. This tennis ball landing point determination is based on homography matrix mapping, avoiding the high costs associated with using complex camera systems and avoiding distortion misjudgments caused by ultra-wide-angle lenses. Therefore, this invention improves the accuracy of tennis ball landing point determination.

[0146] Please see Figure 2 The figure shown is a schematic diagram of a tennis ball landing point determination system based on key point detection on a tennis court, as proposed in the second embodiment of the present invention. The system includes:

[0147] The key point generation module 10 is used to acquire a target tennis court image and generate a key point heatmap, and to perform foreground enhancement processing on the key point heatmap. The key point heatmap is generated based on a Gaussian kernel function, and the key points include the intersection of court landmark lines.

[0148] The camera parameter calibration module 20 is used to calibrate the fisheye camera parameters based on the key point heatmap after foreground enhancement processing. The fisheye camera parameter calibration is used to obtain the internal parameters and distortion coefficients of the fisheye camera.

[0149] The distortion correction module 30 is used to perform distortion correction processing based on the calibrated fisheye camera parameters to obtain key correction points. The distortion correction processing is based on the normalized camera coordinate system.

[0150] The landing point determination module 40 is used to determine the landing point of the tennis ball based on the correction key points to obtain the final landing point of the tennis ball. The tennis ball landing point determination is based on homography matrix mapping.

[0151] The present invention also proposes a computer storage medium storing one or more programs that, when executed by a processor, implement the above-described method for determining the landing point of a tennis ball based on the detection of key points on a tennis court.

[0152] The present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-described method for determining the landing point of a tennis ball based on the detection of key points on a tennis court.

[0153] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0154] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0155] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0156] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0157] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for determining the landing point of a tennis ball based on the detection of key points on a tennis court, characterized in that, include: Acquire an image of the target tennis court and generate a key point heatmap. Perform foreground enhancement processing on the key point heatmap. The key point heatmap is generated based on a Gaussian kernel function. The key points include the intersection of court landmark lines. Fisheye camera parameters are calibrated based on the key point heatmap after foreground enhancement processing. The fisheye camera parameter calibration is used to obtain the internal parameters and distortion coefficients of the fisheye camera. A tennis court coordinate system is constructed based on the target tennis court image. The origin of this coordinate system is any base corner point, and the baseline corresponding to each base corner point is... x In the axial direction, the edge line corresponding to the bottom corner point is y Axial direction; The tennis ball key points in the camera coordinate system are then regenerated. The specific algorithm for generating the tennis ball key points in the camera coordinate system is as follows: ， in, Indicates the camera coordinate system. Represents the coordinate system of the tennis court. Indicates the location of key points in the camera coordinate system. This indicates the spatial location of key points in the tennis court coordinate system. This represents the rotation matrix of the tennis court coordinate system relative to the camera coordinate system. This represents the position vector of the origin of the tennis court coordinate system relative to the origin of the camera coordinate system. Represents the spatial coordinates of key tennis points in the camera coordinate system; Based on the equidistant projection model of a fisheye camera, a relationship function between the incident angle and the projection point is constructed. The specific relationship function between the incident angle and the projection point is as follows: ， ， ， in, This represents the distance from the projection point to the center of the image. f This indicates the focal length of the fisheye camera. Indicates the angle of incidence. , , Represents the spatial position of keypoints in the camera coordinate system using their x, y, and x coordinates. This represents the distance from the undistorted projection point to the center of the image; Distortion correction is performed based on the calibrated fisheye camera parameters to obtain key correction points. The distortion correction is based on the normalized camera coordinate system. The tennis ball landing point is determined based on the correction key points to obtain the final tennis ball landing point. The tennis ball landing point determination is based on homography matrix mapping.

2. The method for determining the landing point of a tennis ball based on the detection of key points on a tennis court according to claim 1, characterized in that, The steps of acquiring the target tennis court image and generating a key point heatmap specifically include: Acquire images of the target tennis court and perform preprocessing, including data cleaning, normalization, and data augmentation. Key points are set based on the image of the target tennis court, including the intersections of court markings. A heatmap of the key points is generated using a Gaussian kernel function. The specific algorithm for generating the heatmap of the key points is as follows: =exp(- ), ， in, This represents the single-point thermal value of a key point heatmap. , These represent the x and y coordinates of a single point on the heatmap, respectively. , These represent the x and y coordinates of the key points, respectively. σ The standard deviation of the Gaussian kernel. The scaling factor representing the Gaussian kernel size. , These represent the width and height of the target tennis court image, respectively. This represents the maximum value function.

3. The method for determining the landing point of a tennis ball based on the detection of key points on a tennis court according to claim 1, characterized in that, The step of performing foreground enhancement processing on the key point heatmap specifically includes: Weights are generated for each pixel in the target tennis court image to perform foreground enhancement processing on the keypoint heatmap. The loss algorithm for the foreground enhancement processing is as follows: ， log(1+ + ε ), in, This indicates the loss from foreground enhancement processing. Indicates the number of key points. Indicates the ordinal number of the key point. Indicates the first i A heatmap of key real-world labels. A heatmap representing the prediction. This represents the weight generation function. ε This represents a small perturbation. ε> 0.

4. The method for determining the landing point of a tennis ball based on the detection of key points on a tennis court according to claim 1, characterized in that, The step of calibrating fisheye camera parameters based on the key point heatmap after foreground enhancement specifically includes: A function relating the distorted angle of refraction to the angle of incidence is constructed based on higher-order distortion coefficients. The specific function relating the distorted angle of refraction to the angle of incidence is as follows: ， in, Indicates the angle of refraction. , , , Indicates the distortion coefficient; The image pixels corresponding to key points are calculated based on the relationship function between the distorted refraction angle and the incident angle. The specific algorithm for calculating the image pixels corresponding to key points is as follows: ， in, Image pixels representing key points, This indicates the internal parameters of the fisheye camera; The optimization is performed based on the relationship function between the tennis ball key points in the camera coordinate system, the incident angle and the projection point, the relationship function between the distorted refraction angle and the incident angle, and the image pixels corresponding to the key points.

5. The method for determining the landing point of a tennis ball based on the detection of key points on a tennis court according to claim 4, characterized in that, The optimization step, which involves optimizing the image based on the relationship between the tennis ball key points in the camera coordinate system, the relationship between the incident angle and the projection point, the relationship between the distorted refraction angle and the incident angle, and the image pixels corresponding to the key points, specifically includes: The objective optimization function is constructed, and the specific algorithm for the objective optimization function is as follows: ， in, Describe the objective optimization function. This represents the rotation matrix of the tennis court coordinate system relative to the camera coordinate system. This represents the position vector of the origin of the tennis court coordinate system relative to the origin of the camera coordinate system. , , , Indicates the distortion coefficient. This indicates the internal parameters of the fisheye camera. Indicates the number of key points. Indicates the ordinal number of the key point. , The x and y coordinates represent the theoretical pixel positions of keypoints. , The x and y coordinates represent the actual pixel positions of key points.

6. The method for determining the landing point of a tennis ball based on the detection of key points on a tennis court according to claim 1, characterized in that, The step of performing distortion correction processing based on the calibrated fisheye camera parameters to obtain key correction points specifically includes: Based on the calibrated fisheye camera parameters, a keypoint coordinate transformation is performed. This keypoint coordinate transformation is based on a normalized camera coordinate system, and the specific algorithm for the keypoint coordinate transformation is as follows: ， in, , Represents the x and y coordinates of key points in the normalized camera coordinate system. This represents the inverse of the intrinsic parameter matrix of the fisheye camera. , The x and y coordinates represent the actual pixel positions of key points; The refraction angle is calculated based on the transformed key point coordinates. The specific algorithm for calculating the refraction angle is as follows: ， in, Indicates the angle of refraction; The incident angle is then optimized based on the refraction angle. The specific algorithm for optimizing the incident angle is as follows: ， in, Indicates the angle of incidence. This represents the optimization solution function. , , , Indicates the distortion coefficient; The distance from the distortion-free projection point to the image center is calculated based on the incident angle. The specific algorithm for calculating the distance from the distortion-free projection point to the image center is as follows: ， in, This represents the distance from the undistorted projection point to the center of the image; The distortion-free pixel coordinates are calculated to obtain the correction key points. The specific algorithm for calculating the distortion-free pixel coordinates is as follows: ， ， in, , Represents the x and y coordinates of a pixel without distortion.

7. The method for determining the landing point of a tennis ball based on the detection of key points on a tennis court according to claim 1, characterized in that, The step of determining the tennis ball's landing point based on the correction key points to obtain the final tennis ball landing point specifically includes: The mapping relationship between the calibration key points and the key points of the actual standard tennis court is constructed. The specific algorithm for the mapping relationship between the calibration key points and the key points of the actual standard tennis court is as follows: ， ， in, , This represents the x and y coordinates of key points on a standard tennis court. , Represents the x and y coordinates of a pixel without distortion. Represents the homography matrix. This represents the homography matrix optimization function. Indicates the number of key points. Indicates the ordinal number of the key point. Represents the function to be minimized. , Indicates the first The x and y coordinates of each key pixel without distortion. , Indicates the first The x and y coordinates of key points on a standard tennis court; Based on the mapping relationship between the corrected key points and the key points of the actual standard tennis court, the final tennis ball landing point is obtained. The specific algorithm for the final tennis ball landing point is as follows: ， in, , The x and y coordinates represent the final landing point of the tennis ball. , The horizontal and vertical coordinates of the image pixels representing the final landing point of the tennis ball.

8. A tennis ball landing point determination system based on the detection of key points on a tennis court, characterized in that, include: A key point generation module is used to acquire an image of a target tennis court and generate a key point heatmap, and to perform foreground enhancement processing on the key point heatmap. The key point heatmap is generated based on a Gaussian kernel function, and the key points include intersections of court landmarks. The camera parameter calibration module is used to calibrate the fisheye camera parameters based on the key point heatmap after foreground enhancement processing. The fisheye camera parameter calibration is used to obtain the internal parameters and distortion coefficients of the fisheye camera. A tennis court coordinate system is constructed based on the target tennis court image. The origin of this coordinate system is any base corner point, and the baseline corresponding to each base corner point is... x In the axial direction, the edge line corresponding to the bottom corner point is y Axial direction; The tennis ball key points in the camera coordinate system are then regenerated. The specific algorithm for generating the tennis ball key points in the camera coordinate system is as follows: ， in, Indicates the camera coordinate system. Represents the coordinate system of the tennis court. Indicates the location of key points in the camera coordinate system. This indicates the spatial location of key points in the tennis court coordinate system. This represents the rotation matrix of the tennis court coordinate system relative to the camera coordinate system. This represents the position vector of the origin of the tennis court coordinate system relative to the origin of the camera coordinate system. Represents the spatial coordinates of key tennis points in the camera coordinate system; Based on the equidistant projection model of a fisheye camera, a relationship function between the incident angle and the projection point is constructed. The specific relationship function between the incident angle and the projection point is as follows: ， ， ， in, This represents the distance from the projection point to the center of the image. f This indicates the focal length of the fisheye camera. Indicates the angle of incidence. , , Represents the spatial position of keypoints in the camera coordinate system using their x, y, and x coordinates. This represents the distance from the undistorted projection point to the center of the image; The distortion correction module is used to perform distortion correction processing based on the calibrated fisheye camera parameters to obtain key correction points. The distortion correction processing is based on the normalized camera coordinate system. The landing point determination module is used to determine the landing point of the tennis ball based on the correction key points to obtain the final landing point of the tennis ball. The tennis ball landing point determination is based on homography matrix mapping.

9. A storage medium, characterized in that, The storage medium stores one or more programs that, when executed by a processor, implement the tennis ball landing point determination method based on the detection of key points on a tennis court as described in any one of claims 1-7.

10. A computer device, characterized in that, The computer device includes a memory and a processor, wherein: The memory is used to store computer programs; When the processor executes the computer program stored in the memory, it implements the tennis ball landing point determination method based on the detection of key points on the tennis court as described in any one of claims 1-7.

Images