Camera self-calibration method and system based on badminton sideline detection

Abstract

This invention belongs to the field of image data processing technology, specifically disclosing a camera self-calibration method and system based on badminton court sideline detection. The method includes the following steps: S1, acquiring court images using a depth camera; S2, detecting the court sidelines in the court images, obtaining the pixel coordinates of the starting and ending points of each sideline segment, as well as the pixel coordinates of some key points, and marking the court sidelines in the image; S3, using the relevant coordinates of each sideline segment and the pixel coordinates of some key points in the court image, combined with the actual distance of the sidelines on a standard court, determining the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image. This invention addresses the problem that existing badminton courts with local viewing angles are not convenient for sideline detection and camera calibration, providing a method that can perform sideline detection and camera calibration when facing a badminton court from any viewing angle, thereby calculating the real-time world coordinates of the badminton shuttlecock.

Description

Technical Field

[0001] This invention belongs to the field of image data processing technology, specifically, it relates to a camera self-calibration method and system based on badminton court sideline detection. Background Technology

[0002] Existing technologies include numerous badminton match analysis systems that utilize cameras to acquire athlete data, perform analysis, and recommend strategies. Based on these systems, before a match, athletes can review and analyze their own and their opponents' past performance, identifying their strengths and weaknesses and developing appropriate training plans and match strategies. After a match, athletes can effectively review the game, summarizing the reasons and patterns for points scored and lost based on specific information such as the type of each shot and its location, and making targeted adjustments in future matches. A crucial step in such a system is the identification and location determination of the badminton shuttlecock. The core prerequisite for this step is the identification of the badminton court and the establishment of a three-dimensional coordinate system, thereby converting the shuttlecock's two-dimensional coordinates to three-dimensional coordinates.

[0003] However, many existing sideline detection and camera calibration solutions are not applicable and perform well when facing badminton courts from any angle. Many existing technologies only work well when facing badminton courts from the perspective of stadium broadcasts, performing camera calibration based on corner detection (sideline detection). While this perspective is common, it is not suitable for ordinary amateur players. For ordinary amateur players, the limitations of available camera mounting positions often limit image acquisition to low-angle or even partial perspectives. These perspectives present the court and shuttlecocks clearly and up close, greatly enhancing convenience, but their partial perspectives also pose challenges for sideline detection and camera calibration. Summary of the Invention

[0004] To address the problem that existing badminton courts with localized views are not convenient for sideline detection and camera calibration, this invention provides a camera self-calibration method based on badminton court sideline detection. This method can perform sideline detection and camera calibration when facing a badminton court from any viewpoint, and then calculate the real-time world coordinates of the badminton shuttlecock.

[0005] To achieve the above-mentioned technical objectives, the technical solution adopted by the present invention is as follows:

[0006] A camera self-calibration method based on badminton court sideline detection includes the following steps:

[0007] S1. Acquire images of the stadium using a depth camera;

[0008] S2. Detect the field sidelines in the field image, obtain the pixel coordinates of the starting and ending points of each sideline segment as well as the pixel coordinates of some key points, and mark the field sidelines in the image.

[0009] S3. By using the relevant coordinates of each sideline and the pixel coordinates of some key points in the court image, and combining them with the actual distance of the sideline on a standard court, determine the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image.

[0010] Furthermore, the detailed steps of step S2 include:

[0011] S201. Traverse each pixel of the image, set a pixel threshold, and if the BGR value of the pixel is higher than the threshold, set it to white; otherwise, set it to black, thereby obtaining the binarization result of the image.

[0012] S202. Use the probabilistic Hough transform to connect the pixels above the threshold into lines.

[0013] S203. Based on the specific location of the court in the detected image, use angle or position information to perform preliminary screening of the edge lines in the set to obtain a preliminary screened binary image.

[0014] S204. After obtaining the set of coordinates of the valid edges after filtering, calculate the coordinates of the intersection points between the two edges and form a rectangle by forming every four intersection points;

[0015] S205. The obtained rectangle is matched with a restoration reference rectangle specified from the multiple rectangles that make up the standard court. Perspective transformation and court restoration are performed on the rectangles obtained in step S204 one by one. The restored court image is compared and scored with the binary image after preliminary screening. The court image with the highest overlap is selected as the restored court image. Then, the pixel coordinates of the starting and ending points of each sideline and the pixel coordinates of some key points are obtained, and the court sidelines are marked in the image.

[0016] Furthermore, in S203, the initial screening method includes manually selecting lines within one or more areas of the image, or initial screening can be performed by setting angle or position parameters.

[0017] Furthermore, in S205, the process of performing perspective transformation and stadium reconstruction on each of the obtained rectangles includes,

[0018] Based on the fact that each rectangle corresponds to the restoration reference rectangle, the perspective relationship between the initially filtered binary image and the reference plane where the restoration reference rectangle is located is obtained using the getPerspectiveTransform function; then, based on the obtained perspective relationship, the PerspectiveTransform function is used to perform perspective transformation on the overall image of the stadium where the restoration reference rectangle is located, mapping the entire stadium onto the plane where the binary image is located, to obtain a stadium image.

[0019] Furthermore, in S205, the images are compared and scored, and the one with the highest overlap is selected as the reconstructed image of the stadium. The process is as follows:

[0020] Iterate through each reproduced court, take a pixel at regular intervals for each sideline of the transformed court, and obtain the coordinates of the selected pixel in the graph;

[0021] Obtain the RGB values ​​of the pixels at the aforementioned coordinates in the binary image after initial screening and sum them;

[0022] If the current RGB sum is greater than the current maximum score, then set the current maximum score to the current RGB sum, and simultaneously set the current court as the selected court.

[0023] Calculate the next reproducible stadium, and continue until all reproducible stadiums have been traversed.

[0024] Furthermore, in step 3, if two sets of mutually orthogonal parallel lines can be found in the badminton court from the current perspective, the vanishing point method of the components is used, combined with the line segment lengths known from the standard court data, to calculate the world coordinate information and the camera's intrinsic parameter matrix K, and then solve for the rotation matrix R and translation vector.

[0025] If two sets of mutually orthogonal parallel lines cannot be found in the badminton court from the current perspective to form two vanishing points, then the N feature points in the world and the N pixels in the image are used;

[0026] The badminton court is treated as a calibration board. The pixel coordinates of some key points are calculated based on the results of the edge detection. The world coordinates of these key points are determined based on the specific dimensions of the badminton court. These key points can include points located on the net and net poles. The camera's intrinsic parameter matrix K is approximately determined by the camera's focal length and image resolution information. Then, the solvePnP algorithm is used to solve for the camera's rotation matrix R and translation vector T.

[0027] Furthermore, the camera calibration results described above are verified by constructing an error loss function, including:

[0028] The projected pixel coordinates of key points are obtained by using the camera's intrinsic and extrinsic parameter matrices. An error loss function is then constructed based on these projected pixel coordinates and pre-determined image pixel coordinates. An optimization algorithm is used to iterate with the goal of minimizing the error loss function, thereby obtaining more accurate camera parameter information.

[0029] Another objective of this invention is to provide a camera self-calibration system based on badminton court sideline detection, comprising a court image acquisition module, a sideline detection module, and a camera calibration module;

[0030] The edge detection module includes an image binarization module, a preliminary screening module for field edge detection, and a perspective transformation module;

[0031] The image binarization module is configured to iterate through each pixel of the image, set a pixel threshold, and set the pixel value to white if the RGB value of the pixel is higher than the threshold, otherwise set it to black to obtain the binarization result of the image.

[0032] The preliminary screening module for detecting the sidelines of the court is set up to first use the HoughLinesP function to connect the pixels that are above the threshold condition to obtain a set of coordinates of all the sidelines.

[0033] Secondly, based on the specific location of the court in the detected images, the sidelines in the set are initially screened using angle or position information;

[0034] The perspective transformation module is configured to, after obtaining the set of coordinates of the filtered valid edges, calculate the coordinates of the intersection points between the two edges and form a rectangle by every four intersection points.

[0035] And, perform perspective transformations and stadium reconstruction on each of the obtained rectangles, including:

[0036] Based on the fact that each rectangle corresponds to the restoration reference rectangle, the perspective relationship between the initially filtered binary image and the reference plane where the restoration reference rectangle is located is obtained using the getPerspectiveTransform function; then, based on the obtained perspective relationship, the PerspectiveTransform function is used to perform perspective transformation on the overall image of the stadium where the restoration reference rectangle is located, mapping the entire stadium onto the plane where the binary image is located, to obtain a stadium image.

[0037] Furthermore, the reconstructed stadium image is compared and scored with the initially screened binary image, and one stadium image is selected as the reconstructed stadium image.

[0038] The camera calibration module is configured to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image by using the relevant coordinates of each sideline and the pixel coordinates of some key points in the field image, combined with the actual distance of each sideline on a standard field.

[0039] Furthermore, the camera calibration module is configured to, if two sets of mutually orthogonal parallel lines can be found in the badminton court from the current perspective, use the vanishing point method of the components, combined with the line segment lengths known from the standard court data, to calculate the world coordinate information and the camera's intrinsic parameter matrix K, and then solve for the rotation matrix R and translation vector T.

[0040] If two sets of mutually orthogonal parallel lines cannot be found in the badminton court from the current perspective to form two vanishing points, then the N feature points in the world and the N pixels in the image are used;

[0041] The badminton court is treated as a calibration board. The pixel coordinates of some key points are calculated based on the results of the edge detection. The world coordinates of these key points are determined based on the specific dimensions of the badminton court. These key points can include points located on the net and net poles. The camera's intrinsic parameter matrix K is approximately determined by the camera's focal length and image resolution information. Then, the solvePnP algorithm is used to solve for the camera's rotation matrix R and translation vector T.

[0042] Furthermore, the perspective transformation module is also coupled with a scoring comparison module, which is configured to...

[0043] Iterate through each reproduced court, take a pixel at regular intervals for each sideline of the transformed court, and obtain the coordinates of the selected pixel in the graph;

[0044] Obtain the RGB values ​​of the pixels at the aforementioned coordinates in the binary image after initial screening and sum them;

[0045] If the current RGB sum is greater than the current maximum score, then set the current maximum score to the current RGB sum, and simultaneously set the current court as the selected court.

[0046] Calculate the next reproducible stadium, and continue until all reproducible stadiums have been traversed.

[0047] Iterate through each reproduced court, take a pixel at regular intervals for each sideline of the transformed court, and obtain the coordinates of the selected pixel in the graph;

[0048] Obtain the RGB values ​​of the pixels at the aforementioned coordinates in the binary image after initial screening and sum them;

[0049] If the current RGB sum is greater than the current maximum score, then set the current maximum score to the current RGB sum, and simultaneously set the current court as the selected court.

[0050] Compared with the prior art, the present invention has the following advantages:

[0051] By combining the sideline detection of the court with camera self-calibration, the results of the sideline detection are combined with a lot of known information about a standard badminton court to locate the court coordinates and construct a world coordinate system.

[0052] Thanks to the fact that multiple sets of mutually orthogonal parallel lines can be easily obtained from the sideline detection results of the court, theoretically, the sidelines of badminton courts can be detected from any angle, without being limited to a specific angle. As long as a part of the court can be seen, the sideline detection and subsequent camera calibration can be achieved. Attached Figure Description

[0053] Figure 1 This is an overall flowchart of the camera self-calibration method based on badminton court sideline detection in an embodiment of the present invention;

[0054] Figure 2 This is a schematic diagram illustrating the camera self-calibration method based on badminton court sideline detection in an embodiment of the present invention.

[0055] Figure 3 This is a schematic diagram showing the perspectives of the camera being placed in the middle and facing sideways and frontally, according to an embodiment of the present invention.

[0056] Figure 4 This is a schematic diagram showing the viewpoint when the camera is placed at the four corners in an embodiment of the present invention;

[0057] Figure 5 This is a schematic diagram illustrating the implementation process of projection perspective transformation in an embodiment of the present invention;

[0058] Figure 6 This is a schematic diagram illustrating the solution of the intrinsic parameter matrix in an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the edge detection results in an embodiment of the present invention;

[0060] Figure 8 This is a schematic diagram of the camera calibration results in an embodiment of the present invention;

[0061] Figure 9 This is a block diagram of the overall structure of the camera self-calibration system based on badminton court sideline detection in an embodiment of the present invention. Detailed Implementation

[0062] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to embodiments and accompanying drawings. The content mentioned in the embodiments is not intended to limit the present invention.

[0063] like Figure 1 As shown, this embodiment provides a camera self-calibration method based on badminton court sideline detection, including the following steps:

[0064] S1. Acquire court images using a depth camera; Determine the acquisition point and orientation of the camera for acquiring court images based on the user's needs for capturing the court's field of view; Considering that cameras in daily badminton match analysis systems can usually only be deployed at lower shooting angles, rather than at the ceiling deployment position of the stadium broadcast view, this example provides two placement schemes: center placement and four corner placement.

[0065] For the center-placement scheme, where the camera is deployed near the pole, it can be further divided into two angles—such as Figure 3 The lateral orientation shown in section (a) and as shown in the figure Figure 3 The frontal orientation shown in section (b) is as follows. The lateral orientation only allows observation of one half of the field, while the frontal orientation allows a barely complete view of the field, but there are blind spots and obstructions in some areas.

[0066] For the four-corner placement scheme, where cameras are deployed at the four corners of the court, such as... Figure 4 As shown, this deployment scheme can observe the entire field, but the images of the far half of the field are relatively blurry, which will affect the detection effect to some extent.

[0067] In addition, the half-court detection can output 8 sidelines, such as Figure 3 As shown in section (a), lines numbered 1-3 are parallel lines, and lines numbered 4-8 are vertical lines; the full-field detection can output 12 edge lines, such as... Figure 3 Part (a) and Figure 4 As shown, lines numbered 1-6 are parallel lines, and lines numbered 7-12 are vertical lines.

[0068] S2. Detect the sidelines of the court, obtain the pixel coordinates of the starting and ending points of each sideline segment as well as the pixel coordinates of some key points, and mark the court sidelines in the image. This step can be completed by the sideline detection module.

[0069] like Figure 2 As shown, the detailed steps of step S2 include:

[0070] S201. Traverse each pixel of the image, set a pixel threshold, and if the RGB value of the pixel is higher than the threshold, set it to white; otherwise, set it to black, thereby obtaining the binarization result of the image.

[0071] S202. Use the HoughLinesP function in the OpenCV library to connect the pixels above the threshold to form a line, and obtain a set of coordinates of all the edges. This function uses probabilistic Huffman transform to find the straight line from the binary image.

[0072] S203. Based on the specific location of the court in the detected image, use angle or position information to perform preliminary screening of the edge lines in the set, obtaining a preliminary screened binary image. Preliminary screening methods include manually selecting lines within one or more regions of the image, or setting angle or position parameters for preliminary selection. The goal of preliminary selection is to include as few straight lines in the image as possible that do not correspond to the court's edge lines. Figure 2 As can be seen intuitively, many of the lines found by using the probability Huffman transform do not correspond to the sidelines of the field. By removing as many of these lines as possible through preliminary screening, the workload of subsequent steps can be reduced and computational efficiency improved.

[0073] S204. After obtaining the set of coordinates of the valid edges after filtering, calculate the coordinates of the intersection points between the two edges and form a rectangle by forming every four intersection points;

[0074] S205. Perform perspective transformation and stadium reconstruction on each rectangle obtained in step S204, and compare and score the reconstructed stadium with the initially screened binary image. Select one stadium image as the reconstructed stadium image, as follows:

[0075] Since the actual dimensions of each rectangle on a standard court, especially the lengths of each sideline and their relative positions, are known; therefore, as Figure 5 As shown, the rectangle marked by the red box in the figure is designated as the restoration reference rectangle beforehand, but the usable restoration reference rectangle is not limited to this, hence the following... Figure 5 As shown on the right, the positions of each side of the complete court relative to the sides of the rectangle are known; based on this, as... Figure 5 As shown on the left, firstly, the rectangle obtained in step S204 is matched with the restored reference rectangle. Based on this correspondence, the perspective relationship function between the binary image and the reference plane where the restored reference rectangle is located is obtained using the getPerspectiveTransform function. Then, using the PerspectiveTransform function, based on the obtained perspective relationship function, the court is mapped from the reference plane to the plane corresponding to the original binary image of each rectangle through perspective transformation, thus completing the restoration of the court in the plane corresponding to the binary image. The getPerspectiveTransform and PerspectiveTransform functions are also from the OpenCV library.

[0076] If one rectangle corresponds to one court, then multiple court coordinates can be obtained. It's not difficult to see that if the rectangle used for perspective function calculations does not actually correspond to... Figure 5 If the restored baseline rectangle shown on the left is not accurate, then the restored stadium will certainly not be the correct stadium. Therefore, the stadium obtained by perspective transformation is compared and scored with the initially filtered binary image to select the required stadium. The specific scoring and filtering method is as follows:

[0077] Iterate through each reproduced court, take a pixel at regular intervals for each sideline of the transformed court, and obtain the coordinates of the selected pixel in the graph;

[0078] Obtain the RGB values ​​of the pixels at the aforementioned coordinates in the binary image after initial screening and sum them;

[0079] If the current RGB sum is greater than the current maximum score, then set the current maximum score to the current RGB sum, and simultaneously set the current court as the selected court.

[0080] Calculate the next reproducible stadium, and continue until all reproducible stadiums have been traversed.

[0081] The principle behind the above method is as follows: considering that the RGB value of black pixels is 0, while that of white is 255, and that the binary image after initial filtering already contains most of the information of the complete stadium; therefore, if the reconstructed stadium has a high degree of overlap with the binary image, the RGB value of the pixels selected from the reconstructed stadium at the same position in the binary image should still be 255. Thus, a higher total score indicates that more points still have a value of 255, meaning a higher degree of overlap. It is worth noting that the calculated RGB sum has a maximum value, meaning that the RGB value at all coordinate points in the binary image is 255. Furthermore, the binary image contains multiple lines corresponding to the same stadium sideline, resulting in multiple rectangles whose reconstructed stadiums all obtain the same RGB sum value in the scoring and filtering process, or even all have the maximum RGB sum value. The above method ensures that only one stadium is output.

[0082] Since the binary image is derived from the original image, with the selection of the court, the pixel coordinates of the starting and ending points of each sideline can be obtained, and the court sidelines can be marked in the original image.

[0083] S3. By using the relevant coordinates of each sideline and the pixel coordinates of key points on the court, and combining them with the actual distance of each sideline on the court, determine the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image. This step can be simply referred to as camera self-calibration, which can be completed by a camera calibration module.

[0084] In this example, camera self-calibration is achieved based on the vanishing point in the image. The vanishing point is the point where two parallel lines in the real world intersect at a distance in an image after perspective transformation. The vanishing point has many useful properties, which can often be used to solve for relevant camera parameters, including:

[0085] (1) Calculate the focal length f and the intrinsic parameter matrix K

[0086] Based on the original image, mark the sidelines of the court and find... Figure 6 The two sets of mutually orthogonal parallel lines shown are used as the two axes of the world coordinate system, with the corresponding unit vectors of the two vanishing points u and v formed by them. The third axis is then determined through the pairwise orthogonal coordinate axis relationships. Figure 6 In the image plane, let the principal point be the center Oi, and let L be the line connecting the principal point and the two vanishing points u and v. uv Construct a triangle ΔuvOi, with the camera projection centers Oc and Oi on L. uv Construct a right triangle ΔpOiOc from the projection point p on the image. Use this triangle to calculate the distance OiOc between the camera projection center Oc and the image center Oi, which is the focal length f. This allows us to approximate the camera's intrinsic parameter matrix.

[0087]

[0088] Where W and H are the width and height of the image, respectively.

[0089] (2) Calculate the rotation matrix R

[0090] First, find two sets of mutually orthogonal parallel lines in the image. The corresponding unit vectors of the two vanishing points u and v formed by these lines can be used as the two axes of the world coordinate system. The third axis can be calculated through the pairwise orthogonal coordinate axis relationships. In this way, the corresponding unit vectors of the three potential vanishing points in the camera coordinate system can be obtained. Furthermore, due to the properties of vanishing points, the coordinate transformation of the vanishing points is independent of the translation vector and only depends on the rotation matrix. Therefore, given that the unit vectors of the three vanishing points in the world coordinate system correspond to the X-axis, Y-axis, and Z-axis respectively (1,0,0), (0,1,0), and (0,0,1), the camera's rotation matrix R can be calculated.

[0091] (3) Calculate the translation vector T:

[0092] Suppose there is a line segment ab of known length in the scene. ab is one of the parallel lines forming a vanishing point, and its endpoint a is located at the origin of the world coordinate system. The vanishing point is v. Without loss of generality, the center of the world can be chosen at any point in the scene. Since the rotation matrix R is known, the line segment ab can be aligned with its image in the camera coordinate system. Assuming O is the camera optical center, it is easy to see that the vector Oa is the translation vector from the camera coordinate system to the world coordinate system, therefore T = -Oa. Using the properties of vanishing points, we know that the line connecting the camera optical center and the vanishing point is parallel to the spatial parallel line forming the vanishing point, i.e., Ov / / ab. Based on this, we can construct similar triangles to solve for the length of vector Oa, and thus obtain the translation vector T.

[0093] For example Figure 7 Taking the scenario shown as an example, the coordinates of the starting and ending points of the 8 sidelines in the half of the court can be obtained, which are:

[0094] L1: (88.9851, 1625.68), (2299.79, 1299.7),

[0095] L2: (167.401, 1703.86), (2477.88, 1324.32),

[0096] L3: (i179.81, 2713.16), (3910.02, 1522.34),

[0097] L4: (88.9851, 1625.68), (1491.6, 3024),

[0098] L5: (355.426, 1586.39), (2430.92, 3024),

[0099] L6: (1477.95, 1420.88), (3174.77, 1843.03),

[0100] L7: (2198.58, 1314.62), (4032, 1586.76),

[0101] L8: (2299.79, 1299.7), (4032, 1539.21);

[0102] The coordinates of the two vanishing points VP1 and VP2 used for camera calibration are as follows:

[0103] VP1:(5503.57, 827.295),

[0104] VP2: (-646.656, 892.294);

[0105] The coordinates of two key points a and b, whose actual distances are known based on standard court dimensions, are as follows:

[0106] a:(3174.77, 1843.03),

[0107] b:(3830.8, 1556.89).

[0108] Therefore, it is possible to construct as follows: Figure 8 The world coordinate system shown, without using a calibration board, estimates the camera's focal length (focal), rotation matrix R, and translation vector T using the court sideline information. In this scene, these are respectively:

[0109] focal = 2976.8915649028936

[0110]

[0111] T=[1.612394350.46061854.142257].

[0112] Furthermore, if it is difficult to find two sets of mutually orthogonal parallel lines to form two vanishing points in a badminton court from a certain perspective, such as the positive orientation angle of a camera placed in the center, it can be transformed into a PerspecTive-n-PoinT problem for solution. That is, by calculating the projection relationship between N feature points in the world and N pixels in the image, the pose of the camera or object can be obtained. The specific steps are as follows:

[0113] (1) The badminton court is regarded as a calibration board. The pixel coordinate information of some key points is calculated by the results of the sideline detection. The world coordinate information corresponding to these key points is determined by the specific size of the badminton court. These key points may include points located on the net and net poles.

[0114] (2) The intrinsic parameter matrix K of the camera is approximately determined by using the camera's focal length information and image resolution information.

[0115] (3) Based on the image pixel coordinates and world coordinates of the key points determined in the above steps, as well as the camera's intrinsic parameter matrix K, the solvePnP algorithm in the OpenCV library is used to solve for the camera's rotation matrix R and translation vector T.

[0116] Alternatively, the camera calibration results can be verified by constructing an error loss function. Specifically, the projected pixel coordinates of keypoints are obtained using the camera's intrinsic parameter matrix and rotation matrix. An error loss function is then constructed based on these projected pixel coordinates and pre-determined image pixel coordinates. An optimization algorithm, such as Newton's method or BFGS, is used iteratively with the goal of minimizing the error loss function to obtain more accurate camera parameter information.

[0117] The self-calibration methods and their optimizations described above are all existing methods that are well-known to those skilled in the art.

[0118] This embodiment also discloses a camera self-calibration system based on badminton court sideline detection, which is basically as follows: Figure 9 As shown, it includes a stadium image acquisition module, a sideline detection module, and a camera calibration module;

[0119] In this embodiment, the image acquisition module is a depth camera.

[0120] The sideline detection module includes an image binarization module, a preliminary screening module for field sideline detection, a perspective transformation module, and a field output module;

[0121] The image binarization module is configured to iterate through each pixel of the image, set a pixel threshold, and set the pixel value to white if the RGB value of the pixel is higher than the threshold, otherwise set it to black to obtain the binarization result of the image.

[0122] The preliminary screening module for detecting the sidelines of the court is set up to first use the HoughLinesP function to connect the pixels that are above the threshold condition to obtain a set of coordinates of all the sidelines.

[0123] Secondly, based on the specific location of the court in the detected images, the edges in the set are initially filtered using angle or positional information to remove the influence of the surrounding environment. Then, the remaining edges are classified according to their location, preparing for the subsequent rectangle construction step between the edges.

[0124] In practice, when using Hough transform for line detection, parameters need to be adjusted according to the actual situation to obtain better detection results. Noise reduction and filtering operations can also be added to assist in detection and filtering. A selection tool can also be introduced, allowing users to view the edges in the set on a display device and then select to retain or remove lines from one or more areas of the image.

[0125] The perspective transformation module is configured to, after obtaining the set of coordinates of the filtered valid edges, calculate the coordinates of the intersection points between the two edges and form a rectangle by every four intersection points.

[0126] Furthermore, perspective transformations and stadium reconstruction are performed on each of the obtained rectangles, as detailed in this embodiment:

[0127] Based on the fact that each rectangle corresponds to the restoration reference rectangle, the perspective relationship between the initially filtered binary image and the reference plane where the restoration reference rectangle is located is obtained using the getPerspectiveTransform function; then, based on the obtained perspective relationship, the PerspectiveTransform function is used to perform perspective transformation on the overall image of the stadium where the restoration reference rectangle is located, mapping the entire stadium onto the plane where the binary image is located, to obtain a stadium image.

[0128] The perspective transformation module also includes a scoring and comparison module. This module compares and scores the reconstructed stadium image with the initially screened binary image, selecting one stadium image as the reconstructed stadium image. The specific implementation in this embodiment is as follows:

[0129] Iterate through each reproduced court, take a pixel at regular intervals for each sideline of the transformed court, and obtain the coordinates of the selected pixel in the graph;

[0130] Obtain the RGB values ​​of the pixels at the aforementioned coordinates in the binary image after initial screening and sum them;

[0131] If the current RGB sum is greater than the current maximum score, then set the current maximum score to the current RGB sum, and simultaneously set the current court as the selected court.

[0132] Calculate the next reproducible stadium, and continue until all reproducible stadiums have been traversed.

[0133] The stadium output module is configured to overlay the selected stadium onto the original image in the display terminal.

[0134] The camera calibration module is configured to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image by using the relevant coordinates of each sideline and the pixel coordinates of key points in the field, combined with the actual distance of the sideline on the field.

[0135] In this embodiment, the module is configured to, if two sets of mutually orthogonal parallel lines can be found in the badminton court from the current perspective, use the vanishing point method of the components, combined with the line segment lengths known from the standard court data, to calculate the world coordinate information and the camera's intrinsic parameter matrix K, and then solve for the rotation matrix R and translation vector.

[0136] If two sets of mutually orthogonal parallel lines cannot be found in the badminton court from the current perspective to form two vanishing points, then the N feature points in the world and the N pixels in the image are used;

[0137] The badminton court is treated as a calibration board. The pixel coordinates of some key points are calculated based on the results of the edge detection. The world coordinates of these key points are determined based on the specific dimensions of the badminton court. These key points can include points located on the net and net poles. The camera's intrinsic parameter matrix K is approximately determined by the camera's focal length and image resolution information. Then, the solvePnP algorithm is used to solve for the camera's rotation matrix R and translation vector T.

[0138] The camera calibration result verification module is configured to verify the camera calibration results by constructing an error loss function.

[0139] In this embodiment, the module is specifically configured to: obtain the projected pixel coordinates of key points through the camera's intrinsic parameter matrix and rotation matrix, and construct an error loss function based on these projected pixel coordinates and pre-determined image pixel coordinates. With the goal of minimizing the error loss function, an optimization algorithm, such as Newton's method and BFGS, is used for iteration to obtain more accurate camera parameter information.

[0140] The above provides a detailed description of a camera self-calibration system based on badminton court sideline detection provided in this application. The specific embodiments are described only to aid in understanding the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A camera self-calibration method based on badminton court sideline detection, characterized in that, Including the following steps: S1. Acquire images of the stadium using a depth camera; S2. Detect the court sidelines in the court image, obtain the pixel coordinates of the start and end points of each sideline segment, as well as the pixel coordinates of some key points, and mark the court sidelines in the image, including: S201. Traverse each pixel of the image, set a pixel threshold, and if the BGR value of the pixel is higher than the threshold, set it to white; otherwise, set it to black, thereby obtaining the binarization result of the image. S202. Use the probabilistic Hough transform to connect the pixels above the threshold into lines. S203. Based on the specific location of the court in the detected image, use angle or position information to perform preliminary screening of the edge lines in the set to obtain a preliminary screened binary image. S204. After obtaining the set of coordinates of the valid edges after filtering, calculate the coordinates of the intersection points between the two edges and form a rectangle by forming every four intersection points; S205. The obtained rectangle is matched with a restoration reference rectangle specified from the multiple rectangles that make up the standard court. Perspective transformation and court restoration are performed on the rectangles obtained in step S204 one by one. The restored court image is compared and scored with the binary image after preliminary screening. The court image with the highest overlap is selected as the restored court image. Then, the pixel coordinates of the starting and ending points of each sideline and the pixel coordinates of some key points are obtained, and the court sidelines are marked in the image. The process of performing perspective transformation and stadium reconstruction on each of the obtained rectangles includes... Based on the fact that each rectangle corresponds to the restoration reference rectangle, the perspective relationship between the initially filtered binary image and the reference plane where the restoration reference rectangle is located is obtained using the getPerspectiveTransform function; then, based on the obtained perspective relationship, the PerspectiveTransform function is used to perform perspective transformation on the overall image of the stadium where the restoration reference rectangle is located, mapping the entire stadium onto the plane where the binary image is located, to obtain a stadium image. S3. By using the relevant coordinates of each sideline and the pixel coordinates of some key points in the court image, and combining them with the actual distance of each sideline on the standard court, determine the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image; if two sets of mutually orthogonal parallel lines can be found in the badminton court from the current perspective, the vanishing point method is used, combined with the line segment lengths known from the standard court data, to calculate the world coordinate information and the camera's intrinsic parameter matrix K, and then solve for the rotation matrix R and translation vector; If two sets of mutually orthogonal parallel lines cannot be found in the badminton court from the current perspective to form two vanishing points, then the N feature points in the world and the N pixels in the image are used; The badminton court is treated as a calibration board. The pixel coordinates of some key points are calculated based on the results of the edge detection. The world coordinates of these key points are determined based on the specific dimensions of the badminton court. These key points can include points located on the net and net poles. The camera's intrinsic parameter matrix K is approximately determined by the camera's focal length and image resolution information. Then, the solvePnP algorithm is used to solve for the camera's rotation matrix R and translation vector T.

2. The method according to claim 1, characterized in that, In S203, the initial screening methods include manually selecting lines within one or more areas of the image, or initial screening by setting angle and position parameters.

3. The method according to claim 1, characterized in that, In S205, the images are compared and scored, and the one with the highest overlap is selected as the reconstructed image of the stadium. The process is as follows: Iterate through each reproduced court, take a pixel at regular intervals for each sideline of the transformed court, and obtain the coordinates of the selected pixel in the graph; Obtain the RGB values ​​of the pixels at the aforementioned coordinates in the binary image after initial screening and sum them; If the current RGB sum is greater than the current maximum score, then set the current maximum score to the current RGB sum, and simultaneously set the current court as the selected court. Calculate the next reproducible stadium, and continue until all reproducible stadiums have been traversed.

4. The method according to claim 1, characterized in that, This also includes S4 verifying the camera calibration results by constructing an error loss function, including: The projected pixel coordinates of key points are obtained by using the camera's intrinsic and extrinsic parameter matrices. An error loss function is then constructed based on these projected pixel coordinates and pre-determined image pixel coordinates. An optimization algorithm is used to iterate with the goal of minimizing the error loss function, thereby obtaining more accurate camera parameter information.

5. A camera self-calibration system based on badminton court sideline detection, characterized in that, It includes a stadium image acquisition module, a stadium sideline detection module, and a camera calibration module; The edge detection module includes an image binarization module, a preliminary screening module for field edge detection, and a perspective transformation module; The image binarization module is configured to iterate through each pixel of the image, set a pixel threshold, and set the pixel value to white if the RGB value of the pixel is higher than the threshold, otherwise set it to black to obtain the binarization result of the image. The preliminary screening module for detecting the sidelines of the court is set up to first use the HoughLinesP function to connect the pixels that are above the threshold condition to obtain a set of coordinates of all the sidelines. Secondly, based on the specific location of the court in the detected images, the sidelines in the set are initially filtered using angle or position information; The perspective transformation module is configured to, after obtaining the set of coordinates of the filtered valid edges, calculate the coordinates of the intersection points between the two edges and form a rectangle by every four intersection points. And, perform perspective transformations and stadium reconstruction on each of the obtained rectangles, including: Based on the fact that each rectangle corresponds to the restoration reference rectangle, the perspective relationship between the initially filtered binary image and the reference plane where the restoration reference rectangle is located is obtained using the getPerspectiveTransform function; then, based on the obtained perspective relationship, the PerspectiveTransform function is used to perform perspective transformation on the overall image of the stadium where the restoration reference rectangle is located, mapping the entire stadium onto the plane where the binary image is located, to obtain a stadium image. Furthermore, the restored stadium is compared and scored with the initially screened binary image, and a stadium image is selected as the restored stadium image. Then, the pixel coordinates of the starting and ending points of each sideline and the pixel coordinates of some key points are obtained, and the stadium sidelines are marked in the image. The camera calibration module is configured to determine the 3D geometric position of a point on the surface of a spatial object and its corresponding point in the image by using the relevant coordinates of each sideline and the pixel coordinates of some key points in the court image, combined with the actual distance of each sideline on the standard court. If two sets of mutually orthogonal parallel lines can be found in the badminton court from the current perspective, the vanishing point method is used, combined with the line segment lengths known from the standard court data, to calculate the world coordinate information and the camera's intrinsic parameter matrix K, and then solve for the rotation matrix R and translation vector T. If two sets of mutually orthogonal parallel lines cannot be found in the badminton court from the current perspective to form two vanishing points, then the N feature points in the world and the N pixels in the image are used; The badminton court is treated as a calibration board. The pixel coordinates of some key points are calculated based on the results of the edge detection. The world coordinates of these key points are determined based on the specific dimensions of the badminton court. These key points can include points located on the net and net poles. The camera's intrinsic parameter matrix K is approximately determined by the camera's focal length and image resolution information. Then, the solvePnP algorithm is used to solve for the camera's rotation matrix R and translation vector T.

6. The system according to claim 5, characterized in that, The phase perspective transformation module also includes a scoring comparison module, which is configured to... Iterate through each reproduced court, take a pixel at regular intervals for each sideline of the transformed court, and obtain the coordinates of the selected pixel in the graph; Obtain the RGB values ​​of the pixels at the aforementioned coordinates in the binary image after initial screening and sum them; If the current RGB sum is greater than the current maximum score, then set the current maximum score to the current RGB sum, and simultaneously set the current court as the selected court. Calculate the next reproducible stadium, and continue until all reproducible stadiums have been traversed.

Images