A multi-view camera calibration method for sports scenes

Abstract

The present invention discloses a multi-view camera calibration method for sports scenes, comprising the following steps: building a multi-view camera system to effectively capture key points of the human body and field in the sports scene; using a pinhole camera model and parametrically representing the camera; respectively providing methods for representing two-dimensional key points and three-dimensional key points of the field and the human body; constructing a key point extraction network, supervising the training of the key point extraction network until convergence; using the trained key point extraction network to detect two-dimensional field key points and two-dimensional human body key points in the video; using static field semantics to initialize camera internal and external parameters, combining static field semantics and dynamic human body semantics as constraints, and performing bundle adjustment optimization on the initialized camera internal and external parameters and human body motion to obtain accurate camera internal and external parameters. The present invention uses field key points to initialize the camera's internal and external parameters, and has better stability and accuracy.

Description

Technical Field

[0001] The present invention relates to the fields of camera calibration and computer vision, and in particular to a multi-view camera calibration method for sports scenes. Background Art

[0002] Multi-view camera calibration is an important preliminary work for many applications such as 3D human reconstruction, motion capture and human-computer interaction. With the continuous development of related technologies such as artificial intelligence, the demand for motion capture and other technologies in sports training and live broadcast of sports events is increasing. As an indispensable prerequisite for these specific applications, multi-view camera calibration in sports scenes has become a very important issue. At present, the technology focusing on multi-view camera calibration mainly faces two problems. First, traditional camera calibration methods rely on specific high-precision calibration tools such as checkerboards, and the equipment is complex. On the other hand, when these methods are applied to large multi-view scenes, the calibration process will be complicated and slow. Therefore, using the scene and human semantic information detected from RGB images to construct the constraints of camera internal and external parameters in multi-view sports scenes has become a challenging and urgent problem to be solved. Summary of the invention

[0003] Purpose of the invention: The present invention proposes a multi-view camera calibration method for sports scenes, which simultaneously estimates the field key points and human body key points in RGB images by training a codec network; obtains the initial intrinsic and extrinsic parameters of the camera by using the field key points and a predefined three-dimensional sports field model; performs bundle adjustment optimization using two-dimensional static field key points, dynamic human body key points and their corresponding three-dimensional points to obtain accurate camera internal and external parameters; and reduces the running time and improves the robustness of calibration by constructing a progressive optimization framework.

[0004] Technical solution: The multi-view camera calibration method for sports scenes described in the present invention comprises the following steps:

[0005] Step 1: Build a multi-view camera system, use hardware triggering for synchronous acquisition, and effectively capture key points of the human body and the field in the sports scene;

[0006] Step 2, adopting a pinhole camera model and parameterizing the camera;

[0007] Step 3, providing a representation method of two-dimensional key points and three-dimensional key points of the human body;

[0008] The coordinates of the two-dimensional key points of the human body are expressed as The coordinates of the three-dimensional key points of the human body are expressed as Where F is the number of video frames and M is the number of human key points.

[0009] Step 4, providing a representation of the key points of the venue; pre-defining a three-dimensional sports venue model, providing a representation method of the two-dimensional key points of the venue and the corresponding three-dimensional key points on the pre-defined sports venue model;

[0010] The coordinates of the two-dimensional key points of the site are expressed as The corresponding three-dimensional key point coordinates on the predefined sports venue model are expressed as Where N is the number of key points in the venue.

[0011] Step 5: Build a key point extraction network to estimate the site key points and human key points from the RGB image at the same time, and train the key point extraction network through true value supervision until convergence;

[0012] Step 6, using the trained key point extraction network to detect the two-dimensional field key points and the two-dimensional human key points in the sports video;

[0013] Step 7, using the obtained two-dimensional field key points and the corresponding three-dimensional points on the predefined three-dimensional sports field model to construct a three-dimensional-two-dimensional correspondence relationship, and then calculate the homography matrix of the camera, and obtain the initial intrinsic and extrinsic parameters of the camera through the conversion relationship;

[0014] Step 8: Combining static site semantics and dynamic human semantics as constraints, perform bundle adjustment optimization on the initialized camera internal and external parameters and human motion.

[0015] Furthermore, in step 2, a pinhole camera model is adopted and the camera is parameterized, specifically:

[0016] Assumptions: 1) The camera distortion is zero 2) The focal lengths of the camera X and Y axes are the same 3) The camera's principal point is located at the center of the image. Therefore, the camera matrix can be expressed as K[RT], where K is the camera's intrinsic parameter matrix, is the rotation matrix and translation matrix of the camera. According to the assumption, the intrinsic parameter K of the camera is:

[0017]

[0018] Where f is the focal length, w and h are the width and height of the image.

[0019] Furthermore, in step 5, a key point extraction network is constructed to simultaneously estimate the site key points and the human key points from the RGB image, and the key point extraction network is trained through true value supervision until convergence. Specifically, the following steps are included:

[0020] Step 5.1: Build a key point extraction network

[0021] The key point extraction network includes an encoder, a first decoder and a second decoder, and the first decoder and the second decoder are used to obtain key point heat maps of the site and the human body respectively.

[0022] The encoder includes a convolutional layer formed by three convolutional cascades, three Resnet modules, a convolutional layer, and an activation function; the first decoder and the second decoder have the same structure, both including a first basic module, a second basic module, a convolutional layer, an activation function, and upsampling connected in sequence; the outputs of the three Resnet modules in the encoder are connected to the first basic module, the second basic module, and the input end of the convolutional layer of the decoder. The first basic module and the second basic module have the same structure, both including a Resnet module, a convolutional layer, an activation function, and upsampling connected in sequence.

[0023] Step 5.2, supervised training of the key point extraction network;

[0024] The L2 loss of the corresponding true value is used to constrain the scene key points and human body key points respectively. The formula is as follows:

[0025]

[0026] Where N s and N h are the number of venue and human body heatmaps, and is the estimated heat map of the site and human body, M s and M h is the corresponding true value heat map obtained from the dataset. ‖·‖ represents the second norm, L scene and L human Represent the L2 loss between the scene key points and human key points and the corresponding true values ​​respectively.

[0027] Finally, the training loss of the key point extraction network is:

[0028] L=ω1L scene +ω2L human

[0029] Where ω1 and ω2 are weight coefficients.

[0030] Furthermore, in step 8, a progressive optimization framework is constructed to segment the video into N batches, each containing the same number of frames. One batch is optimized at a time, and the next batch is added to the optimization target after the optimal solution is obtained. This process is repeated until all frames are optimized. The camera's intrinsic and extrinsic parameters are calibrated by combining static scene semantics and dynamic human semantics.

[0031] For a video with V viewpoints and F frames, the total optimization variables are {(θ,r,t),α 1:V}.in They are the motion parameters, global rotation and translation parameters of the 3D human body model. It is the external parameter of the camera, which includes the three-dimensional rotation vector and the three-dimensional translation parameter.

[0032] Construct the following objective function:

[0033]

[0034] in represents the site term constraint, Represents human constraints, which are:

[0035]

[0036] where ρ is the Geman-McClure term, p h is the coordinates of the two-dimensional human body key points, J h is the coordinate of the key points of the 3D human body, p s is the estimated coordinates of the two-dimensional site key points, J s It is the coordinates of the key points of the three-dimensional venue obtained through the static sports venue model.

[0037] In addition, the regularization term is:

[0038]

[0039] Furthermore, for a basketball court, the key points of the court include the intersection of the center circle and the center line of the basketball court, the two end points of the end lines, the four vertices of the restricted area, the intersection of the three-point shooting area boundary line and the end lines, and two points on the three-point shooting area boundary line excluding the end points, totaling 22 points as the key points of the court.

[0040] Beneficial effects: Compared with the prior art, the beneficial effects of the present invention are: 1. It does not rely on specific camera calibration tools, the process is simple, and it can be applied to multi-view large-scale scenes; 2. The site key points are used to initialize the camera's intrinsic and extrinsic parameters. Compared with initialization using dynamic human motion, the method proposed in the present invention has better stability and accuracy; 3. Using a progressive optimization framework, the running speed is faster and the results obtained are more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of the method of the present invention;

[0042] Figure 2 It is a schematic diagram of the camera acquisition system;

[0043] Figure 3 It is a schematic diagram of a predefined sports venue model;

[0044] Figure 4 It is the key point extraction network structure diagram;

[0045] Figure 5 It is the extracted site key point and human body key point map;

[0046] Figure 6 is the reprojection result image;

[0047] Figure 7 This is the camera calibration result image. DETAILED DESCRIPTION

[0048] The present invention is further described in detail below with reference to the accompanying drawings. The implementation method takes a basketball court as an example, and is also applicable to other sports scenes.

[0049] The present invention provides a multi-view camera calibration method for sports scenes, such as Figure 1 As shown, the following steps are included:

[0050] Step 1, build a multi-view camera system: place the camera position reasonably, use hardware triggering for synchronous acquisition, and effectively capture the key points of the human body and the field in the sports scene while meeting the requirements of event shooting.

[0051] like Figure 2 As shown in the figure, six FLIR Blackfly industrial cameras are placed around the basketball scene and fixed with tripods. The specific camera positions are at the four corners of the field and at both ends of the field center line, so that the placed cameras can effectively collect human body movements and as many key points of the field as possible in the entire sports scene. Hardware triggering is used for synchronous acquisition, and the shooting content is stored in the solid-state hard disk through the USB3.0 and PCIE interfaces.

[0052] The basketball court has some unique properties: there are static marking lines and marking circles on the ground, and N representative points on the sports ground are selected as the key points of the field, such as Figure 3 As shown, 22 representative points on the field are reasonably selected as the field key points. The field key points include the intersection of the center circle and the center line, the two end points of the end line, the four vertices of the restricted area, the intersection of the three-point shooting area boundary line and the end line, and the two points on the three-point shooting area boundary line except the end points. A total of 22 points are used as field key points.

[0053] The key points of the human body include nose, left and right eyes, left and right ears, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right knees and left and right ankles, which are 17 in total.

[0054] Step 2, adopt the pinhole camera model and parameterize the camera according to the assumptions;

[0055] Assumptions: 1) The camera distortion is zero 2) The focal lengths of the camera X and Y axes are the same 3) The camera's principal point is located at the center of the image. Therefore, the camera matrix can be expressed as K[RT], where K is the camera's intrinsic parameter matrix, is the rotation matrix and translation matrix of the camera. According to the assumption, the intrinsic parameter K of the camera is:

[0056]

[0057] Where f is the focal length, w and h are the width and height of the image.

[0058] Step 3, providing a representation method of two-dimensional key points and three-dimensional key points of the human body;

[0059] For a human image sequence containing 256 frames with 17 key points, the coordinates of the two-dimensional key points of the human body can be expressed as The coordinates of the three-dimensional key points of the human body can be expressed as

[0060] Step 4: Give the representation of the key points of the site:

[0061] Predefine a three-dimensional sports venue model, and provide a method for representing two-dimensional key points of the venue and corresponding three-dimensional key points on the predefined sports venue model;

[0062] For an image sequence containing 256 frames with 22 venue key points, the coordinates of the two-dimensional venue key points can be expressed as The corresponding three-dimensional key point coordinates on the predefined sports venue model can be expressed as

[0063] Step 5: construct a key point extraction network, estimate the site key points and human key points from the RGB image at the same time, and train the key point extraction network through true value supervision until convergence. Specifically, the following steps are included:

[0064] Step 5.1: Build a key point extraction network

[0065] The court semantics and human semantics contained in the basketball scene provide rich key point information, which can be used to help estimate camera parameters without relying on additional camera calibration tools. Train a neural network to simultaneously estimate these court key points and human key points from the RGB images collected by the camera in step 1.

[0066] The key point extraction network includes an encoder, a first decoder and a second decoder, and the first decoder and the second decoder are used to obtain key point heat maps of the site and the human body respectively.

[0067] like Figure 4As shown, the encoder includes a convolutional layer formed by three layers of convolution cascade, three layers of Resnet modules, a convolutional layer, and a layer of activation function; the first decoder and the second decoder have the same structure, both including a first basic module, a second basic module, a convolutional layer, a layer of activation function, and upsampling connected in sequence; the outputs of the three layers of Resnet modules in the encoder are connected to the first basic module, the second basic module, and the input end of the convolutional layer of the decoder. The first basic module and the second basic module have the same structure, both including a Resnet module, a convolutional layer, an activation function, and upsampling connected in sequence.

[0068] Step 5.2, supervised training of the key point extraction network;

[0069] The L2 loss of the corresponding true value is used to constrain the scene key points and human body key points respectively. The formula is as follows:

[0070]

[0071] Where N s =22 and N h =17 are the number of venue and human body heat maps, and is the estimated heat map of the site and human body, M s and M h is the corresponding true value heat map obtained from the dataset. ‖·‖ represents the second norm, L scene and L human Represent the L2 loss between the scene key points and human key points and the corresponding true values ​​respectively.

[0072] Finally, the training loss of the key point extraction network is:

[0073] L=ω1L scene +ω2L human

[0074] Among them, ω1=1, ω2=0.5 are weights. Since static venue semantics are used to initialize camera internal and external parameters in subsequent steps, a larger weight is given to the venue key point loss term when designing the training loss of the key point extraction network to obtain more accurate venue key point coordinates.

[0075] Step 6: Input a video of human movement in a sports scene, such as Figure 5 As shown, the trained key point extraction network is used to detect the two-dimensional site key points and the two-dimensional human key points.

[0076] Step 7, camera parameter initialization: Use the 2D field key points obtained in step 6 and the corresponding 3D points on the predefined 3D sports field model to build a 3D-2D correspondence relationship, and then calculate the camera's homography matrix, and obtain the camera's initial intrinsic and extrinsic parameters through the transformation relationship.

[0077] Static field key points are easier to detect than dynamic human key points and the detection results are more accurate. Therefore, using field key points to initialize the camera's internal and external parameters can obtain more reliable initial results. Since different key points are stored in the heat map of different channels, the correspondence between 3D and 2D can be constructed using 2D key points and the corresponding 3D points on the predefined sports field model. If there are more than four pairs of corresponding points, the camera homography matrix can be obtained through the correspondence. The initial camera internal and external parameters are obtained based on the conversion relationship between the homography matrix and the camera's internal and external parameters.

[0078] Using the initial internal and external parameter values ​​of the camera, the two-dimensional human body key points are back-projected to obtain the initial values ​​of the three-dimensional human body key points.

[0079] Step 8, camera parameter optimization: construct a progressive optimization structure framework, combine static site semantics and dynamic human semantics as constraints, and perform bundle adjustment optimization on the initialized camera internal and external parameters and human motion.

[0080] Since the key points of the site can only provide sparse constraints for estimating the homography matrix, the initial internal and external parameters of the camera are not accurate and need to be optimized later. The 2D key point coordinates of the site and the human body and the corresponding 3D point coordinates provide a dense correspondence between 2D and 3D, which can be used to help camera calibration. And since optimizing the entire sequence at one time is a highly non-convex optimization, it is easy to fall into a local minimum. In order to improve the optimization speed and obtain more stable and reliable results, a progressive optimization framework is adopted. X = 16 frames are optimized as a batch, one batch at a time, and the last 16 frames are added to the optimization target after the optimal solution is obtained. Repeat this process until all frames are optimized. This progressive structure has two advantages. First, there are fewer variables to optimize each time, so the optimization speed is faster. Second, the progressive optimization uses a step-by-step approximation method to obtain the optimal value, which is more stable and can obtain more reliable results.

[0081] Combining static scene semantics and dynamic human semantics to help calibrate the camera's internal and external parameters. For a video with 256 frames from six perspectives, the total optimization variables are {(θ, r, t), α 1:6}.in They are the motion parameters, global rotation and translation parameters of the 3D human body model. It is the external parameter of the camera, which includes the three-dimensional rotation vector and the three-dimensional translation parameter.

[0082] Construct the following objective function:

[0083]

[0084] in represents the site term constraint, Represents human constraints, which are:

[0085]

[0086] where ρ is the Geman-McClure term, p h is the coordinates of the two-dimensional human body key points, J h is the coordinate of the key points of the 3D human body, p s is the estimated coordinates of the two-dimensional site key points, J s It is the coordinates of the key points of the three-dimensional venue obtained through the static sports venue model.

[0087] In addition, the regularization term is:

[0088]

[0089] like Figure 6 As shown, the camera internal and external parameters obtained by the method proposed in the present invention are used to reproject the key points on the three-dimensional sports venue model, and the reprojection result has a high degree of overlap with the key points of the venue in the figure. Figure 7 As shown, the method proposed in the present invention can obtain accurate camera calibration results.

Claims

1. A multi-view camera calibration method for sports scenes, characterized in that: The following steps are involved: Step 1: Build a multi-view camera system, use hardware triggering to perform synchronous acquisition, and shoot key points of the human body and the field in the sports scene; Step 2, adopting a pinhole camera model and parameterizing the camera; Step 3, providing a representation method of two-dimensional key points and three-dimensional key points of the human body; The coordinates of the two-dimensional key points of the human body are expressed as The coordinates of the three-dimensional key points of the human body are expressed as Where F is the number of frames in the video, and M is the number of key points of the human body; Step 4: Give the representation of the key points of the site: Predefine a three-dimensional sports venue model, and provide a method for representing two-dimensional key points of the venue and corresponding three-dimensional key points on the predefined sports venue model; The coordinates of the two-dimensional key points of the site are expressed as The corresponding three-dimensional key point coordinates on the predefined sports venue model are expressed as Where N is the number of key points on the site; Step 5: construct a key point extraction network and train the key point extraction network through true value supervision until convergence; Step 6, using the trained key point extraction network to detect the two-dimensional field key points and the two-dimensional human key points in the sports video; Step 7, using the obtained two-dimensional field key points and the corresponding three-dimensional points on the predefined three-dimensional sports field model to construct a three-dimensional-two-dimensional correspondence relationship, and then calculate the homography matrix of the camera, and obtain the initial intrinsic and extrinsic parameters of the camera through the conversion relationship; Step 8: Combining static site semantics and dynamic human semantics as constraints, perform bundle adjustment optimization on the initialized camera internal and external parameters and human motion.

2. The multi-view camera calibration method for sports scenes according to claim 1, characterized in that: In step 2, a pinhole camera model is used and the camera is parameterized as follows: Assumptions: 1) The camera distortion is zero 2) The focal lengths of the camera X and Y axes are the same 3) The principal point of the camera is located at the center of the image; therefore, the camera matrix is ​​represented as K[RT], where K is the intrinsic parameter matrix of the camera, is the rotation matrix and translation matrix of the camera; according to the assumption, the intrinsic parameter K of the camera is: Where f is the focal length, w and h are the width and height of the image.

3. The multi-view camera calibration method for sports scenes according to claim 1, characterized in that: In step 5, a key point extraction network is constructed and trained through true value supervision until convergence; specifically, the following steps are included: Step 5.1: Build a key point extraction network The key point extraction network includes an encoder, a first decoder and a second decoder, wherein the first decoder and the second decoder are used to obtain key point heat maps of the site and the human body respectively; The encoder includes a convolution layer formed by cascading three convolutions, three Resnet modules, a convolution layer, and an activation function; the first decoder and the second decoder have the same structure, both including a first basic module, a second basic module, a convolution layer, an activation function, and upsampling connected in sequence; the outputs of the three Resnet modules in the encoder are connected to the first basic module, the second basic module, and the input end of the convolution layer of the decoder; the first basic module and the second basic module have the same structure, both including a Resnet module, a convolution layer, an activation function, and upsampling connected in sequence; Step 5.2, supervised training of the key point extraction network; The L2 loss of the corresponding true value is used to constrain the scene key points and human body key points respectively. The formula is as follows: Where N s and N h are the number of venue and human body heatmaps, and is the estimated heat map of the site and human body, M s and M h is the corresponding true value heat map obtained from the dataset; ‖·‖ represents the second norm, L scene and L human Respectively represent the L2 loss between the scene key points and human key points and the corresponding true values; Finally, the training loss of the key point extraction network is: L=ω1L scene +ω2L human Where ω1 and ω2 are weight coefficients.

4. The multi-view camera calibration method for sports scenes according to claim 1, characterized in that: In step 8, a progressive optimization structure framework is also constructed to divide the video into N batches, each batch containing the same number of frames. One batch is optimized at a time, and when the optimal solution is obtained, the next batch is added to the optimization target. The process is repeated until all frames are optimized. The camera's intrinsic and extrinsic parameters are calibrated by combining static scene semantics and dynamic human semantics.

5. The multi-view camera calibration method for sports scenes according to claim 1, characterized in that: The specific content of step 8 is: for a video with V perspectives and F frames, the total optimization variables are {(θ, r, t), α 1:V };in They are the motion parameters and global rotation and translation parameters of the 3D human body model; It is the external parameter of the camera, which includes the three-dimensional rotation vector and the three-dimensional translation parameter; The objective function of bundle adjustment optimization is constructed as follows: in represents the site term constraint, Represents human constraints, which are: where ρ is the Geman-McClure term, p h is the coordinates of the two-dimensional human body key points, J h is the coordinate of the key points of the 3D human body, p s is the estimated coordinates of the two-dimensional site key points, J s It is the coordinates of the key points of the three-dimensional field obtained through the static sports field model; In addition, the regularization term is:

6. The multi-view camera calibration method for sports scenes according to claim 1, characterized in that: The key points of the venue include the intersection of the center circle and the center line of the basketball court, the two end points of the end line, the four vertices of the restricted area, the intersection of the three-point shooting area boundary line and the end line, and two points on the three-point shooting area boundary line except the end points, a total of 22 points as the key points of the venue.

Images