Camera internal and external parameter calibration method based on cross-view multi-human semantic matching

Abstract

The application discloses a camera internal and external parameter calibration method based on cross-view multi-human semantic matching, which firstly builds a multi-view camera system and collects multi-human motion videos to obtain multi-human multi-view video sequences; selects a multi-human multi-view video sequence collected, uses an open-source two-dimensional posture estimation method to obtain human two-dimensional joint nodes, uses an open-source three-dimensional posture estimation method to estimate each view separately, and obtains all human three-dimensional joint nodes in a camera coordinate system; then, initial camera internal parameter estimation and initial camera external parameter estimation are carried out respectively; then, a posture-geometry constraint term is constructed, a matching relationship between two-dimensional joint nodes among different views is established; then, three-dimensional joint nodes in a global coordinate system are obtained; finally, human motion and camera parameters are optimized, and more accurate global positions and camera internal and external parameters are obtained. The application can optimize camera parameters and human postures simultaneously, and is more robust to noise input.

Description

Technical Field

[0001] This invention relates to a method for calibrating camera intrinsic and extrinsic parameters based on cross-view multi-human semantic matching, belonging to the fields of computer vision and computer graphics. Background Technology

[0002] Camera calibration is a crucial preliminary step in various applications such as human-computer interaction, sports broadcasting, and virtual reality. With the development of computer vision and artificial intelligence, the market demand for reconstructing multi-person motion from uncalibrated cameras is increasing. Therefore, camera intrinsic and extrinsic parameter calibration, an indispensable prerequisite for these applications, has become a critical issue. Currently, multi-view camera calibration technology faces two main challenges. First, traditional camera calibration methods rely on specific high-precision calibration tools such as checkerboard patterns, resulting in complex equipment. Second, in multi-person scenarios, occlusion between people and missing viewpoints reduce effective information, making it difficult to obtain accurate camera intrinsic and extrinsic parameters from multiple perspectives. Therefore, using human semantic information to initially estimate camera parameters, introducing pose similarity constraints and pose-geometric consistency constraints to match 2D human joints from multiple perspectives, and achieving camera intrinsic and extrinsic parameter calibration based on cross-view multi-human semantic matching, is expected to promote the practical application of current multi-person motion reconstruction technology. Summary of the Invention

[0003] Objective: This invention proposes a camera intrinsic and extrinsic parameter calibration method based on cross-view multi-human semantic matching. The method first proposes an intrinsic parameter estimation approach based on multiple upright human figures from the same viewpoint. It uses initial intrinsic parameters to estimate the 3D joints of the human body from each viewpoint, and then uses these 3D joints from different viewpoints to achieve initial camera extrinsic parameter estimation. Multi-view human body matching is achieved based on pose-geometric consistency constraints. Finally, the camera parameters and human body model are jointly optimized through a reprojection process, thereby achieving accurate camera parameter estimation based on multi-view video sequences.

[0004] Technical solution: The present invention provides a camera intrinsic and extrinsic parameter calibration method based on cross-view multi-human semantic matching, comprising the following steps:

[0005] S1. Build a multi-view camera system and collect multi-person motion videos to obtain a multi-person multi-view video sequence;

[0006] S2. Two-dimensional pose estimation and single-view three-dimensional pose estimation: Select a multi-person, multi-view video sequence acquired in step S1, use an open-source two-dimensional pose estimation method to obtain the two-dimensional joints of the human body, and use an open-source three-dimensional pose estimation method to estimate each viewpoint separately to obtain all the three-dimensional joints of the human body in the camera coordinate system.

[0007] S3. Perform initial camera intrinsic parameter estimation and initial camera extrinsic parameter estimation respectively:

[0008] Initial camera intrinsic parameter estimation: Using the head joints and the midpoints of the two ankle joints obtained in the two-dimensional human body joints in multiple steps S2, a set of parallel lines in three-dimensional space is constructed. The intrinsic parameters are calculated by utilizing the geometric phenomenon that the parallel lines in three-dimensional space will intersect at the vanishing point in the two-dimensional projection space.

[0009] Initial camera extrinsic parameter estimation: First, rigidly align the human 3D joints obtained in step S2 under the camera coordinate system corresponding to different viewpoints to obtain the correspondence, and obtain the camera rotation matrix based on the pose matched by different viewpoints; then use the 2D joints obtained in step S2, the 3D joints corresponding to the 2D joints, the obtained camera rotation matrix, and the camera intrinsic parameters to regress and obtain the camera translation matrix; finally, combine the camera rotation matrix and the camera translation matrix to form the camera extrinsic parameters.

[0010] S4. Using the two-dimensional joints and three-dimensional joints obtained in step S2 and the initial camera extrinsic parameters obtained in step S3, construct the pose-geometric constraint term, calculate the similarity between two-dimensional joints from all viewpoints, and establish the matching relationship between two-dimensional joints from different viewpoints.

[0011] Then, using the initial extrinsic parameters, the matched two-dimensional joints are back-projected back into the three-dimensional space to obtain the three-dimensional joints in the global coordinate system.

[0012] S5. Optimize human motion and camera parameters: Fit the joints of the human model to the 3D joints obtained in S4 as the initial state of the human model, and use the reprojection loss constraint to reconstruct the results, thereby obtaining a more accurate global position and camera intrinsic and extrinsic parameters.

[0013] Furthermore, the multi-view camera system described in step S1 includes using multiple FLIR Blackfly industrial cameras for multi-view acquisition, fixing the industrial cameras with a tripod, placing them around the area to be acquired, performing synchronous acquisition through hardware triggering, and storing the data on a solid-state drive using USB 3.0 and PCIe interfaces.

[0014] Furthermore, the specific method of step S3 includes:

[0015] S31: Initial Camera Intrinsic Estimation: Assuming the camera principal point coincides with the image coordinate system, the camera intrinsic matrix is ​​simplified to a diagonal matrix. Where f x f is the focal length along the x-axis of the camera. y Let be the camera's y-axis focal length, and diag(·) denote the diagonal matrix;

[0016] Using the line connecting the midpoints of the head joint and the two ankle joints of an upright human body as parallel lines, and knowing that parallel lines in two-dimensional image and three-dimensional space intersect at vanishing points, construct a projection matrix based on projection relationships:

[0017]

[0018]

[0019] Where λ H,i Let λ be the unknown depth of the i-th human head joint. B,i Let be the unknown depth of the midpoint between the two ankle joints of the i-th human body. Let be the homogeneous representation of the joints of the i-th human head. Let X be the homogeneous representation of the midpoint of the two ankle joints of the i-th human body. H,i Let X be the coordinates of the three-dimensional joint of the i-th human head. B,i Let be the coordinates of the midpoints of the two ankle joints of the i-th human body. The height of the human body is obtained from the initially estimated three-dimensional coordinates of the human body, thus a height-related constraint is constructed:

[0020]

[0021] Where N is the normal to the ground plane, h i Let λ be the height of the i-th person, and λ be the unknown depth. H,i and λ B,i Cross product Elimination, that is:

[0022]

[0023] Define vanishing point When there are at least two upright human figures, a unique solution is obtained using the least squares method. The midpoints of the ankles of different human figures approximately form the ground plane, perpendicular to the normal to the ground plane. This constraint is used to solve for the intrinsic parameters.

[0024]

[0025] The superscript T indicates matrix transpose. A unique solution can be found when the number of participants is at least three.

[0026] When the number of upright human figures in the same frame is insufficient, upright human figures from different frames are used to solve the camera intrinsic parameters together.

[0027] S32: Initial extrinsic parameter estimation of the camera: First, the three-dimensional human joints obtained in the camera coordinate system corresponding to different viewpoints in step S2 are rigidly aligned. Based on the error between the joints after alignment, the matching relationship between the three-dimensional human joints and the corresponding two-dimensional human joints between different viewpoints is established. The rotation relationship between the matched three-dimensional joints of different viewpoints is regarded as the inverse transformation of the rotation relationship between cameras. Therefore, the first camera rotation matrix is ​​set as the identity matrix. The rotation relationship between the other cameras and this camera is obtained, and the rotation matrix of all cameras can be obtained.

[0028] Then, using the two-dimensional joints obtained in step S2, the corresponding three-dimensional joints, the obtained camera rotation matrix, and the camera intrinsic parameters obtained in step S31, the camera translation matrix is ​​regressed, thus constructing the camera extrinsic parameters from the rotation and translation matrices:

[0029]

[0030] in This represents the value of t when the expression following it reaches its minimum, ‖·‖2 is the L2 norm, π is the projection operation, R and t are the camera rotation and translation matrices respectively, J is the 3D joint coordinates of the first viewpoint, p is the 2D joint pixel coordinates, and t * To obtain the optimal camera translation matrix.

[0031] Furthermore, the specific method of step S4 includes:

[0032] S41: The 3D human body joints estimated in the camera coordinate system in S2 are used with the camera extrinsic parameters ε of the estimated viewpoint i in S3. i Transform to global coordinate system:

[0033] X w,i =ε i -1 X c,i

[0034] Among them, X w,i Let ε be the coordinates of the three-dimensional joints of the human body in the world coordinate system. i X is the external parameter of the camera. c,i The coordinates of the three-dimensional joints of the human body in the camera coordinate system;

[0035] Due to the large estimation error of the camera translation matrix, the root joints of the human body's 3D joints are aligned with the origin of the world coordinate system. The aligned 3D joint coordinates are represented as follows: Calculate the similarity of 3D joints and match the corresponding 2D joints from different viewpoints based on the similarity. The similarity consists of three components: temporal pose similarity, spatial pose similarity, and geometric similarity.

[0036] S42: Temporal Pose Similarity: Temporal pose similarity of viewpoint i at frame t Defined as:

[0037]

[0038] Among them, X t-1 The coordinates of the three-dimensional joints in the previous frame are obtained by averaging the coordinates of the three-dimensional joints matched in the previous frame.

[0039] S43: Spatial pose similarity: Spatial pose similarity between viewpoint i and viewpoint j Defined as:

[0040]

[0041] S44: Geometric Similarity: Constructing Geometric Similarity

[0042]

[0043] Wherein, the rays at viewpoints i and j are represented in the Plück coordinate system as (n i ,l i ) and (n j ,l j );

[0044] S45: Use the temporal pose similarity obtained in step S42, the spatial pose similarity obtained in step S43, and the geometric similarity obtained in step S44 to construct pose loss and geometric loss from different viewpoints:

[0045]

[0046] in, and The pose loss and geometric loss are for viewpoints i and j, using positive semidefinite matrices. To represent the correctness of matching between different viewpoints, solve... To minimize the attitude-geometry consistency loss:

[0047]

[0048] in, This is the sum of pose loss across all viewpoints. c is the sum of geometric losses across all viewpoints. g and c p The values ​​are 0.7 and 0.3 respectively, and <> represents the Hadamard product;

[0049] S46: The positive semi-definite matrix obtained in step S46 is used to represent the similarity of two-dimensional joints of the human body between different viewpoints. The two-dimensional joints of a certain human body in the current viewpoint and the two-dimensional joints of another viewpoint are sorted according to the similarity. The matched two-dimensional joints are back-projected according to the sorting order to obtain three-dimensional joints. Then, the three-dimensional joints are re-projected using the camera parameters of the current viewpoint. The error between the two-dimensional joints of the current viewpoint and the re-projected two-dimensional joints is calculated. After comparing all the matched two-dimensional joints, a set of errors is obtained. The two-dimensional joint with the smallest error can be regarded as the two-dimensional joint that matches the current human body in the current viewpoint, thereby establishing the matching relationship between two-dimensional joints of different viewpoints. The two-dimensional joints are back-projected back into three-dimensional space according to the camera parameters estimated in S3 to obtain the global three-dimensional joints.

[0050] Furthermore, the specific method of step S5 includes:

[0051] S51: Fit the joints of the human body model to the 3D joint points obtained in S4 as the initial state of the human body model. Optimize the camera parameters and human body model using the initial human body model, the initial camera parameters obtained in step S5, and the 2D joint points matched in step S4.

[0052] S52: Optimization variables are derived from videos from V viewpoints, including... Individual body, namely the body shape parameter β of the human body model and the global rotation parameter of the human body model. Global translation parameters of the human body model Given the camera's extrinsic parameter ε and intrinsic parameter K, the objective function is expressed as:

[0053]

[0054] in, For collision items, data items For two-dimensional reprojection error:

[0055]

[0056] Where ρ is the Geman-McClure function, and J represents the matched 2D pose and its corresponding confidence level. n The three-dimensional joint coordinates, ε, are generated from the parameters of the human body model. v K represents the camera extrinsic parameters from the v-th viewpoint. v Here are the camera intrinsic parameters for the v-th viewpoint, where V is the total number of viewpoints. The number of human bodies;

[0057] We employ a collision term based on a differentiable symbolic distance field (SDF) to prevent artifacts caused by multi-user interaction:

[0058]

[0059] Among them, SDF i (vt) is the distance from the sampling vertex vt to the surface of the human body mesh.

[0060] Compared with existing technologies, the beneficial effects of this invention are as follows: 1. The camera calibration method proposed in this invention does not require additional calibration tools, thus having a wider range of applications. 2. The pose-geometric constraints proposed in this invention can calculate the similarity of human poses between different viewpoints, thereby achieving human body matching between different viewpoints and thus achieving more accurate camera calibration. 3. The joint optimization process can simultaneously optimize camera parameters and human body models, making it more robust to noise input. Attached Figure Description

[0061] Figure 1 This is a flowchart of the present invention;

[0062] Figure 2 This is a schematic diagram of attitude matching based on attitude-geometric constraints;

[0063] Figure 3 It is a schematic diagram of a human body model;

[0064] Figure 4 This is a schematic diagram of the matched human body numbers, where (a) to (i) are schematic diagrams of human body numbers from 9 different perspectives;

[0065] Figure 5 This is a diagram showing the camera calibration results;

[0066] Figure 6 This is a diagram showing the results of human motion reconstruction.

[0067] Figure 7 It is a result image of the motion trajectory and camera posture. Detailed Implementation

[0068] The implementation process of the present invention will be described in detail below with reference to the embodiments and the accompanying drawings.

[0069] The camera intrinsic and extrinsic parameter calibration method based on cross-view multi-human semantic matching described in this invention includes the following steps:

[0070] Build a multi-view camera system and capture multi-person motion videos

[0071] (1) Use multiple FLIR Blackfly industrial cameras to acquire data from multiple perspectives. Use a tripod to fix the industrial cameras and place them around the area to be acquired. Acquire data synchronously through hardware triggering and store the data on a solid-state drive using USB 3.0 and PCIE interfaces.

[0072] Two-dimensional pose estimation and single-view three-dimensional pose estimation

[0073] (2) Take a segment of the multi-person, multi-view human motion video sequence collected in (1), use the open-source two-dimensional pose estimation method to obtain the coordinates of the two-dimensional joints of the human body, and use the open-source three-dimensional pose estimation method to estimate each viewpoint separately to obtain the coordinates of all three-dimensional joints of the human body in the camera coordinate system.

[0074] Camera initial parameter estimation

[0075] (3) Assuming the camera principal point coincides with the origin of the image coordinate system, the camera intrinsic parameter matrix is ​​simplified to a diagonal matrix. Where f x f is the focal length along the x-axis of the camera. y Let y be the focal length of the camera along the y-axis, and diag(·) denote the diagonal matrix.

[0076] (4) Utilizing the head joint X of an upright human body H,i and the midpoint X of the ankle joint B,i The lines connecting the points are considered as parallel lines. These parallel lines perpendicular to the ground intersect at the vertical vanishing point in the 2D image. Based on the projection relationship in the camera coordinate system, a projection matrix can be constructed:

[0077]

[0078]

[0079] Where λ H,i Let λ be the unknown depth of the i-th human head joint. B,i Let be the unknown depth of the midpoint between the two ankle joints of the i-th human body. Let be the homogeneous representation of the joints of the i-th human head. Let X be the homogeneous representation of the midpoint of the two ankle joints of the i-th human body. H,i Let X be the coordinates of the three-dimensional joint of the i-th human head. B,i Let be the coordinates of the midpoints of the two ankle joints of the i-th human body. The height of the human body is obtained from the initially estimated three-dimensional coordinates of the human body, thus a height-related constraint is constructed:

[0080]

[0081] Where N is the normal to the ground plane, h i Let λ be the height of the i-th person, and λ be the unknown depth. H,i and λ B,i Cross product Elimination, that is:

[0082]

[0083] Define vanishing point When there are at least two upright human figures, a unique solution is obtained using the least squares method. The midpoints of the ankles of different human figures approximately form the ground plane, perpendicular to the normal to the ground plane. This constraint is used to solve for the intrinsic parameters.

[0084]

[0085] The superscript T indicates matrix transpose. A unique solution can be found when the number of participants is at least three.

[0086] When the number of upright human figures in the same frame is insufficient, the 3D coordinates of upright human figures from different frames at the same viewpoint are used to jointly solve the camera intrinsic parameters, thereby obtaining a simplified intrinsic parameter representation. The actual camera principal point is offset from the image coordinate system. Therefore, the image coordinate system is shifted back to the upper left corner of the image, and the final initial intrinsic parameter estimate is:

[0087]

[0088] Among them, c x and c y Using the image center coordinates, this method solves the intrinsic parameters by requiring only two-dimensional human joints and human height, significantly simplifying the calibration process. It is also applicable to scenarios where tool calibration is difficult, such as large-scale scenes and publicly available online images.

[0089] (5) Rigidly align the three-dimensional human joints obtained from different viewpoints in the camera coordinate system in (2). Based on the error between the joints after alignment, establish the matching relationship between the three-dimensional human joints and the corresponding two-dimensional human joints from different viewpoints. Set the first camera rotation matrix as the identity matrix, that is, set the global coordinate system to coincide with the first camera coordinate system. Sequentially calculate the rotation matrices between the remaining cameras and the first camera to obtain the rotation matrices of all cameras. Use the two-dimensional joints estimated in (2), the three-dimensional joints corresponding to the two-dimensional joints, the camera intrinsic parameters estimated in (4), and the obtained camera rotation matrices to regress the camera translation matrix. Finally, combine the camera rotation matrix and the camera translation matrix to form the camera extrinsic parameters:

[0090]

[0091] in This represents the value of t when the expression following it reaches its minimum, ‖·‖2 is the L2 norm, π is the projection operation, R and t are the camera rotation and translation matrices respectively, J is the 3D joint coordinates of the first viewpoint, p is the 2D joint pixel coordinates, and t * To obtain the optimal camera translation matrix.

[0092] Multi-view human pose matching

[0093] (6) The three-dimensional human body joints estimated in (2) in the camera coordinate system are used with the camera extrinsic parameters ε of the initial estimated viewpoint i in (3). i Transform to global coordinate system:

[0094] X w,i =ε i -1 X c,i

[0095] Among them, X w,i Let ε be the coordinates of the three-dimensional joints of the human body in the world coordinate system. i X is the external parameter of the camera. c,i The coordinates of the three-dimensional joints of the human body in the camera coordinate system;

[0096] Due to the large estimation error of the camera translation matrix, the root joints of the human body's 3D joints are aligned with the origin of the world coordinate system. The aligned 3D joint coordinates are represented as follows: The similarity of 3D joints is calculated, and the corresponding 2D joints are matched according to the similarity. The similarity includes three components: temporal pose similarity, spatial pose similarity, and geometric similarity.

[0097] (7) Figure 2 As shown, the 3D joint points of the previous frame are used as input, and the calculations are performed using the 3D joint points of the current frame's viewpoints i and j. Specifically, the calculations are performed using the 3D joint point X from the previous frame. t-1 The difference between the 3D joints of the current frame and the current viewpoint is used as the temporal pose similarity of viewpoint i in frame t. Mathematically represented as:

[0098]

[0099] Among them, X t-1 These are the 3D joints from the previous frame, obtained by averaging the coordinates of the 3D joints matched in the previous frame.

[0100] (8) Posture similarity also includes spatial posture similarity. The difference between the 3D joint coordinates of viewpoints i and j is mathematically represented as:

[0101]

[0102] (9) Using only pose constraints cannot achieve the expected results. When the poses of different human bodies are very similar, mismatches are likely to occur. Therefore, a set of optical rays emitted from the optical center of the camera is added. Each set of rays passes through the two-dimensional joint coordinates within the viewpoint. Ideally, the optical rays of viewpoints i and j at the same two-dimensional joint coordinates are coplanar, thereby constructing geometric similarity.

[0103]

[0104] Wherein, the rays at viewpoints i and j are represented in the Plück coordinate system as (n i ,l i ) and (n j ,l j The pose loss and geometric loss under different viewpoints are expressed as follows:

[0105]

[0106] in, and These are the pose loss and geometric loss for viewpoints i and j.

[0107] (10) To integrate the above three similarities, a positive semidefinite matrix is ​​used. To represent the correctness of matching between different viewpoints, solve... To minimize the attitude-geometry consistency loss:

[0108]

[0109] Among them, c g and c p The values ​​are 0.7 and 0.3 respectively, and <> represents the Hadamard product. Positive semi-definite matrix. It can be used to represent the similarity between the current human body from the current perspective and the human body from other perspectives. The greater the similarity, the greater the probability that the two human body's two-dimensional joints belong to the same person.

[0110] (11) The positive semidefinite matrix obtained in (10) This is used to represent the similarity between the current human body from the current viewpoint and human bodies from other viewpoints. The higher the similarity, the greater the probability that the two human bodies' two-dimensional joints belong to the same person. Figure 2 As shown, the two-dimensional joints of a human body in the current view and the two-dimensional joints of another view are sorted according to the similarity. The two-dimensional joints are back-projected according to the sorting order to obtain three-dimensional joints. Then, the three-dimensional joints are re-projected using the camera parameters of the current view. The error between the two-dimensional joints of the current view and the two-dimensional joints obtained by reprojection is calculated. After comparing all the two-dimensional joints, a set of errors is obtained. The two-dimensional joint with the smallest error can be considered as the two-dimensional joint that matches the current human body in the current view. Thus, the matching relationship between two-dimensional joints between different viewpoints is established. The two-dimensional joints are back-projected back into three-dimensional space according to the camera parameters estimated by (5) and (6) to obtain global three-dimensional joints.

[0111] Jointly optimize human motion and camera parameters

[0112] (12) Fit the three-dimensional joint points obtained in (11) to the joint pairs of the human body model, and use them as the initial state of the human body model to achieve... Figure 1 The initialization process of the human body model in the video uses the initial human body model, the initial camera parameters obtained from (5) and (6), and the two-dimensional joints matched in (11) to optimize the camera parameters and the human body model. The optimization variables come from the video from V perspectives, including Individual body, namely the body shape parameter β of the human body model and the global rotation parameter of the human body model. Global translation parameters of the human body model Given the camera's extrinsic parameter ε and intrinsic parameter K, the objective function is expressed as:

[0113]

[0114] Among them, among them, For collision items, data items For two-dimensional reprojection error:

[0115]

[0116] Where ρ is the Geman-McClure function, and J represents the matched two-dimensional key points and their corresponding confidence levels. n The three-dimensional joint coordinates, ε, are generated from the parameters of the human body model. v K represents the camera extrinsic parameters from the v-th viewpoint. v Here are the camera intrinsic parameters for the v-th viewpoint, where V is the total number of viewpoints. The number of human figures is used to constrain camera parameters through the projection process, forcing the results of the 3D human joint coordinate projection to approximate the 2D joint points, thereby optimizing camera parameters and the human model.

[0117] We employ a collision term based on a differentiable symbolic distance field (SDF) to prevent artifacts caused by multi-user interaction:

[0118]

[0119] Among them, SDF i (vt) represents the distance from the sampling vertex vt to the surface of the human body mesh. By fitting the motion information using a skinned human body model, the final multi-view multi-human motion reconstruction result can be obtained, such as... Figure 6 As shown, jointly optimizing human motion and camera parameters can yield more accurate human motion trajectories and camera poses, such as... Figure 7 As shown.

[0120] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of the invention. Various changes and modifications can be made to the present invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the claims and their equivalents.

Claims

1. A camera intrinsic and extrinsic parameter calibration method based on cross-view multi-person semantic matching, characterized in that, The method comprises the following steps: S1. Build a multi-view camera system and collect multi-person motion video to obtain a multi-person multi-view video sequence; S2. Two-dimensional pose estimation and single-view three-dimensional pose estimation: select a section of the multi-person multi-view video sequence collected in step S1, use an open-source two-dimensional pose estimation method to obtain human two-dimensional joint nodes, and use an open-source three-dimensional pose estimation method to estimate each view separately to obtain all human three-dimensional joint nodes in the camera coordinate system; S3. Respectively perform initial camera intrinsic parameter estimation and initial camera extrinsic parameter estimation: Initial camera intrinsic parameter estimation: use the joint nodes of the head of the human body and the midpoint of the two ankle joint nodes in the human two-dimensional joint nodes obtained in step S2 to construct a set of parallel lines in three-dimensional space, and use the geometric phenomenon that parallel lines in three-dimensional space intersect at a vanishing point in two-dimensional projection space to solve the intrinsic parameters; Initial camera extrinsic parameter estimation: first, rigidly align the human three-dimensional joint nodes obtained in step S2 in the camera coordinate system corresponding to different views to obtain the corresponding relationship, obtain the camera rotation matrix based on the matched poses of different views, then use the two-dimensional joint nodes obtained in step S2, the three-dimensional joint nodes corresponding to the two-dimensional joint nodes, the obtained camera rotation matrix, and the camera intrinsic parameter regression to obtain the camera translation matrix; finally, the camera rotation matrix and the camera translation matrix form the camera extrinsic parameter; S4. Use the two-dimensional joint nodes obtained in step S2, the three-dimensional joint nodes, and the initial camera extrinsic parameter obtained in step S3 to construct a pose-geometry constraint term, calculate the similarity between all two-dimensional joint nodes of different views, and establish the matching relationship between the two-dimensional joint nodes of different views; Then use the initial extrinsic parameter to project the matched two-dimensional joint nodes back to three-dimensional space to obtain three-dimensional joint nodes in the global coordinate system; S5. Optimize human motion and camera parameters: fit the joints of the human model to the three-dimensional joint nodes obtained in step S4 as the initial state of the human model, and use the reprojection loss to constrain the reconstruction result to obtain more accurate global positions and camera intrinsic and extrinsic parameters.

2. The camera intrinsic and extrinsic parameter calibration method based on cross-view multi-person semantic matching according to claim 1, characterized in that, The multi-view camera system of step S1 comprises using multiple FLIR Blackfly industrial cameras for multi-view acquisition, using a tripod to fix the industrial cameras, placing them around the area to be collected, synchronously collecting through hardware triggering, and storing to a solid state disk through a USB3.0 and a PCIE interface.

3. The camera intrinsic and extrinsic parameter calibration method based on cross-view multi-person semantic matching according to claim 1, characterized in that, The specific method of step S3 comprises: S31: Initial camera intrinsic estimation: assuming the principal point of the camera coincides with the image coordinate system, the camera intrinsic matrix is simplified as a diagonal matrix where f x is the focal length of the camera x-axis, f y is the focal length of the camera y-axis, and diag(·) represents a diagonal matrix. Use the line connecting the head joint node of the upright human body and the midpoint of the two ankle joint nodes as a parallel line, and known that parallel lines in three-dimensional space intersect at a vanishing point in a two-dimensional image, construct a projection matrix according to the projection relationship: where λ H,i is the unknown depth of the i-th human head joint, λ B,i is the unknown depth of the midpoint of the i-th human ankle joints, is the homogeneous representation of the i-th human head joint, is the homogeneous representation of the midpoint of the i-th human ankle joints, X H,i is the three-dimensional coordinate of the i-th human head joint, X B,i is the three-dimensional coordinate of the midpoint of the i-th human ankle joints, the human height is obtained from the initial estimated human three-dimensional coordinates, thus a height-related constraint is constructed: where N is the normal to the ground plane, h i is the height of the ith person, and the unknown depth λ H,i and λ B,i can be eliminated by the cross product i.e.: Defining vanishing points When the number of upright human bodies is at least two, a unique solution is obtained by least squares, the midpoints of the ankles of different human bodies approximately constitute the ground plane, and the normal line of the ground plane is perpendicular to the ground plane. The intrinsic parameters are solved by using this constraint: where the superscript T denotes matrix transposition, A unique solution can be obtained when the number of people is at least three; When the number of upright human bodies in the same frame is not enough, use upright human bodies in different frames to solve the camera intrinsic parameters together; S32: Camera initial extrinsic parameter estimation: first, rigidly align the human three-dimensional joint points obtained in step S2 in the camera coordinate system corresponding to different angles of view, and establish the matching relationship between the three-dimensional human joint points and the corresponding two-dimensional human joint points according to the error between the joints after alignment. The rotation relationship between the matching three-dimensional joint points of different angles of view is regarded as the inverse transformation of the rotation relationship between the cameras. Therefore, the rotation matrix of the first camera is set as the unit matrix, and the rotation relationship between the remaining cameras and the camera is obtained, that is, the rotation matrix of all cameras is obtained. Then, the camera translation matrix is regressed by using the two-dimensional joint points obtained in step S2, the three-dimensional joint points corresponding to the two-dimensional joint points, the obtained camera rotation matrix, and the camera intrinsic parameter obtained in step S31, so as to construct the camera extrinsic parameter by the rotation matrix and the translation matrix: wherein represents solving the value of t when the expression behind it reaches a minimum, ||·||2 is the two-norm, π is the projection operation, R, t are the camera rotation matrix and the camera translation matrix respectively, J is the three-dimensional joint coordinate of the first view, p is the two-dimensional joint pixel coordinate, t * is the optimal camera translation matrix obtained by solving.

4. The camera intrinsic and extrinsic parameter calibration method based on cross-view multi-person semantic matching according to claim 1, characterized in that, The specific method of step S4 includes: S41: using the camera extrinsic parameter ε of the view i estimated in S3, the 3D human joint points in the camera coordinate system estimated in S2 are converted into the global coordinate system. i Convert to global coordinate system: X w,i = ε i -1 X c,i wherein X w,i is the coordinate of the three-dimensional human body joint in the world coordinate system, ε i is the camera extrinsic parameter, X c,i is the coordinate of the three-dimensional human body joint in the camera coordinate system; Due to the large error of camera translation matrix estimation, the root joint in the three-dimensional human body joint is aligned with the origin of the world coordinate system, and the three-dimensional joint coordinates of the aligned human body are represented as The similarity of the three-dimensional joint is calculated, and the corresponding two-dimensional joint under different perspectives is matched according to the similarity, wherein the similarity includes three items, that is, the time posture similarity, the space posture similarity and the geometric similarity. S42: Time Pose Similarity: Time pose similarity of view i t-th frame is defined as: wherein X t-1 is the three-dimensional joint node of the previous frame, which is obtained by averaging the three-dimensional joint node coordinates of the previous frame after matching; S43: Spatial pose similarity: the spatial pose similarity between view i and view j is defined as: S44: Geometric similarity: Construct geometric similarity wherein the rays of view i and j are represented in the Plücker coordinate system as (n i ,l i ) and (n j ,l j ) respectively; S45: Use the time pose similarity obtained in step S42, the space pose similarity obtained in step S43, and the geometric similarity obtained in step S44 to construct the pose loss and the geometric loss under different angles of view: where, and are pose loss and geometry loss for view i and view j, using positive semi-definite matrix denotes the correctness of matching between different views, solving minimizing the pose-geometry consistency loss: wherein, is the sum of all inter-view pose losses, is the sum of all inter-view geometry losses, c g and c p are 0.7 and 0.3, respectively, and < > denotes the Hadamard product; S46: Use the positive semi-definite matrix obtained in step S46 to represent the similarity of the two-dimensional joint points between different angles of view. According to the size of the similarity, the two-dimensional joint points of the current human and the matching two-dimensional joint points of another angle of view are sorted, and the matched two-dimensional joint points are back projected in order, and the three-dimensional joint points are obtained. Then, the camera parameters of the current angle of view are used for re-projection, and the error between the two-dimensional joint points of the current angle of view and the re-projected two-dimensional joint points is calculated. After comparing all the matched two-dimensional joint points, a group of errors is obtained. The two-dimensional joint point with the smallest error is considered to be the matching two-dimensional joint point of the current human of the current angle of view, so as to establish the matching relationship between the two-dimensional joint points between different angles of view. The two-dimensional joint points are back projected into three-dimensional space according to the camera parameters estimated in S3 to obtain global three-dimensional joint points.

5. The camera intrinsic and extrinsic parameter calibration method based on cross-view multi-person semantic matching according to claim 1, characterized in that, The specific method of step S5 includes: S51: Fit the joints of the human model to the three-dimensional joint points obtained in step S4 as the initial state of the human model. Use the initial human model, the initial camera parameters obtained in step S5, and the two-dimensional joint points matched in step S4 to optimize the camera parameters and the human model. S52: the optimization variables are from videos of V views, containing personal body, respectively, the body model size parameter β, the global rotation parameter of the body model the global translation parameter of the body model the camera extrinsic parameter ε and the camera intrinsic parameter K, and the objective function is represented as: where is the collision term, the data term is the two-dimensional re-projection error: where p is the Geman-McClure function, and is the matched two-dimensional pose and its corresponding confidence, J n is the three-dimensional joint coordinates generated by the human model parameters, ε v is the camera extrinsic parameter under the vth view, K v is the camera intrinsic parameter under the vth view, and V is the total number of views, is the number of people; A collision term based on a differentiable signed distance field (SDF) is used to prevent artifacts caused by multi-person interaction: where SDF i (vt) is the distance of the sampling vertex vtto the surface of the human mesh.

Images