A multi-view image based motion capture method and system

Abstract

This invention relates to the field of image data processing technology, and more particularly to a motion capture method and system based on multi-view images. The method includes: acquiring all views containing the target marker and the gaze direction vector; constructing a spatial information matrix based on the gaze direction vector and identifying the principal uncertainty axis to determine geometric weights; acquiring the imaging distance and spot circularity of the target marker to determine imaging quality weights; constructing a comprehensive weight matrix based on the geometric weights and imaging quality weights, and combining this with a regularization term constructed from the eigenvalues ​​of the weighted Hessian approximation matrix to construct a system of linear equations; and solving the system of linear equations to obtain the three-dimensional coordinates of the target marker. This invention reduces depth oscillations in narrow baseline or highly occluded scenes by identifying the principal uncertainty axis and combining it with physical imaging quality for weighting, and by utilizing spectral damping technology to apply adaptive constraints in the direction of missing information.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a motion capture method and system based on multi-view images. Background Technology

[0002] In digital simulation training and virtual reality interactive applications, motion capture technology based on multi-view images captures the movement trajectories of key human body parts, providing fundamental data support for highly interactive tasks such as multi-person collaborative tactical exercises and the simulated assembly and disassembly of precision instruments. This technology senses personnel posture in real time and drives the virtual model; its calculation accuracy determines the immersion and realism of the operation feedback in the training system, making it a core component of high-end simulation and human-computer interaction applications.

[0003] Existing large-space motion capture systems typically employ a passive positioning scheme based on the target markers. The system deploys multiple high-speed positioning devices around the perimeter of the training area, simultaneously acquiring images of reflective markers attached to the trainee's motion capture suit and gloves. The data processing layer receives multiple video streams, extracts and matches the centroids of the two-dimensional light spots in each view, and then establishes a set of spatial projection equations using multi-view geometry principles. Finally, a triangulation algorithm is used to inversely calculate the three-dimensional spatial coordinates of the target markers frame by frame. This method relies on redundant observation from multiple angles and achieves good positioning results in environments with a wide field of view and no obstructions.

[0004] However, in practical applications involving dense multi-person collaboration or complex component assembly, existing triangulation algorithms have limitations: the number of effective cameras capable of simultaneously observing a given marker point is drastically reduced due to operator limb occlusion or structural obstruction by precision equipment. Furthermore, the remaining effective cameras may be limited to a single lateral viewpoint, resulting in small angles between observation lines of sight. Under such narrow baseline observation conditions, the coefficient matrix behavior of the 3D reconstruction equations is poor, making it highly sensitive to image noise. Tiny pixel extraction errors on the image plane are amplified along the depth direction of the line of sight, causing severe oscillations in the calculated 3D coordinates on the depth axis, failing to meet the stringent positioning stability requirements of precision operations. Summary of the Invention

[0005] To address the technical problem of oscillations in the three-dimensional coordinate depth of the target marker points caused by equation instability and noise sensitivity under narrow baseline observation configurations, this invention provides solutions in the following aspects.

[0006] In a first aspect, the present invention provides a motion capture method based on multi-view images, the method comprising the steps of:

[0007] All views containing the target marker are acquired, and the observation coordinates of the target marker in each view are obtained. The line-of-sight direction vector corresponding to each view is determined. A spatial information matrix of the target marker is constructed based on the line-of-sight direction vector. Eigenvalue decomposition is performed on the spatial information matrix to determine the principal uncertainty axis. The geometric weight of the target marker in each view is determined according to the positional relationship between the line-of-sight direction vector and the principal uncertainty axis. The imaging distance and spot circularity of the target marker in each view are acquired. The imaging quality weight of the target marker in each view is determined based on the imaging distance and the spot circularity. A comprehensive weight matrix is ​​constructed based on the geometric weight and the imaging quality weight. A system of linear equations is constructed by combining the regularization term constructed by the eigenvalues ​​of the weighted Hessian approximation matrix. The system of linear equations is solved to obtain the three-dimensional coordinates of the target marker, thereby realizing motion capture based on multi-view images.

[0008] This invention acquires the line-of-sight vector and constructs a spatial information matrix. Using eigenvalue decomposition, it identifies the principal uncertainty axis of the current observation configuration, thereby determining the direction with the largest spatial positioning error for the target marker. Based on this, a comprehensive weight matrix is ​​constructed by combining geometric and imaging quality weights. During the iterative solution process, a regularization term based on the eigenspectrum of the weighted Hessian approximation matrix is ​​introduced to specifically suppress coordinate updates in restricted directions. This design addresses common occlusion or narrow baseline scenarios in multi-view motion capture. It not only considers the spatial distribution advantages of the camera but also incorporates physical imaging quality factors such as imaging distance and spot circularity. Through spectral damping technology, constraints are applied in directions with missing information, reducing depth direction oscillations caused by restricted observation configurations while maintaining sensitivity, thus improving the stability and accuracy of motion capture data in complex environments.

[0009] Preferably, the geometric weights satisfy the following relationship:

[0010] ;

[0011] in, It is the first The number of test markers at the th ... Geometric weights in each view; , They are the first The minimum and maximum eigenvalues ​​of the spatial information matrix of the test markers; It is the first The number of test markers at the th ... The gaze direction vector in each view; It is the first The principal uncertainty axis vector of the test marker points; It is the geometric gain constant; It is the modulo operator; It is the preset first minute value, and its range is [value range missing]. .

[0012] This invention introduces a geometric weighting relationship that includes the eigenvalue ratio and the magnitude of the vector cross product. It uses the eigenvalue ratio to assess the constraint degree of the observation configuration and the cross product of the line-of-sight vector and the principal uncertainty axis to assess the lateral observation capability of the camera's line of sight. This setup allows for assigning higher weights to views perpendicular to the principal uncertainty axis that provide effective lateral constraints during motion capture, thereby enhancing the equation's ability to suppress depth errors and improving the solution stability under narrow baseline configurations.

[0013] Preferably, constructing the spatial information matrix of the marker point to be tested based on the line-of-sight vector includes: calculating a projection matrix for each of the views using its line-of-sight vector for the marker point to be tested; and summing the projection matrices corresponding to all views to obtain the spatial information matrix of the marker point to be tested.

[0014] Preferably, the step of performing eigenvalue decomposition on the spatial information matrix to determine the principal uncertainty axis includes: performing eigenvalue decomposition on the spatial information matrix to obtain several eigenvalues ​​and corresponding eigenvectors; extracting the minimum eigenvalue and recording its corresponding unit eigenvector as the principal uncertainty axis vector of the test marker point.

[0015] Preferably, the imaging quality weights satisfy the following relationship:

[0016] ;

[0017] in, It is the first The number of test markers at the th ... Image quality weights in each view; It is a reference distance; It is the first The number of test markers at the th ... Imaging distance in each view; It is the distortion suppression coefficient; It is the first The number of test markers at the th ... Roundness of the light spot in each view; It is the preset second minute value, and its range is [value range missing]. .

[0018] This invention utilizes a relationship between a distance-squared ratio term and a roundness exponential attenuation term to calculate the imaging quality weight. Based on the physical imaging laws, the observation data is filtered and cleaned. This setting reduces the negative impact of large spatial discrete errors in distant views and severe light spot distortion in large-angle oblique views on the calculation accuracy. It ensures that the data participating in motion capture calculation mainly comes from high-quality views with clear imaging and small distortion, thereby improving the physical fidelity of the final three-dimensional coordinates.

[0019] Preferably, obtaining the imaging distance and spot roundness of the target marker point in each view includes: extracting the camera optical center coordinates corresponding to each view; using the line-of-sight vectors of each view, performing intersection calculations using linear triangulation to obtain the initial three-dimensional coordinates of the target marker point; calculating the Euclidean distance between the camera optical center corresponding to each view and the initial three-dimensional coordinates, as the imaging distance of the target marker point in each view; performing ellipse fitting on the spot corresponding to the target marker point in each view to obtain a fitted ellipse; and recording the ratio of the minor axis length to the major axis length of the fitted ellipse as the spot roundness of the target marker point.

[0020] Preferably, the step of constructing a comprehensive weight matrix based on the geometric weights and the imaging quality weights includes: calculating the product of the geometric weights and imaging quality weights of the marker point to be tested in each view to obtain the comprehensive weights of each view; constructing a diagonal matrix based on the comprehensive weights of each view, and filling each comprehensive weight onto the diagonal to obtain the comprehensive weight matrix of the marker point to be tested.

[0021] This invention constructs a comprehensive weight matrix by calculating the product of geometric weights and imaging quality weights, and establishes a unified evaluation system that integrates the advantages of geometric configuration and physical imaging quality. This prevents a single-dimensional high-quality view, which is only good in location but blurry in imaging or only clear in imaging but limited in viewing angle, from dominating the solution. It ensures that the method ultimately adopts view data that has both good observation angle and high signal-to-noise ratio, and further improves the reliability of multi-view image motion capture results.

[0022] Preferably, the linear equation satisfies the following relation:

[0023] ;

[0024] in, It is the first An iterative correction vector for the three-dimensional coordinates of the test marker points; It is the first Jacobian matrix of the test markers; It is the vector transpose symbol; It is the first The comprehensive weight matrix of the test markers; It is the first The reprojection residual vector of each marker point to be tested; It is the first of the weighted Hessian approximation matrices. The weighted Hessian approximation matrix is ​​represented by eigenvalues. ; It is related to eigenvalues The corresponding unit eigenvector; It is an adaptive damping parameter; It is the preset third minute value, and its range is [value range missing]. ; It is the number of characteristic directions in three-dimensional space.

[0025] This invention constructs a linear equation containing a regularization term and applies adaptive constraints to the solution process using the eigenvalues ​​and eigenvectors of the weighted Hessian approximation matrix. The linear equation applies greater damping through the regularization term in the direction with smaller eigenvalues ​​to limit the coordinate update amplitude, while maintaining smaller damping in the direction with larger eigenvalues ​​to maintain sensitivity. This achieves a balance between anti-oscillation stability and dynamic response sensitivity in motion capture solution.

[0026] Preferably, solving the linear equation system to obtain the three-dimensional coordinates of the target marker includes: using the Gauss-Newton method as an iterative strategy, updating the reprojection residual vector and Jacobian matrix in each iteration to construct the linear equation system; solving the linear equation system using Cholesky decomposition or LDLT decomposition to obtain the coordinate correction amount for the current iteration; updating the iterative three-dimensional coordinates of the target marker based on the coordinate correction amount until the iteration condition is met, and determining the iterative three-dimensional coordinates at the time of stopping iteration as the three-dimensional coordinates of the target marker.

[0027] In a second aspect, the present invention provides a motion capture system based on multi-view images. The motion capture system based on multi-view images includes a memory and a processor. The memory stores computer program instructions, and when the computer program instructions are executed by the processor, a motion capture method based on multi-view images according to the first aspect of the present invention is implemented.

[0028] By adopting the above technical solution, a computer program is generated from the motion capture method based on multi-view images of the first aspect of the present invention, and stored in a memory so that it can be loaded and executed by a processor, thereby creating a terminal device based on the memory and the processor for convenient use.

[0029] The beneficial effects of this invention are as follows: This invention can improve positioning stability in highly occluded environments. By constructing a spatial information matrix and identifying the principal uncertainty axis, this invention can accurately identify the direction with the weakest constraint in the observation geometry and use geometric weights to increase the contribution of the lateral view. This allows for the extraction of remaining lateral view information in scenarios with large-area occlusion, such as four-person collaborative training, reducing the scattering of virtual limbs. This invention achieves high-precision measurement of physical perception. By introducing imaging quality weights based on imaging distance and spot roundness, distance quantization error and spot distortion are taken into consideration. Essentially, a dynamic data cleaning mechanism is constructed, ensuring that poor-quality data from long distances or large angles does not contaminate precise measurements at close range, improving the accuracy of tasks such as equipment disassembly and assembly. This invention achieves a balance between anti-vibration and high sensitivity. By introducing a feature spectrum damping regularization term, the algorithm changes the action delay caused by traditional global filtering. High damping is applied only in feature directions with missing information to resist vibration, while low damping is maintained in information-rich directions to maintain high sensitivity, meeting the dual requirements of real-time performance and stability in simulation training. Attached Figure Description

[0030] Figure 1 A flowchart of a motion capture method based on multi-view images provided in an embodiment of the present invention;

[0031] Figure 2 This is a schematic diagram comparing the stability of three-dimensional coordinate depth direction coordinate calculation provided in an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram showing the distribution of three-dimensional coordinate calculation results provided in an embodiment of the present invention.

[0033] Figure 4 This is a structural block diagram of a motion capture system based on multi-view images provided in an embodiment of the present invention. Detailed Implementation

[0034] The first aspect of this invention provides a motion capture method based on multi-view images, such as... Figure 1 As shown, the method includes steps S100-S400:

[0035] Step S100: Obtain all views containing the marker point to be measured, obtain the observation coordinates of the marker point to be measured in each view, and determine the line-of-sight direction vector corresponding to each view.

[0036] It should be noted that motion capture calculation relies on multi-angle spatial intersection of the same target. Therefore, multiple motion capture cameras distributed around the site need to cover the capture area from different directions to provide the necessary observation redundancy for the calculation of 3D coordinates. At the same time, the original images captured by the cameras contain complex environmental backgrounds. Thresholding segmentation using the high reflectivity of the target markers can quickly separate the high-brightness foreground targets from the background. In addition, in order to eliminate false matches caused by environmental noise or similar textures when fusing all views, geometric constraints need to be introduced to ensure that the data input into the solver all originate from the real reflection of the same target marker.

[0037] Specifically, firstly, multiple motion capture cameras distributed around the site are controlled to synchronously acquire raw images containing test markers attached to target objects, such as trainees or equipment. For each acquired raw image frame, a preset brightness threshold is used to filter bright areas, and the two-dimensional centroid coordinates of the test markers on the image plane are extracted through connected component analysis and defined as the observation pixel coordinates. Simultaneously, the morphological data of the light spot connected components of the test markers are extracted and saved. Based on the calibration parameters of each camera, the vertical distance from candidate points to the corresponding epipolar lines, i.e., the epipolar distance, is calculated between different views, and matching point pairs with epipolar distances less than a preset tolerance threshold are retained, thus obtaining all views belonging to each test marker in the scene.

[0038] For any given marker point, its spatial geometric description is obtained based on the camera parameter matrix acquired during the calibration phase: the extrinsic translation vector of the camera is read and transformed to the world coordinate system as the three-dimensional coordinate position of the camera's optical center; the intrinsic matrix of the camera is read and the observed pixel coordinates of the marker point are back-projected onto the normalized imaging plane, and combined with the inverse transformation of the camera's extrinsic rotation matrix, the direction vector on the imaging plane is transformed to the world coordinate system. Finally, it is normalized to obtain the line-of-sight direction vector corresponding to each view. It should be noted that after normalization, the line-of-sight direction vector is a unit vector with a magnitude of 1 to eliminate the influence of different observation distances on the vector scale.

[0039] At this point, all views of each target marker and the corresponding line-of-sight vector for each view have been obtained.

[0040] Step S200: Construct a spatial information matrix of the marker point to be tested based on the line-of-sight vector, perform eigenvalue decomposition on the spatial information matrix to determine the principal uncertainty axis, and determine the geometric weight of the marker point to be tested in each view according to the positional relationship between the line-of-sight vector of each view and the principal uncertainty axis.

[0041] It should be noted that in scenarios with multiple obstructions, the number of effective cameras capable of observing the target marker decreases sharply, and the lines of sight of the remaining cameras tend to be parallel, forming a restricted configuration with narrow intersection angles. This results in significant uncertainty in spatial positioning along the line-of-sight direction. This invention uses mathematical methods to identify the direction of greatest error in the observation geometry, i.e., the principal uncertainty axis, and specifically increases the weight of cameras that can observe this direction from the side. This improves the behavior of the equations at the geometric level and effectively suppresses depth oscillations along the line-of-sight direction.

[0042] Specifically, firstly, for any given marker point, a spatial information matrix is ​​constructed using the line-of-sight vectors of the marker point in each view. It should be noted that a single view can only provide ray constraints in that direction. To quantitatively evaluate the overall locking capability of the multiphase configuration to the marker point in three-dimensional space, the geometric constraint information of all views needs to be aggregated into a matrix that reflects the stiffness distribution of the observed configuration in various spatial directions.

[0043] Specifically, initialize one Using the zero matrix as an accumulator, the projection matrix is ​​calculated for any of the views using its line-of-sight vector toward the target marker. ,in for The identity matrix, It is the first The number of test markers at the th ... The gaze direction vector in each view; It is the vector transpose symbol; by summing the projection matrices corresponding to all views, the spatial information matrix of the target marker point is obtained. .

[0044] Then, eigenvalue decomposition is performed on the spatial information matrix to obtain the principal uncertainty axes. It should be noted that the eigenstructure of the spatial information matrix describes the shape of the positioning error ellipsoid. Through spectral analysis, the complex spatial observation geometry can be decoupled into orthogonal principal eigendirections. The direction with the smallest eigenvalue corresponds to the direction with the least information and the weakest constraint, i.e., the axis where the positioning error is most likely to diverge. Identifying this axis is a prerequisite for targeted compensation.

[0045] Specifically, eigenvalue decomposition is performed on the spatial information matrix to obtain three non-negative eigenvalues ​​and their corresponding eigenvectors. The eigenvalues ​​are arranged in descending order, and the largest and smallest eigenvalues ​​are extracted. Simultaneously, the unit eigenvector corresponding to the smallest eigenvalue is obtained and defined as the principal uncertainty axis vector of the multi-view observation configuration of the target marker point.

[0046] Finally, the geometric weights of the marker points to be measured in each view are calculated. It should be noted that the traditional least squares method treats all views equally. However, in height-constrained configurations such as narrow baselines, lines of sight parallel to the principal uncertainty axis contribute little to depth calculation and may even introduce noise. Only lines of sight perpendicular to this axis, i.e., those observed from the side, can effectively constrain errors. Therefore, this invention constructs an adaptive weighting model that automatically adjusts the sensitivity based on the degree of geometric constraint of the observation and assigns higher weights to views that can patch up observational gaps from the side, thereby mathematically suppressing oscillating depth values.

[0047] Based on the above logic, the first The number of test markers at the th ... Geometric weights in each view Satisfying the relation:

[0048] ;

[0049] in, , They are the first The minimum and maximum eigenvalues ​​of the spatial information matrix of the test markers; It is the first The number of test markers at the th ... The gaze direction vector in each view; It is the first The principal uncertainty axis vector of the test markers is a unit vector. It is the geometric gain constant; It is the modulo operator; It is a preset first tiny value used to prevent The value is 0, and the preferred value range is... .

[0050] In this relation, Used to measure the degree of constraint of the observed geometry, when the minimum eigenvalue is much smaller than the maximum eigenvalue, the ratio between the two is small, the value of its negative square increases, and the weighting mechanism is triggered. Used to measure the degree of orthogonality between the camera's line of sight and the principal uncertainty axis, when the line of sight direction vector With the principal uncertainty axis When the view is perpendicular, the cross product modulus is close to 1, indicating that the camera is observing the direction of uncertainty from the side, which can provide the most effective constraint. At this time, the term takes a maximum value, thereby effectively increasing the weight of the view. Conversely, if the line of sight is parallel to the uncertainty axis, the term approaches 0, and the weight remains at the basic level.

[0051] It should be added that, regarding the geometric gain constant... The value needs to be set based on the total number of all cameras. If If the value is too small, it cannot effectively differentiate the weights of the side view and the same-direction view when occluded; if An excessively large value may cause the solution to rely too heavily on a single view. Preferably, this value is set to 0.5 to 1.0 times the total number of all cameras. In this embodiment, for a hardware environment with 26 cameras... The preferred value is set to 15.0.

[0052] At this point, the geometric weights of each test point in each view have been obtained.

[0053] Step S300: Obtain the imaging distance and spot circularity of the target marker in each view, and determine the imaging quality weight of the target marker in each view based on the imaging distance and the spot circularity.

[0054] It should be noted that the physical laws of imaging dictate that the farther the observation distance, the larger the spatial physical range represented by a unit pixel, meaning an increase in spatial discretization error. Simultaneously, the more tilted the camera's observation angle, the more the projection of the spherical marker point onto the imaging plane tends to be a flattened ellipse, leading to a decrease in the accuracy of the centroid extraction algorithm, i.e., increased projection distortion. This invention cleans the data from the perspective of physical imaging quality, reducing the weight of views with low imaging quality or severe distortion to ensure that the data participating in subsequent calculations has a high signal-to-noise ratio.

[0055] Specifically, firstly, for any target marker, the three-dimensional coordinates of the camera optical centers corresponding to all views are extracted; then, using the line-of-sight vectors of all views, a preliminary intersection calculation is performed using linear triangulation to obtain the initial three-dimensional coordinates of the target marker; finally, the Euclidean distance between the camera optical centers of each view and the initial three-dimensional coordinates is calculated as the imaging distance of the target marker in each view.

[0056] Secondly, the morphological data of the light spot connected region of the test marker point in all views are processed using the image moment algorithm or the least squares ellipse fitting method to obtain the best fitting ellipse of the light spot connected region, thereby obtaining the major axis length and minor axis length of the light spot. The ratio of the minor axis length to the major axis length is defined as the light spot roundness of the test marker point in all views, and the value ranges from 0 to 1.

[0057] Then, the imaging quality weights of the marker points to be tested in each view are calculated. It should be noted that, in order to map the two physical characteristics of imaging distance and spot roundness into mathematical weight indicators, this invention uses an inverse square model to characterize the constraint of distance on the weights based on the characteristic that optical resolution decays with distance; and uses an exponential decay model to characterize the constraint of spot roundness on the weights based on the nonlinear response characteristics of imaging quality to shape distortion. This composite model can retain high-quality data while exhibiting a highly sensitive suppression capability for poor-quality data.

[0058] Based on the above logic, the first The number of test markers at the th ... Image quality weights in each view Satisfying the relation:

[0059] ;

[0060] in, It is a reference distance; It is the first The number of test markers at the th ... Imaging distance in each view; It is the distortion suppression coefficient; It is the first The number of test markers at the th ... Roundness of the light spot in each view; It is a preset second tiny value used to prevent The value is 0, and the preferred value range is [missing information]. .

[0061] In this relation, Utilizing the inverse square property of distance, when the observed distance Greater than the reference distance At that time, this term decays rapidly, thereby suppressing the influence of spatial discrete errors of long-distance cameras; Using the exponential decay property to address spot distortion, when the spot roundness When the value is close to 1, the exponential term is close to 0 and has a higher weight; when the light spot becomes flattened due to large-angle oblique viewing, it leads to... When the exponent term is reduced, it quickly becomes negative, causing the image quality weight to decrease. It decreases exponentially, thus effectively suppressing the centroid extraction error caused by projection distortion.

[0062] It should be noted that the reference distance Typically set based on the size of the site, it is preferably directly defined as the optimal focusing distance during calibration; in this embodiment, it is set to 4.0 meters. Distortion suppression coefficient The tolerance of the algorithm for non-circular light spots is used to control the value, which is recommended to be in the range of 3.0 to 5.0. In this embodiment, 4 is preferred to achieve a balance between retaining a sufficient amount of data and eliminating poor data.

[0063] It should be noted that the reference distance With distortion suppression coefficient The value needs to be set according to the spatial scale of the actual capture site and the dynamic characteristics of the measured action: for precision operation scenarios with compact space and high requirements for measurement accuracy, such as the disassembly and assembly of precision instruments, It can be set to a smaller value, such as 2 meters, and at the same time... Set to a larger value, such as 5, to construct a highly sensitive suppression mechanism for imaging distortion and distance attenuation, ensuring that the data involved in each frame of the calculation is close to the ideal form, thereby guaranteeing accuracy; for tactical training scenarios with open spaces and violent movements, such as multi-person combat, The depth of field can be set to a larger value to suit the site, such as 8 meters, and then appropriately reduced. To point 3, this enhances the algorithm's tolerance for motion blur or large field-of-view edge imaging caused by rapid movement, preventing data discontinuity due to overly strict filtering. In this embodiment, both conventional laboratory environments and standard training actions are comprehensively considered. The preferred setting is 4 meters. The preferred value is 4.

[0064] At this point, the imaging quality weights of each test marker point in each view have been obtained.

[0065] Step S400: Construct a comprehensive weight matrix based on the geometric weights and the imaging quality weights, and construct a system of linear equations by combining the regularization terms constructed from the eigenvalues ​​of the weighted Hessian approximation matrix; solve the system of linear equations to obtain the three-dimensional coordinates of the target marker, thereby realizing motion capture based on multi-view images.

[0066] It should be noted that even after weighting by geometric and physical dimensions, under severe occlusion, the amount of information contained in the data itself may still be insufficient to determine a unique solution, leading to numerical overflow or oscillations in traditional solvers. This invention maps the linear system to a feature subspace and applies adaptive regularization constraints only to the feature directions lacking information, while maintaining low damping in the information-rich directions. This mathematically suppresses oscillations while preserving the algorithm's high sensitivity and avoiding the action delay caused by global filtering.

[0067] First, for any target marker, a comprehensive weight matrix is ​​constructed. It should be noted that the geometric weight focuses on evaluating the spatial configuration advantages of the observation line of sight, while the imaging quality weight focuses on evaluating the signal-to-noise ratio of the imaging quality. A single indicator cannot comprehensively reflect the reliability of the data. Therefore, this invention integrates the two to construct a unified evaluation system, ensuring that subsequent optimization processes are biased towards views that combine superior observation geometry with high imaging quality.

[0068] Specifically, based on geometric weights and imaging quality weights, the first... The number of test markers at the th ... Overall weight in each view The comprehensive weights satisfy the relational expression ;in, It is the first The number of test markers at the th ... Geometric weights in each view; It is the first The number of test markers at the th ... Image quality weights for each view. Construct a diagonal weight matrix that matches the observation dimension, and assign weights to each view. Fill the diagonal lines to obtain the comprehensive weight matrix of the marker points to be tested. ,in For the first The total number of views corresponding to each test marker point.

[0069] Secondly, a linear equation including a regularization term is constructed and solved iteratively. It should be noted that the projection process from three-dimensional spatial coordinates to two-dimensional image pixel coordinates involves perspective division and distortion correction, which is a non-convex nonlinear mapping. Directly solving this nonlinear equation system is difficult to obtain an analytical solution. Therefore, this invention adopts the Gauss-Newton method as an iterative linearization strategy. The Gauss-Newton method is a classic algorithm for solving nonlinear least squares problems. Its core idea is to use Taylor series expansion to locally approximate the nonlinear residual function as a linear function, and to approximate the complex Hessian matrix using the product of Jacobian matrices, i.e., constructing a weighted Hessian approximation matrix. This invention chooses the Gauss-Newton method because it has high computational efficiency and convergence speed when dealing with visual projection problems. It can effectively cope with the strong nonlinear characteristics brought by perspective projection and avoid calculating the complex Hessian matrix, thus ensuring the high-frequency response requirements of motion capture in real-time solution scenarios.

[0070] Specifically, in any iteration, for any target marker, based on its estimated 3D coordinates, the theoretical reprojected pixel coordinates of the target marker projected onto the imaging plane of each view in all views are calculated using the intrinsic and extrinsic parameter matrices of each camera. The theoretical reprojected pixel coordinates are then differiated from the observed pixel coordinates of the target marker in the corresponding view, and the deviations of all views are integrated to construct a reprojection residual vector. Simultaneously, the partial derivatives of the projection functions of each view with respect to the 3D coordinates of the target marker are calculated and stacked to form a Jacobian matrix. Based on the Jacobian matrix, the integrated weight matrix, and the reprojection residual vector, a linear equation for the correction amount of the 3D coordinates of the target marker is constructed.

[0071] Based on the above logic, the linear equation satisfies the following relation:

[0072] ;

[0073] in, It is the first An iterative correction vector for the three-dimensional coordinates of the test marker points; It is the first Jacobian matrix of the test markers; It is the vector transpose symbol; It is the first The comprehensive weight matrix of the test markers; It is the first The reprojection residual vector of each marker point to be tested; It is the first of the weighted Hessian approximation matrices. The weighted Hessian approximation matrix is ​​represented by eigenvalues. ; It is related to eigenvalues The corresponding unit eigenvector; It is an adaptive damping parameter; It is a preset third micro value used to prevent The value is 0, and the preferred value range is... ; It is the number of characteristic directions in three-dimensional space.

[0074] In this relation, For regularization, the term in this term It is a weighted Hessian approximation matrix, and in this term... By dynamically adjusting the damping using eigenspectral decomposition technology, adaptive constraints on coordinate updates are achieved, where the eigenvalues ​​in a certain characteristic direction... When the value is small, meaning that the direction is a restricted direction where observational information is missing, the denominator... Approaching 0, leading to The corresponding increase is equivalent to applying a higher impedance regularization constraint in that direction in the equation, limiting the coordinate update in that direction and preventing violent oscillations caused by noise amplification; conversely, when the eigenvalue of a certain characteristic direction... When the value is large, it means that the direction is clear and reliable. Approaching zero means there is almost no additional damping in that direction, allowing the algorithm to respond effectively to residual data and maintain highly sensitive position updates.

[0075] It should be noted that the adaptive damping parameters The value of needs to be set according to the resolution of the input image and the system's stability requirements: for scenes using ultra-high resolution cameras such as 4K and with low environmental noise, the value can be appropriately reduced. The value can be set to 0.1 to reduce damping and fully utilize the positioning accuracy advantage brought by high pixel count; for low-resolution scenes or scenes with poor ambient lighting conditions resulting in high image noise, the value should be increased appropriately. A value, such as 1, is set to enhance the regularization constraint, prioritizing the smoothness and stability of the solution. In this embodiment, for standard 2K or 4K motion capture cameras, it is recommended... Set it to 0.5.

[0076] Then, the linear equations are solved and the coordinates are updated. It should be noted that the coefficient matrix of the linear equation system usually has symmetric positive definiteness. Existing numerical linear algebra methods, such as Cholesky decomposition or LDLT decomposition, are used to solve the equations. These classical algorithms can make full use of the structural characteristics of the matrix for efficient computation and have good numerical stability, avoiding solution divergence caused by computer floating-point errors. Based on this, the above decomposition algorithm is used to solve the equations in this step.

[0077] Specifically, the linear equation is solved using Cholesky decomposition or LDLT decomposition to obtain the coordinate correction for the current iteration step. The correction is then superimposed onto the current estimated 3D coordinates to obtain the updated 3D coordinates. The specific solution method is existing technology and will not be elaborated upon here. This process of constructing the equation, solving for the correction, and updating the coordinates is repeated until the correction amount is reached. If the modulus is less than the preset convergence threshold or the maximum number of iterations is reached, the iteration stops and the final converged 3D coordinates are output.

[0078] like Figure 2 The diagram shows a comparison of the stability of 3D depth coordinate calculations. The horizontal axis represents the sampling frame number, and the vertical axis represents the depth coordinate value. The dashed line represents the depth value of the 3D coordinates calculated by the traditional method, while the solid line represents the depth value of the 3D coordinates calculated by the method of this invention. The dotted lines represent the true depth reference. Observing the curves in the diagram, it can be seen that during continuous sampling, the curve corresponding to the traditional method exhibits high-frequency and large-amplitude fluctuations relative to the reference line, indicating the existence of depth direction oscillations. The curve corresponding to the method of this invention remains smooth and fits the true depth reference well, with the fluctuation amplitude significantly suppressed. This demonstrates that by introducing geometric spectral analysis and physical fidelity weights, this invention effectively eliminates the numerical instability caused by the narrow baseline observation configuration, achieving stable tracking of the depth coordinates.

[0079] like Figure 3 The diagram shows the distribution of the 3D coordinate solution results. The three coordinate axes represent the spatial positions in the X, Y, and Z dimensions, respectively. It can be seen from the diagram that the data point set of traditional methods exhibits significant dispersion along the Z-axis, failing to achieve effective convergence. In contrast, the data points of the method in this invention exhibit a tight clustering feature in 3D space, with the cluster centers coinciding with the actual spatial locations. This demonstrates that, even in complex occlusion environments, this invention can ensure the consistency and reliability of spatial positioning results across all dimensions through the organic coupling of multi-dimensional weights.

[0080] It should be further noted that the convergence threshold and maximum number of iterations need to be set in a balance between real-time requirements and accuracy requirements: For interactive scenarios with high real-time requirements, it is recommended to set a larger convergence threshold, such as 0.1, and a smaller maximum number of iterations, such as 5, to ensure that the calculation time for each frame is strictly controlled within the frame interval; for offline data processing or precision measurement scenarios with high accuracy requirements, it is recommended to set a smaller convergence threshold, such as 0.001, and a larger maximum number of iterations, such as 20, to obtain higher geometric accuracy. In this embodiment, to balance real-time performance and accuracy, the convergence threshold is preferably set to 0.05, and the maximum number of iterations is preferably set to 10.

[0081] This completes the motion capture and calculation based on multi-view images. This method transmits the high-precision 3D coordinate data to the virtual reality engine in real time to drive the limb movements of the virtual character or to perform precise operation judgments, thereby achieving stable and accurate human-computer interaction in complex occlusion environments.

[0082] The second aspect of this embodiment provides a motion capture system based on multi-view images, such as Figure 4 As shown, the motion capture system based on multi-view images includes a memory and a processor. The memory stores computer program instructions, which, when executed by the processor, implement a motion capture method based on multi-view images according to the first aspect of the present invention.

[0083] The motion capture system based on multi-view images also includes other components well known to those skilled in the art, such as communication buses and communication interfaces. Their settings and functions are known in the art and will not be described in detail here.

[0084] In this invention, the aforementioned memory can be any tangible medium containing or storing a program that can be used or combined with an instruction execution system, apparatus, or device. For example, a computer-readable storage medium can be any suitable magnetic or magneto-optical storage medium, such as resistive random access memory (DRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (DRAM), high-bandwidth memory, hybrid memory cube, etc., or any other medium that can be used to store desired information and can be accessed by an application, module, or both. Any such computer storage medium can be part of a device or accessible to or connected to a device.

[0085] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A motion capture method based on multi-view images, characterized in that, include: Obtain all views containing the marker point to be measured, obtain the observation coordinates of the marker point to be measured in each view, and determine the line-of-sight direction vector corresponding to each view; Based on the line-of-sight vector, a spatial information matrix of the marker point to be tested is constructed. Eigenvalue decomposition is performed on the spatial information matrix to determine the principal uncertainty axis, including: performing eigenvalue decomposition on the spatial information matrix to obtain several eigenvalues ​​and corresponding eigenvectors; extracting the minimum eigenvalue and recording its corresponding unit eigenvector as the principal uncertainty axis vector of the marker point to be tested; and determining the geometric weight of the marker point to be tested in each view according to the positional relationship between the line-of-sight vector of each view and the principal uncertainty axis. The imaging distance and spot circularity of the target marker point in each view are obtained, and the imaging quality weight of the target marker point in each view is determined based on the imaging distance and the spot circularity. A comprehensive weight matrix is ​​constructed based on the geometric weights and the imaging quality weights. A regularization term is constructed by combining the eigenvalues ​​of the weighted Hessian approximation matrix to construct a system of linear equations. The system of linear equations is solved to obtain the three-dimensional coordinates of the target marker, thereby realizing motion capture based on multi-view images.

2. The motion capture method based on multi-view images according to claim 1, characterized in that, The geometric weights satisfy the following relationship: ； in, It is the first The number of test markers at the th ... Geometric weights in each view; , They are the first The minimum and maximum eigenvalues ​​of the spatial information matrix of the test markers; It is the first The number of test markers at the th ... The gaze direction vector in each view; It is the first The principal uncertainty axis vector of the test marker points; It is the geometric gain constant; It is the modulo operator; It is the preset first minute value, and its range is [value range missing]. .

3. The motion capture method based on multi-view images according to claim 1, characterized in that, The construction of the spatial information matrix of the marker point to be measured based on the line-of-sight vector includes: For each of the views, the projection matrix is ​​calculated using its line-of-sight vector toward the marker to be measured. The projection matrices corresponding to all the views are summed to obtain the spatial information matrix of the marker point to be tested.

4. The motion capture method based on multi-view images according to claim 1, characterized in that, The imaging quality weights satisfy the following relationship: ； in, It is the first The number of test markers at the th ... Image quality weights in each view; It is a reference distance; It is the first The number of test markers at the th ... Imaging distance in each view; It is the distortion suppression coefficient; It is the first The number of test markers at the th ... Roundness of the light spot in each view; It is the preset second minute value, and its range is [value range missing]. .

5. The motion capture method based on multi-view images according to claim 1, characterized in that, The process of obtaining the imaging distance and spot roundness of the target marker point in each view includes: Extract the camera optical center coordinates corresponding to each view, use the line-of-sight vector of each view, and perform intersection calculation using linear triangulation to obtain the initial three-dimensional coordinates of the marker point to be measured. Calculate the Euclidean distance between the camera optical center corresponding to each view and the initial three-dimensional coordinates, and use it as the imaging distance of the marker point to be measured in each view. Ellipse fitting is performed on the light spot corresponding to the test mark point in each view to obtain the fitted ellipse. The ratio of the minor axis length to the major axis length of the fitted ellipse is recorded as the light spot roundness of the test mark point.

6. The motion capture method based on multi-view images according to claim 1, characterized in that, The construction of a comprehensive weight matrix based on the geometric weights and the imaging quality weights includes: The product of the geometric weight and the imaging quality weight of the target marker point in each view is calculated to obtain the comprehensive weight of each view; A diagonal matrix is ​​constructed based on the comprehensive weights of each view, and the comprehensive weights are filled onto the diagonal to obtain the comprehensive weight matrix of the marker point to be tested.

7. The motion capture method based on multi-view images according to claim 1, characterized in that, The linear equation satisfies the following relation: ； in, It is the first An iterative correction vector for the three-dimensional coordinates of the test marker points; It is the first Jacobian matrix of the test markers; It is the vector transpose symbol; It is the first The comprehensive weight matrix of the test markers; It is the first The reprojection residual vector of each marker point to be tested; It is the first of the weighted Hessian approximation matrices. The weighted Hessian approximation matrix is ​​represented by eigenvalues. ; It is related to eigenvalues The corresponding unit eigenvector; It is an adaptive damping parameter; It is the preset third minute value, and its range is [value range missing]. ; It is the number of characteristic directions in three-dimensional space.

8. The motion capture method based on multi-view images according to claim 7, characterized in that, Solving the system of linear equations to obtain the three-dimensional coordinates of the target marker includes: The Gauss-Newton method is used as an iterative strategy to update the reprojection residual vector and Jacobian matrix in each iteration to construct the linear equation system. The linear equations are solved using Cholesky decomposition or LDLT decomposition to obtain the coordinate correction for the current iteration. The iterative three-dimensional coordinates of the marker point to be tested are updated based on the coordinate correction amount until the iteration condition is met, and the iterative three-dimensional coordinates at the point where iteration stops are determined as the three-dimensional coordinates of the marker point to be tested.

9. A motion capture system based on multi-view images, characterized in that, The motion capture system based on multi-view images includes a processor and a memory. The memory stores computer program instructions, which, when executed by the processor, implement a motion capture method based on multi-view images according to any one of claims 1-8.

Images