3D human motion digital twin method and system based on multi-camera hardware synchronization

Abstract

This invention discloses a 3D human motion digital twin method and system based on multi-camera hardware synchronization, belonging to the field of digital twin technology. It solves the technical problems of high cost of traditional optical motion capture, lack of depth information in monocular vision, and insufficient accuracy of 3D pose acquisition under limited viewpoints. The key points of its technical solution are that it only requires a small number of ordinary industrial cameras, combined with a low-cost TTL synchronization triggering scheme, to achieve reconstruction accuracy close to that of expensive optical motion capture systems, greatly reducing the system deployment threshold. Through virtual reprojection technology, it breaks through the limitations of physical site and number of cameras, and can simulate data from multiple virtual observation perspectives, effectively solving the pain points of single data perspective and poor generalization in traditional methods. The high-precision reconstructed 3D human skeleton data sequence obtained can be transmitted to the Unity 3D virtual engine environment to realize real-time motion driving and motion state reproduction of the digital twin virtual human model.

Description

Technical Field

[0001] This application relates to the technical fields of computer vision, human kinematics and digital twins, and in particular to a 3D human motion digital twin method and system based on multi-camera hardware synchronization. Background Technology

[0002] For limb function recovery after orthopedic surgery, doctors need to obtain millimeter-level quantitative data on the patient's joint range of motion, limb movement trajectory, and changes in the body's center of gravity. [4] Traditional rehabilitation assessments primarily rely on doctors' visual estimation or goniometer measurements, which suffer from significant subjective errors and a lack of continuous dynamic data recording. With the rapid development of artificial intelligence and the Internet of Things (IoT), precise capture and three-dimensional quantitative analysis of human movement have become core common requirements in the field of computer vision. Human posture assessment aims to reconstruct the spatial position of human joints and skeletal structure through sensor data. This technology has been widely applied to several key industries, including smart healthcare, public safety, competitive sports, and virtual reality. [1] However, different application scenarios place different and stringent requirements on the real-time performance, accuracy, and environmental adaptability of attitude capture systems, and traditional single-modal monitoring methods are difficult to meet the needs of increasingly complex scenarios.

[0003] In the field of public safety and health monitoring, the recognition of abnormal human behavior (such as falls, sudden illness with convulsions, violent conflicts, etc.) is characterized by its extreme suddenness and dynamic non-steady-state nature. According to the World Health Organization, falls are the second leading cause of accidental death, especially among elderly people living alone or those in high-risk occupations. Millisecond-level posture capture and anomaly detection are crucial for gaining golden rescue time. [2] Most existing monitoring systems rely on two-dimensional video playback and lack the ability to perceive three-dimensional spatial depth. This makes it difficult to achieve accurate real-time early warnings in complex and obstructed environments, often resulting in missed or false alarms and failing to meet the reliability requirements of 24 / 7 security monitoring. [3] .

[0004] In the fields of medical rehabilitation and sports science, quantitative assessment of human movement is the core basis for developing treatment plans and optimizing training strategies. This is particularly relevant for gait correction in stroke sequelae and Parkinson's disease patients, and for addressing the lack of dynamic data support. Furthermore, the analysis of minute deviations in movement precision in competitive sports poses extremely high challenges to the spatial accuracy of motion capture systems. [5] .

[0005] In the fields of human-computer interaction and digital twins, building high-fidelity virtual digital humans requires real-time, smooth, and jitter-free 3D skeletal data streams. With the rise of the metaverse concept, users' demands for immersive experiences in replicating realistic body movements in virtual space are constantly increasing. However, current contactless interaction technologies often suffer from data loss when handling large movements or special perspectives (such as overhead views or blind spots), leading to distorted virtual avatars and severely impacting the interactive experience and system usability. [6] .

[0006] In conclusion, although the application scenarios vary, all fields urgently require a universal 3D human motion digital twin method. This digital twin method needs to have the ability to capture high-speed motion (to meet safety requirements), achieve medical-grade measurement accuracy (to meet rehabilitation requirements), and maintain low cost and wearable characteristics (to meet interaction requirements).

[0007] In terms of data acquisition, while active optical motion capture systems, such as Vicon, can provide high-precision 3D data at the sub-millimeter level, they heavily rely on expensive infrared camera arrays (typically costing hundreds of thousands to millions of yuan) and specific indoor environments. [7][8] Furthermore, participants must wear specially designed clothing with reflective markings, a highly invasive requirement that is extremely difficult to implement in daily care of the elderly or in non-cooperative scenarios. [9] .

[0008] While deep learning-based monocular vision solutions (such as MediaPipe and OpenPose) offer advantages like low cost and wearable devices, they essentially infer three-dimensional structures from two-dimensional images, resulting in severe depth information ambiguity and difficulty in obtaining true physical spatial absolute coordinates.

[10] Especially in complex scenarios such as falls and lying down, there is severe self-occlusion between human limbs. Monocular algorithms are prone to missing key points or misaligning the spatial positions of limbs, resulting in a high false alarm rate.

[11] .

[0009] Existing low-cost multi-camera monitoring solutions mostly rely on software triggering or network synchronization (NTP, Network Time Protocol), with synchronization errors typically in the millisecond range. For high-speed, unsteady-state movements (within 0.5 seconds) that occur instantaneously, such as falls, this millisecond-level time difference can lead to mismatches in multi-view images, resulting in "skeleton disintegration" or high-frequency noise in the 3D reconstruction results.

[12] Furthermore, due to limitations in the physical acquisition environment, training data typically only covers a limited number of fixed viewpoints and lacks samples for special viewpoints (such as overhead views or blind spots), which limits the algorithm's generalization ability in complex environments.

[0010] On the one hand, in multi-view geometry-based 3D reconstruction, strict temporal alignment of images from different viewpoints is the physical basis for ensuring the accuracy of triangulation. However, existing multi-camera acquisition schemes have significant limitations in balancing cost and accuracy.

[0011] To reduce hardware costs, most mainstream security monitoring or low-cost motion capture solutions currently use Network Time Protocol (NTP) or software command-triggered synchronization. Limited by operating system scheduling latency and network transmission fluctuations, the synchronization error of these methods is typically in the millisecond range (10ms–100ms). This error is acceptable for stationary or slow-moving motion, but it is insufficient for high-precision measurement requirements.

[13] .

[0012] A human fall is an extremely rapid, non-steady-state process (typically completed within 0.3 to 0.5 seconds). If there is a millisecond-level time difference between cameras, the "same frame" image may record postures at different physical moments (e.g., camera A captures the person just losing balance, while camera B captures the person already on the ground). This spatiotemporal mismatch directly violates the epipolar constraints of multi-view geometry, causing triangulation algorithms to fail to calculate convergent intersection points.

[14] This results in severe motion artifacts, high-frequency jitter of the skeleton, and even disintegration of the three-dimensional structure in the reconstruction results, rendering them unusable as effective detection criteria.

[15] .

[0013] On the other hand, the performance of deep learning algorithms is highly dependent on the scale and diversity of training data. However, in the field of human fall detection and posture analysis, high-quality training samples generally suffer from a severe scarcity of viewing angles. Limited by physical space and hardware costs, actual data acquisition systems typically only cover 3-4 fixed horizontal viewing angles. This results in existing fall databases generally having large visual blind spots, such as large-angle overhead views, upward views, or specific blind spots. Furthermore, when the installation height or angle of the camera in the actual application scenario is inconsistent with the distribution of the training dataset (for example, home surveillance often uses ceiling-mounted overhead cameras, while the training data is mostly eye-level), the problem becomes even more pronounced.

[16] The algorithm's generalization ability will decrease sharply.

[17] This leads to missed detections.

[0014] To expand data, existing technologies typically employ two methods, but both have significant drawbacks: Firstly, there's traditional 2D image enhancement, which involves simple rotation, scaling, or cropping of the image. However, planar rotation of a 2D image cannot simulate perspective transformations in real 3D space, resulting in samples that don't conform to physical imaging principles and thus contributing little to improving the model's 3D perception capabilities.

[18] .

[0015] Secondly, virtual human data can be directly generated using game engines (such as Unity). Although the field of view is unlimited, the synthesized images have significant differences between the real and virtual domains in terms of texture, lighting, and background, making it difficult to directly transfer models trained on virtual data to real-world applications.

[19] .

[0016] In summary, further research is needed to address the challenges of high costs associated with traditional optical motion capture, lack of depth information in monocular vision, and insufficient accuracy in acquiring 3D pose under limited field of view.

[0017] References: [1]Gong, W., Zhang, X., Gonzàlez, J., et al. Human pose estimation from monocular images: A comprehensive survey. Sensors, 2016, 16(12): 1966. [2]World Health Organization. WHO global report on falls prevention in older age. World Health Organization, 2007. [3]Noury, N., Rumeau, P., Bourke, AK, et al. A proposal for the classification of fall detection systems. IRBM, 2024. [4]Muro-de-la-Herran, A., Garcia-Zapirain, B.,&Mendez-Zorrilla, A.Gait analysis methods: An overview of wearable and non-wearable systems,highlighting clinical applications. Sensors, 2014, 14(2): 3362-3394. [5]Cappozzo, A., Della Croce, U., Leardini, A.,&Chiari, L. Humanmovement analysis using stereophotogrammetry. Gait&posture, 2005, 21(2): 186-219. [6]Menolotto, M., Komaris, D. S., Tedesco, S., et al. Motion capturetechnology in industrial applications: A systematic review. Sensors, 2020, 20(19): 5687. [7]Rodrigues T B, Catháin C Ó, Devine D, et al. An evaluation of a 3Dmultimodal marker-less motion analysis system[C] / / Proceedings of the 10th ACMmultimedia systems Conference. 2019: 213-221. [8]Wade L, Needham L, McGuigan P, et al. Applications and limitationsof current markerless motion capture methods for clinical gait biomechanics[J]. PeerJ, 2022, 10: e12995. [9]Scataglini S, Abts E, Van Bocxlaer C, et al. Accuracy, validity,and reliability of markerless camera-based 3D motion capture systems versusmarker-based 3D motion capture systems in gait analysis: a systematic reviewand meta-analysis[J]. Sensors, 2024, 24(11): 3686.

[10] Wang J, Tan S, Zhen X, et al. Deep 3D human pose estimation: Areview[J]. Computer Vision and Image Understanding, 2021, 210: 103225.

[11] Cheng Y, Yang B, Wang B, et al. Occlusion-aware networks for 3dhuman pose estimation in video[C] / / Proceedings of the IEEE / CVF internationalconference on computer vision. 2019: 723-732.

[12] Takahashi, K., et al. (2018). "3D Human Pose Estimation WithMultiple Unsynchronized and Uncalibrated Cameras." CVPR Workshops.

[13] Seyfu M K, Yang Y S. A Stereo Synchronization Method forConsumer-Grade Video Cameras to Measure Multi-Target 3D Displacement UsingImage Processing in Shake Table Experiments[J]. Sensors, 2025, 25(17): 5535.

[14] Vo M, Sheikh Y, Narasimhan S G. Spatiotemporal bundle adjustmentfor dynamic 3d human reconstruction in the wild[J]. IEEE Transactions onPattern Analysis and Machine Intelligence, 2020, 44(2): 1066-1080.

[15] Zhou X, Dai Y, Qin H, et al. Subframe-level synchronization in multi-camera system using time-calibrated video[J]. Sensors, 2024, 24(21): 6975.

[16] Purkrabek M, Matas J. Improving 2D human pose estimation in rare camera views with synthetic data[C] / / 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG). IEEE, 2024: 1-9.

[17] Lee H, Ryu J. Toward Efficient Generalization in 3D Human Pose Estimation via a Canonical Domain Approach[J]. IEEE Access, 2025.

[18] Gong K, Zhang J, Feng J. Poseaug: A differentiable pose augmentation framework for 3d human pose estimation[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 8575-8584.

[19] Zakharov S. Learning to Estimate 3D Object Pose from Synthetic Data[D]. Technische Universität München, 2020. Summary of the Invention

[0018] This application provides a 3D human motion digital twin method and system based on multi-camera hardware synchronization. Its technical objective is to achieve high-precision 3D human skeleton reconstruction with microsecond-level synchronization through a low-cost hardware solution. Based on a virtual camera model and spatial reprojection mechanism, it realizes multi-view mapping and visualization of the human 3D skeleton in virtual space, thereby improving the stability of human posture analysis under complex observation conditions and supporting virtual human driving and digital twin display in application scenarios such as public safety monitoring and medical rehabilitation assessment.

[0019] The above-mentioned technical objective of this application is achieved through the following technical solution: A 3D human motion digital twin method based on multi-camera hardware synchronization includes: Step S1: Construct a multi-camera synchronous acquisition array; Step S2: Acquire multi-view image sequences of the calibration reference object using a multi-camera synchronous acquisition array, obtain the corner features of the calibration reference object based on the multi-view image sequences, and complete the absolute alignment of the multi-camera world coordinate system based on the corner features and the horizontal reference plane to obtain the final coordinate system; Step S3: Acquire time-aligned multi-view human image sequences through a multi-camera synchronous acquisition array, extract features from the multi-view human image sequences, and obtain the two-dimensional pixel coordinates and confidence scores of each joint point; Step S4: In the final coordinate system, based on the two-dimensional pixel coordinates and their confidence levels of each joint, multi-view triangulation is performed using the weighted DLT-SVD method to reconstruct the three-dimensional skeleton vector of the real human body, thus obtaining the reconstructed three-dimensional skeleton vector. Step S5: Construct a three-dimensional spatial perspective projection model of the virtual camera based on the reconstructed three-dimensional skeleton vector, and perform spatial reprojection on the reconstructed three-dimensional skeleton vector based on the three-dimensional spatial perspective projection model to generate two-dimensional pose features from different viewing angles. Step S6: Perform coordinate system transformation on the reconstructed 3D skeleton vector to map the reconstructed 3D skeleton vector to the standard bone coordinates of the virtual engine, and realize high-fidelity driving of the virtual digital human based on the standard bone coordinates.

[0020] Preferably, the construction of the multi-camera synchronous acquisition array includes: selecting N industrial cameras equipped with global shutter sensors, the N cameras being networked in a star topology, connecting the GPIO optocoupler isolated trigger interfaces of all cameras in parallel to a signal generator via shielded twisted-pair cables, the signal generator being programmed to output a TTL level rectangular wave pulse signal with an adjustable frequency and a duty cycle of 50%, and all cameras being uniformly configured to rise-edge trigger mode.

[0021] Preferably, step S2 includes: A calibration reference object is freely moved within the common field of view of N cameras to acquire a multi-view image sequence of the calibration reference object; among the N cameras, one main camera is selected and the rest are auxiliary cameras; Based on the multi-view image sequence, corner features are extracted from the synchronized images of each camera viewpoint, and then feature point pairs corresponding to the corner features are extracted. Based on the feature point pairs, with the optical center of the main camera as the initial origin, the relative rotation matrix and relative translation vector of each auxiliary camera relative to the main camera are calculated iteratively, and the relative spatial topology of N cameras is established according to the relative rotation matrix and the relative translation vector. The calibration reference is placed on a preset horizontal reference surface. Multiple coplanar feature points on the horizontal reference surface are extracted. The spatial plane equation is fitted based on the coplanar feature points. The normal vector perpendicular to the horizontal reference surface is calculated based on the spatial plane equation. The rotation angle between the normal vector and the main camera relative to the Z-axis of the coordinate system is calculated. A global rotation matrix and a global translation vector are constructed based on the rotation angle. The relative coordinate system of the main camera is subjected to a rigid body transformation based on the relative rotation matrix, the relative translation vector, the global rotation matrix, and the global translation vector, so that the origin of the final coordinate system is in contact with the horizontal reference plane, and the Z-axis of the final coordinate system is parallel to the direction of gravity and upward, thus completing the absolute alignment of the world coordinate systems of the multi-camera system. Wherein, any point in the final coordinate system Rather than in the Two-dimensional pixel coordinates in the camera image plane The following pinhole perspective projection model is represented as: ; in, Represents the global projection matrix. Represents the global rotation matrix. Represents the global translation vector. Indicates the first The relative rotation matrix of each camera, Indicates the first The relative translation vector of each camera, Indicates the first The intrinsic parameter matrix obtained from camera calibration; Indicates the first Depth scale factor of each camera projection Indicates the first Homogeneous pixel coordinates of each camera; A vector representing the actual three-dimensional skeleton of the human body; Let the absolute extrinsic parameter matrix after calibration and integration be The pinhole perspective projection model is then represented as: ; in, Indicates the first Rotation matrix of each camera, Indicates the first The absolute extrinsic translation vector of each camera.

[0022] Preferably, step S4 includes: Two-dimensional pixel coordinates of human joints with the same name in N camera images and their corresponding confidence levels Extract; In the final coordinate system, for any unknown three-dimensional spatial joint... In its first The homogeneous pixel coordinates in the camera projection plane are The nonlinear mapping relationship obtained from the pinhole perspective projection model is as follows: ; in, For the first Global projection matrix of each camera; The nonlinear mapping relationship is converted into a linear constraint by using the three-dimensional vector outer product, and is expressed as follows: ; make Represents the global projection matrix The first in The row vectors, where the linear constraints are expanded by cross product to construct a system of linear equations, are expressed as follows: ; Construct an overdetermined linear matrix based on the system of linear equations, denoted as: ;in, Denotes the overdetermined linear matrix; The overdetermined linear matrix is ​​left-multiplied by a diagonal weight matrix. The weighted overdetermined linear matrix is ​​obtained, denoted as: ;in, This represents a diagonal weight matrix, where the diagonal elements are the keypoint confidence scores from each camera's viewpoint. constitute; Factoring the weighted overdetermined linear matrix using singular value decomposition, we can express the following: ;in, Describes a left singular matrix. Represents a right singular matrix; Then, if the constraints are satisfied Under the premise that the global optimal solution of the weighted overdetermined linear matrix is ​​the right singular matrix. The right singular vector corresponding to the smallest singular value in the solution is the homogeneous coordinate vector of the global optimal solution. Dividing each term of the homogeneous coordinate vector by its fourth component yields the three-dimensional spatial joint. Three-dimensional absolute coordinates with millimeter-level precision in the real physical world; The three-dimensional skeleton vector of the real human body is reconstructed based on the three-dimensional absolute coordinates of each joint, resulting in the reconstructed three-dimensional skeleton vector.

[0023] Preferably, step S5 includes: The intrinsic parameter matrix of the virtual camera is configured according to the target application scenario. A 3D spatial perspective projection model of the virtual camera is constructed based on the intrinsic parameter matrix and the reconstructed 3D skeleton vector. This 3D spatial perspective projection model is represented as follows: ; in, Indicates the virtual projection depth; , where is the homogeneous planar coordinate of a point in the generated two-dimensional image; Represents the reconstructed 3D skeleton vector; The rotation matrix represents the virtual camera; Represents the translation vector of the virtual camera; The intrinsic parameter matrix representing the virtual camera; Using the three-dimensional centroid of the human body as the center of a sphere, the sampling space of the virtual camera surrounding the three-dimensional centroid of the human body is constructed according to the aforementioned three-dimensional spatial perspective projection model; By discretizing and traversing the extrinsic parameter matrix of the virtual camera within the sampling space to dynamically change the spatial observation position and rotation / pitch angle of the virtual camera, the virtual two-dimensional projection feature point sequence containing ceiling-view, ground-view, and arbitrary blind-angle orientations in the reconstructed three-dimensional skeleton vector is simulated and generated. Two-dimensional pose features are generated from different viewing angles based on the virtual two-dimensional projection feature point sequence.

[0024] Preferably, step S6 includes: By using matrix transformation, the single-frame 3D points in the reconstructed 3D skeleton vector are mapped to the standard skeleton coordinates of the virtual engine, as follows: ; in, Represents standard skeletal coordinates. Represents a single frame of 3D points. The scaling factor represents the conversion factor from physical units to the virtual engine's spatial scale. The standard skeletal coordinates of the left and right joints of the pelvis are extracted to calculate the horizontal width vector, and the standard skeletal coordinates of the pelvic center and neck nodes are extracted to calculate the vertical height vector. The orthogonal orientation vector of the torso is derived from the cross product of the horizontal width vector and the vertical height vector. A reference rotation quaternion of the root node is then constructed based on this orthogonal orientation vector. The orthogonal orientation vector is expressed as: , Represents a horizontally wide vector. Represents the vertical height vector; The root node implements high-fidelity driving of the virtual digital human based on the reference quaternion.

[0025] Preferably, step S6 further includes: for the curvature of multiple vertebrae, the reference rotation quaternion of the root node is evenly distributed to the corresponding sub-bone nodes of the lumbar and thoracic vertebrae through piecewise interpolation weights to achieve natural human torso movement restoration.

[0026] Preferably, step S6 further includes: The reconstructed 3D skeleton vectors extract the direction vectors of the upper arm, thigh, forearm, and lower leg. Then, the normal vectors of the joints in the arms and legs are calculated and represented as follows: , This represents the direction vector of the upper arm or the direction vector of the thigh. This represents the forearm direction vector or the lower leg direction vector. when When the rotation exceeds the preset safe rotation threshold, a standard fixed smoothing coefficient is used; when... When the rotation speed is less than the preset safe rotation threshold, the smoothing coefficient will be adjusted accordingly. The square relationship undergoes nonlinear decay, i.e. , Represents the smoothing coefficient. Represents a proportionality constant; The three-dimensional target rotation posture of the corresponding joint is constructed based on the upper arm direction vector, the thigh direction vector and their respective normal vectors, to obtain the target rotation quaternion, i.e. the target rotation amount; In each frame of pose update, the dynamically calculated smoothing coefficient is used as a dynamic interpolation weight to drive the current rotation of the model skeleton to a non-linear smooth transition to the target rotation.

[0027] Preferably, N=4.

[0028] A 3D human motion digital twin system based on multi-camera hardware synchronization, the 3D human motion digital twin system is used to implement the aforementioned 3D human motion digital twin method, the 3D human motion digital twin system includes: The camera array construction module is used to build a multi-camera synchronous acquisition array. The calibration and alignment module acquires multi-view image sequences of the calibration reference object through a multi-camera synchronous acquisition array, obtains the corner features of the calibration reference object based on the multi-view image sequences, and completes the absolute alignment of the multi-camera world coordinate system based on the corner features and the horizontal reference plane to obtain the final coordinate system. The feature extraction module acquires time-aligned multi-view human image sequences through a multi-camera synchronous acquisition array, extracts features from the multi-view human image sequences, and obtains the two-dimensional pixel coordinates and confidence scores of each joint point. The 3D reconstruction module, in the final coordinate system, based on the two-dimensional pixel coordinates and their confidence levels of each joint, performs multi-view triangulation using the weighted DLT-SVD method to reconstruct the 3D skeleton vector of the real human body, thus obtaining the reconstructed 3D skeleton vector. The virtual reprojection module constructs a three-dimensional spatial perspective projection model of the virtual camera based on the reconstructed three-dimensional skeleton vector, and performs spatial reprojection on the reconstructed three-dimensional skeleton vector based on the three-dimensional spatial perspective projection model to generate two-dimensional pose features from different viewing angles. The digital twin module performs coordinate system transformation on the reconstructed 3D skeleton vector to map the reconstructed 3D skeleton vector to the standard skeletal coordinates of the virtual engine, and realizes high-fidelity driving of the virtual digital human based on the standard skeletal coordinates.

[0029] The above technical solution can achieve at least some of the following technical effects: (1) This application only requires a small number (e.g., 4) of ordinary industrial cameras, combined with a low-cost TTL synchronous triggering scheme, to achieve reconstruction accuracy close to that of expensive optical motion capture systems, which greatly reduces the system deployment threshold.

[0030] (2) Hardware-level microsecond synchronization is adopted, which effectively solves the problem of 3D reconstruction failure caused by camera asynchrony during rapid movements such as falls in the elderly.

[0031] (3) By using virtual reprojection technology, the limitations of physical site and number of cameras are overcome, and data from multiple virtual observation perspectives can be simulated, effectively solving the pain points of single data perspective and poor generalization in traditional methods.

[0032] (4) The high-precision reconstructed three-dimensional human skeleton data sequence obtained by the solution can be transmitted to the Unity three-dimensional virtual engine environment to realize the real-time motion driving and motion state reproduction of the digital twin virtual human model. Attached Figure Description

[0033] Figure 1 This is a flowchart of the 3D human motion digital twin method based on multi-camera hardware synchronization described in the embodiments of this application. Detailed Implementation

[0034] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0035] like Figure 1 As shown, the 3D human motion digital twin method based on multi-camera hardware synchronization described in this application includes: Step S1: Construct a multi-camera synchronous acquisition array.

[0036] Specifically, to completely eliminate motion blur and spatiotemporal misalignment caused by high-speed motion at the physical level, a high-precision hardware synchronous acquisition system was constructed in step S1. This includes selecting N industrial cameras equipped with global shutter sensors. Compared to rolling shutters, global shutters can achieve simultaneous exposure of all pixels, fundamentally eliminating the rolling shutter effect of fast-moving objects. In terms of networking, a star topology is used, connecting the GPIO optocoupler-isolated trigger interfaces of all cameras in parallel to a high-precision signal generator via shielded twisted-pair cables. The signal generator, serving as the master clock source, is programmed to output a TTL-level rectangular wave pulse signal with an adjustable frequency (e.g., 20Hz) and a 50% duty cycle. All cameras are uniformly configured to use rising-edge triggering mode.

[0037] The moment the rectangular wave pulse signal voltage transitions from low to high (rise time < 1 μs), the electronic shutters of all cameras are forcibly triggered to open simultaneously. Oscilloscope measurements show that this mechanism can control the exposure time difference between multiple cameras to within 10 microseconds, ensuring that the images captured from the four perspectives are strictly aligned on the time axis at the high-speed instant of the fall, providing a reliable frozen frame for subsequent multi-view geometry calculations.

[0038] Step S2: Acquire multi-view image sequences of the calibration reference object using a multi-camera synchronous acquisition array, obtain the corner features of the calibration reference object based on the multi-view image sequences, and complete the absolute alignment of the multi-camera world coordinate system based on the corner features and the horizontal reference surface to obtain the final coordinate system.

[0039] In order to establish the spatial topological relationship between multiple cameras and provide a unified metric for human pose analysis, this application implements a step-by-step global spatial calibration.

[0040] In one embodiment, the camera's intrinsic parameters are first calibrated and its phase pose is calculated. A multi-view image sequence of the calibration reference (such as a ChArUco calibration array, which does not need to be fixed in a specific spatial position) is obtained by freely moving a high-precision calibration reference object within the common field of view of N cameras. Compared to traditional checkerboard patterns, the ChArUco calibration array can still provide a globally unique identifier using its internal ArUco QR code, even under local occlusion or edge blurring conditions in multi-camera viewpoints, ensuring extremely high feature matching robustness.

[0041] Then, based on the multi-view image sequence, corner features are extracted from the synchronized images of each camera viewpoint, and then feature point pairs corresponding to the corner features are extracted.

[0042] Based on the aforementioned feature point pairs, using bundle adjustment and epipolar geometry constraints, with the optical center of the main camera as the initial origin, the relative rotation matrix and relative translation vector of each auxiliary camera relative to the main camera are iteratively calculated. The relative spatial topology of the N cameras is then established based on the relative rotation matrix and the relative translation vector.

[0043] Based on this, since the origin and orientation of the relatively calibrated coordinate system are arbitrary (dependent on the main camera's perspective), it lacks the scale and gravity direction perception of the real world, and cannot meet the stringent measurement requirements for the absolute height (Z-axis) of human joints in fall detection and early warning or medical rehabilitation assessment.

[0044] Therefore, a calibration reference is placed on a preset horizontal reference plane (or the physical ground plane of the real scene). Multiple coplanar feature points on the horizontal reference plane are extracted. Based on the coplanar feature points, a random sampling consensus algorithm is used to fit the spatial plane equation. The normal vector perpendicular to the horizontal reference plane is calculated based on the spatial plane equation. The rotation angle between the normal vector and the main camera relative to the Z-axis of the coordinate system is calculated. A global rotation matrix and a global translation vector are constructed based on the rotation angle. A rigid body transformation is performed on the main camera's relative coordinate system based on the relative rotation matrix, the relative translation vector, the global rotation matrix, and the global translation vector. This process ensures that the origin of the final coordinate system is aligned with the horizontal reference plane (or the physical ground plane, Z=0), and the Z-axis of the final coordinate system is parallel to the direction of gravity and points upwards, thus achieving absolute alignment of the multi-camera world coordinate systems.

[0045] After the two-step correction of relative solution and global alignment described above, any point in the physical space under the final coordinate system Rather than in the Two-dimensional pixel coordinates in the camera image plane The following pinhole perspective projection model is represented as: ; in, Represents the global projection matrix. Represents the global rotation matrix. Represents the global translation vector. Indicates the first The relative rotation matrix of each camera, Indicates the first The relative translation vector of each camera, Indicates the first The intrinsic parameter matrix obtained from camera calibration; Indicates the first Unknown depth scale factor of each camera projection Indicates the first Homogeneous pixel coordinates of each camera; A vector representing the actual three-dimensional skeleton of the human body.

[0046] Let the absolute extrinsic parameter matrix after calibration and integration be The pinhole perspective projection model is then represented as: ; in, Indicates the first Rotation matrix of each camera, Indicates the first The absolute extrinsic translation vector of each camera.

[0047] The above-mentioned multi-camera world coordinate system alignment method not only breaks through the limitation of traditional optical motion capture systems that heavily rely on specific calibration sites, ensuring sub-millimeter-level high-precision calculation of the relative positions of multiple cameras, but more importantly, it provides a zero-cost physical coordinate system absolute alignment scheme, laying the physical measurement foundation for subsequent digital twin models.

[0048] Step S3: Acquire time-aligned multi-view human image sequences through a multi-camera synchronous acquisition array, extract features from the multi-view human image sequences, and obtain the two-dimensional pixel coordinates and confidence scores of each joint point.

[0049] Step S4: In the final coordinate system, based on the two-dimensional pixel coordinates and their confidence levels of each joint point, multi-view triangulation is performed using the weighted DLT-SVD (Direct Linear Transform-Singular Value Decomposition) method to reconstruct the three-dimensional skeleton vector of the real human body, thus obtaining the reconstructed three-dimensional skeleton vector.

[0050] To address the depth blur problem inherent in monocular vision and the common problem of severe limb occlusion during complex movements (such as falls, Tai Chi turns, etc.), this application employs a confidence-weighted direct linear transformation (DLT) combined with a singular value decomposition (SVD) algorithm to perform optimal solution of occlusion-resistant 3D physical coordinates.

[0051] In one embodiment, a deep learning pose estimation network (such as the MediaPipe architecture) is first used to calculate the two-dimensional pixel coordinates of corresponding human joints in N camera images. and their corresponding confidence levels Extraction, confidence level The detection uncertainty caused by limited viewpoint or object occlusion was quantified.

[0052] In the final coordinate system, for any unknown three-dimensional spatial joint... In its first The homogeneous pixel coordinates in the camera projection plane are The nonlinear mapping relationship obtained from the pinhole perspective projection model is as follows: ; in, For the first The global projection matrix of each camera. In this application, we take an example where N is 4, i.e., a total of 4 cameras. It is a 3×4 global projection matrix.

[0053] To eliminate unknown depth scale factors This application utilizes the collinear cancellation property of the cross product of three-dimensional vectors, i.e. This transforms the nonlinear mapping relationship into a strict linear constraint.

[0054] make Represents the global projection matrix The first in The row vectors, where the linear constraints are expanded by cross product to construct a system of linear equations, are expressed as follows: .

[0055] Furthermore, to address the errors caused by blind spots in some viewpoints and to fully utilize the redundancy features of multiple viewpoints, this application simultaneously solves the linear constraint equations of all N visible cameras, constructing a system of the form... of An overdetermined linear matrix; where, That is, to represent the overdetermined linear matrix.

[0056] Considering measurement noise and severe occlusion interference, this application innovatively introduces a diagonal weight matrix. The overdetermined linear matrix is ​​left-multiplied by a diagonal weight matrix. The weighted overdetermined linear matrix is ​​obtained, denoted as: ;in, The diagonal elements are determined by the confidence of the key points from each camera's perspective. constitute.

[0057] Based on the least squares principle, this application uses singular value decomposition (SVD) to factorize the weighted overdetermined linear matrix, i.e. ;in, Describes a left singular matrix. Let represent a right singular matrix. At this point, the constraints are satisfied. Under the premise that the global optimal solution of the weighted overdetermined linear matrix is ​​the right singular matrix. The right singular vector (i.e., matrix) corresponding to the smallest singular value in the middle The last line), the global optimal solution is the homogeneous coordinate vector.

[0058] Finally, by performing perspective division on the homogeneous coordinate vector (i.e., dividing each term by its fourth component), the three-dimensional spatial joint points can be accurately recovered. Three-dimensional absolute coordinates with millimeter-level precision in the real physical world Then, based on the three-dimensional absolute coordinates of each joint, the three-dimensional skeleton vector of the real human body is reconstructed to obtain the reconstructed three-dimensional skeleton vector.

[0059] The aforementioned DLT-SVD method, through a weighted SVD mechanism, automatically reduces or isolates camera viewpoints with obstructed or extremely low confidence during matrix operations, thereby enabling it to output a highly robust 3D skeleton flow even in complex environments with high occlusion.

[0060] Step S5: Construct a three-dimensional spatial perspective projection model of the virtual camera based on the reconstructed three-dimensional skeleton vector, and perform spatial reprojection on the reconstructed three-dimensional skeleton vector based on the three-dimensional spatial perspective projection model to generate two-dimensional pose features from different viewing angles.

[0061] To address the fundamental flaw in practical applications where physical acquisition arrays are limited by site space and hardware costs, resulting in training data with a single perspective and significant blind spots (such as large-angle overhead, overhead, or specific physical blind spots), thus causing poor generalization ability of pose estimation algorithms, this application proposes a virtual perspective generation method based on the reprojection of reconstructed 3D skeleton vectors. In one embodiment, a high-precision 3D skeleton vector of a real human body is reconstructed based on the above reconstruction solution. Construct a parametric virtual camera model in digital three-dimensional space.

[0062] First, the intrinsic parameter matrix of the virtual camera is configured according to the target application scenario to accurately simulate the imaging characteristics of different focal lengths and optical centers. Then, a 3D spatial perspective projection model of the virtual camera is constructed based on the intrinsic parameter matrix and the reconstructed 3D skeleton vector. This 3D spatial perspective projection model is represented as follows: ; in, Indicates the virtual projection depth; , where is the homogeneous planar coordinate of a point in the generated two-dimensional image; Represents the reconstructed 3D skeleton vector; The rotation matrix represents the virtual camera; Represents the translation vector of the virtual camera; This represents the intrinsic parameter matrix of the virtual camera.

[0063] Furthermore, utilizing the three-dimensional spatial perspective projection principle of computer graphics, with the three-dimensional centroid of the human body as the center of a sphere, the sampling space of the virtual camera surrounding the three-dimensional centroid of the human body is constructed according to the aforementioned three-dimensional spatial perspective projection model.

[0064] By discretizing and traversing the extrinsic parameter matrix of the virtual camera within the sampling space, the spatial observation position and rotation / pitch angle of the virtual camera can be dynamically changed (i.e., the extrinsic parameter matrix of the virtual camera can be dynamically changed). The reconstructed 3D skeleton vector contains a sequence of virtual 2D projection feature points from ceiling-view, ground-view, and arbitrary blind-angle orientations. The simulation is then performed. Finally, based on the sequence of virtual two-dimensional projection feature points, two-dimensional pose features are generated from different viewing angles.

[0065] Through the aforementioned reprojection mechanism based on physical laws, this application can realize the skeletal projection mapping of realistically captured, high-quality movements that conform to human biodynamics (such as complex Tai Chi routines or rapid falls) from different viewing angles in virtual space. This overcomes the inherent defect of traditional 2D image enhancement (such as planar rotation and scaling) in failing to simulate perspective deformation in real 3D space, thus establishing a correspondence between virtual 2D projection features and the real 3D skeleton from different perspectives. This method can adapt to special installation viewing conditions such as ceiling mounting, complex obstructions, or vehicle mounting, improving the stability of human posture analysis under different observation angles, thereby enhancing its environmental adaptability in application scenarios such as public safety monitoring and medical rehabilitation assessment.

[0066] Step S6: Perform coordinate system transformation on the reconstructed 3D skeleton vector to map the reconstructed 3D skeleton vector to the standard bone coordinates of the virtual engine, and realize high-fidelity driving of the virtual digital human based on the standard bone coordinates.

[0067] Specifically, after obtaining the physical coordinate sequence in the smoothed, high-precision reconstructed 3D skeleton vector (such as the joint coordinates of consecutive frames calculated through multi-view vision), the skeleton data sequence is imported into the 3D virtual reality engine (Unity3D is used in this application) through a data synchronization interface. In order to drive the virtual digital human model containing a standard skeletal topology, this application constructs an inverse kinematics mapping engine that includes spatial chirality transformation, root node constraints, and adaptive smoothing to prevent deadlock.

[0068] Since the world coordinate system calculated by the visual algorithm is a right-handed coordinate system, while the virtual engine is usually a left-handed coordinate system, a strict spatial chirality transformation and scale normalization must first be performed at the receiving end. Through matrix transformation, the single-frame 3D points in the reconstructed 3D skeleton vector are mapped to the standard skeleton coordinates of the virtual engine, expressed as: ; in, Represents standard skeletal coordinates. Represents a single frame of 3D points. This represents the conversion factor from physical units to the virtual engine's spatial scale, i.e., the scaling ratio. This transformation not only completes the mapping and interchange between the Z-axis and Y-axis, but also perfectly adapts to the requirements of mirror-driven systems through sign reversal.

[0069] Furthermore, when performing skeletal pose repositioning, this application abandons the direct dragging of local coordinates, which easily leads to skeletal stretching and clipping, and instead adopts a rotational skeleton-driven method based on vector cross product. In the pose calculation of the root node and spine, the standard skeletal coordinates of the left and right pelvic joints are extracted to calculate the horizontal width vector, and the standard skeletal coordinates of the pelvic center and neck nodes are extracted to calculate the vertical height vector. The orthogonal orientation vector of the torso is derived from the cross product of the horizontal width vector and the vertical height vector, and the reference rotation quaternion of the root node is constructed based on the orthogonal orientation vector; wherein, the orthogonal orientation vector is expressed as: , Represents a horizontally wide vector. This represents the vertical height vector.

[0070] The root node implements high-fidelity driving of the virtual digital human based on the reference rotation quaternion. That is, firstly, the coordinate positions of the real human joints (such as the pelvis) are extracted, then the overall rotation posture of the human body is derived by calculating the vector cross product, and finally, this overall rotation posture data is sent as an instruction to the root node of the virtual engine, thereby driving the entire virtual digital human to make synchronous movements.

[0071] Furthermore, for the curvature of multiple vertebrae, the reference rotation quaternion of the root node is evenly distributed to the corresponding sub-bone nodes of the lumbar and thoracic vertebrae through piecewise interpolation weights to achieve natural human trunk movement restoration.

[0072] Furthermore, addressing the gimbal lock-up and normal vector abrupt jitter issues that easily occur in limb joints when they tend to be extended, this application proposes an adaptive smooth rotation algorithm based on the sagittal plane. For any limb, the direction vectors of the upper arm, thigh, forearm, and lower leg are extracted using the reconstructed 3D skeleton vector. Then, the normal vectors of the joints in the arm and leg are calculated, and expressed as: , This represents the direction vector of the upper arm or the direction vector of the thigh. This represents the direction vector of the forearm or the direction vector of the lower leg.

[0073] when When the rotation exceeds the preset safe rotation threshold (i.e., the joint is in a significantly bent state and the normal vector is stable), a standard fixed smoothing coefficient is used; when... When the rotation is less than the preset safe rotation threshold (i.e., the limb tends to straighten, and the normal vector has a high risk of sudden change in three-dimensional space), the smoothing coefficient will be adjusted accordingly. The square relationship undergoes nonlinear decay, i.e. , Represents the smoothing coefficient. This represents the proportionality constant.

[0074] The 3D target rotational pose of the corresponding joints is constructed based on the upper arm direction vector, thigh direction vector, and their respective normal vectors, resulting in a target rotation quaternion, i.e., the target rotation amount. Specifically, the normal vector is used as the forward reference axis of the local coordinate system, and the upper arm / thigh direction vector is used as the upward reference axis. The target rotation quaternion is generated through orthogonalization calculation. In the pose update of each frame, the spherical linear interpolation (Slerp) algorithm is used, and the dynamically calculated smoothing coefficient is used as the dynamic interpolation weight to drive the current rotation amount of the model skeleton to transition non-linearly and smoothly to the target rotation amount.

[0075] Through this underlying control logic of "normal response during bending and nonlinear suppression of abrupt changes during straightening," this application, based on principles of graphics and kinematics, essentially eliminates the skeletal flipping and high-frequency twitching commonly seen in virtual characters during extreme limb extension moments such as punching and kicking. Whether in the evaluation of Tai Chi movements requiring extreme smoothness or in the monitoring of highly dynamic instantaneous falls, this application can guarantee high fidelity, clipping-free animation, and extremely low visual latency in the digital twin model's movements.

[0076] The 3D human motion digital twin system based on multi-camera hardware synchronization described in this application includes a camera array construction module, a calibration and alignment module, a feature extraction module, a 3D reconstruction module, a virtual reprojection module, and a digital twin module.

[0077] The camera array building module is used to build multi-camera synchronous acquisition arrays.

[0078] The calibration alignment module is used to acquire multi-view image sequences of the calibration reference object through a multi-camera synchronous acquisition array, obtain the corner features of the calibration reference object based on the multi-view image sequences, and complete the absolute alignment of the multi-camera world coordinate system based on the corner features and the horizontal reference surface to obtain the final coordinate system.

[0079] The feature extraction module is used to acquire time-aligned multi-view human image sequences through a multi-camera synchronous acquisition array, and to extract features from the multi-view human image sequences to obtain the two-dimensional pixel coordinates and confidence scores of each joint point.

[0080] The 3D reconstruction module is used to reconstruct the 3D skeleton vector of the real human body by performing multi-view triangulation using the weighted DLT-SVD method based on the 2D pixel coordinates and confidence levels of each joint in the final coordinate system.

[0081] The virtual reprojection module is used to construct a three-dimensional spatial perspective projection model of the virtual camera based on the reconstructed three-dimensional skeleton vector, and to perform spatial reprojection on the reconstructed three-dimensional skeleton vector based on the three-dimensional spatial perspective projection model to generate two-dimensional pose features from different viewing angles.

[0082] The digital twin module is used to perform coordinate system transformation on the reconstructed 3D skeleton vector to map the reconstructed 3D skeleton vector to the standard bone coordinates of the virtual engine, and realize high-fidelity driving of the virtual digital human based on the standard bone coordinates.

[0083] The above are exemplary embodiments of this application, and the scope of protection of this application is defined by the claims and their equivalents.

Claims

1. A method for 3D human motion digital twin based on multi-camera hardware synchronization, characterized in that, include: Step S1: Construct a multi-camera synchronous acquisition array; Step S2: Acquire multi-view image sequences of the calibration reference object using a multi-camera synchronous acquisition array, obtain the corner features of the calibration reference object based on the multi-view image sequences, and complete the absolute alignment of the multi-camera world coordinate system based on the corner features and the horizontal reference plane to obtain the final coordinate system; Step S3: Acquire time-aligned multi-view human image sequences through a multi-camera synchronous acquisition array, extract features from the multi-view human image sequences, and obtain the two-dimensional pixel coordinates and confidence scores of each joint point; Step S4: In the final coordinate system, based on the two-dimensional pixel coordinates and their confidence levels of each joint, multi-view triangulation is performed using the weighted DLT-SVD method to reconstruct the three-dimensional skeleton vector of the real human body, thus obtaining the reconstructed three-dimensional skeleton vector. Step S5: Construct a three-dimensional spatial perspective projection model of the virtual camera based on the reconstructed three-dimensional skeleton vector, and perform spatial reprojection on the reconstructed three-dimensional skeleton vector based on the three-dimensional spatial perspective projection model to generate two-dimensional pose features from different viewing angles. Step S6: Perform coordinate system transformation on the reconstructed 3D skeleton vector to map the reconstructed 3D skeleton vector to the standard bone coordinates of the virtual engine, and realize high-fidelity driving of the virtual digital human based on the standard bone coordinates.

2. The 3D human motion digital twin method as described in claim 1, characterized in that, The construction of the multi-camera synchronous acquisition array includes: selecting N industrial cameras equipped with global shutter sensors, networking the N cameras in a star topology, connecting the GPIO optocoupler isolated trigger interfaces of all cameras in parallel to a signal generator via shielded twisted-pair cables, programming the signal generator to output a TTL level rectangular wave pulse signal with adjustable frequency and a duty cycle of 50%, and configuring all cameras uniformly in rising edge trigger mode.

3. The 3D human motion digital twin method as described in claim 2, characterized in that, Step S2 includes: A calibration reference object is freely moved within the common field of view of N cameras to acquire a multi-view image sequence of the calibration reference object; among the N cameras, one main camera is selected and the rest are auxiliary cameras; Based on the multi-view image sequence, corner features are extracted from the synchronized images of each camera viewpoint, and then feature point pairs corresponding to the corner features are extracted. Based on the feature point pairs, with the optical center of the main camera as the initial origin, the relative rotation matrix and relative translation vector of each auxiliary camera relative to the main camera are calculated iteratively, and the relative spatial topology of N cameras is established according to the relative rotation matrix and the relative translation vector. The calibration reference is placed on a preset horizontal reference surface. Multiple coplanar feature points on the horizontal reference surface are extracted. The spatial plane equation is fitted based on the coplanar feature points. The normal vector perpendicular to the horizontal reference surface is calculated based on the spatial plane equation. The rotation angle between the normal vector and the main camera relative to the Z-axis of the coordinate system is calculated. A global rotation matrix and a global translation vector are constructed based on the rotation angle. The relative coordinate system of the main camera is subjected to a rigid body transformation based on the relative rotation matrix, the relative translation vector, the global rotation matrix, and the global translation vector, so that the origin of the final coordinate system is in contact with the horizontal reference plane, and the Z-axis of the final coordinate system is parallel to the direction of gravity and upward, thus completing the absolute alignment of the world coordinate systems of the multi-camera system. Wherein, any point in the final coordinate system Rather than in the Two-dimensional pixel coordinates in the camera image plane The following pinhole perspective projection model is represented as: ； in, Represents the global projection matrix. Represents the global rotation matrix. Represents the global translation vector. Indicates the first The relative rotation matrix of each camera, Indicates the first The relative translation vector of each camera, Indicates the first The intrinsic parameter matrix obtained from camera calibration; Indicates the first Depth scale factor of each camera projection Indicates the first Homogeneous pixel coordinates of each camera; A vector representing the actual three-dimensional skeleton of the human body; Let the absolute extrinsic parameter matrix after calibration and integration be The pinhole perspective projection model is then represented as: ； in, Indicates the first Rotation matrix of each camera, Indicates the first The absolute extrinsic translation vector of each camera.

4. The 3D human motion digital twin method as described in claim 3, characterized in that, Step S4 includes: Two-dimensional pixel coordinates of human joints with the same name in N camera images and their corresponding confidence levels Extract; In the final coordinate system, for any unknown three-dimensional spatial joint... In its first The homogeneous pixel coordinates in the camera projection plane are The nonlinear mapping relationship obtained from the pinhole perspective projection model is as follows: ； in, For the first Global projection matrix of each camera; The nonlinear mapping relationship is converted into a linear constraint by using the three-dimensional vector outer product, and is expressed as follows: ; make Represents the global projection matrix The first in The row vectors, where the linear constraints are expanded by cross product to construct a system of linear equations, are expressed as follows: ； Construct an overdetermined linear matrix based on the system of linear equations, denoted as: ;in, Denotes the overdetermined linear matrix; The overdetermined linear matrix is ​​left-multiplied by a diagonal weight matrix. The weighted overdetermined linear matrix is ​​obtained, denoted as: ;in, This represents a diagonal weight matrix, where the diagonal elements are the keypoint confidence scores from each camera's viewpoint. constitute; Factoring the weighted overdetermined linear matrix using singular value decomposition, we can express the following: ;in, Describes a left singular matrix. Represents a right singular matrix; Then, if the constraints are satisfied Under the premise that the global optimal solution of the weighted overdetermined linear matrix is ​​the right singular matrix. The right singular vector corresponding to the smallest singular value in the solution is the homogeneous coordinate vector of the global optimal solution. Dividing each term of the homogeneous coordinate vector by its fourth component yields the three-dimensional spatial joint. Three-dimensional absolute coordinates with millimeter-level precision in the real physical world; The three-dimensional skeleton vector of the real human body is reconstructed based on the three-dimensional absolute coordinates of each joint, resulting in the reconstructed three-dimensional skeleton vector.

5. The 3D human motion digital twin method as described in claim 4, characterized in that, Step S5 includes: The intrinsic parameter matrix of the virtual camera is configured according to the target application scenario. A 3D spatial perspective projection model of the virtual camera is constructed based on the intrinsic parameter matrix and the reconstructed 3D skeleton vector. This 3D spatial perspective projection model is represented as follows: ； in, Indicates the virtual projection depth; , where is the homogeneous planar coordinate of a point in the generated two-dimensional image; Represents the reconstructed 3D skeleton vector; The rotation matrix represents the virtual camera; Represents the translation vector of the virtual camera; The intrinsic parameter matrix representing the virtual camera; Using the three-dimensional centroid of the human body as the center of a sphere, the sampling space of the virtual camera surrounding the three-dimensional centroid of the human body is constructed according to the aforementioned three-dimensional spatial perspective projection model; By discretizing and traversing the extrinsic parameter matrix of the virtual camera within the sampling space to dynamically change the spatial observation position and rotation / pitch angle of the virtual camera, the virtual two-dimensional projection feature point sequence containing ceiling-view, ground-view, and arbitrary blind-angle orientations in the reconstructed three-dimensional skeleton vector is simulated and generated. Two-dimensional pose features are generated from different viewing angles based on the virtual two-dimensional projection feature point sequence.

6. The 3D human motion digital twin method as described in claim 5, characterized in that, Step S6 includes: By using matrix transformation, the single-frame 3D points in the reconstructed 3D skeleton vector are mapped to the standard skeleton coordinates of the virtual engine, as follows: ； in, Represents standard skeletal coordinates. Represents a single frame of 3D points. The scaling factor represents the conversion factor from physical units to the virtual engine's spatial scale. The standard skeletal coordinates of the left and right joints of the pelvis are extracted to calculate the horizontal width vector, and the standard skeletal coordinates of the pelvic center and neck nodes are extracted to calculate the vertical height vector. The orthogonal orientation vector of the torso is derived from the cross product of the horizontal width vector and the vertical height vector. A reference rotation quaternion of the root node is then constructed based on this orthogonal orientation vector. The orthogonal orientation vector is expressed as: , Represents a horizontally wide vector. Represents the vertical height vector; The root node implements high-fidelity driving of the virtual digital human based on the reference quaternion.

7. The 3D human motion digital twin method as described in claim 6, characterized in that, Step S6 further includes: for the curvature of multiple vertebrae, the reference rotation quaternion of the root node is evenly distributed to the corresponding sub-bone nodes of the lumbar and thoracic vertebrae through piecewise interpolation weights to achieve natural human trunk movement restoration.

8. The 3D human motion digital twin method as described in claim 7, characterized in that, Step S6 further includes: The reconstructed 3D skeleton vectors extract the direction vectors of the upper arm, thigh, forearm, and lower leg. Then, the normal vectors of the joints in the arms and legs are calculated and represented as follows: , This represents the direction vector of the upper arm or the direction vector of the thigh. This represents the forearm direction vector or the lower leg direction vector. when When the rotation exceeds the preset safe rotation threshold, a standard fixed smoothing coefficient is used; when... When the rotation speed is less than the preset safe rotation threshold, the smoothing coefficient will be adjusted accordingly. The square relationship undergoes nonlinear decay, i.e. , Represents the smoothing coefficient. Represents a proportionality constant; The three-dimensional target rotation posture of the corresponding joint is constructed based on the upper arm direction vector, the thigh direction vector and their respective normal vectors, to obtain the target rotation quaternion, i.e. the target rotation amount; In each frame of pose update, the dynamically calculated smoothing coefficient is used as a dynamic interpolation weight to drive the current rotation of the model skeleton to a non-linear smooth transition to the target rotation.

9. The 3D human motion digital twin method as described in claim 8, characterized in that, The value of N is 4.

10. A 3D human motion digital twin system based on multi-camera hardware synchronization, wherein the 3D human motion digital twin system is used to implement the 3D human motion digital twin method according to any one of claims 1-9, characterized in that, This 3D human motion digital twin system includes: The camera array construction module is used to build a multi-camera synchronous acquisition array. The calibration and alignment module acquires multi-view image sequences of the calibration reference object through a multi-camera synchronous acquisition array, obtains the corner features of the calibration reference object based on the multi-view image sequences, and completes the absolute alignment of the multi-camera world coordinate system based on the corner features and the horizontal reference plane to obtain the final coordinate system. The feature extraction module acquires data through a multi-camera synchronous acquisition array. Time-aligned A multi-view human image sequence is used to extract features from the multi-view human image sequence to obtain the two-dimensional pixel coordinates and confidence scores of each joint point. The 3D reconstruction module, in the final coordinate system, based on the two-dimensional pixel coordinates and their confidence levels of each joint, performs multi-view triangulation using the weighted DLT-SVD method to reconstruct the 3D skeleton vector of the real human body, thus obtaining the reconstructed 3D skeleton vector. The virtual reprojection module constructs a three-dimensional spatial perspective projection model of the virtual camera based on the reconstructed three-dimensional skeleton vector, and performs spatial reprojection on the reconstructed three-dimensional skeleton vector based on the three-dimensional spatial perspective projection model to generate two-dimensional pose features from different viewing angles. The digital twin module performs coordinate system transformation on the reconstructed 3D skeleton vector to map the reconstructed 3D skeleton vector to the standard skeletal coordinates of the virtual engine, and realizes high-fidelity driving of the virtual digital human based on the standard skeletal coordinates.

Images