A label-free real-time human pose reconstruction system based on multimodal sensor fusion

Abstract

This invention relates to the field of human pose estimation technology and discloses a label-free real-time human pose reconstruction system based on multimodal sensor fusion. The system integrates a depth camera and millimeter-wave radar, achieving spatiotemporal alignment of multi-source data through hardware trigger pulses. Addressing the challenge of visual occlusion in industrial scenarios, the system utilizes a mask projection mechanism to perceive failed nodes in real time and performs state dimensionality reduction prediction based on radar radial observation features. A tangential relaxation compensation mechanism extracts the traction velocity of adjacent parent nodes to correct motion uncertainties. Simultaneously, the system introduces human anatomical topology to construct zero-order Euclidean distance and first-order rotational constraints, employing a Lagrange space projection algorithm to forcibly truncate the unconstrained estimate to a feasible solution manifold conforming to physiological characteristics. This invention eliminates trajectory divergence and limb distortion caused by visual blind spots, improving the accuracy and physical consistency of human pose reconstruction in complex environments.

Description

Technical Field

[0001] This invention relates to the field of human pose estimation technology, specifically to a label-free real-time human pose reconstruction system based on multimodal sensor fusion. Background Technology

[0002] With the deep integration of industrial automation and human-machine collaboration technologies, real-time perception and reconstruction of human pose within the workspace has become a crucial prerequisite for ensuring operational safety and achieving efficient collaboration. Existing label-free human pose estimation technologies mostly rely on single-modal optical sensors (such as depth cameras) to infer human skeletal nodes through feature point detection algorithms. However, in actual industrial assembly scenarios, the frequent reciprocating movements of industrial robot arms, conveyor belts, and moving workpieces can easily cause physical obstruction of the optical line of sight, resulting in discontinuities in visual observation information in the time domain or the generation of severe nondeterministic noise. This, in turn, causes the reconstructed limb coordinates to diverge abruptly in the blind zone or the skeletal structure to disappear, making it difficult to meet the continuous and robust monitoring requirements of industrial applications.

[0003] Furthermore, although some improved solutions introduce millimeter-wave radar with penetrating capabilities to assist perception, in workshops with dense metal components and complex electromagnetic environments, radar echoes are highly susceptible to multipath reflections and clutter interference, resulting in a large number of artifact nodes in the observation point cloud. Existing sensor fusion frameworks mostly focus on statistical data weighting, lacking deep integration with underlying perception mechanisms and human motion characteristics. This processing logic ignores inherent biomechanical and physical characteristics of the human body, such as constant link lengths and joint rotation limits. As a result, when observation quality deteriorates, the system cannot effectively constrain the movement of the solution space, often exhibiting non-physical reconstruction phenomena that violate physiological common sense, such as limb stretching, joint over-limits, or bone fractures.

[0004] Therefore, how to achieve real-time reconstruction of human posture with both high-precision motion compensation and real physical logic constraints under occlusion and multipath clutter interference is a technical bottleneck that urgently needs to be solved in the field of human-machine collaborative safety. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a labelless real-time human posture reconstruction system based on multimodal sensor fusion. This system solves the problems of decreased accuracy in human skeleton reconstruction, divergent motion trajectories, and limb postures violating biomechanical logic caused by the lack of visual observation due to dynamic occlusion of mechanical structures in complex industrial environments.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a label-free real-time human pose reconstruction system based on multimodal sensor fusion. The core of this invention lies in: By establishing a spatiotemporal coupling mechanism between visual depth mask and radar Doppler vector, the originally divergent three-dimensional search space is dynamically reduced to a specific motion manifold based on Doppler physical characteristics during the state estimation stage. Prior information of human kinematic chain is introduced to perform kinematic relaxation compensation for radar observation blind zone. Finally, the physical consistency of the output results is ensured through geometric projection operator.

[0007] The label-free real-time human pose reconstruction system based on multimodal sensor fusion provided by this invention includes the following core functional units: Data Acquisition and Registration Module: This module establishes the heterogeneous data benchmark for the system. By receiving external high-precision trigger pulses, it enables the depth camera and the frequency-modulated continuous wave millimeter-wave radar to achieve microsecond-level alignment on the time axis. Simultaneously, it uses a calibrated extrinsic parameter matrix to map the polar coordinate system point cloud output by the radar to the visual pixel coordinate system, establishing a physical relationship between the point cloud and image pixels.

[0008] Masking Projection Module: This module acts as a logic switch for system mode switching. Leveraging the spatial resolution advantage of depth images, it compares the measured depth map with the depth coordinates of the system's predicted skeleton positions to determine in real time whether each node of the human body is obstructed by external physical entities. The spatial occlusion mask output by this module is used to indicate which mode's observation weights should be emphasized during subsequent filter updates.

[0009] State dimensionality reduction prediction module: This is the core processing unit of the invention, used to achieve high-precision state extrapolation within visual blind spots. When a node is occluded, this module no longer performs conventional full-dimensional spatial prediction, but instead extracts the Doppler velocity components of the radar point cloud. By constructing an orthogonal projection operator, the uncertainty (i.e., process noise) in the state transition process is non-uniformly distributed: the necessary search width is retained in the radar line-of-sight direction, while it is suppressed through a dynamic damping mechanism in the subspace orthogonal to the radar line-of-sight. This dimensionality reduction prediction mechanism ensures that, even without visual observation, the limb trajectory remains constrained by the Doppler physical manifold.

[0010] State Update Constraint Module: This module is responsible for performing the fusion update of multimodal observations. It dynamically adjusts the Kalman gain based on the mask state and integrates a set of second-order kinematic constraint operators. Utilizing the fixed length of the human skeleton and the biological limits of joint rotation angles, this module performs Lagrange space projection on the initially calculated posterior coordinates to ensure that the reconstructed posture always conforms to the anatomical structure of the human body.

[0011] In one specific embodiment, the data acquisition and registration module utilizes the hardware clock synchronization interface of the programmable logic controller to ensure that the time axis deviation between depth image acquisition and radar linear frequency modulated pulse sequence transmission is no greater than 1 millisecond, thereby eliminating spatial inaccuracies under dynamic motion.

[0012] Preferably, the system further includes a gated pre-filtering module for filtering out interference using spatiotemporal joint features during the data input stage. The specific logic of this module includes: Spatial thickness filtering: Compare the depth value of the radar point with the physical depth of the occluder of the corresponding pixel point to remove multipath reflection artifacts with abnormal penetration thickness; Kinematic Mahalanobis Filtering: Based on kinematic priors, the deviation of radar observations from the predicted human position is calculated, and statistical thresholds are used to filter out random noise that does not conform to the continuity of human movement.

[0013] In one specific embodiment, the dynamic damping constraint performed by the state dimensionality reduction prediction module includes the reconstruction of the state prediction covariance matrix. By reducing the eigenvalues ​​of the orthogonal subspace, the system can evolve its state only along the Doppler vector direction when occlusion occurs, preventing the coordinates from drifting randomly within the occlusion area.

[0014] Preferably, the system incorporates a tangential relaxation compensation mechanism to address radar blind spots. When the radial motion of the target node relative to the radar approaches zero, the state dimensionality reduction prediction module reads the motion vector of the parent node connected to that node, utilizes the angular velocity traction effect of the skeleton chain to calculate the prior tangential velocity of the node, and accordingly increases the relaxation coefficient of the orthogonal subspace, enabling the system to handle tangential motion trajectories.

[0015] In one specific embodiment, the state update constraint module amplifies the visual observation noise covariance to more than three orders of magnitude above the base value when the occlusion mask is triggered by applying a penalty multiplier, thereby guiding the system to enter a narrow window search mode guided by radar observation and kinematic prediction within the blind zone.

[0016] Preferably, the biomechanical boundary conditions are specifically manifested as spatial geometric equality constraints and inequality constraints. The equality constraints are used to anchor the Euclidean distance (link length) between adjacent nodes, while the inequality constraints are used to limit the achievable rotation angle range of the joint in three-dimensional space.

[0017] In one specific embodiment, the spatial geometric truncation solves a constrained second-order optimization problem using the Lagrange multiplier method, that is, mapping the original output of the filter onto a convex manifold defined by a biomechanical boundary to obtain a physically true solution with minimal bias.

[0018] In one specific embodiment, the system further includes a pose output module, which maps the parsed three-dimensional node coordinate sequence into a hierarchical quaternion rotation matrix. This module sends data packets containing human topological pose to an external robot controller or digital factory platform via an industrial Ethernet interface at a preset sampling frequency.

[0019] This invention provides a label-free real-time human pose reconstruction system based on multimodal sensor fusion. It has the following beneficial effects: 1. This invention, through the collaborative operation of a mask projection module and a state dimensionality reduction prediction module, utilizes the Doppler radial velocity of millimeter-wave radar to construct an orthogonal projection operator under occlusion conditions where optical observation information is lacking, thus confining the uncertainty of state estimation within the physical manifold. This mechanism effectively solves the problem of state divergence in blind zones caused by a single visual sensor, ensuring continuous tracking of the human skeleton trajectory in complex spatial environments.

[0020] 2. This invention utilizes a gated pre-filtering module to achieve joint verification of depth geometric features and kinematic information. By comparing the radar point cloud depth with the physical thickness of environmental obstructions, and combining Mahalanobis distance to eliminate abnormal signal points that violate human kinematic continuity, the system can effectively identify and filter multipath reflection clutter generated by dense metal environments, improving the signal purity of the data fusion front end.

[0021] 3. This invention introduces biomechanical prior constraints such as zero-order Euclidean distance and first-order rotation angle into the state update constraint module, and uses the Lagrange multiplier method to map the original output of the nonlinear state estimation into a constraint space that conforms to the human anatomical structure. This processing eliminates limb proportion misalignment and joint over-limit distortion caused by sensor noise or algorithm convergence deviation, ensuring that the reconstructed three-dimensional posture coordinates have true physical meaning. Attached Figure Description

[0022] Figure 1 This is the main flowchart of the system of the present invention; Figure 2 This is a flowchart of the mask projection and spatial occlusion determination process of the present invention; Figure 3 This is a flowchart of the core algorithm for state dimensionality reduction prediction in this invention; Figure 4 This is a flowchart illustrating the execution of the tangential relaxation compensation mechanism of the present invention. Figure 5 This is a flowchart of the measurement noise weighting adjustment strategy of the present invention; Figure 6 This is a flowchart illustrating the construction of biomechanical boundary conditions for this invention. Figure 7 This is a flowchart of the spatial geometric truncation process based on Lagrange projection of the present invention; Figure 8 This is a flowchart of the pose data conversion and output process of the present invention; Figure 9 This is a curve comparing the accuracy of various algorithms in this invention under dynamic occlusion. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] See attached document Figure 1 The label-free real-time human pose reconstruction system based on multimodal sensor fusion consists of sensing devices, a synchronization controller, and a computing terminal at the hardware level. The sensing devices include a depth camera and a frequency-modulated continuous wave millimeter-wave radar. The synchronization controller employs a programmable logic controller. The computing terminal is used to deploy and run data processing algorithms.

[0025] The data acquisition and registration module operates on the aforementioned hardware and is used to simultaneously acquire depth images and millimeter-wave radar point clouds of the target scene, and to unify the millimeter-wave radar point clouds into the visual coordinate system of the depth images. This process is implemented in two parts: time synchronization and spatial registration.

[0026] The time synchronization process specifically includes the following steps: The programmable logic controller generates a reference trigger pulse signal according to a preset sampling frequency, and sends the reference trigger pulse signal to the external trigger input pin of the depth camera and the synchronous receiving port of the frequency modulated continuous wave millimeter-wave radar respectively through hardware wiring.

[0027] When the depth camera detects the rising edge of the reference trigger pulse signal, it triggers the global exposure cycle of the underlying layer, records the optical information and depth distance information within the current field of view, and generates a depth image.

[0028] When a frequency-modulated continuous wave (FM-CVT) millimeter-wave radar detects the rising edge of the same reference trigger pulse signal, it triggers the FM transmission cycle of the radio frequency front-end, transmitting a linear FM-CVT continuous wave into the target space and receiving the reflected echo. After processing by Fast Fourier Transform, a millimeter-wave radar point cloud containing three-dimensional spatial coordinates and Doppler radial velocity is generated. Through the above hardware-level pulse distribution, the global exposure cycle of the depth camera and the FM transmission cycle of the FM-CVT millimeter-wave radar are forcibly synchronized, eliminating the timestamp misalignment caused by the independent clock drift of heterogeneous sensors.

[0029] The spatial registration process specifically includes the following steps: Before the system runs, it loads a pre-calibrated rigid body transformation matrix. For the corner reflector calibration plate fabrication, image feature extraction, and specific extrinsic parameter optimization processes involved in the heterogeneous sensor extrinsic parameter calibration, those skilled in the art can use existing vision-radar joint calibration algorithms for calculation. The specific implementation methods are well-known technologies in this field and will not be elaborated upon here.

[0030] The data acquisition and registration module reads the millimeter-wave radar point cloud data one by one at the current moment, and uses a rigid body transformation matrix to map the point cloud coordinates in the radar local coordinate system to the visual coordinate system where the depth image is located. The specific spatial mapping calculation logic is shown in the following formula: ; In the formula, This represents the three-dimensional coordinate vector of a millimeter-wave radar point cloud mapped to the visual coordinate system. This represents the original three-dimensional coordinate vector of the millimeter-wave radar point cloud in the radar local coordinate system. This represents the rotation matrix component in the preset rigid body transformation matrix; Represents the translation vector components in the predefined rigid body transformation matrix. The system stores the millimeter-wave radar point cloud and depth image with completed coordinate mapping, which serves as a unified data source for subsequent modules to perform physical occlusion judgment and state nonlinear estimation.

[0031] See attached document Figure 1 The labelless real-time human pose reconstruction system based on multimodal sensor fusion operates within a computational framework of nonlinear state estimation, such as extended Kalman filtering, and performs continuous recursive estimation and physical correction on the input multimodal data stream. After acquiring spatiotemporally aligned depth images and millimeter-wave radar point clouds, the system executes the pose reconstruction task according to the following process.

[0032] The system invokes a gated pre-filtering module to perform joint spatial and kinematic filtering on the input data. This module is connected in series between the data acquisition and registration module and the state dimensionality reduction and prediction module in the data flow direction. The system extracts the 3D coordinates of the millimeter-wave radar point cloud, projects them onto the pixel grid of the depth image, and reads the depth value of the corresponding pixel as the physical occlusion thickness. The system compares the depth coordinates of the millimeter-wave radar point cloud with this physical occlusion thickness, directly discarding radar points whose depth coordinates exceed the physical occlusion thickness to eliminate multipath reflection radar points caused by metal surfaces or walls.

[0033] For the remaining millimeter-wave radar points after spatial filtering, the system extracts the prior prediction distribution output by the nonlinear state estimation framework in the previous calculation cycle. It then calculates the innovation vector by representing the difference between the current radar observation and the expected observation, and combines this with the system's prediction covariance and observation noise covariance to calculate the innovation covariance matrix. Based on these parameters, the system solves for the Mahalanobis distance using the following formula: ; In the formula, Represents the squared value of the Mahalanobis distance; Represents the innovation vector; Represent the new information covariance matrix; This represents the transpose of the information vector.

[0034] The system sets a confidence interval threshold that conforms to the chi-square distribution and performs a chi-square test. When the calculated Mahalanobis distance exceeds the confidence interval threshold, the corresponding radar point is determined to violate human kinematic continuity and is removed as an abnormal radar point, thereby constructing a high-confidence subset of radar observations.

[0035] The system extracts the estimated human skeleton node positions from the previous time step, back-projects them onto the camera's pixel plane using the camera's intrinsic parameter matrix, and obtains the theoretical projected pixel coordinates of each skeleton node. The system reads the measured depth value at the corresponding pixel coordinates from the depth image and subtracts this measured depth value from the depth axis component of the estimated human skeleton node position. If the measured depth value is less than the difference between the depth axis component and the preset depth tolerance margin, it indicates the presence of an opaque physical occlusion between the camera and the skeleton node. The system determines that the human skeleton node has entered a visual blind spot and assigns the corresponding spatial occlusion mask a physical occlusion state. Conversely, it assigns a non-occluded state.

[0036] In the prediction phase of nonlinear state estimation, the system reads the logical value of the spatial occlusion mask. When the spatial occlusion mask indicates that a human skeleton node is physically occluded, the system extracts the Doppler radial velocity and its direction vector from the millimeter-wave radar point cloud. The system uses this direction unit vector to construct parallel projection and orthogonal projection operators, orthogonally decomposing the three-dimensional motion state space into a parallel subspace parallel to the radar line of sight and an orthogonal subspace perpendicular to the radar line of sight. The system divides the process noise covariance matrix of the nonlinear filter into a first part controlled by the parallel projection operator and a second part controlled by the orthogonal projection operator. The system multiplies the second part by a dynamic damping coefficient, significantly reducing the prediction uncertainty in the orthogonal direction to suppress the random walk degrees of freedom of the human skeleton node in the direction orthogonal to the radar line of sight, thus reducing the three-dimensional motion uncertainty to within a local manifold constrained by the Doppler radial velocity.

[0037] The dynamic damping coefficient is controlled by a tangential relaxation mechanism. The system extracts the angular velocity vectors of the adjacent parent nodes of the physically obscured human skeleton nodes at the previous moment and obtains the skeleton link vectors between parent and child nodes. The system performs a spatial cross product operation on the angular velocity vectors and skeleton link vectors to calculate the prior tangential predicted velocity generated by the mechanical traction of the skeletal chain at the physically obscured human skeleton nodes. The system projects this prior tangential predicted velocity onto an orthogonal subspace and solves for the square of its vector magnitude as a tangential energy scalar. When the measured Doppler radial velocity is lower than a preset velocity noise floor threshold, the system uses this tangential energy scalar to positively compensate the dynamic damping coefficient, relaxes the covariance constraint of the orthogonal subspace, and allows the system's predicted state to evolve tangentially, thus outputting the state prediction result.

[0038] The system dynamically adjusts the measurement noise covariance weights of visual observations based on the spatial occlusion mask. When the spatial occlusion mask indicates that the visual observations are not physically occluded, the system maintains the basic calibration noise covariance of the visual observations. When the spatial occlusion mask indicates that the visual observations are physically occluded, the system injects a very large preset penalty multiplier parameter into the measurement noise covariance matrix of the visual observations. This operation causes the Kalman gain calculated by the nonlinear state estimation to approach a zero matrix, forcing the system to ignore the currently failed optical observations. The system combines the state prediction results with the adjusted Kalman gain to complete unconstrained state updates.

[0039] The system further defines biomechanical boundary conditions for the human body, including zero-order Euclidean distance constraints and first-order rotational constraints. Based on common knowledge of human anatomy, the system limits the spatial distance between adjacent skeletal nodes to equal the prior calibration rod length, forming a zero-order Euclidean distance constraint; it also limits the bending angle of a joint formed by three adjacent skeletal nodes to within a preset physiological limit angle range, forming a first-order rotational constraint. If the unconstrained state update coordinates violate any of the above constraints, the system uses the updated state as the initial value to establish an objective function, using the zero-order Euclidean distance constraint as the equality constraint and the first-order rotational constraint as the inequality constraint.

[0040] For this constrained nonlinear optimization problem, those skilled in the art can use the Lagrange multiplier method or interior point method for iterative solution. The specific optimization calculation process is well-known in the field and will not be elaborated here. The system solves this optimization problem by finding the optimal projected solution with the minimum Mahalanobis distance from the original state estimate within the solution space that satisfies the human biomechanical boundary conditions. The system uses this optimal projected solution to replace the unconstrained update result as the final output of the current-moment three-dimensional coordinates of the human posture after spatial geometric truncation.

[0041] The system receives the three-dimensional coordinate sequence of human posture output by the state update constraint module. Based on a predefined human kinematics hierarchy tree, the system calculates the relative rotation transformation relationship between the coordinate systems of adjacent parent and child nodes, and transforms it into a relative rotation quaternion matrix without singularity defects. The system packages the data containing the absolute pose of the global root node and the relative rotation quaternions of each child node into a standard communication message, and continuously outputs it at a fixed frequency to an external industrial robot controller or digital twin platform via an industrial Ethernet bus, completing a single reconstruction cycle.

[0042] The data acquisition and registration module physically comprises a programmable logic controller, a depth camera, and a frequency-modulated continuous wave millimeter-wave radar. This module establishes a unified spatiotemporal reference for multimodal observation data through underlying electrical signals and coordinate transformation parameters.

[0043] The programmable logic controller configures a fixed-period pulse width modulation signal through an internal high-frequency counter, and sends the pulse width modulation signal as a reference trigger pulse through a general-purpose input / output pin to the external trigger pin of the depth camera and the synchronous receiving pin of the frequency modulated continuous wave millimeter-wave radar through parallel electrical wiring.

[0044] When the image signal processor at the bottom layer of the depth camera detects the rising edge of the reference trigger pulse, it interrupts the current standby loop, drives the global shutter to open, and performs a global exposure operation with a set integration time to obtain the depth image of the target scene at the current moment.

[0045] When the phase-locked loop and voltage-controlled oscillator circuit inside the frequency-modulated continuous wave (FM-CVT) millimeter-wave radar detect the rising edge of the same reference trigger pulse, they initiate the radio frequency (RF) transmission process, transmitting a linear frequency-modulated (LFM) pulse sequence into space and receiving the corresponding reflected echo signals. After analog-to-digital conversion and multidimensional fast Fourier transform processing, the output is a millimeter-wave radar point cloud containing spatial coordinates and Doppler radial velocity. This hardware-level parallel triggering mechanism forcibly synchronizes the global exposure period of the depth camera with the FM transmission period of the FM-CVT millimeter-wave radar, avoiding asynchronous sampling deviations caused by the drift of the device's independent internal clock.

[0046] The system's underlying driver records the system timestamp when each sensor hardware interrupt is triggered, and calculates the absolute time deviation between the two. The calculation formula is as follows: ; In the formula, This represents the absolute value of the time synchronization deviation between heterogeneous sensors; Indicates the global shutter opening timestamp recorded by the depth camera; This indicates the start transmission timestamp of the frequency-modulated pulse sequence recorded by the frequency-modulated continuous wave millimeter-wave radar.

[0047] The system uses the level-flipping characteristic of the programmable logic controller to limit the absolute value of the aforementioned time synchronization deviation to within the range of the hardware cable propagation delay, thus establishing a time alignment reference for data acquisition.

[0048] The data acquisition and registration module reads a preset rigid body transformation matrix from the system memory. This rigid body transformation matrix defines the spatial geometric transformation relationship between the radar local coordinate system and the visual coordinate system. For the specific solution process of the heterogeneous sensor extrinsic parameter calibration matrix, those skilled in the art can use a corner reflector calibration plate with known geometric dimensions to perform multi-viewpoint joint calibration calculations. The extrinsic parameter calibration calculation is a well-known technique in this field and will not be elaborated upon here.

[0049] The data acquisition and registration module extracts the three-dimensional coordinate vectors of each data point in the millimeter-wave radar point cloud in the radar's local coordinate system. The module then calls a linear algebra library to perform matrix multiplication on this three-dimensional coordinate vector with the rotation component of the rigid body transformation matrix, and adds the result to the translation component of the rigid body transformation matrix, directly mapping the point cloud in the radar's local coordinate system to the visual coordinate system. This computational processing ensures that the physical observation field of view of the depth image and the spatial dispersion region of the millimeter-wave radar point cloud achieve a unified coordinate coefficient benchmark.

[0050] See attached document Figure 1 The system includes a gated pre-filtering module, which is connected in series between the data acquisition and registration module and the state dimensionality reduction and prediction module in the data flow direction. This module is used to address the multipath reflection artifacts and kinematic abrupt noise problems that are easily generated by a single radar sensor in complex industrial environments. The system achieves high-confidence screening of radar point clouds through two stages: spatial geometric truncation and probabilistic statistical verification.

[0051] The system extracts millimeter-wave radar point clouds mapped to the visual coordinate system and reads the 3D spatial coordinates of individual radar points in the point cloud set. The system calls the intrinsic parameter matrix of the depth camera, projects the 3D spatial coordinates onto the 2D pixel plane of the depth image, and calculates the target pixel index corresponding to the radar point in the image space.

[0052] The system reads the measured depth value at the corresponding location from the acquired depth image matrix based on the target pixel index. The system then performs a scalar addition operation on this measured depth value and a preset physical thickness margin. This physical thickness margin represents the maximum physical extension thickness of a human limb or a known environmental obstruction. The system uses the summed value as the physical occlusion thickness corresponding to the pixel on the current line of sight.

[0053] The system extracts the depth coordinate component from the original radar point's three-dimensional spatial coordinates. This depth coordinate component is then compared numerically with the previously calculated physical obstruction thickness. When the depth coordinate component of a radar point is greater than the physical obstruction thickness, it indicates that the radar point's spatial location is inside or behind an opaque physical entity. Based on the physical laws of electromagnetic wave propagation, such measurement points generate false echoes after the radar beam undergoes secondary or even multiple reflections through walls or metal equipment. The system directly eliminates multipath reflection radar points whose depth coordinates exceed the physical obstruction thickness, thus completing the filtering of spatial geometric dimensions.

[0054] For the remaining millimeter-wave radar points after spatial comparison, the system extracts the prior prediction distribution of the system state derived from the nonlinear state estimation framework in the previous calculation cycle. This prior prediction distribution specifically includes the prior state estimation vector and the prior error covariance matrix.

[0055] The system utilizes the radar's observation model to map the prior state estimation vector into the observation space, obtaining the system's expected radar observation vector. The system then performs vector subtraction between the actual measurement vector of the remaining millimeter-wave radar point cloud and the expected radar observation vector to calculate the innovation vector. Finally, combining the observation partial derivative matrix in the state space, the prior error covariance matrix, and the radar's fundamental measurement noise covariance matrix, the system calculates the innovation covariance matrix through matrix multiplication and addition. The specific calculation logic is shown in the following equation: ; ; In the formula, Represents the innovation vector; Represents the actual measurement vector of the millimeter-wave radar point; Represents a nonlinear observation function; Represents the prior state estimation vector; Represent the new information covariance matrix; This represents the observation Jacobian matrix obtained by taking the partial derivative of the nonlinear observation function; Represents the prior error covariance matrix; This represents the radar measurement noise covariance matrix.

[0056] The system uses the calculated innovation vector and innovation covariance matrix to solve for the Mahalanobis distance. The specific formula for calculating this Mahalanobis distance is consistent with the description in the overall system workflow section above, and the original formula is directly used for calculation. The system performs a chi-square test on the calculated Mahalanobis distance. The system pre-defines the chi-square distribution test threshold with a specific confidence level.

[0057] For the chi-square test lookup table and threshold parameter settings under specific degrees of freedom, those skilled in the art can set them according to the actual fault tolerance rate of the system. The specific statistical verification process is a well-known technology in this field and will not be elaborated here.

[0058] The system compares the Mahalanobis distance with a chi-square test threshold. When the Mahalanobis distance exceeds the threshold, it determines that the probability of the radar measurement vector occurring exceeds the reasonable physical boundary of human motion coherence. The system then removes abnormal radar points that violate human kinematic coherence and outputs the radar point cloud that has passed all the above joint filtering to the subsequent state estimation algorithm module.

[0059] See attached document Figure 2 The mask projection module establishes a mapping relationship between three-dimensional spatial nodes and two-dimensional image pixels, and uses the geometric characteristics of optical occlusion to generate visibility masks for each skeleton node, providing a logical basis for the subsequent weight allocation of heterogeneous sensors.

[0060] The system acquires the estimated positions of the human skeleton nodes from the previous moment. This estimated position is a set of multiple three-dimensional coordinate points, covering the major joints of the human body. The system extracts the position vector of each node in this coordinate set in the current visual coordinate system.

[0061] The system utilizes the intrinsic parameter matrix of the depth camera to perform a backprojection operation on the estimated positions of the acquired human skeleton nodes, mapping them from 3D space to the camera's 2D pixel plane. The specific geometric backprojection calculation logic is shown in the following equation: ; In the formula, and These represent the horizontal and vertical pixel coordinates of the human skeleton nodes projected onto the camera pixel plane, respectively. This represents the projection scaling factor, which is numerically equal to the depth axis component of the human skeleton node in the camera coordinate system. This represents the intrinsic parameter matrix of the depth camera; This represents the estimated position vector of the human skeleton nodes at the previous moment.

[0062] The system retrieves the measured depth value of the corresponding pixel position from the depth image matrix acquired at the current moment based on the calculated pixel coordinates. For abnormal projection points that exceed the boundary of the pixel plane, those skilled in the art can perform preprocessing by boundary truncation or assigning maxima. The specific coordinate validity verification is a well-known technique in the field and will not be elaborated here.

[0063] The mask projection module performs visibility logic determination. The system compares the retrieved measured depth value with the depth axis component estimated from the human skeleton node positions. During this process, the system introduces a preset depth tolerance margin to eliminate comparison deviations caused by depth sensor measurement noise, system calibration errors, and the physical thickness of the human limbs themselves. The specific determination expression is as follows: ; In the formula, The logical state value representing the spatial occlusion mask, where Defined as a physical occlusion state, Defined as an unoccluded state; This represents the measured depth value at the corresponding position on the pixel plane. The depth axis component represents the estimated location of nodes in the human skeleton. This indicates the preset depth tolerance allowance.

[0064] When the calculated depth value is less than the difference between the depth axis component and the preset depth tolerance margin, it is determined that there are other opaque physical entities blocking the target's line of sight along the camera's optical axis. This means the human skeleton node has entered a visual blind spot, and the corresponding spatial occlusion mask is assigned a physical occlusion state. The system stores the spatial occlusion masks generated by each node into a mask vector in a memory buffer for real-time use by the state dimensionality reduction prediction module and the state update constraint module, guiding the system to adjust its state estimation strategy in the event of missing observations.

[0065] See attached document Figure 3 The state dimensionality reduction prediction module plays a crucial role in the derivation and calculation of nonlinear state estimation. It aims to address the divergence of three-dimensional spatial coordinates caused by relying solely on extrapolation of the system's kinematic model when visual sensors fail. This module incorporates the spatial geometric characteristics of the radar beam to physically constrain and mathematically decompose the covariance space of the motion state.

[0066] Before executing the prediction update equation of the nonlinear filter, the system reads the spatial occlusion mask output by the mask projection module. When the logical state of the spatial occlusion mask indicates that the target human skeleton node is physically occluded, the system blocks the conventional full-dimensional state covariance inference path and triggers and executes the state dimensionality reduction prediction branch.

[0067] The system retrieves the current millimeter-wave radar point cloud and extracts radar measurement points spatially mapped and associated with physically obscured human skeleton nodes. The system reads the three-dimensional spatial coordinate vector of this radar measurement point and calculates its unit direction vector relative to the origin of the radar coordinate system. This unit direction vector physically defines the radar line-of-sight direction and simultaneously calibrates the collinear action axis of the Doppler radial velocity.

[0068] The system constructs a spatial projection operator matrix using the extracted Doppler radial velocity direction unit vector. The system performs matrix multiplication on the direction unit vector and its transpose to generate a parallel projection operator. This operator algebraically projects any three-dimensional spatial vector or matrix onto a straight axis parallel to the radar line of sight. Using the three-dimensional identity matrix as a reference, the system subtracts the aforementioned parallel projection operator to calculate and generate an orthogonal projection operator. This orthogonal projection operator algebraically projects the target onto a two-dimensional orthogonal plane perpendicular to the radar line of sight. The specific operator construction logic is shown in the following equation: ; ; In the formula, Represents the parallel projection operator; The unit vector representing the direction of the Doppler radial velocity; The transpose of the unit vector of direction; Represents the orthogonal projection operator; Represents a three-dimensional identity matrix.

[0069] The system extracts the initial process noise covariance matrix set in the nonlinear state estimation model. The system performs a spatial orthogonal decomposition operation on this initial process noise covariance matrix. The system multiplies the initial process noise covariance matrix on the left and right by the parallel projection operator and its transpose, respectively, generating a first part controlled by the parallel projection operator. The system then multiplies the initial process noise covariance matrix on the left and right by the orthogonal projection operator and its transpose, respectively, generating a second part controlled by the orthogonal projection operator. After operator decomposition, the first part characterizes the uncertainty of the system's motion evolution along the Doppler radial velocity axis, and the second part characterizes the uncertainty of the system's motion evolution in the orthogonal plane where radar Doppler observations are lacking.

[0070] The system applies dynamic damping constraints to the second part controlled by the orthogonal projection operator. The system performs a scalar matrix multiplication operation on the generated dynamic damping coefficients and the decomposed second part, numerically compressing the eigenvalues ​​of this covariance submatrix. The system then performs matrix addition and reconstruction on the undamped first part and the dynamically damped second part to calculate the dimensionality-reduced target process noise covariance matrix. The specific covariance decomposition and reconstruction dimensionality reduction logic is shown in the following equation: ; In the formula, This represents the process noise covariance matrix after dimensionality reduction and reconstruction. This represents the initial process noise covariance matrix; This represents the dynamic damping coefficient, and its value ranges within a closed interval between zero and one.

[0071] The system replaces the original matrix parameters with the reconstructed process noise covariance matrix and substitutes it into the prior prediction equation for nonlinear state estimation for state extrapolation. By applying dynamic damping constraints, the system significantly reduces the state evolution variance of the human skeleton nodes on the orthogonal tangent plane where direct physical observation is lacking. This computational operation mathematically suppresses the random walk degrees of freedom of the human skeleton nodes in the direction orthogonal to the radar line of sight, forcibly compressing their divergent three-dimensional motion uncertainty and reducing the dimension to a local one-dimensional manifold defined by the single physical quantity of Doppler radial velocity, and outputs stable state prediction results to the next level of the system.

[0072] For the specific derivation of the Jacobian matrix and the calculation of the prior state vector recursive equation in the nonlinear state estimation framework, those skilled in the art can write the code according to the conventional implementation specifications of the extended Kalman filter. The specific prior state recursive calculation is a well-known technology in this field and will not be elaborated here.

[0073] See attached document Figure 4During the orthogonal decomposition and reconstruction of the noise covariance in the state dimensionality reduction prediction module, the value of its dynamic damping coefficient is controlled in real time by an internal tangential relaxation mechanism. This mechanism is used to solve the problem of blind zone tracking failure caused by radar sensors being able to measure only radial velocity and unable to sense tangential motion. By introducing the kinematic traction physical correlation of the human skeletal chain, the constraint strength of the orthogonal subspace is adaptively adjusted.

[0074] Based on a pre-defined human topology hierarchy tree, the system determines the adjacent parent nodes corresponding to physically occluded human skeleton nodes. The system extracts the system posterior state vector output by the nonlinear state estimation framework in the previous calculation cycle, and parses the three-dimensional spatial angular velocity vectors of the adjacent parent nodes from this vector. At the same time, it extracts the three-dimensional skeleton link vectors between the adjacent parent node and the physically occluded human skeleton node.

[0075] The system utilizes the rigid body linkage transmission characteristics in human kinematics to calculate the traction velocity of physically occluded human skeleton nodes affected by the motion of adjacent parent nodes. The system performs a three-dimensional cross product operation on the extracted 3D spatial angular velocity vector and the 3D skeleton linkage vector to derive the prior tangential predicted velocity vector of the physically occluded human skeleton node at the current moment. The specific cross product calculation logic is shown in the following equation: ; In the formula, This represents the prior tangential predicted velocity vector of the human skeleton node that is physically occluded; Represents the three-dimensional angular velocity vector of the adjacent parent node; This represents a 3D skeleton link vector that points from an adjacent parent node to a physically occluded human skeleton node.

[0076] The system invokes the orthogonal projection operator generated and saved by the state dimensionality reduction prediction module during the orthogonal decomposition stage. The system left-multiplies the generated prior tangential predicted velocity vector by this orthogonal projection operator matrix, projecting it from three-dimensional space onto a two-dimensional orthogonal plane perpendicular to the radar line of sight. The system calculates the squared Euclidean norm of this projected vector and uses it as the tangential energy scalar. This tangential energy scalar quantifies, at a physical level, the expected intensity of motion of human skeletal nodes within an orthogonal tangential plane lacking direct radar observation. The specific projection and energy calculation logic is shown in the following equation: ; In the formula, Represents the tangential energy scalar; Represents the orthogonal projection operator matrix; This represents the operation of solving for the Euclidean norm of a vector.

[0077] The system reads the absolute value of the current target's Doppler radial velocity measured by millimeter-wave radar and compares it with a preset velocity noise floor threshold. This velocity noise floor threshold is calibrated based on the signal-to-noise ratio and Doppler resolution characteristics of the millimeter-wave radar hardware. When the absolute value of the Doppler radial velocity is lower than the preset velocity noise floor threshold, it indicates that the target's motion relative to the radar is mainly concentrated on the orthogonal tangential plane. In this case, simply using radar radial observation will lead to divergence in the tangential motion trajectory. When this condition is met, the system uses the tangential energy scalar to perform positive compensation calculation on the dynamic damping coefficient. The system extracts the preset lower bound value of the basic damping, combines the linear gain coefficient with the tangential energy scalar to perform algebraic mapping calculation, and generates the real-time dynamic damping coefficient. The specific mapping function is shown in the following formula: ; In the formula, This represents the lower limit of the basic damping, used to set the maximum constraint force when the target is stationary; This represents the linear gain coefficient, used to adjust the mapping ratio of energy to the damping constant; This represents the minimum function, used to strictly limit the dynamic damping coefficient to a range not exceeding one.

[0078] The system outputs the calculated dynamic damping coefficient to the reconstruction equation of the process noise covariance matrix. Through the above algebraic mapping association, when the human body produces violent tangential arm swings and other movements that cause an increase in the tangential energy scalar, the dynamic damping coefficient approaches one. The system uses this coefficient to relax the covariance constraint of the orthogonal subspace, maintain the necessary prediction variance weights on the orthogonal tangential surface, and allow the system state to evolve and be derived towards the tangential physical components. When the tangential energy scalar approaches zero, the dynamic damping coefficient falls back to the lower bound of the basic damping. In the absence of kinematic traction, the system restores the strong compression of the prediction uncertainty of the orthogonal subspace, preventing the node coordinates from drifting randomly within the physical occlusion area.

[0079] See attached document Figure 5 Before executing the measurement update equations for multimodal data, the state update constraint module dynamically adjusts the covariance weights to cut off the negative impact of failed sensors on system state estimation, thereby achieving adaptive fusion of heterogeneous observation data under different environmental conditions.

[0080] The system reads the spatial occlusion mask generated by the mask projection module from the memory buffer. The system extracts the current-moment visual observation vector provided by the depth camera and reads the visual fundamental measurement noise covariance matrix configured during the initialization phase. The diagonal elements of this matrix represent the inherent measurement uncertainty of the depth camera in each coordinate axis direction in three-dimensional space under unobstructed ideal optical conditions.

[0081] The system parses the logical state value of the spatial occlusion mask. When the logical state of the spatial occlusion mask indicates that the target human skeleton node is not physically occluded, the system maintains the original observation confidence assignment and directly uses the extracted visual basic measurement noise covariance matrix as the effective measurement noise matrix of the current nonlinear filtering cycle, inputting it into the Kalman gain solution equation.

[0082] When the logical state of the spatial occlusion mask indicates that the target human skeleton node is physically occluded, the system determines that the current visual observation vector is an invalid feature. The system extracts a preset penalty multiplier parameter, performs a scalar matrix multiplication operation on this penalty multiplier parameter and the visual baseline measurement noise covariance matrix, proportionally amplifying each variance element in the original matrix to calculate and generate an adjusted visual measurement noise covariance matrix. The specific covariance penalty adjustment logic is shown in the following formula: ; In the formula, This represents the adjusted visual measurement noise covariance matrix; This represents the penalty multiplier parameter, whose value is usually set to a constant on the order of 10³; This represents the pre-calibrated visual baseline measurement noise covariance matrix.

[0083] The system combines the prior error covariance matrix output by the state dimensionality reduction prediction module and the Jacobian matrix of the observation model to calculate the Kalman gain matrix required for multimodal observation updates. The specific gain calculation logic is shown in the following equation: ; In the formula, This represents the calculated Kalman gain matrix; This represents the prior error covariance matrix of the system's previous output. The Jacobian matrix represents the observation model; The transpose of the Jacobian matrix of the observation model; This represents the effective visual measurement noise covariance matrix determined by conditional branching.

[0084] The system utilizes the aforementioned matrix operation logic to achieve adaptive adjustment of multi-sensor weights. When the system is in a physically occluded state, the norm of the adjusted visual measurement noise covariance matrix increases sharply due to the penalty multiplier. Since this matrix is ​​located inside the expression for matrix inversion, its norm surge causes the solved Kalman gain matrix to numerically approach zero. The system uses this near-zero Kalman gain matrix, combined with the visual observation vector and the predicted innovation vector, to perform a state update operation. This mechanism completely eliminates the influence of failed visual observations on the posterior state update at the algebraic level, making the calculated unconstrained posterior state coordinates entirely dependent on radar Doppler observations and kinematic dimensionality reduction prediction results based on the tangential relaxation mechanism.

[0085] For the conventional state vector addition and fusion and posterior covariance recursion in nonlinear state estimation, those skilled in the art can use the update equation of the standard extended Kalman filter for code computation. The specific code for the recursive operation is well-known in the field and will not be elaborated upon here. The system uses the generated unconstrained posterior state vector as an intermediate variable, which is then passed to the subsequent biomechanical truncation calculation stage. See attached document Figure 6 After obtaining the unconstrained state update coordinates, the state update constraint module constructs a geometric constraint cluster that reflects the real physical limits by introducing a priori human anatomical topology, providing a mathematical domain for subsequent spatial geometric truncation.

[0086] The system loads a human kinematic model from a preset configuration file. This model defines the hierarchical connections between nodes of the human skeleton and the lengths of skeletal links calibrated during the initialization phase. Simultaneously, the system reads the physiologically achievable angle range for each joint. For the topological definition of the human skeleton model and the selection of general biomechanical parameters, those skilled in the art can refer to publicly available clinical anatomical data for setting; the specific parametric modeling process is well-known in the field and will not be elaborated upon here.

[0087] Based on the rigid physical properties of the human skeleton, the system constructs a zero-order Euclidean distance constraint. The system identifies all node pairs with parent-child connections, extracts the real-time 3D coordinate estimates of each pair, and sets the Euclidean distance to be equal to the corresponding calibrated skeletal link length. This constraint is used to prevent non-physical distortions such as limb stretching or compression under sensor noise interference. The specific equality constraint equation is shown below: ; In the formula, Represents a node With nodes The equality constraint function between them; Represents a node 3D coordinate vector; Represents a node 3D coordinate vector; This represents the L2 norm calculation operation; This indicates the preset calibration length of the skeleton link.

[0088] The system constructs a first-order rotational constraint based on the anatomical rotational limits of human joints. The system extracts adjacent bone links sharing the same central node and calculates the cosine of the angle between the two link vectors to limit the joint's bending angle to within a preset physiological safety range. This constraint exists in the form of an inequality to eliminate joint overturning caused by nonlinear estimation bias. The specific inequality constraint equation is shown below: ; In the formula, Indicates the first First-order rotational constraint functions for each joint; Indicates that it is the central node Point to adjacent nodes skeletal link vectors; Indicates that it is caused by the central node Point to another adjacent node skeletal link vectors; Indicates the first The maximum physiological limit angle preset for each joint; Indicates the first The minimum physiological limit angle preset for each joint.

[0089] The state update constraint module vectorizes and stacks the generated equality and inequality constraint equations to form a complete set of biomechanical boundary conditions. Mathematically, this set defines a non-convex constraint manifold representing all physically feasible human posture spaces. The system inputs this set of boundary conditions, along with the unconstrained state update coordinates, into the optimization solver to obtain a final reconstruction result conforming to anatomical logic through Lagrange space projection.

[0090] See attached document Figure 7 After fusing heterogeneous observation data and extracting biomechanical boundary conditions, the state update constraint module performs optimization and truncation calculations based on Lagrange projection for estimation results that violate physical laws. This process maps the unconstrained state space at the purely mathematical level back to a feasible solution manifold that conforms to human physiological characteristics, thereby eliminating limb pose distortions caused by severe sensor noise or model linearization errors.

[0091] The system receives the unconstrained posterior state vector output after Kalman filtering gain update and simultaneously obtains the corresponding unconstrained posterior error covariance matrix. The system substitutes this unconstrained posterior state vector into a pre-constructed set of biomechanical boundary conditions and calculates the values ​​of each zero-order Euclidean distance constraint equation and first-order rotation constraint inequality.

[0092] The system determines whether the unconstrained posterior state vector satisfies all biomechanical boundary conditions. If all equality constraint residuals are within the set numerical tolerance range and all inequality constraints are met, the system determines that the current estimate has not produced physical distortion and directly outputs the unconstrained posterior state vector as the final attitude coordinates at the current moment, skipping subsequent spatial truncation calculations. If any constraint condition is not met, the system triggers the Lagrange spatial projection truncation mechanism.

[0093] The system uses the unconstrained posterior state vector as the initial reference point to construct an objective function for finding the optimal spatial projection solution. To ensure that the projected coordinates are statistically closest to the original system observation fusion intention, the system uses non-uniformly weighted Mahalanobis distance as the metric for the objective function. The inverse of the unconstrained posterior error covariance matrix is ​​introduced into the objective function as the weight matrix. The specific quadratic objective function is shown in the following equation: ; In the formula, This represents the objective function measured by the squared Mahalanobis distance. This represents the projection vector of the target state to be solved; This represents the unconstrained posterior state vector output after filter fusion calculation; The matrix representing the inverse of the unconstrained posterior error covariance matrix; This represents the transpose operation of a vector or matrix.

[0094] The system utilizes the Lagrange multiplier method to transform the conditional extremum problem with physical boundary constraints into an unconstrained optimization problem. The system introduces two independent sets of Lagrange multiplier vectors, which are used to perform inner product operations with the sets of equality and inequality constraints in the biomechanical boundary conditions, respectively. These inner product vectors are then algebraically added to the previously constructed objective function to construct the generalized Lagrange function. The specific function construction logic is shown in the following equation: ; In the formula, Represents the constructed generalized Lagrange function; This represents the Lagrange multiplier vector corresponding to the set of equality constraints; This represents the equality constraint vector function formed by stacking all zero-order Euclidean distance constraint equations; Let represent the Carlow-Kun-Tucker multiplier vector corresponding to the set of inequality constraints; Let f(x) represent an inequality constraint vector function consisting of a stack of all first-order rotation constraint equations, and all its elements must be less than or equal to zero.

[0095] The system performs differentiation and optimization on the constructed generalized Lagrangian function, calculating the optimal extreme point within the solution space satisfying the Caro-Kun-Tucker conditions. In practical engineering calculations, for this type of highly nonlinear constrained state projection problem, those skilled in the art can use sequential quadratic programming algorithms or interior-point penalty function methods to iteratively approximate the optimal solution. The specific numerical optimization solution and iterative convergence condition determination are well-known techniques in this field and will not be elaborated here.

[0096] The system obtains the optimal target state projection vector output by the optimizer. This vector represents, in algebraic space, the set of coordinates that are closest to the original invalid estimate via the Mahalanobis distance and strictly fall on the geometric manifold surface allowed by the human body's physical structure. The system completely replaces the original unconstrained posterior state vector, which violates physical constraints, with this optimal target state projection vector. Through the above numerical replacement operation, the system achieves spatial geometric truncation, forcibly discarding non-physical walk components caused by observation degradation, and outputs a three-dimensional coordinate sequence of the human body's posture at the current moment with true anatomical significance.

[0097] See attached document Figure 8 The pose output module is located at the end of the system data stream and receives the three-dimensional coordinate sequence of human posture output by the state update constraint module. This module converts the discrete coordinate point cloud into hierarchical rotation parameters that meet the inverse kinematics requirements of the downstream controller through spatial geometric transformation and communication protocol encapsulation, and performs fixed-frequency transmission of data packets.

[0098] The system determines the parent-child connection hierarchy and membership relationship of each human skeleton node in the topology based on a predefined human kinematics hierarchy tree. The system uses the pelvic node or the root node of the spine as the global root node and extracts the global absolute translation of the human body using the three-dimensional coordinates of this node in the visual coordinate system.

[0099] The system traverses all child nodes sequentially along a predefined human kinematics hierarchy tree, constructing a local spatial coordinate system at each node. The system extracts the 3D coordinate vectors of the currently traversed node and its adjacent parent nodes, subtracts the two to obtain the longitudinal direction vector of the skeletal link, and normalizes it to serve as the principal axis of the local coordinate system. Combining a preset reference orientation vector or vectors from other adjacent skeletal links, the system uses the Schmidt orthogonalization method to derive mutually perpendicular normal and tangential vectors, thereby establishing a local node coordinate system with three orthogonal bases.

[0100] The system calculates the relative rotational transformation relationship between the coordinate systems of adjacent parent and child nodes in the human kinematics hierarchy tree. The system multiplies the basis matrix of the child node's local coordinate system by the inverse of the basis matrix of the parent node's local coordinate system to generate a three-dimensional rotation matrix representing the relative spatial orientation of the two nodes.

[0101] The system converts the calculated 3D rotation matrix into relative rotation quaternions. Using quaternions for attitude representation avoids gimbal lock-up defects that may occur when using Euler angles under specific physical poses. The system extracts the diagonal elements of the 3D rotation matrix to calculate the trace of the matrix, and then solves for the real part and three imaginary parameters of the quaternion based on the value of the trace. The conventional positive trace conversion logic is shown in the following equation: ; ; ; ; In the formula, Represents the real part of a relative rotation quaternion; , , These represent the imaginary components of the relative rotation quaternion on the three spatial axes, respectively. The trace of a three-dimensional rotation matrix is ​​the sum of the elements on the main diagonal of the matrix. Represents the position of the 3D rotation matrix at the th position. Line number The elements of the column. For the quaternion piecewise solution logic when the trace of the matrix is ​​less than or equal to the negative one boundary condition, those skilled in the art can use the corresponding numerically stable branch algorithm for calculation. The specific quaternion boundary transformation is a well-known technology in this field and will not be elaborated here.

[0102] The system combines the acquired absolute pose of the global root node and the relative rotation quaternion sequences of each child node, and encapsulates them into a predefined data structure according to the traversal order of the human kinematics hierarchy tree. The system then converts this data structure into a byte stream sequence corresponding to a standard communication message.

[0103] The system configures the communication port and transmission parameters of the industrial Ethernet bus. The system reads the target receiving frequency set by the lower-level industrial robot controller or digital twin platform. Based on the set target receiving frequency, the system periodically starts a network transmission thread, sending the generated standard communication messages externally via the physical network port. Through continuous output at a fixed frequency, the system provides real-time rigid body rotation states of various joints in the human body to upper-level applications.

[0104] For the handshake connection and message verification mechanism of the underlying communication protocol of industrial Ethernet, those skilled in the art can configure it in combination with specific user datagram protocols or bus standards. The network communication implementation process is well known in the field and will not be described in detail here. To more clearly illustrate the collaborative working mechanism and technical advantages of the label-free real-time human posture reconstruction system based on multimodal sensor fusion and its various functional modules in a real industrial environment, this section combines a specific human-machine collaborative assembly application embodiment to conduct detailed experimental verification and comparative analysis of the technical effects of the present invention.

[0105] This embodiment uses an automated assembly line for automotive parts as the experimental background. In this scenario, an operator performs the positioning and inspection of precision parts at a workstation, while a six-DOF industrial robot collaborative arm is responsible for transporting heavy components and performing palletizing operations around the operator. Due to the high overlap of the workspaces, the collaborative arm frequently and randomly obstructs the depth camera mounted on the support during its movement, causing instantaneous loss of visual observation data or severe noise interference.

[0106] The system operates according to the following steps: Deployment phase: The depth camera and the frequency-modulated continuous wave millimeter-wave radar are mounted 15cm apart horizontally above the workstation. A 60Hz synchronization pulse is sent through the programmable logic controller to ensure that the sampling phases of the two are completely consistent.

[0107] Benchmark establishment: In a laboratory environment, using the Vicon high-precision optical motion capture system as a true reference, infrared reflective markers are pasted on the operator's key joints (such as elbows, wrists, knees, etc.) to establish a millimeter-level true reference coordinate system.

[0108] Dynamic occlusion simulation: During assembly, the collaborative arm sweeps across the front of the camera at a speed of 1.5 m / s. At this time, the mask projection module senses the occurrence of visual blind spots in real time and sets the spatial occlusion mask of the corresponding joint to a physical occlusion state.

[0109] To quantify the reliability of the invention under extreme occlusion conditions, three representative technical paths were selected for comparative analysis.

[0110] Comparison with Option A: Using only a single depth camera for pose estimation.

[0111] Comparison Scheme B: adopts a conventional weighted fusion scheme of vision and radar, but does not include the state dimensionality reduction prediction and biomechanical truncation module of this invention.

[0112] The present invention employs a complete system that includes tangential relaxation compensation and Lagrange projection truncation.

[0113] See attached document Figure 9 The figure reflects the trend of human wrist joint reconstruction error over time during the complete cycle (approximately 2000 ms) of the collaborative arm sweeping across the visual field.

[0114] Accuracy performance analysis: In the critical interval where occlusion occurs (800ms to 1300ms in the figure), the average joint position error of Comparative Solution A spikes sharply to over 300mm, exhibiting significant coordinate divergence and limb disappearance. While Comparative Solution B avoids complete loss by incorporating radar data, its reconstruction error remains around 120mm, and the trajectory displays irregular high-frequency jitter. In contrast, the solution of this invention, through state dimensionality reduction prediction module to compress the covariance space and tangential relaxation compensation mechanism to compensate for the arm movement, keeps the error consistently within 45mm, resulting in an extremely smooth curve that closely matches the true value.

[0115] The system uses the average position error per joint as the core accuracy evaluation index, and its calculation formula is as follows: ; In the formula, This represents the average positional error of each joint; This indicates the total number of frames in the test sequence; This indicates the total number of key joints in the human body. Indicates the first The first frame reconstructed by the system The three-dimensional coordinates of each joint; This represents the baseline coordinates output by the corresponding motion capture system.

[0116] Biomechanical consistency verification: Comparison revealed that in the output of Comparison Scheme B, the length of the human upper arm skeleton fluctuated by ±15% due to radar noise during the obstruction period. In contrast, the present invention, through the spatial geometry truncation module, forcibly maps the output back to a geometric manifold surface conforming to human physiological characteristics, ensuring that the variation rate of the skeleton link length is less than 0.5% throughout the reconstruction process. This physically and logically eliminates unnatural phenomena such as severed arms or limb stretching.

[0117] Real-time assessment: Experimental data shows that the end-to-end latency of the proposed solution on mainstream computing platforms is consistently 12.8ms with a standard deviation of only 1.2ms. This means that the system can stably output high-precision pose information at a frame rate of over 70fps, fully meeting the performance requirements for industrial-grade real-time collision avoidance monitoring and collaborative control.

[0118] In summary, this embodiment fully verifies that the present invention solves the core technical pain points of low reconstruction accuracy, divergent motion trajectories, and lack of physical logic caused by visual occlusion in complex industrial scenarios through multimodal deep fusion and biomechanical constraints.

Claims

1. A label-free real-time human pose reconstruction system based on multimodal sensor fusion, characterized in that, include: The data acquisition and registration module is used to simultaneously acquire depth images and millimeter-wave radar point clouds of the target scene, and to unify the millimeter-wave radar point clouds to the visual coordinate system of the depth image. The mask projection module is used to determine whether each human skeleton node is physically occluded based on the depth contour of the depth image and the estimated position of the human skeleton node output by the system at the previous moment, and to generate the corresponding spatial occlusion mask. The state dimensionality reduction prediction module is used in the prediction stage of nonlinear state estimation. When the spatial occlusion mask indicates that the human skeleton node is physically occluded, it extracts the Doppler radial velocity and direction in the millimeter-wave radar point cloud, orthogonally decomposes the motion state space of the human skeleton node into parallel subspaces and orthogonal subspaces, and applies dynamic damping constraints to the process noise covariance of the orthogonal subspace to output the state prediction result. The state update constraint module is used to dynamically adjust the measurement noise covariance weight of visual observation according to the spatial occlusion mask, update the state in combination with the state prediction result, and perform spatial geometric truncation on the updated state based on human biomechanical boundary conditions, and output the three-dimensional coordinates of the human posture at the current moment.

2. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 1, characterized in that, The data acquisition and registration module includes a programmable logic controller, a depth camera, and a frequency-modulated continuous wave millimeter-wave radar. The programmable logic controller is used to output a reference trigger pulse to force synchronization between the global exposure period of the depth camera and the frequency modulation transmission period of the frequency-modulated continuous wave millimeter-wave radar. The data acquisition and registration module maps the point cloud in the radar local coordinate system to the visual coordinate system through a preset rigid body transformation matrix.

3. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 1, characterized in that, The specific operating logic of the mask projection module is as follows: Extract the estimated position of the human skeleton nodes from the previous moment and back-project it onto the pixel plane of the camera; The measured depth value at the corresponding position of the pixel plane is compared with the depth axis component of the estimated position of the human skeleton node. If the measured depth value is less than the difference between the depth axis component and the preset depth tolerance margin, the human skeleton node is determined to have entered the visual blind zone, and the spatial occlusion mask is assigned a physical occlusion state.

4. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 1, characterized in that, It also includes a gated pre-filtering module, which is used for: The depth coordinates of the millimeter-wave radar point cloud are compared with the physical occlusion thickness of the corresponding pixels in the depth image, and multipath reflection radar points whose depth coordinates exceed the physical occlusion thickness are removed. For the remaining millimeter-wave radar points, the innovation vector and innovation covariance matrix are calculated by combining the prior prediction distribution of nonlinear state estimation. By solving the Mahalanobis distance and performing the chi-square test, abnormal radar points that violate the kinematic coherence of the human body are eliminated.

5. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 1, characterized in that, The specific implementation method for applying dynamic damping constraints to the process noise covariance of the orthogonal subspace in the state dimensionality reduction prediction module is as follows: Parallel projection operators and orthogonal projection operators are constructed using the direction unit vector of the Doppler radial velocity; The process noise covariance matrix for system state prediction is divided into a first part controlled by the parallel projection operator and a second part controlled by the orthogonal projection operator. The second part is multiplied by a dynamic damping coefficient to suppress the random walk degrees of freedom of the human skeleton nodes in the direction orthogonal to the radar line of sight, thereby reducing the three-dimensional motion uncertainty to a local manifold constrained by the Doppler radial velocity.

6. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 5, characterized in that, The dynamic damping coefficient is controlled by a tangential relaxation mechanism, the specific logic of which is as follows: Obtain the angular velocity vector of the adjacent parent node of the physically occluded human skeleton node at the previous moment, and combine it with the skeleton link vector to calculate the prior tangential predicted velocity of the physically occluded human skeleton node by cross product. The prior tangential predicted velocity is projected onto the orthogonal subspace to obtain the tangential energy scalar. When the Doppler radial velocity is lower than the preset velocity noise floor threshold, the dynamic damping coefficient is increased by the tangential energy scalar to relax the constraints of the orthogonal subspace and allow the predicted state to evolve tangentially.

7. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 1, characterized in that, The logic for dynamically adjusting the measurement noise covariance weights of visual observations in the state update constraint module is as follows: When the spatial occlusion mask indicates that the human skeleton node is not physically occluded, the basic calibration noise covariance for visual observation is maintained. When the spatial occlusion mask indicates that the human skeleton node is physically occluded, a preset penalty multiplier parameter is applied to the measurement noise covariance of the visual observation, so that the Kalman gain of the nonlinear state estimation ignores the optical observation that is currently invalid.

8. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 1, characterized in that, The human biomechanical boundary conditions in the state update constraint module include zero-order Euclidean distance constraints and first-order rotation constraints. The zero-order Euclidean distance constraint limits the spatial distance between adjacent human skeleton nodes to the prior calibration rod length. The first-order rotational constraint limits the joint bending angle formed by three adjacent human skeletal nodes to within a preset physiological limit angle range.

9. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 8, characterized in that, The specific logic for spatial geometric truncation of the updated state based on human biomechanical boundary conditions is as follows: If the unconstrained state update coordinates violate the zero-order Euclidean distance constraint or the first-order rotation constraint, then the updated state is used as the objective function, the human biomechanical boundary conditions are used as equality or inequality constraints, the Lagrange multiplier method is used to solve for the optimal projection solution with the minimum Mahalanobis distance from the original state estimate, and the optimal projection solution is used as the final output three-dimensional coordinates of the human posture.

10. The label-free real-time human pose reconstruction system based on multimodal sensor fusion according to claim 1, characterized in that, It also includes a pose output module, which receives the three-dimensional coordinates of the human body posture output by the state update constraint module, converts them into a relative rotation quaternion matrix of parent and child nodes containing the human body hierarchy, and outputs them continuously to an external receiving device at a fixed frequency via an industrial Ethernet bus.

Images