Burn hand training posture reconstruction and correction method based on vision and inertial navigation fusion

Abstract

The invention relates to a post-burn hand training posture reconstruction and correction method based on vision and inertial navigation fusion. The method comprises the following steps: acquiring a hand vision sequence and IMU angular velocity or acceleration; detecting starting, stopping and turning boundary events in the two paths of data, pairing in sequence, and aligning by taking the pairing events as anchor points to form action fragments; performing kinematics consistency constraint test on the visual postures in the segments to output visual evidences, wherein the visual evidences comprise consistency test passing or non-passing marks and illegal joint sets; judging drifting of the IMU attitude increment, outputting inertial navigation evidences including a drifting mutation judgment result and a freezing state mark, and freezing inertial navigation contribution when drifting mutation exceeds a threshold value; performing evidence priority arbitration based on visual evidence, inertial navigation evidence and freezing state according to joints; when the vision does not pass the constraint and the inertial navigation contribution is frozen, maintaining the degradation output of the fusion attitude passing the consistency test of the previous frame; carrying out local correction on the unconstrained joints according to the evidence; and outputting the corrected fusion attitude and the dominant label.

Description

Technical Field

[0001] This invention relates to the field of computer vision and inertial sensor fusion technology, specifically a method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion. Background Technology

[0002] In existing motion capture and attitude reconstruction technologies based on vision and inertial navigation fusion, most solutions remain at the level of data alignment and basic fusion. For example, patent document CN116264621A discloses a method for aligning IMU data and video data. Its core lies in initially aligning the first IMU data with a video I-frame and establishing a correspondence between the video frame number and the IMU identifier number. In subsequent sampling processes, verification and correction are performed based on the sampling frequency ratio to ensure the synchronization of video and IMU data on the time axis. This type of technology addresses the time alignment problem, not the attitude quality control problem. Its focus is on whether the correspondence between frames and data entries matches, without deeply addressing the impact of visual attitude misidentification, occlusion errors, abnormal joint shaking, or sudden IMU drift on the fusion results. Especially in post-burn hand training scenarios, patients' hands exhibit characteristics such as scar contracture, limited movement, irregular posture, and small, non-periodic motion amplitude. Simply relying on frame number alignment and frequency ratio verification is insufficient to guarantee the accuracy and reliability of attitude reconstruction. Existing alignment techniques typically assume that visual and IMU data are physically equivalent and can be directly fused. However, when visual keypoints experience local drift or IMU experiences transient changes, the system lacks a mechanism to identify and isolate the source of the anomaly. This can easily lead to the continued input of erroneous data into the fusion model, resulting in overall pose distortion and affecting the training and evaluation results.

[0003] Furthermore, existing vision-inertial navigation fusion methods mostly employ uniform weights or filtering for posture fusion. Abnormal data processing typically involves smoothing, filtering, or simple threshold removal, lacking a mechanism for localization and responsibility separation for individual joints. During post-burn hand training, due to limited joint movement or significant compensatory actions, some joint postures may frequently violate normal kinematic constraints. However, current technologies often propagate errors across the entire kinetic chain for overall correction, lacking a violation localization strategy based on single-joint exclusion, and failing to accurately distinguish between local and global anomalies. Simultaneously, for IMU drift issues, current technologies mainly rely on frequency verification or data addition / reduction correction, lacking a freeze and hysteresis release mechanism based on drift mutation detection. When sudden posture changes occur, the system may continue to output erroneous postures, lacking a degradation output strategy under double unreliability conditions. This deficiency is particularly evident in rehabilitation training scenarios; once posture continuity is disrupted, the stability and reliability of training feedback will significantly decrease. Therefore, although existing solutions based on vision and inertial navigation fusion have solved the data synchronization problem, they still have significant shortcomings in anomaly recognition, evidence classification, joint-level responsibility localization, freeze control and phased threshold adjustment, making it difficult to meet the high stability and high reliability requirements of fine-grained training posture reconstruction and correction of the hand after burns. Summary of the Invention

[0004] The purpose of this invention is to provide a method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion, thereby addressing some of the drawbacks and shortcomings pointed out in the background art.

[0005] The present invention adopts the following technical solution to solve the above-mentioned technical problems: a method for reconstructing and correcting the training posture of the hand after burns based on vision and inertial navigation fusion, including: acquiring the visual sequence of the hand and the angular velocity or acceleration of the IMU; detecting start, stop, and turning boundary events in the two data channels and pairing them in sequence, and aligning the paired events as anchor points to form action segments;

[0006] The kinematic consistency constraint test is performed on the visual pose within the segment and visual evidence is output. The visual evidence includes a pass or fail flag for the consistency test and a set of non-compliant joints. The drift of the IMU pose increment is determined and inertial navigation evidence is output. The inertial navigation evidence includes the drift mutation determination result and a frozen state flag. When the drift mutation exceeds a threshold, the inertial navigation contribution is frozen.

[0007] The joints are arbitrated based on visual evidence, inertial navigation evidence, and frozen state, and either visual or inertial navigation is selected as the dominant force to obtain the fused pose. When the visual pose fails the constraint and the inertial navigation contribution is frozen, the fused pose that passed the consistency check in the previous frame is maintained and downgraded for output. The joints that fail the constraint are locally corrected based on the evidence to obtain the corrected fused pose. The corrected fused pose and the dominant label are output.

[0008] Furthermore, the action segment alignment includes event despurification: boundary events are paired only when there are candidate events of the same type and with the same relative order in another data within a preset time window, and the event time difference is within the preset time window; when there is repeated starting or stopping, stopping within a preset duration threshold after starting, or a turning point before starting, the boundary event is determined to be invalid and not used as an anchor point.

[0009] Furthermore, when the kinematic consistency constraint check fails, a set of non-compliant joints is output. The set of non-compliant joints is determined by single-joint exclusion: each joint is excluded from the pose solution using its current frame observation value and its consistency is rechecked. If the recheck changes from failing to passing, the joint is recorded as a non-compliant joint. The set of non-compliant joints is used to limit the objects of the arbitration fusion and local correction.

[0010] Furthermore, the frozen inertial navigation contribution is released using hysteresis: during freezing, the contribution of the inertial navigation in the fusion is set to zero or maintained at the reliable value before freezing; after freezing, the visual system must pass consistency for multiple consecutive frames and there must be no evidence of sudden drift in the inertial navigation during the freezing period before it can be released; during release, the reliable attitude before freezing is used as a reference, and the increment of the freezing period is compensated once before participating in the evidence priority arbitration.

[0011] Furthermore, the re-inspection of consistency includes comparing the change in bone segment length, the increment of joint angle, and the deviation of the position of the endpoint of the joint chain with the corresponding preset thresholds. If all of them are satisfied, the kinematic consistency constraint test is deemed to have passed. If the test changes from failing to passing after a certain joint does not use the current frame observation value to participate in the attitude solution, the joint is recorded as a violation joint and added to the violation joint set.

[0012] Furthermore, the single joint exclusion process sequentially excludes each joint from using the current frame observation value in attitude solving according to a preset proximal-to-far order and performs the re-inspection consistency check; after a joint is marked as a violation joint, the joint is kept from using the current frame observation value in attitude solving during the re-inspection of the remaining joints, so as to determine the set of violation joints corresponding to the one that passed the inspection.

[0013] Furthermore, the set of non-compliant joints is determined by temporal filtering: the number of consecutive frames in which a joint is recorded as non-compliant within an action segment is counted, and only joints with a consecutive frame count not less than a preset frame count threshold are retained as the final set of non-compliant joints, which is used to limit the objects of the arbitration fusion and local correction.

[0014] Furthermore, the preset thresholds are switched according to the action segment stage: a first set of thresholds is used between start and stop, and a second set of thresholds is used within a preset number of frames before and after the transition event. The second set of thresholds is less than the corresponding thresholds in the first set of thresholds. The re-inspection consistency is determined to pass or fail based on the comparison result of the switched thresholds.

[0015] Furthermore, before adding the violating joint to the set of violating joints, reverse verification is performed: after the test changes from failing to passing by not using the current frame observation value of the joint, the current frame observation value of the joint is restored and retested; if the retest still fails, the joint is added to the set of violating joints, otherwise it is not added.

[0016] Furthermore, the stage switching of the preset threshold adopts hysteresis determination: when a turning event is detected, it enters the second set of thresholds and remains there until no new turning event is detected for a consecutive preset number of frames, then exits the second set of thresholds; when a turning event is detected again during the holding period, the count of the preset number of frames is reset; the re-inspection consistency is compared with the threshold group determined by the hysteresis determination and determined to pass or fail.

[0017] The beneficial effects of this invention are as follows: By constructing a semantic-level alignment mechanism based on action boundary events, this invention achieves accurate matching of visual data and inertial navigation data at the action segment level, avoiding the cumulative errors and misalignments caused by relying solely on time synchronization. By introducing kinematic consistency constraint checks and single-joint exclusion mechanisms, it can accurately locate violating joints when visual posture is abnormal, and only restrict the processing of abnormal joints, preventing errors from spreading throughout the entire kinematic chain. Simultaneously, combined with inertial navigation drift discrimination and freezing mechanisms, it promptly suppresses erroneous data from participating in fusion when abrupt posture changes are detected, improving the stability and reliability of posture reconstruction results.

[0018] Furthermore, this invention achieves dynamic complementarity between visual and inertial navigation data through evidence priority arbitration, degraded output, and local correction strategies. This ensures continuous attitude output even when both source data exhibit uncertainty, preventing attitude jumps. Combined with threshold stage switching and hysteresis judgment mechanisms, it improves consistency verification accuracy during complex action stages such as turning points, reducing the false positive rate. The overall method, while maintaining real-time performance, enhances resistance to occlusion, drift, and transient anomalies, significantly improving the accuracy, stability, and traceability of attitude reconstruction and correction during post-burn hand training. Attached Figure Description

[0019] Figure 1 This is a flowchart of the attitude arbitration logic of the vision and inertial navigation fusion of the present invention.

[0020] Figure 2 This is a schematic diagram of the event anchor point pairing and pseudo-alignment effect in Embodiment 1 of the present invention.

[0021] Figure 3 This is a schematic diagram of the drift score Dt and the freeze hysteresis release process in Embodiment 1 of the present invention.

[0022] Figure 4 This is a schematic diagram showing the comparison of fingertip error changes and comprehensive evaluation before and after one-time compensation in Embodiment 1 of the present invention.

[0023] Figure 5 This is a schematic diagram of the switching process between two sets of thresholds and hysteresis in Embodiment 2 of the present invention.

[0024] Figure 6 This is an example diagram of the three criteria for consistency verification, single-joint exclusion, and reverse verification in Embodiment 2 of the present invention.

[0025] Figure 7 This is a schematic diagram showing the comparison of the timing screening results and the effects before and after local correction in Embodiment 2 of the present invention, as well as the evaluation results. Detailed Implementation

[0026] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] Combined with appendix Figure 1 This invention relates to a method for reconstructing and correcting post-burn hand training posture based on the fusion of vision and inertial navigation. First, visual sequence data of the hand is acquired. Then, image frames of key points of the hand are continuously acquired using a camera device, and the spatial position sequence of each joint in the image coordinate system is extracted. Simultaneously, angular velocity or acceleration data output from an inertial measurement unit fixedly connected to the hand is acquired and time-stamped, ensuring that the visual data and inertial navigation data are on the same time axis.

[0028] After data preprocessing, motion boundary events are monitored in both visual and inertial navigation sequences. For visual data, start-up events, stop-up events, and turning events caused by changes in motion direction are identified based on the overall displacement trend of key points. For inertial navigation data, corresponding start-up, stop-up, and turning events are identified based on changes in angular velocity or acceleration amplitude. The boundary events monitored in the two datasets are matched in order of occurrence, and events that are of the same type and whose time difference is within a preset time window are considered valid pairings.

[0029] Using the boundary events that complete the pairing as anchor points, the visual and inertial navigation data are segmented, and the data between two adjacent anchor points is defined as the same action segment. This segmentation method achieves alignment of the two data streams at the action semantic level, providing a consistent segment basis for subsequent pose fusion and correction.

[0030] After aligning the motion segments, a kinematic consistency constraint check is performed on the visual pose results within each motion segment. Based on the pre-defined skeletal topology of the hand, the changes in bone segment length between adjacent joints, the angular increments between adjacent frames for each joint, and the positional deviations of the joint chain endpoints relative to the reference coordinate system are calculated. These calculation results are compared with corresponding pre-defined thresholds. If all comparison results meet the threshold conditions, the visual pose of that frame is considered to have passed the kinematic consistency constraint check; otherwise, it is considered to have failed. For cases that fail the constraint check, a single-joint exclusion method is used to individually mask the current frame observations of each joint in the visual pose and the consistency check is re-executed. When the check result changes from failing to passing after masking a certain joint, that joint is identified as a violation joint. The output visual evidence includes a consistency check pass / fail flag and the identified set of violation joints.

[0031] Simultaneously, drift detection is performed on the inertial navigation attitude increments within the same action segment. The trend of attitude increment changes is calculated based on the integral results of the angular velocity or acceleration output by the inertial navigation system, and any abrupt amplitude changes are monitored. When the change in attitude increment within a preset time window exceeds a preset drift abruptness threshold, a drift abruptness is determined, and a drift abruptness detection result is generated. When a drift abruptness occurs, the inertial navigation contribution in the fusion calculation is maintained at the inertial navigation contribution value at the moment the drift was triggered, and a freeze state flag is set. During the freeze state, no new inertial navigation attitude increments are used in the fusion calculation. The output inertial navigation evidence includes the drift abruptness detection result and the corresponding freeze state flag, which are used for subsequent attitude arbitration and correction processing.

[0032] After obtaining visual and inertial navigation (INS) evidence, evidence priority arbitration is performed on each joint within each action segment. For each joint, the visual consistency check flag, the set of non-compliant joints, the INS drift judgment result, and the freeze status flag of the corresponding frame are read. If the visual consistency check passes and the joint does not belong to the set of non-compliant joints, the visual pose is used as the dominant data for that joint; if the visual consistency check fails and the INS is not in a frozen state, the INS pose increment is used as the dominant data for that joint; when the visual consistency check fails and the INS is in a frozen state, the current frame's visual or INS data is not used for updates.

[0033] Under the aforementioned dual unreliability conditions, the system calls the fused pose from the previous frame that passed the kinematic consistency check as the output result of that joint in the current frame, forming a degraded output to ensure the continuity and stability of the pose sequence. After determining the dominant source of each joint, the selected dominant data are combined to form the initial fused pose of the current frame.

[0034] For joints that fail the visual consistency check and are identified as violations, local corrections are performed based on visual and inertial navigation (INS) evidence. If the violation stems from visual observation anomalies and the INS is not frozen, the joint is corrected using INS attitude increments; if the INS exhibits drift abruptly and is frozen, the previous reliable fused attitude is retained as the joint's current value. Local corrections only apply to violating joints and do not alter the data of other joints that have passed the check. The corrected fused attitude is then obtained, and the dominant source label for each joint is simultaneously output for subsequent state recording and processing.

[0035] To avoid misjudgments caused by noise or instantaneous fluctuations during motion segment alignment, event desmearing is further performed after boundary event monitoring. For start, stop, and turn events detected in visual and inertial navigation data respectively, a preset time window is first used as a constraint to search for candidate events of the same type in the other data. Only when the event types in the two data streams are consistent, the order of occurrence is consistent, and the time difference between the two events is within the preset time window, are they considered valid pairings. If no candidate event meeting the conditions is found within the preset time window, the boundary event will not participate in subsequent segment division.

[0036] Simultaneously, logical consistency checks are performed on the boundary event sequences within a single data stream. When two consecutive start events or two consecutive stop events are detected, the latter event is determined to be a duplicate and marked as invalid. When a stop event is detected within a preset duration threshold after a start event, it is determined that the start and stop events do not constitute a valid action range, and both events are marked as invalid. When a turning event is detected without a start event, it is determined that the turning event lacks a valid premise and is also marked as invalid. All boundary events marked as invalid are not used as anchor points for action segment alignment; only valid paired events that have undergone false positive removal are retained to determine the action segment range, thereby improving the reliability of visual and inertial navigation data alignment.

[0037] When the inertial navigation attitude increment is determined to have experienced a drift abrupt change, the inertial navigation contribution freeze mechanism is triggered. During freeze, the inertial navigation contribution in the fusion calculation is fixed to the inertial navigation contribution value at the moment the freeze is triggered, and the use of inertial navigation attitude increments from subsequent frames in the fusion calculation is stopped. A freeze state flag is also generated. During the freeze period, the visual attitude continues to undergo kinematic consistency constraint checks to determine the system recovery conditions.

[0038] To avoid attitude jitter caused by frequent switching, this embodiment employs a hysteresis release strategy. Unfreezing requires two conditions to be met simultaneously: first, the visual attitude passes the kinematic consistency constraint test for multiple consecutive frames not less than a preset frame threshold; second, no further evidence of inertial navigation drift abrupt change is detected during the freeze period. The freeze state can only be lifted when both conditions are met.

[0039] Upon unfreezing, the reliable fused attitude of the last frame before the freeze was triggered is used as the reference attitude. The accumulated inertial navigation attitude increments during the freeze period are then superimposed onto this reference attitude to obtain the updated inertial navigation attitude result. Subsequently, the inertial navigation data is restored to participate in the evidence priority arbitration, jointly determining the dominant source of each joint with visual evidence, thereby suppressing abrupt errors caused by drift while ensuring attitude continuity.

[0040] When the kinematic consistency constraint check for visual pose fails, the system enters the violation joint localization process. The system records the current frame's failure to pass the consistency check and initiates a single-joint exclusion process. Following a preset joint order, exclusion operations are performed on each joint sequentially. This means that the visual observation value of the joint in the current frame is not used during pose calculation; instead, the joint pose value from the previous reliable frame is used. Then, the kinematic consistency constraint check is re-executed.

[0041] If, after excluding a certain joint, the consistency constraint check result changes from failing to passing, then that joint is identified as a non-compliant joint causing inconsistency and added to the set of non-compliant joints. The same exclusion and re-check steps are then performed on the remaining joints to identify multiple potentially non-compliant joints. This ultimately forms the set of non-compliant joints corresponding to the current frame.

[0042] The set of non-compliant joints serves as the limiting object for subsequent evidence priority arbitration and local correction. During the arbitration process, inertial navigation evidence or reliable historical values ​​are prioritized for non-compliant joints; during the local correction process, correction operations are performed only on non-compliant joints, while the original fusion results are left unchanged for joints not included in the set of non-compliant joints, thereby achieving local control of errors and avoiding unnecessary disturbances to the overall attitude structure.

[0043] During single-joint exclusion localization, the posture results after each exclusion are re-checked for consistency. This includes calculating the change in length of each bone segment relative to the baseline length within the current motion segment, the angle increment of the same joint in adjacent frames, and the positional deviation of the joint chain endpoints from the wrist to the fingertips, and comparing each with a corresponding preset threshold. If all the above indicators meet the corresponding threshold conditions, the kinematic consistency constraint test is considered passed; if any indicator fails the threshold condition, it is considered failed. When the consistency test changes from failed to passed after excluding a joint from its current frame observations in the posture solution, that joint is identified as a violation joint and added to the violation joint set.

[0044] Single joint elimination is performed sequentially from proximal to distal according to a preset order. For each joint, the current frame's observations are not used in the pose calculation, and a consistency check is performed. Once a joint is marked as a violation joint, when eliminating and checking the remaining joints, the identified violation joint remains in a state where the current frame's observations are not used in the pose calculation. This ensures that the check result corresponds to the consistency pass state, thereby determining the complete set of violation joints.

[0045] To improve the stability of violation detection, the obtained set of violating joints undergoes further temporal filtering. The number of consecutive frames in which each joint is recorded as a violating joint within the same motion segment is counted. Only joints with a consecutive frame count not less than a preset frame threshold are retained as members of the final set of violating joints. This final set of violating joints is used to define the objects for subsequent arbitration fusion and local correction, thus avoiding the impact of transient fluctuations on the overall pose reconstruction results.

[0046] To improve the adaptability of kinematic consistency constraint testing across different movement stages, preset thresholds are switched in stages. Based on the alignment results of completed movement segments, initiation events, stopping events, and transition events within each segment are identified. The interval between the initiation event and the stopping event is defined as the stable movement stage. Within this stage, the first set of thresholds is used to compare and determine the changes in bone segment length, joint angle increments, and deviations in the position of the articular chain endpoints.

[0047] When a transition event is detected, the transition phase is defined as the frame centered on the transition event and the range of a preset number of frames before and after it. Within the transition phase, a second set of thresholds is used for consistency comparison. The second set of thresholds is lower than the corresponding thresholds in the first set of thresholds to improve sensitivity to posture anomalies. Through this phase division, the inspection criteria are adaptively adjusted under different motion states.

[0048] During the consistency check, the system automatically selects the corresponding threshold group based on the stage to which the current frame belongs, and compares the calculated changes in bone segment length, joint angle increments, and endpoint position deviations with the selected threshold group. When all indicators meet the corresponding threshold conditions, the kinematic consistency constraint check is deemed passed; otherwise, it is deemed failed. This enhances the ability to identify anomalies during transition phases while ensuring the continuity of movement.

[0049] After identifying candidate erroneous joints through single-joint elimination, a reverse verification process is performed before adding the joint to the erroneous joint set to avoid misjudgment. Specifically, when a joint is not used for pose solving based on its current frame observations, and the kinematic consistency constraint check changes from failing to passing, the joint is first recorded as a candidate erroneous joint, but is not immediately added to the erroneous joint set.

[0050] The current frame observations of the joint are then used for pose determination, and the kinematic consistency constraint check is re-executed. If the consistency check fails again after restoration, it indicates that the current frame observations of the joint do indeed cause pose inconsistency, and the joint is formally added to the set of illegal joints. If the consistency check passes after restoration, it indicates that the previous consistency changes were unstable or affected by other factors, and the joint is not added to the set of illegal joints. Through the reverse verification step, a secondary confirmation of candidate illegal joints is achieved, reducing the probability of misjudgment caused by instantaneous noise or calculation errors.

[0051] To avoid fluctuations in consistency test results due to frequent threshold switching during transition phases, a hysteresis judgment mechanism is introduced during phase switching. When a transition event is detected, the system immediately switches the current threshold group to the second threshold group and enters a hold state. During the hold state, the system does not immediately revert to the first threshold group if no transition event is detected in a single frame; instead, it performs statistical analysis on consecutive frames.

[0052] Only when no new transition events are detected for a consecutive preset number of frames is the current action determined to have exited the transition phase, and the system switches the threshold group from the second group back to the first group. During the state holding period, if a transition event is detected again, the consecutive frame count restarts, and the second group of thresholds is maintained.

[0053] During the consistency review, the system selects the currently valid threshold group based on the hysteresis judgment result, and compares the changes in bone segment length, joint angle increment, and joint chain endpoint position deviation with the corresponding thresholds in the selected threshold group. If each indicator meets the corresponding threshold condition, the consistency check is considered passed; otherwise, it is considered failed. This hysteresis mechanism ensures stability in threshold switching, reducing misjudgments caused by short-term fluctuations.

[0054] Example 1:

[0055] In this embodiment, the rehabilitation scenario is the fifth week of rehabilitation training for patient A's hand contracture after burns. The therapist requires the patient to complete three cycles of opening and closing the fist and three cycles of thumb-to-finger opposition within 2 minutes, and to add a wrist pronation-to-reversion transition movement at the end of each cycle to train forearm rotation ability. The patient wears a miniature inertial navigation unit on the back of the hand for 100Hz sampling and a ring inertial navigation unit for 100Hz sampling. A fixed camera in front captures visual sequences of the hand at 30fps, and the visual side outputs a sequence of 21 key points of the hand and their confidence levels.

[0056] The system monitors both visual and inertial navigation data for start, stop, and transition events, respectively. Start indicates the movement speed exceeds a threshold from rest; stop indicates the movement speed drops below the threshold and remains there; and transition indicates a stable flip of the wrist angular velocity sign within a short window. The monitoring results are sorted chronologically and cross-modal pairing is attempted. Paired events are used as anchor points to divide the 2-minute data into multiple action segments, and pose reconstruction, fusion, and correction are performed within each segment.

[0057] To avoid mismatches and misslices, this embodiment performs a descaling rule on boundary events. Only when another data stream is within a preset time window... Memory pairs are only used when candidate events of the same type have the same relative order. If repeated starts or stops occur, or if the startup time exceeds the minimum duration threshold... If the event stops within the specified range, or if a turning point is detected before any start-up occurs, the boundary event is considered invalid and not used as an anchor point.

[0058] Taking a sequence of opening and closing punches combined with wrist turning as an example, the visual system originally detected a start of 12.430s, a turning of 37.210s, and a stop of 44.980s, while the inertial navigation system originally detected a start of 12.455s, a turning of 37.185s, and a stop of 44.955s. Without artifact removal, there is also a visual artifact of 12.590s caused by light reflection and an inertial navigation artifact of 12.640s, which would cause the segment to be incorrectly cut. After artifact removal, these artifacts are eliminated and anchor point pairing is completed. The start and end points of the segment are maintained so that the start of 12.430s is aligned with 12.455s and the stop of 44.980s is aligned with 44.955s, as shown in the attached figure of the embodiment. Figure 2 As shown.

[0059] Alignment error is defined as the absolute value of the time difference between paired events of the same type, denoted as . Statistically analyze all paired events in 2-minute data sets, and average the results before removing false positives. And the largest After removing false positives, the average And the largest The alignment stability is significantly improved, making the time consistency of subsequent cross-modal fusion more reliable. Figure 2 The pairing diagram of the fragment-level event timeline and anchor point connection is given, and the pseudo-event removal process is marked. The above statistical results before and after pseudo-event removal are the statistics of all paired events in 2 minutes, which are used to illustrate the improvement of event pseudo-event removal on the overall alignment stability, thereby reducing the impact of cross-modal time mispairing fusion and improving the reliability of action segment segmentation.

[0060] Within a segment, each frame on the visual side first undergoes a kinematic consistency constraint check to form visual evidence, outputting a pass or fail flag. If a pass is not achieved, a set of violating joints is provided for subsequent local correction object limitation. The constraint check can be implemented using a combination of bone segment length stability and joint angular velocity upper limit and endpoint position residuals. In this embodiment, the pass flag of this check is used as the visual evidence input arbitration.

[0061] The inertial navigation system (INS) uses abrupt changes in attitude increment drift to identify INS evidence and freezes the INS contribution when the abrupt change exceeds a threshold. Let the wrist attitude increment estimated visually in frame (t) be... The wrist attitude increment obtained by inertial navigation integration is Define drift score:

[0062]

[0063] And take the drift mutation threshold During the stable phase of this segment, The values ​​are concentrated between 0.03 rad and 0.06 rad, which is within the confidence range. The relationship between the drift score curve, the threshold line, and the freeze interval is shown in the appendix. Figure 3 .

[0064] Around 63.480s, the sensor on the back of the patient's hand momentarily loosened, causing a spike in the inertial angular velocity, which led to... A jump occurs within one frame. Taking the corresponding frame as an example, , Calculated The system outputs a drift mutation detection as true and sets the freeze flag to true, setting the inertial navigation system's contribution to the fusion to zero or maintaining it at the reliable value before freezing, thereby preventing drift from further contaminating the fusion results. Combined with... Figure 3 The relationship between mutation point markers and thresholds is calculated using example frames. and ,because The determination of drift mutations and the triggering of freeze demonstrate that mutation-triggered freeze can effectively isolate the impact of spike drift on the fusion attitude.

[0065] The joint-by-joint evidence priority arbitration rule is as follows: if visual consistency is approved, the joint is visually dominated and labeled with the dominant tag "V"; if visual consistency is not approved but inertial navigation is not frozen, and... If the visual input fails and the inertial navigation system freezes, a degraded output is triggered, maintaining the fusion pose that passed the consistency check in the previous frame and labeling it as H. This arbitration is performed independently on the wrist and metacarpophalangeal joints, allowing the inertial navigation system to take over visual anomalies caused by occlusion, while maintaining output stability even when the inertial navigation system drifts and freezes.

[0066] A hysteresis condition is used to unfreeze the system to avoid frequent switching. Assume the visual continuity count is... When the freeze flag is true and continuous The freeze is lifted when all frames pass visual consistency and no further evidence of drift or abrupt change appears during the freeze period. Taking frames 1901 to 1905 after the freeze begins as an example, a continuous pass in the visual consistency flag indicates that the freeze is lifted. The value increased from 1 to 5, and no further occurrences were observed on the inertial navigation side. The system unfroze and resumed inertial navigation participation in frame 1906. Figure 3 The unfreezing point is illustrated to show the trigger location for hysteresis release. Combined with... Figure 3 Hysteresis release requires stable visual evidence across multiple consecutive frames, which helps reduce frequent jittering of the frozen state in noisy environments and improves the continuity and reliability of the fusion output.

[0067] Upon release, a one-time compensation is performed to eliminate the real motion increments ignored during the freeze period. Let the freeze start frame be... Release frame as The last reliable fusion posture before freezing is The cumulative increment of the inertial navigation system during the freezing period is:

[0068]

[0069] The compensated reference attitude is:

[0070]

[0071] The freeze in this segment lasts for 0.33 seconds, corresponding to approximately 33 inertial navigation sampling periods, and the estimated... The main focus is on wrist rotation degrees of freedom. Before compensation, the peak error of the fingertip position based on vision was 18 mm, and after compensation, the peak value decreased to 9 mm. At the same time, the wrist angle error decreased from 7.4° to 3.1°. The change in error before and after compensation and the comparison with the comprehensive evaluation are shown in the attached figure of the example. Figure 4 Combining Figure 4 One-time compensation example The attitude reference compensation used during unfreezing reduced the peak fingertip error from 18mm to 9mm, a 50% decrease, indicating that the real motion increment during the freezing period can significantly reduce the cumulative deviation caused by freezing and improve end-effector error performance after compensation and recirculation.

[0072] In this embodiment, six motion segments were selected from two minutes of patient A's data for comparative evaluation. Indicators included mean joint angle error (based on therapist-annotated offline reconstruction), mean fingertip position error, and the proportion of abnormal frames (frames that failed visual consistency testing or were frozen due to inertial navigation drift abrupt changes). The results are shown in Table 1 below. Figure 4The differences between the three schemes were visualized through a comprehensive comparison, demonstrating that the scheme integrating freezing and compensation is superior to the schemes with only vision and only inertial navigation in all three indicators.

[0073] method Mean joint angle error Average fingertip error Abnormal frame ratio Visual only 4.6° 14mm 9% Inertial navigation only 6.1° 17mm 5% Integration including freezing and compensation 2.9° 9mm 3%

[0074] In summary, this embodiment ensures cross-modal time consistency through event anchor point alignment and anti-spoofing, and the related effects are achieved by... Figure 2 As shown, after desmearing, the average alignment error was reduced to 8ms and the maximum alignment error was reduced to 34ms. Inertial navigation system abrupt freeze was achieved through drift scoring and then released under hysteresis conditions. The relevant process was... Figure 3 As shown, and the example frame satisfies Triggering a freeze, a one-time compensation is used to make up for the actual increase in motion during the freeze period; the related effects are determined by... Figure 4 As shown, the peak fingertip error was reduced from 18mm to 9mm. Combined with the joint-by-joint evidence priority arbitration and degraded output mechanism, the hand posture reconstruction after burn training can still maintain stability and improve accuracy under typical interferences such as occlusion and sensor loosening. The output dominant label facilitates subsequent interpretable feedback and training evaluation.

[0075] Example 2:

[0076] In this embodiment, the rehabilitation scenario is the 7th week of rehabilitation training for patient B's hand scar contracture after burns. The therapist requires the patient to complete a cycle of grasping, relaxing, and then thumb-to-finger contact within 90 seconds. After each thumb-to-finger contact, a transition from wrist pronation to wrist supination is added, followed by a second transition from wrist supination to pronation after 1 second, and then a grasping motion is performed to form at least two transition events. The patient wears a back-of-hand IMU and a ring IMU sampling at 100Hz, and a fixed camera in front collects visual sequence sampling of the hand at 30fps. The visual side outputs 21 key points and confidence scores. During the training, the patient wears thin gloves, and there is palm occlusion and glare during the thumb-to-finger contact phase, causing some joint key points to jump.

[0077] The system processes the cross-modal aligned data into action segments. For each frame, it estimates joint angles and bone segment lengths based on visual key points and calculates the endpoint positions of the joint chain for consistency constraint checks. The consistency check includes three criteria: bone segment length change, joint angle increment, and endpoint deviation. Each criterion is compared with a threshold. If all three criteria are met, the system passes; otherwise, it fails and outputs a failure flag to trigger subsequent illegal joint localization and local correction processes.

[0078] The change in bone segment length is defined as the relative change between the current frame and the calibration reference, denoted as:

[0079]

[0080] in The length of the bone segment in the current frame is in mm. The static calibration length unit is mm, and the threshold is... The joint angle increment is defined as the absolute change in joint angle between adjacent frames, denoted as:

[0081]

[0082] in The unit is °, and the threshold is The joint chain endpoint position deviation is defined as the Euclidean distance between the endpoint position calculated from the current joint angle and the observed endpoint position, denoted as:

[0083]

[0084] in The endpoint positions calculated from the kinematic chain are in mm. The visual observation endpoint position is in mm, and the threshold is... .when and and If both conditions are met, the agreement is considered valid; otherwise, it is considered invalid.

[0085] To balance the rapid changes in action during transition phases with stability during non-transition phases, this embodiment employs two sets of thresholds and switches between them around transition events. The first set of thresholds is used for the normal phase between start and stop. , , When a wrist twisting event is detected, the system moves to a second, more stringent threshold group. , , To improve sensitivity to occlusion transitions, a hysteresis mechanism is used for threshold switching. Upon detecting a transition, the threshold enters the second group and remains there until a continuous transition occurs. If no new transition event is detected in the frame, the frame exits the second group. If a transition is detected again during the holding period, the count is reset, thereby avoiding judgment jitter caused by frequent switching of threshold groups.

[0086] Taking a segment containing two transitions as an example, the segment is sampled at 30fps and labeled with frame numbers. The first transition occurs at frame 540. The system enters the second set of thresholds and starts counting. When the second transition is detected again at frame 548, the count is reset and the second set of thresholds is maintained. Afterward, no new transitions occur for nine consecutive frames starting from frame 549. The system exits the second set of thresholds at frame 558 and reverts to the first set of thresholds. This hysteresis process ensures that the strict thresholds near the transitions cover the perturbation range of the two transitions, thereby reducing missed detections and false positives during the transition phase. The correspondence between the threshold set maintenance interval and the transition event frames is shown in [link to documentation]. Figure 5 Combining Figure 5 Hysteresis parameters The transition event frames are frames 540 and 548. The system enters the second set of thresholds at frame 540 and resets the count at frame 548 to maintain it. Finally, it exits the second set of thresholds at frame 557 and restores the first set of thresholds. This means that when a transition is detected, the system enters the second set of strict thresholds and maintains it. If another transition occurs during the maintenance period, the no-transition count is reset. The system exits the second set of thresholds only after there are no new transitions for H consecutive frames. This allows the strict threshold to cover the range of two transition disturbances and reduces threshold jitter.

[0087] In frame 545, during the thumb-to-finger phase, glove reflection caused a jump in the key point of the thumb metacarpophalangeal joint, exceeding the threshold for all three criteria and thus triggering a consistency failure. The calculation for this frame yielded... and and At that time, it was in the second threshold stage, therefore and and If the consistency is deemed unsuccessful and a failure flag is output, the system enters the violation joint set localization process. The relationship between the three criteria for this frame and the second set of thresholds is shown below. Figure 6 .

[0088] The localization of non-compliant joint assemblies employs a single-joint exclusion strategy, executed in a proximal-to-distal order. Proximal joints, such as the wrist and metacarpophalangeal joints, are fixed first, followed by sequential exclusion checks on distal joints. During each check, the current frame observation of the candidate joint is removed from the pose calculation and replaced with the previous reliable frame or fused estimate, before recalculation. and and A consistency check is performed. If the re-inspection changes from failing to passing, the joint is temporarily marked as a violation joint, and this joint will not use the current frame observation value when re-inspecting other joints in the future, so as to obtain the set of violation joints corresponding to the ones that pass the inspection.

[0089] Taking frame 545 as an example, the system first excludes the thumb metacarpophalangeal joint observation and then re-examines it, obtaining... and and The system satisfies all three criteria under the second set of thresholds, thus changing the re-examination from failing to passing. The thumb metacarpophalangeal joint is temporarily recorded as a violation joint. Subsequently, while keeping the thumb metacarpophalangeal joint excluded, the system continues to re-examine the index and middle finger metacarpophalangeal joints. The re-examination is still passed and no further exclusion is needed. Therefore, the set of violation joints for this frame is temporarily recorded as the thumb metacarpophalangeal joint.

[0090] To reduce the randomness of single-joint exclusion, this embodiment performs reverse verification before adding the temporarily recorded joint to the set of non-violation joints. The reverse verification process involves restoring the joint's current frame observation value after the exclusion passes the re-examination and performing another re-examination. If the re-examination still fails, the joint is confirmed as a genuine violation and added to the set; otherwise, it is not added to suppress false alarms. In frame 545, the thumb metacarpophalangeal joint observation value is restored and re-examined. and and The second threshold is still not met, therefore the reverse verification fails, and the joint is added to the set of violating joints and used as a subsequent local correction target. Figure 6 The keyframe is frame 545 and it is in the second threshold stage. The second threshold is... and and The original observation was and and The test failed; a single joint was excluded and a retest was conducted. and and The test was passed; reverse verification and retesting were performed. and and The failure to pass the initial test indicates that the three criteria for the original observation exceeded the threshold, resulting in a failure to achieve consistency. After excluding the thumb metacarpophalangeal joint, the retest passed, but the test still failed after observation was resumed. This satisfies the reverse verification condition, confirming that the thumb metacarpophalangeal joint is a real non-compliant joint and adding it to the set of non-compliant joints.

[0091] To avoid short-term false detections caused by single-frame jumps, this embodiment performs temporal screening on the set of violating joints. For each joint, the number of consecutive frames within the action segment that are recorded as violating is counted. Only retain The joints are taken as the final set of illegal joints. A frame is approximately 200ms. Statistical results show that the thumb metacarpophalangeal joint had 8 consecutive frames that met the threshold and were retained. The proximal interphalangeal joint of the index finger was only temporarily recorded as a violation in 2 frames due to short-term jumps caused by glare, failing to meet the threshold and being removed. This resulted in the final set of violation joints focusing more on stable anomalies rather than transient noise. The relationship between this selection result and the threshold line is shown in [link to relevant documentation]. Figure 7 .

[0092] After obtaining the final set of non-compliant joints, local correction is performed, and the correction result is used for the arbitration fusion output. This embodiment adopts a simple and implementable strategy of weighted pullback of the non-compliant joint angles, bringing the joint angles of the current frame closer to the previous reliable fusion angle. The correction is defined as:

[0093]

[0094] in The fused joint angles, in degrees, represent the joint angles that passed the consistency check in the previous frame. The joint angle for the current frame visual observation is in degrees. The pullback factor is set to 0.7. After correction, the endpoint deviation is recalculated and the fused pose is updated. The dominant label output rule is to prioritize visual dominant label V for non-violation joints, and label violation joints as C in the frame where correction takes effect to indicate that correction is dominant. When visual correction passes and there are no abnormalities, the label V is restored to facilitate interpretive display for training system.

[0095] In a 90-second training session for patient B, eight motion segments containing transition events were selected to evaluate the consistency pass rate, endpoint error, and joint angle error. The results were compared with a baseline approach that did not employ stage threshold switching and did not use reverse validation and temporal screening. Evaluation metrics included mean endpoint error (mm), mean thumb metacarpophalangeal joint angle error (°), and consistency pass rate (%). The baseline approach used a single threshold. and and Furthermore, there is no delayed switching, and no reverse verification or timing filtering is performed on temporarily recorded violation joints.

[0096] The comparison results are shown in Table 2 below. The method represents a complete process that uses two sets of thresholds and hysteresis switching, including single-joint exclusion, reverse verification, and temporal screening, and performs local correction on the final non-compliant joints. The results show that the average endpoint error decreased from 13mm to 8mm, the average angle error of the thumb metacarpophalangeal joint decreased from 5.2° to 3.0°, and the consistency pass rate increased from 88% to 95%. This indicates that under the interference of turning and occlusion, the phased threshold and non-compliant joint localization and correction can significantly improve the reconstruction stability and accuracy. The changes in endpoint error and evaluation trends before and after temporal screening and local correction are shown in Table 2. Figure 7 Combining Figure 7 Time series filtering parameters Frame, thumb metacarpophalangeal joint consecutive violations for 8 frames satisfy The index finger was retained, but was removed due to insufficient consecutive frames (2 frames or less) of violation at the proximal interphalangeal joint. Local correction parameters were used. The peak value of the corrected front-end point error is approximately the peak value of the code generation curve. The peak value after correction is approximately 70% of the peak value before correction, and the peak value decreases by about 30%. This indicates that the timing screening suppresses occasional false detections caused by short-term flickering and only retains continuously abnormal joints for correction. Performing weighted pullback correction on the final non-compliant joint can reduce the peak value of the endpoint error and improve the overall consistency pass rate, which is in line with the comparative evaluation trend.

[0097] method Endpoint average error Thumb palmar angle error Consistency pass rate Baseline scheme 13mm 5.2° 88% Method of this embodiment 8mm 3.0° 95%

[0098] In summary, this embodiment employs three criteria—bone segment length variation, joint angle increment, and endpoint deviation—to construct a kinematic consistency constraint test. A stricter threshold is used around turning events, and a hysteresis mechanism ensures stable switching, making the turning phase sensitive to visual jump changes without introducing frequent jitter. Single-joint exclusion localization from proximal to distal, combined with reverse verification, reduces false alarms. Temporal filtering retains only persistently abnormal joints. Finally, weighted pullback local correction is applied to violating joints, and a dominant label is output. This achieves a more stable and reliable hand posture reconstruction and correction effect under complex training conditions such as glove occlusion, glare, and multiple wrist turns.

[0099] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion, characterized in that... include: Acquire hand visual sequences and IMU angular velocity or acceleration; detect start, stop, and turning boundary events in the two data streams and pair them in sequence, using the paired events as anchor points to align and form motion segments; The kinematic consistency constraint test is performed on the visual pose within the segment and visual evidence is output. The visual evidence includes a pass or fail flag for the consistency test and a set of non-compliant joints. The drift of the IMU pose increment is determined and inertial navigation evidence is output. The inertial navigation evidence includes the drift mutation determination result and a frozen state flag. When the drift mutation exceeds a threshold, the inertial navigation contribution is frozen. Arbitration is performed based on visual evidence, inertial navigation evidence, and frozen state at each joint, and the fused posture is obtained by selecting either visual or inertial navigation as the dominant factor. When vision fails the constraint and inertial navigation contribution is frozen, maintain the fusion pose that passed the consistency check in the previous frame and output it in a downgraded manner; perform local correction on the joints that failed the constraint based on evidence to obtain the corrected fusion pose; output the corrected fusion pose and the dominant label.

2. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 1, characterized in that... The action segment alignment includes event despurification: boundary events are only paired when there are candidate events of the same type and with the same relative order in another data within a preset time window, and the event time difference is within the preset time window; when there is repeated start or stop, start and stop within a preset duration threshold, or change of direction before start, the boundary event is determined to be invalid and not used as an anchor point.

3. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 1, characterized in that... When the kinematic consistency constraint check fails, a set of non-compliant joints is output. The set of non-compliant joints is determined by single-joint exclusion: each joint is excluded from the pose solution using its current frame observation value and its consistency is rechecked. If the recheck changes from failing to passing, the joint is recorded as a non-compliant joint. The set of non-compliant joints is used to limit the objects of the arbitration fusion and local correction.

4. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 1, characterized in that... The frozen inertial navigation contribution is released using hysteresis: when frozen, the contribution of the inertial navigation in the fusion is set to zero or maintained at the reliable value before freezing; after freezing, the visual consistency must be passed for multiple consecutive frames and there must be no evidence of sudden drift in the inertial navigation during the freezing period before it can be released; when releasing, the reliable attitude before freezing is used as a reference, and the increment of the freezing period is compensated once before participating in the evidence priority arbitration.

5. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 3, characterized in that... The re-inspection of consistency includes comparing the change in bone segment length, the increase in joint angle, and the deviation of the position of the end point of the joint chain with the corresponding preset thresholds. If all of them are satisfied, the kinematic consistency constraint test is deemed to have passed. If the test for a certain joint changes from failing to passing after the current frame observation is not used in the attitude solution, the joint is recorded as a violation joint and added to the violation joint set.

6. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 3, characterized in that... The single joint exclusion process involves sequentially eliminating the current frame observation values ​​from each joint in the attitude solution according to a preset proximal-to-far order, and then performing the aforementioned consistency check. Once a joint is identified as a violation, when re-examining the remaining joints, the joint is kept out of the current frame observation value for pose solving, in order to determine the set of violation joints that pass the test.

7. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 3, characterized in that... The set of non-compliant joints is determined by temporal filtering: the number of consecutive frames in which a joint is recorded as non-compliant within a motion segment is counted, and only joints with a consecutive frame count not less than a preset frame count threshold are retained as the final set of non-compliant joints, which is used to limit the objects of the arbitration fusion and local correction.

8. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 5, characterized in that... The preset thresholds are switched according to the action segment stage: a first set of thresholds is used between start and stop, and a second set of thresholds is used within a preset number of frames before and after the transition event. The second set of thresholds is less than the corresponding thresholds in the first set of thresholds. The re-inspection consistency is determined to pass or fail based on the comparison result of the switched thresholds.

9. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 5, characterized in that... Before adding a violating joint to the set of violating joints, reverse verification is performed: after the test changes from failing to passing by not using the current frame observation value of the joint, the current frame observation value of the joint is restored and retested; if the retest still fails, the joint is added to the set of violating joints, otherwise it is not added.

10. The method for reconstructing and correcting post-burn hand training posture based on vision and inertial navigation fusion according to claim 8, characterized in that... The stage switching of the preset threshold adopts hysteresis determination: when a turning event is detected, it enters the second set of thresholds and remains there until no new turning event is detected for a consecutive preset number of frames, then exits the second set of thresholds; when a turning event is detected again during the holding period, the count of the preset number of frames is reset; the re-inspection consistency is compared with the threshold group determined by the hysteresis determination and determined to pass or fail.

Images