Posture skeleton guided human-computer interaction comfort degree evaluation method and system thereof
By reconstructing 3D skeleton data using depth cameras and neural networks, and combining it with an inverse dynamics model, the problem of insufficient accuracy and dimensionality in existing human-computer interaction comfort assessment methods is solved, achieving high-precision, multi-dimensional comfort assessment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAN UNIV OF TECH
- Filing Date
- 2026-04-08
- Publication Date
- 2026-06-09
AI Technical Summary
Existing human-computer interaction comfort assessment methods are difficult to achieve high-precision three-dimensional posture reconstruction and temporal motion analysis in non-contact, natural working conditions. They cannot accurately reflect key fatigue-causing factors such as joint torque, muscle load, and mechanical work, resulting in a single assessment dimension and limited interpretability.
A depth camera was used to simultaneously acquire RGB and depth information. Combined with instance segmentation and pose estimation neural networks, three-dimensional skeleton data was reconstructed. The net joint torque and muscle load were solved by inverse dynamics model to construct a comfort evaluation function. The comfort was evaluated by integrating kinematic and biomechanical indicators.
It improves the accuracy of 3D pose estimation in complex scenarios, provides multi-dimensional and mechanistic comfort assessment, reveals the intrinsic load of fatigue and discomfort, and provides clear physical evidence.
Smart Images
Figure CN121982225B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, and more specifically to a method and system for evaluating human-computer interaction comfort guided by posture skeleton. Background Technology
[0002] The deep integration of human-computer interaction and ergonomics in fields such as industrial manufacturing, rehabilitation medicine, and virtual reality has placed higher demands on operator comfort and occupational health. Traditional human-computer interaction comfort assessments often rely on subjective questionnaires, expert judgment, or simple motion video playback, making it difficult to achieve objective, quantitative, and real-time assessments. With the development of computer vision and sensor technology, vision-based human posture estimation methods are gradually being applied to motion analysis and ergonomics assessment, providing the possibility of non-contact and automated comfort assessment. However, existing methods are mostly focused on two-dimensional posture analysis or static posture evaluation, lacking in-depth modeling of three-dimensional motion temporal characteristics and the intrinsic biomechanical load of the human body. This leads to a discrepancy between the assessment results and actual physiological sensations, making it difficult to fully reflect the cumulative fatigue and discomfort under long-term, repetitive work.
[0003] In existing technologies, some studies attempt to collect motion and muscle activity data through wearable sensors. While this can obtain relatively direct biomechanical information, the devices are cumbersome to wear, easily interfere with normal operations, and are costly, making large-scale application in real production environments difficult. Another type of visual method based on a monocular RGB camera achieves contactless assessment, but it is limited by issues such as missing depth information, weak occlusion handling capabilities, and insufficient 3D reconstruction accuracy, especially exhibiting poor robustness in complex backgrounds and multi-person scenarios. Furthermore, most assessment models only focus on kinematic parameters such as joint angles and movement frequency, without effectively combining biomechanical mechanisms such as inverse dynamics and muscle force distribution. This fails to accurately reflect key fatigue-causing factors such as joint torque, muscle load, and mechanical work, resulting in a single assessment dimension and limited interpretability.
[0004] Therefore, there is an urgent need for a comfort assessment method that can integrate high-precision three-dimensional posture reconstruction, temporal motion analysis and multi-rigid-body biomechanical simulation, so as to achieve an objective, accurate and interpretable evaluation of the local joint load and comfort of the human body in a non-contact, natural working state, and provide a quantitative basis for work posture optimization and human-machine interface design. Summary of the Invention
[0005] In view of the above-mentioned shortcomings of the prior art, the present invention provides a posture skeleton-guided human-computer interaction comfort assessment method and system, which can effectively solve the problems mentioned in the prior art.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] This invention provides a posture skeleton-guided method for evaluating human-computer interaction comfort, comprising the following steps:
[0008] S100: Simultaneously acquire RGB image sequences and depth image sequences of the work scene using a depth camera, and perform time alignment.
[0009] S200. Based on the RGB image sequence, perform human instance segmentation and two-dimensional joint detection, fuse the depth image of the corresponding frame, and use camera calibration parameters to map the two-dimensional joint coordinates to three-dimensional spatial coordinates in the camera coordinate system to generate temporal three-dimensional skeleton data.
[0010] S300. For a preset target joint, calculate the joint angle timing curve within a single operation cycle based on the temporal three-dimensional skeleton data. With angular velocity time series curve The cumulative time during which the joint angle continuously deviated from the ergonomic neutral position range was recorded. ;
[0011] S400, Construct a human biomechanical model and generate joint angle time-series curves. Angular velocity time series curve The inverse dynamics submodel is used to calculate the limb segment centroid acceleration based on the temporal 3D skeleton data, and the net torque of the target joint is solved. ;Will Input the muscle load distribution sub-model and solve for the equivalent torques of the main muscle groups driving the joint. And calculate the mechanical work done by the joint. ;
[0012] S500, construct a comfort evaluation function based on joint angle extreme values. Cumulative time , peak Given the input parameters, output the local comfort score of the target body part. The specific formula is as follows:
[0013] ;
[0014] in, For neutral angles, For the allowable angle range, For operation cycle duration, and These are the reference thresholds for the equivalent torque of the major muscle groups and the mechanical work done by the joints, respectively. , , and Weighting coefficients, local comfort scores A higher value indicates a higher level of comfort.
[0015] S600, based on local comfort rating Generate a dynamic comfort heatmap overlaid on a 3D human body model and generate an optimization suggestion report;
[0016] The single operation cycle is the complete time interval for performing a single repetitive human-computer interaction task, and the target joint is pre-specified according to the task type.
[0017] Furthermore, the specific steps of S200 include:
[0018] S201. Perform camera distortion correction on the RGB image and median filtering to denoise the depth image; spatially align the depth image and the RGB image based on calibration parameters.
[0019] S202. The instance segmentation neural network is used to process the RGB image after distortion correction, and the mask and confidence score of each human instance are output. Instances with confidence scores lower than the first threshold are removed.
[0020] S203. For each preserved human instance mask region, a high-resolution pose estimation network is used to output the two-dimensional coordinates and detection confidence of preset joints; for joints with detection confidence below the second threshold, the coordinates are fine-tuned based on the heatmap distribution.
[0021] S204. Query the depth value of the corresponding position in the aligned depth image according to the two-dimensional coordinates; if the depth value is invalid or the confidence of the joint detection is lower than the third threshold, the weighted median of the effective depth values in the neighborhood centered on the two-dimensional coordinates is used as the compensation depth value, where the weight is inversely proportional to the Euclidean distance from the neighboring pixel to the center point.
[0022] S205. Based on the camera intrinsic parameter matrix, back-project the two-dimensional coordinates and corresponding depth values to the camera coordinate system to obtain the three-dimensional joint coordinates:
[0023] ;
[0024] in, Let i be the image coordinates of the i-th joint. The depth value corresponding to the image coordinates. For the camera intrinsic parameter matrix, , Focal length Principal point coordinates;
[0025] S206. Apply Kalman filtering to the three-dimensional coordinate sequence of the same joint point in consecutive frames for temporal smoothing, and use an interpolation algorithm based on kinematic constraints to complete the trajectory of joint points that are continuously occluded for more than a preset number of frames.
[0026] S207. Encapsulate the smoothed 3D joint coordinates into structured data containing timestamps, instance identifiers, and joint coordinates according to frame order to generate temporal 3D skeleton data.
[0027] Furthermore, the calculation of the limb segment centroid acceleration includes: determining the position coordinate sequence of the limb segment centroid in the three-dimensional skeleton based on anthropometry proportion parameters, performing time-domain numerical differentiation on the position coordinate sequence to obtain the centroid acceleration vector.
[0028] Furthermore, the single operation cycle is determined by the Hilbert-Huang transform analysis of the joint angle time series curve. When the standard deviation of the cycle length of three consecutive cycles is less than 3%, the average value is taken as the operation cycle length. When the fluctuation exceeds 15%, manual review is triggered.
[0029] The joint angle timing curve Calculations are differentiated based on joint type, where joint types include hinge joints and ball-and-socket joints, and the angular velocity time-series curves are... Through the The cumulative deviation time is calculated using the central difference method after applying a filter with a preset window length. By traversing a single cycle The total time during which the joint angle exceeds the preset neutral position range is counted.
[0030] Furthermore, the construction of the human biomechanical model is based on the height and weight of the target being measured, and the mass, center of mass position and moment of inertia of each limb segment are determined using the Winter anthropometry standard, thus constructing a multi-rigid-body dynamic model with 14 rigid body segments.
[0031] The inverse dynamics sub-model is solved using the Newton-Euler recursive algorithm, calculating segment by segment from the distal end of the limb to the proximal end to determine the net torque of the target joint. ,Will The specific formula is as follows:
[0032] ( , ;
[0033] in, For the force of the k-th muscle, The maximum isometric contractile force of the k-th muscle. This is the transpose of the lever arm vector. This represents the total number of muscles involved in joint movement.
[0034] Furthermore, the specific formula for calculating the mechanical work done by the joint in step S500 is as follows:
[0035] ;
[0036] in, and These represent the start and end times of the operation cycle.
[0037] A posture-skeleton-guided human-computer interaction comfort assessment system includes:
[0038] The data acquisition module is used to simultaneously acquire RGB image sequences and depth image sequences of the work scene through a depth camera, and perform time alignment.
[0039] The three-dimensional skeleton generation module is used to perform human instance segmentation and two-dimensional joint detection based on the RGB image sequence, fuse the depth image of the corresponding frame, and use camera calibration parameters to map the two-dimensional joint coordinates to three-dimensional spatial coordinates in the camera coordinate system to generate temporal three-dimensional skeleton data.
[0040] The joint motion analysis module is used to calculate the joint angle time-series curve within a single operation cycle based on the time-series three-dimensional skeleton data for a preset target joint. With angular velocity time series curve The cumulative time during which the joint angle continuously deviated from the ergonomic neutral position range was recorded. ;
[0041] The biomechanical analysis module is used to construct human biomechanical models and analyze the time-series curves of joint angles. Angular velocity time series curve The inverse dynamics submodel is used to calculate the limb segment centroid acceleration based on the temporal 3D skeleton data, and the net torque of the target joint is solved. ;Will Input the muscle load distribution sub-model and solve for the equivalent torques of the main muscle groups driving the joint. And calculate the mechanical work done by the joint. ;
[0042] The comfort assessment module is used to construct a comfort assessment function, based on the extreme values of joint angles. Cumulative time , peak Given the input parameters, output the local comfort score of the target body part. ;
[0043] The visualization and report generation module is used for local comfort rating. Generate a dynamic comfort heatmap overlaid on a 3D human body model and generate an optimization suggestion report.
[0044] Furthermore, the three-dimensional skeleton generation module specifically includes:
[0045] The preprocessing unit performs camera distortion correction on the RGB image and median filtering to denoise the depth image; it also spatially aligns the depth image with the RGB image based on calibration parameters.
[0046] The instance segmentation unit is used to process the distortion-corrected RGB image using an instance segmentation neural network, output the mask and confidence score of each human instance, and remove instances with confidence scores lower than the first threshold.
[0047] The two-dimensional joint detection unit is used to output the two-dimensional coordinates and detection confidence of preset joints for each preserved human instance mask region using a high-resolution pose estimation network; for joints with detection confidence below the second threshold, the coordinates are fine-tuned based on the heatmap distribution.
[0048] The depth fusion unit is used to query the depth value of the corresponding position in the aligned depth image based on the two-dimensional coordinates. If the depth value is invalid or the confidence of the joint detection is lower than the third threshold, the weighted median of the effective depth values in the neighborhood centered on the two-dimensional coordinates is used as the compensation depth value, where the weight is inversely proportional to the Euclidean distance from the neighboring pixel to the center point.
[0049] The 3D backprojection unit is used to backproject the 2D coordinates and corresponding depth values to the camera coordinate system based on the camera intrinsic parameter matrix to obtain the 3D joint coordinates.
[0050] The timing processing unit is used to apply Kalman filtering to the three-dimensional coordinate sequence of the same joint point in consecutive frames for timing smoothing, and to use a kinematic constraint-based interpolation algorithm to complete the trajectory of joint points that are continuously occluded for more than a preset number of frames.
[0051] The data encapsulation unit is used to encapsulate the smoothed and completed 3D joint coordinates into structured data containing timestamps, instance identifiers, and joint coordinates in frame order, generating temporal 3D skeleton data.
[0052] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the posture skeleton-guided human-computer interaction comfort assessment method.
[0053] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the posture skeleton-guided human-computer interaction comfort assessment methods described above.
[0054] The technical solution provided by this invention has the following advantages compared with the known prior art:
[0055] This invention employs a depth camera to simultaneously acquire RGB and depth information, combined with advanced instance segmentation and pose estimation neural networks, to achieve stable and accurate reconstruction of the operator's 3D skeleton temporal data under natural working conditions. This avoids the interference and burden caused to the operator by traditional wearable sensors or electromyography devices, making it more suitable for real production environments. Furthermore, by integrating depth information with kinematic constraints in temporal processing, it effectively overcomes problems such as occlusion and jitter, significantly improving the accuracy of 3D pose estimation in complex scenes.
[0056] This invention not only analyzes kinematic parameters such as joint angles and deviation time, but more importantly, it introduces biomechanical modeling based on multi-rigid-body dynamics and muscle force distribution. By solving for the net torque of the joint through inverse dynamics, and optimizing the calculation of the equivalent torque and mechanical work of the main muscle groups, it reveals the intrinsic load that leads to fatigue and discomfort from a mechanical perspective. Finally, by integrating the key indicators of the four dimensions of kinematics and biomechanics, a local comfort score is obtained through a weighted function. This multi-dimensional and mechanistic evaluation method provides clear physical evidence and is more accurate. Attached Figure Description
[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.
[0058] Figure 1 This is a schematic diagram of the method flow of the present invention;
[0059] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation
[0060] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0061] The present invention will be further described below with reference to embodiments.
[0062] Example:
[0063] Reference Figure 1 A posture-skeleton-guided method for evaluating human-computer interaction comfort includes the following steps:
[0064] S100: Simultaneously acquire RGB image sequences and depth image sequences of the work scene using a depth camera, and perform time alignment.
[0065] Specifically, in this invention, an Intel RealSense D455 depth camera is used, equipped with a global shutter RGB sensor and an infrared depth sensing module, enabling hardware-level synchronization. The acquisition frame rate is set to 30 fps, the RGB image resolution is 1920×1080, and the depth image resolution is 1280×720. Timestamp-aligned image pairs are directly obtained by calling the `synced_frames` interface in the camera SDK, ensuring strict correspondence between RGB and depth information at the same moment. During acquisition, the camera is mounted in a fixed position approximately 2 meters above and to the side of the work area, covering the operator's entire range of motion.
[0066] S200: Human instance segmentation and 2D joint detection are performed based on RGB image sequences. The depth images of the corresponding frames are fused, and the 2D joint coordinates are mapped to 3D spatial coordinates in the camera coordinate system using camera calibration parameters to generate temporal 3D skeleton data.
[0067] S300. For a preset target joint, calculate the joint angle time-series curve within a single operation cycle based on the time-series three-dimensional skeleton data. With angular velocity time series curve The cumulative time during which the joint angle continuously deviated from the ergonomic neutral position range was recorded. ;
[0068] In one specific embodiment, using the right elbow joint as the target joint, firstly, the intrinsic mode function (IMF) is extracted by performing a Hilbert-Huang transform on the elbow joint flexion-extension angle sequence, identifying the periodic boundaries corresponding to the dominant frequency components, thereby automatically defining the repetitive operation cycle of "reaching out - grabbing - installing - returning". The system is configured such that when the standard deviation of three consecutive identified cycle durations is less than 3% of their mean, the mean is taken as the operation cycle duration; if the standard deviation exceeds 15%, the data for that period is automatically labeled and a manual review is prompted.
[0069] Joint angle calculation: The elbow joint is a hinge joint, and the angle is obtained by calculating the angle between the upper arm vector (from shoulder joint to elbow joint) and the forearm vector (from elbow joint to wrist joint). The angle range is defined as 0° to 150°.
[0070] Angular velocity calculation: The sequence is first smoothed using Savitzky-Golay filtering with a window length of 5 frames, and then the angular velocity is calculated using the center difference method. .
[0071] Based on ergonomic standards, the comfortable neutral position range of the elbow joint is set to -10° extension to 30° flexion. This is followed by a complete operational cycle. , total of The sum of time segments exceeding the range of [-10°, 30°] yields the cumulative deviation time. .
[0072] S400, Construct a human biomechanical model and generate joint angle time-series curves. Angular velocity time series curve The inverse dynamics sub-model is constructed by inputting the limb segment center-of-mass acceleration calculated based on time-series 3D skeleton data, and then solving for the net torque of the target joint. ;Will Input the muscle load distribution sub-model and solve for the equivalent torques of the main muscle groups driving the joint. And calculate the mechanical work done by the joint. ;
[0073] S500, construct a comfort evaluation function based on joint angle extreme values. Cumulative time , peak Given the input parameters, output the local comfort score of the target body part. The specific formula is as follows:
[0074] ;
[0075] in, For neutral angles, For the allowable angle range, For operation cycle duration, and These are the reference thresholds for the equivalent torque of the major muscle groups and the mechanical work done by the joints, respectively. , , and Weighting coefficients, local comfort scores A higher value indicates a higher level of comfort.
[0076] In one specific embodiment, based on the height of the target being measured (175cm) and weight (70kg), and according to the Winter anthropometry data table, the mass ratio, center of mass position ratio, and radius of rotation around the center of mass of each limb segment are determined, and a dynamic model is constructed that includes 14 rigid body segments: head and neck, upper torso, lower torso, left and right upper arms, left and right forearms, left and right hands, left and right thighs, left and right calves, and left and right feet.
[0077] Calculation of centroid acceleration of limb segments: Based on the coordinates of the joints at both ends of each limb segment and the proportional coefficient of the centroid position, linear interpolation is used to obtain the three-dimensional coordinate sequence of the centroid, and the acceleration vector is obtained by performing second-order central difference on it.
[0078] Inverse dynamics solution: The Newton-Euler recursive algorithm is used. Starting from the distal limbs, such as the forearm and hand, given their motions, including angles, angular velocities, angular accelerations, center-of-mass accelerations, and external loads, the forces and moments at adjacent joints are calculated. This process is then repeated proximally until the target joint (elbow joint) is reached, where the net moment at the elbow joint is obtained. .
[0079] Muscle load distribution: A lever arm matrix and a database of maximum isometric contractile forces were established for the six major muscles around the elbow joint, including the biceps brachii, brachialis, brachioradialis, long head, lateral head, and medial head of the triceps brachii. The force f_k(t) and equivalent torque of each muscle were obtained by solving the following optimization problem. :
[0080] Minimize the objective function: ;
[0081] The constraints are: , .
[0082] The mechanical work done by the joint is calculated by integration: Discrete numerical integration is performed using the trapezoidal rule.
[0083] S600, based on local comfort rating Generate a dynamic comfort heatmap overlaid on a 3D human body model and generate an optimization suggestion report;
[0084] In one specific embodiment, the elbow joint comfort score C is mapped to a Jet color map, with low scores mapped to red and high scores to green. On a WebGL-rendered 3D SMPL human body model, a semi-transparent color patch is rendered in the elbow joint area, with its color and size dynamically changing according to the C value. Simultaneously, the system automatically generates a structured report stating: "The right elbow joint's peak flexion angle reached 105° during the installation phase, exceeding the comfort range for 30% of the time, indicating a high equivalent torque load. Recommendations: Increase the height of the material rack by 15cm to maintain the elbow joint operating angle within the range of 60°±20°; also optimize the tool handle design to reduce grip torque." The report supports the display of text, images, and video clips.
[0085] A single operation cycle is defined as the complete time interval for executing a single repetitive human-computer interaction task, with the target joint pre-specified based on the task type.
[0086] Furthermore, the specific steps of S200 include:
[0087] S201. Perform camera distortion correction on the RGB image and median filtering to denoise the depth image; spatially align the depth image and the RGB image based on calibration parameters.
[0088] S202. The instance segmentation neural network is used to process the RGB image after distortion correction, and the mask and confidence score of each human instance are output. Instances with confidence scores lower than the first threshold are removed.
[0089] S203. For each preserved human instance mask region, a high-resolution pose estimation network is used to output the two-dimensional coordinates and detection confidence of preset joints; for joints with detection confidence below the second threshold, the coordinates are fine-tuned based on the heatmap distribution.
[0090] S204. Query the depth value of the corresponding position in the aligned depth image based on the two-dimensional coordinates; if the depth value is invalid or the confidence of the joint detection is lower than the third threshold, the weighted median of the effective depth values in the neighborhood centered on the two-dimensional coordinates is used as the compensation depth value, where the weight is inversely proportional to the Euclidean distance from the neighboring pixels to the center point.
[0091] S205. Based on the camera intrinsic parameter matrix, back-project the two-dimensional coordinates and corresponding depth values to the camera coordinate system to obtain the three-dimensional joint coordinates:
[0092] ;
[0093] in, Let i be the image coordinates of the i-th joint. The depth value corresponding to the image coordinates. For the camera intrinsic parameter matrix, , Focal length Principal point coordinates;
[0094] S206. Apply Kalman filtering to the three-dimensional coordinate sequence of the same joint point in consecutive frames for temporal smoothing, and use an interpolation algorithm based on kinematic constraints to complete the trajectory of joint points that are continuously occluded for more than a preset number of frames.
[0095] S207. Encapsulate the smoothed 3D joint coordinates into structured data containing timestamps, instance identifiers, and joint coordinates according to frame order to generate temporal 3D skeleton data.
[0096] Specifically, firstly, distortion correction is performed on the RGB image, using the Zhang Zhengyou calibration method to obtain the distortion coefficients. Inverse mapping correction is performed by applying a 5×5 median filter to the depth image to fill holes and suppress noise. Then, based on the camera calibration extrinsic parameters, the depth image pixels are mapped to the RGB image coordinate system to achieve spatial alignment.
[0097] Subsequently, the Mask R-CNN model pre-trained on the COCO dataset was used to segment the corrected RGB images, outputting human masks and confidence scores. A first threshold of 0.8 was set, and only instances with confidence scores higher than this value were retained.
[0098] For each preserved human instance region, the HRNet-w32 pose estimation network is used to predict the two-dimensional coordinates and confidence of 17 COCO standard joints. For joints with confidence below the second threshold (set to 0.7), the coordinates are fine-tuned to the region with the highest response by using 2D Gaussian kernel weighted interpolation based on their heatmap distribution.
[0099] During the deep fusion stage, the depth value at the corresponding position in the aligned depth map is queried based on the 2D coordinates of the joint. If the depth of that point is invalid or the confidence of the joint is lower than the third threshold (0.5), a 7×7 neighborhood is taken centered on that point, and the weighted median of the effective depth values in the neighborhood is calculated as the compensation depth. The weight is inversely proportional to the Euclidean distance from the pixel to the center.
[0100] Using camera intrinsic matrix , Two-dimensional pixel coordinates and corresponding depth Three-dimensional coordinates projected back onto the camera coordinate system The calculation formula is: .
[0101] The three-dimensional coordinate sequence of the same joint point in consecutive frames is smoothed by applying a Kalman filter. The state vector includes position and velocity. If a joint point is lost for more than 10 consecutive frames, the trajectory is completed by cubic spline interpolation based on human kinematic constraints, such as the invariance of bone length and the continuity of joint angles.
[0102] Finally, the smoothed 3D coordinates of all joints are encapsulated into JSON format data in frame order. Each record contains a timestamp, instance ID, and XYZ coordinates of each joint, forming temporal 3D skeleton data.
[0103] Furthermore, the calculation of the limb segment center of mass acceleration includes: determining the position coordinate sequence of the limb segment center of mass in the three-dimensional skeleton based on anthropometric proportion parameters, performing time-domain numerical differentiation on the position coordinate sequence, and obtaining the center of mass acceleration vector.
[0104] Furthermore, the single operation cycle is determined by the Hilbert-Huang transform analysis of the joint angle time series curve. When the standard deviation of the cycle length of three consecutive cycles is less than 3%, the mean is taken as the operation cycle length. When the fluctuation exceeds 15%, manual review is triggered.
[0105] Joint angle time sequence curve Calculations are differentiated based on joint type, including hinge joints and ball-and-socket joints, and angular velocity time-series curves are provided. Through the The cumulative deviation time is calculated using the central difference method after applying a filter with a preset window length. By traversing a single cycle The total time during which the joint angle exceeds the preset neutral position range is counted.
[0106] Furthermore, a biomechanical model was constructed based on the height and weight of the target being measured. The mass, center of mass position and moment of inertia of each limb segment were determined using the Winter anthropometry standard, and a multi-rigid-body dynamic model with 14 rigid body segments was constructed.
[0107] The inverse dynamics sub-model is solved using the Newton-Euler recursive algorithm, calculating segment by segment from the distal end of the limb to the proximal end to determine the net torque of the target joint. ,Will The specific formula is as follows:
[0108] ( , ;
[0109] in, For the force of the k-th muscle, The maximum isometric contractile force of the k-th muscle. This is the transpose of the lever arm vector. This represents the total number of muscles involved in joint movement.
[0110] Furthermore, the specific formula for calculating the mechanical work done by the joint in step S500 is as follows:
[0111] ;
[0112] in, and These represent the start and end times of the operation cycle.
[0113] Reference Figure 2 A posture-skeleton-guided human-computer interaction comfort assessment system, including:
[0114] The data acquisition module is used to simultaneously acquire RGB image sequences and depth image sequences of the work scene through a depth camera, and perform time alignment.
[0115] The 3D skeleton generation module is used for human instance segmentation and 2D joint detection based on RGB image sequences, fuses the depth images of the corresponding frames, and uses camera calibration parameters to map the 2D joint coordinates to 3D spatial coordinates in the camera coordinate system to generate temporal 3D skeleton data.
[0116] The joint motion analysis module is used to calculate the joint angle time-series curve within a single operation cycle for a preset target joint based on time-series 3D skeleton data. With angular velocity time series curve The cumulative time during which the joint angle continuously deviated from the ergonomic neutral position range was recorded. ;
[0117] The biomechanical analysis module is used to construct human biomechanical models and analyze the time-series curves of joint angles. Angular velocity time series curve The inverse dynamics sub-model is constructed by inputting the limb segment center-of-mass acceleration calculated based on time-series 3D skeleton data, and then solving for the net torque of the target joint. ;Will Input the muscle load distribution sub-model and solve for the equivalent torques of the main muscle groups driving the joint. And calculate the mechanical work done by the joint. ;
[0118] The comfort assessment module is used to construct a comfort assessment function, based on the extreme values of joint angles. Cumulative time , peak Given the input parameters, output the local comfort score of the target body part. ;
[0119] The visualization and report generation module is used for local comfort rating. Generate a dynamic comfort heatmap overlaid on a 3D human body model and generate an optimization suggestion report.
[0120] Furthermore, the 3D skeleton generation module specifically includes:
[0121] The preprocessing unit performs camera distortion correction on the RGB image and median filtering to denoise the depth image; it also spatially aligns the depth image with the RGB image based on calibration parameters.
[0122] The instance segmentation unit is used to process the distortion-corrected RGB image using an instance segmentation neural network, output the mask and confidence score of each human instance, and remove instances with confidence scores lower than the first threshold.
[0123] The two-dimensional joint detection unit is used to output the two-dimensional coordinates and detection confidence of preset joints for each preserved human instance mask region using a high-resolution pose estimation network; for joints with detection confidence below the second threshold, the coordinates are fine-tuned based on the heatmap distribution.
[0124] The depth fusion unit is used to query the depth value of the corresponding position in the aligned depth image based on the two-dimensional coordinates. If the depth value is invalid or the confidence of the key point detection is lower than the third threshold, the weighted median of the effective depth values in the neighborhood centered on the two-dimensional coordinates is used as the compensation depth value, where the weight is inversely proportional to the Euclidean distance from the neighboring pixels to the center point.
[0125] The 3D backprojection unit is used to backproject the 2D coordinates and corresponding depth values to the camera coordinate system based on the camera intrinsic parameter matrix to obtain the 3D joint coordinates.
[0126] The timing processing unit is used to apply Kalman filtering to the three-dimensional coordinate sequence of the same joint point in consecutive frames for timing smoothing, and to use a kinematic constraint-based interpolation algorithm to complete the trajectory of joint points that are continuously occluded for more than a preset number of frames.
[0127] The data encapsulation unit is used to encapsulate the smoothed and completed 3D joint coordinates into structured data containing timestamps, instance identifiers, and joint coordinates in frame order, generating temporal 3D skeleton data.
[0128] A computer device includes a memory and a processor, the memory storing a computer program, and the processor implementing a posture skeleton-guided human-computer interaction comfort assessment method when executing the computer program.
[0129] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a posture skeleton-guided human-computer interaction comfort assessment method for any one of the following.
[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.
Claims
1. A posture-skeleton-guided method for evaluating human-computer interaction comfort, characterized in that, Includes the following steps: S100: Simultaneously acquire RGB image sequences and depth image sequences of the work scene using a depth camera, and perform time alignment. S200. Based on the RGB image sequence, perform human instance segmentation and two-dimensional joint detection, fuse the depth image of the corresponding frame, and use camera calibration parameters to map the two-dimensional joint coordinates to three-dimensional spatial coordinates in the camera coordinate system to generate temporal three-dimensional skeleton data. S300. For a preset target joint, calculate the joint angle timing curve within a single operation cycle based on the temporal three-dimensional skeleton data. With angular velocity time series curve The cumulative time during which the joint angle continuously deviated from the ergonomic neutral position range was recorded. ; S400, Construct a human biomechanical model and generate joint angle time-series curves. Angular velocity time series curve The inverse dynamics submodel is used to calculate the limb segment centroid acceleration based on the temporal 3D skeleton data, and the net torque of the target joint is solved. ;Will Input the muscle load distribution sub-model and solve for the equivalent torques of the main muscle groups driving the joint. And calculate the mechanical work done by the joint. ; S500, construct a comfort evaluation function based on joint angle extreme values. Cumulative time , peak Given the input parameters, output the local comfort score of the target body part. The specific formula is as follows: ； in, For neutral angles, For the allowable angle range, For operation cycle duration, and These are the reference thresholds for the equivalent torque of the major muscle groups and the mechanical work done by the joints, respectively. , , and Weighting coefficients, local comfort scores A higher value indicates a higher level of comfort. S600, based on local comfort rating Generate a dynamic comfort heatmap overlaid on a 3D human body model and generate an optimization suggestion report; The single operation cycle is the complete time interval for performing a single repetitive human-computer interaction task, and the target joint is pre-specified according to the task type.
2. The posture skeleton-guided human-computer interaction comfort evaluation method according to claim 1, characterized in that, The specific steps of S200 include: S201. Perform camera distortion correction on the RGB image and median filtering to denoise the depth image; spatially align the depth image and the RGB image based on calibration parameters. S202. The instance segmentation neural network is used to process the RGB image after distortion correction, and the mask and confidence score of each human instance are output. Instances with confidence scores lower than the first threshold are removed. S203. For each preserved human instance mask region, a high-resolution pose estimation network is used to output the two-dimensional coordinates and detection confidence of preset joints; for joints with detection confidence below the second threshold, the coordinates are fine-tuned based on the heatmap distribution. S204. Query the depth value of the corresponding position in the aligned depth image according to the two-dimensional coordinates; if the depth value is invalid or the confidence of the joint detection is lower than the third threshold, the weighted median of the effective depth values in the neighborhood centered on the two-dimensional coordinates is used as the compensation depth value, where the weight is inversely proportional to the Euclidean distance from the neighboring pixel to the center point. S205. Based on the camera intrinsic parameter matrix, back-project the two-dimensional coordinates and corresponding depth values to the camera coordinate system to obtain the three-dimensional joint coordinates: ； in, Let i be the image coordinates of the i-th joint. The depth value corresponding to the image coordinates. For the camera intrinsic parameter matrix, , Focal length Principal point coordinates; S206. Apply Kalman filtering to the three-dimensional coordinate sequence of the same joint point in consecutive frames for temporal smoothing, and use an interpolation algorithm based on kinematic constraints to complete the trajectory of joint points that are continuously occluded for more than a preset number of frames. S207. Encapsulate the smoothed 3D joint coordinates into structured data containing timestamps, instance identifiers, and joint coordinates according to frame order to generate temporal 3D skeleton data.
3. The posture skeleton-guided human-computer interaction comfort evaluation method according to claim 1, characterized in that, The calculation of the limb segment centroid acceleration includes: determining the position coordinate sequence of the limb segment centroid in the three-dimensional skeleton based on anthropometry proportion parameters, performing time-domain numerical differentiation on the position coordinate sequence to obtain the centroid acceleration vector.
4. The posture skeleton-guided human-computer interaction comfort evaluation method according to claim 1, characterized in that, The single operation cycle is determined by the Hilbert-Huang transform analysis of the joint angle time series curve. When the standard deviation of the cycle length of three consecutive cycles is less than 3%, the average value is taken as the operation cycle length. When the fluctuation exceeds 15%, manual review is triggered. The joint angle timing curve Calculations are differentiated based on joint type, where joint types include hinge joints and ball-and-socket joints, and the angular velocity time-series curves are... Through the The cumulative deviation time is calculated using the central difference method after applying a filter with a preset window length. By traversing a single cycle The total time during which the joint angle exceeds the preset neutral position range is counted.
5. The posture skeleton-guided human-computer interaction comfort evaluation method according to claim 1, characterized in that, The construction of the human biomechanical model is based on the height and weight of the target being measured. The mass, center of mass position and moment of inertia of each limb segment are determined by the Winter anthropometry standard, and a multi-rigid-body dynamic model with 14 rigid body segments is constructed. The inverse dynamics sub-model is solved using the Newton-Euler recursive algorithm, calculating segment by segment from the distal end of the limb to the proximal end to determine the net torque of the target joint. ,Will The specific formula is as follows: （ , ； in, For the force of the k-th muscle, The maximum isometric contractile force of the k-th muscle. This is the transpose of the lever arm vector. This represents the total number of muscles involved in joint movement.
6. The posture skeleton-guided human-computer interaction comfort evaluation method according to claim 1, characterized in that, The specific formula for calculating the mechanical work done by the joint in step S500 is as follows: ； in, and These represent the start and end times of the operation cycle.
7. A posture-skeleton-guided human-computer interaction comfort assessment system, characterized in that, include: The data acquisition module is used to simultaneously acquire RGB image sequences and depth image sequences of the work scene through a depth camera, and perform time alignment. The three-dimensional skeleton generation module is used to perform human instance segmentation and two-dimensional joint detection based on the RGB image sequence, fuse the depth image of the corresponding frame, and use camera calibration parameters to map the two-dimensional joint coordinates to three-dimensional spatial coordinates in the camera coordinate system to generate temporal three-dimensional skeleton data. The joint motion analysis module is used to calculate the joint angle time-series curve within a single operation cycle based on the time-series three-dimensional skeleton data for a preset target joint. With angular velocity time series curve The cumulative time during which the joint angle continuously deviated from the ergonomic neutral position range was recorded. ; The biomechanical analysis module is used to construct human biomechanical models and analyze the time-series curves of joint angles. Angular velocity time series curve The inverse dynamics submodel is used to calculate the limb segment centroid acceleration based on the temporal 3D skeleton data, and the net torque of the target joint is solved. ;Will Input the muscle load distribution sub-model and solve for the equivalent torques of the main muscle groups driving the joint. And calculate the mechanical work done by the joint. ; The comfort assessment module is used to construct a comfort assessment function, based on the extreme values of joint angles. Cumulative time , peak Given the input parameters, output the local comfort score of the target body part. ; The visualization and report generation module is used for local comfort rating. Generate a dynamic comfort heatmap overlaid on a 3D human body model and generate an optimization suggestion report.
8. The posture skeleton-guided human-computer interaction comfort assessment system according to claim 7, characterized in that, The three-dimensional skeleton generation module specifically includes: The preprocessing unit performs camera distortion correction on the RGB image and median filtering to denoise the depth image; it also spatially aligns the depth image with the RGB image based on calibration parameters. The instance segmentation unit is used to process the distortion-corrected RGB image using an instance segmentation neural network, output the mask and confidence score of each human instance, and remove instances with confidence scores lower than the first threshold. The two-dimensional joint detection unit is used to output the two-dimensional coordinates and detection confidence of preset joints for each preserved human instance mask region using a high-resolution pose estimation network; for joints with detection confidence below the second threshold, the coordinates are fine-tuned based on the heatmap distribution. The depth fusion unit is used to query the depth value of the corresponding position in the aligned depth image based on the two-dimensional coordinates. If the depth value is invalid or the confidence of the joint detection is lower than the third threshold, the weighted median of the effective depth values in the neighborhood centered on the two-dimensional coordinates is used as the compensation depth value, where the weight is inversely proportional to the Euclidean distance from the neighboring pixel to the center point. The 3D backprojection unit is used to backproject the 2D coordinates and corresponding depth values to the camera coordinate system based on the camera intrinsic parameter matrix to obtain the 3D joint coordinates. The timing processing unit is used to apply Kalman filtering to the three-dimensional coordinate sequence of the same joint point in consecutive frames for timing smoothing, and to use a kinematic constraint-based interpolation algorithm to complete the trajectory of joint points that are continuously occluded for more than a preset number of frames. The data encapsulation unit is used to encapsulate the smoothed and completed 3D joint coordinates into structured data containing timestamps, instance identifiers, and joint coordinates in frame order, generating temporal 3D skeleton data.
9. A computer device, comprising a memory and a processor, characterized in that, The memory stores a computer program, and when the processor executes the computer program, it implements the posture skeleton-guided human-computer interaction comfort assessment method according to any one of claims 1 to 6.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the posture skeleton-guided human-computer interaction comfort evaluation method according to any one of claims 1 to 6.