Joint movement detection method and system, computer equipment and storage medium

Abstract

The invention relates to the technical field of auxiliary rehabilitation, and discloses a joint movement detection method and system, computer equipment and a storage medium, and the method comprises the steps: obtaining joint movement video data of a to-be-measured user; joint two-dimensional coordinate information in one-to-one correspondence with the joint points is obtained; converting the joint two-dimensional coordinate information into joint three-dimensional coordinate information based on depth information in the joint activity video data; determining joint motion range measurement information of the user to be measured based on the joint three-dimensional coordinate information corresponding to each frame of image; judging whether the to-be-measured user has an abnormal posture or action or not based on the joint motion range measurement information; and when the user to be measured has the abnormal posture or action, generating an abnormal analysis report. According to the invention, full automation of joint motion detection is effectively realized, the joint motion range measurement efficiency and the judgment efficiency of abnormal user motion conditions are improved, personal errors are reduced, and the accuracy of joint motion range and abnormal posture or motion detection is ensured.

Description

Technical Field

[0001] This invention relates to the field of assistive rehabilitation technology, specifically to a method, system, computer device, and storage medium for detecting joint movement. Background Technology

[0002] In the assessment of motor dysfunction, limb range of motion measurement is a key clinical examination method, widely used in the diagnosis and rehabilitation of various diseases, such as stroke. By measuring the range of motion of a patient's joints, doctors can accurately understand the degree of functional impairment, thereby developing a more appropriate treatment plan.

[0003] Taking upper limb range of motion measurement as an example, current clinical practice commonly uses scales to evaluate patients' upper limb function. These scales typically involve quantitative measurements of upper limb range of motion, including parameters such as the patient's maximum angle and angular velocity during voluntary movement and weight-bearing training. However, traditional upper limb range of motion measurement methods mainly rely on manual operation, manually recording joint angles using tools such as goniometers and angle meters, and are mostly based on the operator's experience to determine whether the patient has abnormal postures or movements. Faced with a large patient population and repetitive assessment tasks, physicians bear enormous workload pressure, and efficiency is low. Summary of the Invention

[0004] In view of this, the present invention provides a method, system, computer device and storage medium for joint activity detection, in order to solve the problem of low efficiency in the prior art when measuring limb range of motion and judging abnormal postures or movements.

[0005] In a first aspect, the present invention provides a method for detecting joint movement, the method comprising: Acquire joint activity video data of the user to be measured. The joint activity video data is continuously acquired by a depth camera. The joint activity video data is input into a pre-established machine learning model to identify the joint points corresponding to each frame of the image, and the joint points corresponding to each frame of the image are calibrated to obtain the two-dimensional coordinate information of the joint corresponding to each joint point. Based on the depth information in the joint activity video data, the two-dimensional coordinate information of the joint is converted into the three-dimensional coordinate information of the joint. Based on the joint three-dimensional coordinate information corresponding to each frame of the image, the joint range of motion measurement information of the user to be measured is determined. This includes: determining a first frame image and a second frame image adjacent to the first frame image, wherein the first frame image is any frame image in the joint activity video data; determining the joint range of motion measurement information of the user to be measured based on the joint three-dimensional coordinate information of the first frame image and the joint three-dimensional coordinate information of the second frame image; wherein the joint range of motion measurement information includes: joint movement angle and joint movement angular velocity; Based on joint range of motion measurement information, determine whether the user under measurement has abnormal postures or movements; When the user being measured has abnormal postures or movements, an anomaly analysis report is generated, which includes the anomaly type identification results and rehabilitation training suggestions.

[0006] In this embodiment, the present invention proposes an automated joint activity detection method. This method uses a depth camera as the primary sensor to capture real-time video data of the patient. Through a machine learning model, the system automatically identifies and calibrates the large and small joints of the shoulder, elbow, wrist, and hand in the video frames, thereby obtaining the coordinates of these joints on a two-dimensional plane. While ensuring measurement accuracy, the two-dimensional coordinates are converted into true three-dimensional spatial coordinates with the camera as a reference, based on the depth data from the depth camera. Then, using a spatial vector angle calculation method, the true angles and angular velocities of each joint in the patient's upper limb are measured. This effectively achieves full automation of joint activity detection without manual intervention by a doctor. It significantly improves the efficiency of joint range of motion measurement and the efficiency of judging abnormal user movements, reduces human error, and ensures the accuracy of joint range of motion and abnormal posture or movement detection.

[0007] In one optional implementation, joint range of motion measurement information of the user to be measured is determined based on the joint three-dimensional coordinate information of the first frame image and the joint three-dimensional coordinate information of the second frame image, including: The first target joint point is determined from the joint three-dimensional coordinate information of the first frame image. The first target joint point is any joint point among the joint points corresponding to the first frame image. Identify the two first adjacent joints that are adjacent to the first target joint; The first bone vector is determined based on the first target joint and the first adjacent joint. Based on the first bone vector, determine the first bone joint angle formed by the first target joint point and the first adjacent joint point; The second target joint point corresponding to the first target joint point and the two second adjacent joint points corresponding to the two first adjacent joint points are determined from the joint three-dimensional coordinate information of the second frame image. The second bone vector is determined based on the second target joint and the second adjacent joint. Based on the second bone vector, determine the second bone joint angle formed by the second target joint point and the second adjacent joint point; Based on the angles of the first and second joints, the joint range of motion and the joint angular velocity are determined.

[0008] In this embodiment, the dynamic range of motion and angular velocity of the joint can be determined by comparing the three-dimensional coordinate information of the joint in two consecutive frames. Real-time video capture ensures the immediacy and efficiency of the detection, effectively realizing the dynamic measurement of joint range of motion information.

[0009] In one optional implementation, after acquiring the joint movement video data of the user to be measured, the method further includes: Analyze the motion trajectories of adjacent frames in joint activity video data to determine if there are any missing data. Perform time-series interpolation in the presence of missing data; The joint activity video data after time-series interpolation is input into the machine learning model.

[0010] In this embodiment, the video-based system effectively fills in the data gaps caused by frame breaks or noise by introducing a temporal proximity compensation algorithm, ensuring the continuity of dynamic motion and thus providing a highly efficient and accurate method for measuring upper limb range of motion in clinical practice.

[0011] In one optional implementation, based on joint range of motion measurement information, determining whether the user being measured has abnormal postures or movements includes: Acquire the joint range of motion measurement information sequence of the user to be measured. The joint range of motion measurement information sequence includes the three-dimensional coordinate information of the joint, the joint range of motion angle and the joint range of motion angular velocity corresponding to each of the multiple frames of images. Dynamic motion feature parameters are extracted from the joint range of motion measurement information sequence. These dynamic motion feature parameters include: the relationship between joint angle and time series, the relationship between joint angular velocity and time series, the smoothness of the motion trajectory, and the symmetry of the motion period. Based on a pre-established database of standard movements of healthy individuals, it is determined whether there are abnormality indicators in the dynamic motion characteristic parameters. The database of standard movements of healthy individuals is a dynamic database that can characterize the normal activity patterns of various joints of healthy individuals after standardized data collection and data cleaning. If at least one anomaly index is found among the dynamic motion characteristic parameters, it is determined that the user to be measured has an abnormal posture or action.

[0012] In one optional implementation, based on a pre-established database of standard movements of healthy individuals, it is determined whether there are any abnormalities in the relationship between joint angles and time series characteristics or the relationship between joint angular velocity and time series characteristics, including: Obtain the joint-time features corresponding to the test sequence. The test sequence is any time sequence in the joint range of motion measurement information sequence. The joint-time features are the relationship between joint angle and time sequence or the relationship between joint angular velocity and time sequence. Obtain standard norm data from the database of standard movements of healthy individuals; Determine the distance information between each node in the joint-time feature to be tested and each node in the standard norm data, and determine the distance matrix; Based on the distance matrix, the minimum cost path between each node in the joint-time feature under test and each node in the standard norm data is determined; the minimum cost path is used to align each node in the joint-time feature under test with each node in the standard norm data. Based on the minimum cost path, the similarity between the joint-time features to be tested and the standard norm data is determined. Based on similarity, determine whether there are any abnormalities in the joint-time features to be tested.

[0013] In this embodiment, by comprehensively analyzing the user's joint range of motion measurement information sequence, dynamic motion characteristic parameters are extracted and compared with a database of standard movements of healthy people. By analyzing features such as joint angle, angular velocity, smoothness of motion trajectory, and symmetry of motion cycle, abnormalities in the user's movement can be effectively identified, and an abnormality analysis report can be generated in a timely manner. This can help the user carry out targeted rehabilitation training and improve the effect of motor function recovery.

[0014] In a second aspect, the present invention provides a joint range of motion measurement system, the system comprising: The depth camera module is used to acquire video data of the joint movements of the user being measured. The information processing module is used to execute the joint activity detection method in any of the above real-time modes; The user interface module is used to display joint activity video data and joint range of motion measurement information of the user being measured.

[0015] In one alternative implementation, the user interface module includes a graphical user interface; the graphical user interface includes: The video playback window is used to display joint activity video data with mapping node information; The dynamic change display window is used to display a line graph showing the dynamic changes in joint range of motion measurement information; The measurement results display window shows the hand position map and hand position information.

[0016] In one optional implementation, the measurement result display window includes: The sub-interface switching bar includes a hand position map button and a joint switching button corresponding to different parts of the body. When the hand point map button or joint switching button in the sub-interface switching bar is triggered, the measurement result display area displays the corresponding hand point map or joint measurement result. Each measurement result also has a corresponding line graph selection button. When the line graph selection button is triggered, the dynamic change display window displays the corresponding dynamic change line graph.

[0017] This invention presents an automated upper limb joint range of motion measurement system, providing clinicians with an efficient, accurate, and easy-to-use tool to effectively assist them in assessing the functional abilities of patients with upper limb dysfunction. The system also offers a user-friendly interface that displays real-time measurement data and video feedback, helping clinicians assess rehabilitation progress for detailed analysis and reference.

[0018] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the joint movement detection method of the first aspect or any corresponding embodiment described above.

[0019] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the joint movement detection method of the first aspect or any corresponding embodiment described above. Attached Figure Description

[0020] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of a joint movement detection method according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating the analysis of abnormal postures or movements according to an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the process of constructing a database of standard movements for healthy individuals according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the node-aligned sequence according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the execution flow of a joint range of motion measurement system according to an embodiment of the present invention; Figure 6This is a schematic diagram of the joint range of motion measurement system according to an embodiment of the present invention; Figure 7 This is a schematic diagram of a graphical user interface according to an embodiment of the present invention; Figure 8 This is a schematic diagram illustrating the content displayed in a video playback window according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the measurement result display area interface according to an embodiment of the present invention; Figure 10 This is a schematic flowchart illustrating the use of a joint range of motion measurement system according to an embodiment of the present invention; Figure 11 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Most current goniometers suffer from the following problems: 1. Require manual operation by doctors; traditional mechanical goniometers rely on manual intervention by clinicians, leading to low measurement efficiency and affecting accuracy. Furthermore, individual differences among operators often result in a lack of objectivity, impacting data reliability. 2. Low level of automation; while devices like the Goniometer Pro digitize the functions of mechanical goniometers and implement them on mobile devices, improving efficiency to some extent, they still require manual measurement by doctors, failing to achieve full automation. 3. Wearable devices increase patient burden; for example, some goniometers are designed to be worn on the wrist, but this design is unsuitable for assessing the range of motion in patients with upper limb dysfunction. The patient's already limited joint movement, coupled with the added weight of the device, leads to a significant difference between their mobility and normal voluntary movement, thus affecting the accuracy of clinical assessments of upper limb function. 4. Projection error exists in two-dimensional planar measurements; some researchers' inventions involve measuring joint angles by identifying joint points in images through manual annotation or AI models. Although this method achieves a degree of automation and intelligence, an inherent measurement bias remains: the joint points in the image represent projections on a two-dimensional plane, not their actual physical locations in three-dimensional space. This projection error inevitably affects the clinical assessment of patients' upper limb function. 5. Lack of real-time dynamic measurement; the methods mentioned above generally face a common limitation: they are all limited to measuring joint motion angles at a static, single point in time. This approach does not meet the needs of dynamic assessment of patients with upper limb dysfunction because it cannot effectively quantify the patient's motor speed ability. 6. Lack of comprehensive or detailed hand joint range of motion testing; existing technologies do not cover key hand joints (such as metacarpophalangeal joints and interphalangeal joints). In upper limb rehabilitation assessment, hand function (such as grip strength and fine motor skills) is an important indicator, and these methods may not meet the clinical needs for hand testing, failing to accurately assess the hand rehabilitation progress of stroke patients (such as finger flexion and extension angles and opposition ability). For patients with hand injuries or arthritis, the system cannot provide effective quantitative data support.

[0024] In view of this, according to an embodiment of the present invention, a method for detecting joint movement is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0025] This embodiment provides a joint movement detection method, which can be executed by devices such as servers, terminals, and mobile terminals. Figure 1 This is a flowchart of a joint motion detection method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps: Step S101: Obtain joint activity video data of the user to be measured. The joint activity video data is obtained by continuous acquisition by a depth camera.

[0026] In this embodiment, a high-precision RealSense-D455 depth camera can be used as the main sensor to capture real-time video data of the user being measured. The depth camera can simultaneously record two-dimensional and depth information of the image, providing a foundation for subsequent three-dimensional coordinate transformation.

[0027] Step S102: Input the joint activity video data into the pre-established machine learning model, identify the joint points corresponding to each frame of the image, and calibrate the joint points corresponding to each frame of the image to obtain the two-dimensional coordinate information of the joint corresponding to each joint point.

[0028] In this embodiment, the open-source machine learning model MediaPipe can be used. This model can automatically identify and label key joints of the human upper limb in video frames, including four major joints (shoulder and elbow) and 42 minor joints (metacarpophalangeal joints (MCP), proximal interphalangeal joints (PIP), distal interphalangeal joints (DIP), fingertips, and wrists). Through this step, the pixel coordinates of the joints on the two-dimensional video frame can be accurately obtained.

[0029] Step S103: Based on the depth information in the joint activity video data, convert the two-dimensional coordinate information of the joint into three-dimensional coordinate information of the joint.

[0030] Provided that the measurement accuracy is ensured, the depth data from the depth camera can be used to convert the joint coordinates on the two-dimensional plane into real three-dimensional spatial coordinates with the camera as a reference.

[0031] Step S104: Based on the joint three-dimensional coordinate information corresponding to each frame of image, determine the joint range of motion measurement information of the user to be measured.

[0032] In this embodiment, the actual angles of each joint of the user's upper limb can be determined based on the three-dimensional coordinate information of the joints using the spatial vector angle calculation method. This includes static angles, dynamic angle changes, and angular velocities during dynamic movement.

[0033] Step S104 above, which is to determine the joint range of motion measurement information of the user to be measured based on the joint three-dimensional coordinate information corresponding to each frame of image, includes: Step S1041: Determine the first frame image and the second frame image adjacent to the first frame image. The first frame image is any frame image in the joint activity video data.

[0034] Step S1042: Based on the joint three-dimensional coordinate information of the first frame image and the joint three-dimensional coordinate information of the second frame image, determine the joint range of motion measurement information of the user to be measured; wherein, the joint range of motion measurement information includes: joint range of motion angle and joint range of motion angular velocity.

[0035] In this embodiment, the dynamic range of motion and angular velocity of the joint can be determined by comparing the three-dimensional coordinate information of the joint in two consecutive frames. Real-time video capture ensures the immediacy and efficiency of the detection, effectively realizing the dynamic measurement of joint range of motion information.

[0036] Step S105: Based on the joint range of motion measurement information, determine whether the user to be measured has any abnormal posture or movement.

[0037] Specifically, dynamic motion characteristic parameters such as the relationship between joint angle and time series, the relationship between joint angular velocity and time series, the smoothness of motion trajectory, and the symmetry of motion cycle can be used to determine whether the user being measured has abnormal movements or postures, such as uncoordinated or interrupted movements, or shrugging.

[0038] Step S106: If the user to be measured has abnormal posture or movement, generate an abnormality analysis report. The abnormality analysis report includes the abnormality type identification results and rehabilitation training suggestions.

[0039] In this embodiment, an anomaly analysis report can be automatically generated based on anomaly score and specific pattern rules, for example: The variation in standard movements is analyzed. If a user undergoes upper limb joint range of motion detection and variation analysis through the system, specifying their gender, age group, and the joint to be tested, and follows the movement guidelines in the subject movement paradigm table to complete the range of motion detection for the tested joint, the results are obtained. The system automatically compares it with the standard norm sequence. Curve similarity calculation is performed to obtain (simplified to) )and (simplified to) ),like The analysis report indicates that the function is normal; if The analysis report indicates that a certain joint has a level one functional impairment; if The analysis report indicated that a certain joint had a level 2 functional impairment.

[0040] The functional differences between the left and right sides are analyzed. If a user undergoes upper limb left and right joint range of motion detection and difference analysis through the system, and follows the movement guidelines in the subject movement paradigm table to complete the range of motion detection of the joint to be tested, the left upper limb sequence is obtained. and right upper limb sequence The system automatically calculates the similarity between the two sequences using curve similarity. (simplified to) ) and anomaly (simplified to) ),like The analysis report showed that the function of a certain joint on both sides of the upper limb was symmetrical. The analysis report indicated poor functional symmetry of a certain joint on the left and right sides of the upper limb, requiring intensive symmetry adjustment training.

[0041] The anomaly analysis report also includes some suggestions, such as recommending "low-speed control training" or similar functions if the motion smoothness is low.

[0042] The anomaly analysis report can also combine historical data to generate rehabilitation progress trend charts and quantitatively display rehabilitation effects, among other functions.

[0043] In some alternative implementations, specific compensatory abnormal postures or movements are identified, including shoulder shrugging compensation and trunk tilt compensation; when the shoulder joint range of motion is detected to be below a threshold and the ipsilateral scapular girdle joint range of motion is abnormally increased, shoulder shrugging compensation is determined to exist; when the trunk tilt angle is detected to exceed a threshold and is not matched with the direction of upper limb movement, trunk tilt compensation is determined to exist.

[0044] This invention proposes an automated joint activity detection method. This method uses a depth camera as the primary sensor to capture real-time video data of the patient. Through a machine learning model, the system automatically identifies and calibrates the large and small joints of the shoulder, elbow, wrist, and hand in the video frames, thereby obtaining the coordinates of these joints on a two-dimensional plane. While ensuring measurement accuracy, the two-dimensional coordinates are converted into true three-dimensional spatial coordinates with the camera as a reference, based on the depth data from the depth camera. Then, using a spatial vector angle calculation method, the true angles and angular velocities of each joint in the patient's upper limb are measured. Simultaneously, this embodiment also provides automatic detection of abnormal postures or movements, which can generate abnormality analysis reports in a timely manner, helping users to conduct targeted rehabilitation training and improve the effectiveness of motor function recovery. The joint activity detection method provided by this invention effectively achieves full automation of joint activity detection without manual intervention by doctors. It significantly improves the efficiency of joint range of motion measurement and the efficiency of judging abnormal user movements, reduces human error, and ensures the accuracy of joint range of motion and abnormal posture or movement detection.

[0045] Compared with related technologies, this invention has the following advantages: 1. High degree of automation; This invention adopts advanced technology, combining depth sensors and machine learning models, and designs a program to automatically acquire three-dimensional coordinate information, achieving fully automated measurement without manual intervention by doctors. This not only significantly improves measurement efficiency but also reduces human error, ensuring the accuracy of measurement results. 2. Improved measurement accuracy; Through intelligent algorithms and precision sensors, it can capture joint angle changes in real time, thereby improving measurement accuracy and providing reliable data support for clinical diagnosis and treatment. 3. Three-dimensional spatial measurement; This method uses three-dimensional measurement technology, avoiding projection errors caused by two-dimensional planar measurement, and can more accurately reflect the actual range of motion of the joint in three-dimensional space.

[0046] In some optional implementations, step S1042 above, namely determining the joint range of motion measurement information of the user to be measured based on the joint three-dimensional coordinate information of the first frame image and the joint three-dimensional coordinate information of the second frame image, includes: Step a1: Determine the first target joint point from the joint 3D coordinate information of the first frame image. The first target joint point is any joint point among the joint points corresponding to the first frame image.

[0047] Step a2: Determine the two first adjacent joints adjacent to the first target joint. Both the target joint and the adjacent joints carry corresponding three-dimensional coordinate information.

[0048] Step a3: Determine the first bone vector based on the first target joint and the first adjacent joint.

[0049] Step a4: Based on the first bone vector, determine the first bone joint angle formed by the first target joint point and the first adjacent joint point.

[0050] Step a5: Determine the second target joint point corresponding to the first target joint point and the two second adjacent joint points corresponding to the two first adjacent joint points from the joint three-dimensional coordinate information of the second frame image.

[0051] Step a6: Determine the second bone vector based on the second target joint and the second adjacent joint.

[0052] Step a7: Based on the second bone vector, determine the second bone joint angle formed by the second target joint point and the second adjacent joint point.

[0053] Step a8: Based on the first joint angle and the second joint angle, determine the joint range of motion and the joint angular velocity.

[0054] This embodiment takes the calculation of one bone joint angle as an example: From the joint 3D coordinate information of the first frame image, determine the first target joint and two adjacent first neighboring joints, denoted as three bony markers in space: P1(x1,y1,z1), P2(x2,y2,z2), and P3(x3,y3,z3). Define two bone vectors. and , representing the two skeletal segments of the joint: ; ; Calculate bone vector and dot product: ; Calculate bone vector and The model: ; ; Calculate bone vector and The angle between : ; ; Using the above method, the angle of the joint formed by three bony landmarks in space can be accurately calculated.

[0055] From the joint 3D coordinate information of the second frame image, determine the second target joint point corresponding to the first target joint point, and the two second adjacent joint points corresponding to the two first adjacent joint points. Repeating the above calculation process, the bone joint angles corresponding to the second frame image can be determined. By comparing the calculated bone joint angles, the joint range of motion angles can be determined. And based on the joint range of motion time, the joint range of motion angular velocity can be determined.

[0056] This embodiment provides a method for calculating the angle of a bone joint formed by three specific bony landmarks in space. This method is applicable to fields such as medical imaging, biomechanical analysis, and sports rehabilitation, and can accurately measure the range of motion of a joint in three-dimensional space. Through intelligent algorithms, it can capture changes in joint angles in real time, thereby improving measurement accuracy and providing reliable data support for clinical diagnosis and treatment. It also ensures the immediacy and efficiency of detection, effectively realizing the dynamic measurement of joint range of motion information.

[0057] In some optional implementations, after acquiring video data of the joint movements of the user to be measured, the method further includes: Analyze the motion trajectories of adjacent frames in the joint movement video data to determine if there is any missing data.

[0058] Perform time-series interpolation in the presence of missing data.

[0059] The joint activity video data after time-series interpolation is input into the machine learning model.

[0060] Due to the inherent frame loss problem in depth camera recording, and the use of time-domain filters in the camera driver design, noise and artifact points are filtered out, resulting in blank images and causing breakpoints. To ensure measurement continuity and reduce data errors, this embodiment employs a temporal nearest neighbor compensation algorithm. This algorithm analyzes the motion trajectories of adjacent frames and uses this information to estimate the joint positions of the missing frames. When a missing frame is detected, the system performs temporal interpolation based on the keypoint positions and motion trends of adjacent frames to fill in the missing keypoint coordinates. Specifically, if all values ​​in the Ti vector are 0, it is considered a breakpoint, and a vector with non-zero values ​​in Ti-1 is used to ensure the continuity of the joint motion temporal sequence and guarantee the real-time performance of the system measurement. By using the nearest neighbor algorithm to fill in data breakpoints caused by the common video frame loss problem, dynamic measurement of upper limb joint range of motion is effectively achieved.

[0061] In this embodiment, the video-based system effectively fills in data gaps caused by frame breaks or noise by introducing a temporal proximity compensation algorithm, ensuring the continuity of dynamic motion and thus providing a highly efficient and accurate method for measuring upper limb range of motion in clinical practice. Furthermore, by introducing the temporal proximity compensation algorithm, joint range of motion measurements become more real-time, dynamic angle / angular velocity measurements become more stable, suitable for real-time feedback in rehabilitation training, and also have strong anti-interference capabilities, significantly reducing data loss due to occlusion or camera shake, thereby improving clinical reliability.

[0062] In some alternative implementations, the joint movement video data includes the upper limb movements of the user to be measured; the joints identified by the machine learning model include 42 hand joints, 2 elbow joints, and 2 shoulder joints.

[0063] In this embodiment, a depth camera is used to directly output high-precision 3D point cloud data. Combined with the MediaPipe open-source model, 46 upper limb joint points are identified, including 42 detail points of the hand. This meets the clinical practice requirements for joint mobility detection during the rehabilitation of stroke patients (focusing on hand joints).

[0064] This application utilizes MediaPipe's Pose and Hand modules, excluding non-upper limb joints (such as the lower limbs) and focusing solely on upper limb motion recognition. The model processes only relevant joints, reducing computational load and achieving a measured frame rate improvement of 10 frames per second (further optimized by a dual-thread design), resulting in better computational efficiency. It is also more clinically adaptable, specifically designed for upper limb rehabilitation, and focuses on 42 detailed joints in the hand, effectively avoiding interference from irrelevant joints.

[0065] This embodiment focuses on the detailed recognition of hand joints, and can identify delicate parts such as metacarpophalangeal joints, which is more in line with clinical needs and has a wider functional coverage.

[0066] Reference Figure 2 As shown, in some optional embodiments, step S105 above, namely, determining whether there is an abnormality index in the dynamic motion characteristic parameters based on a pre-established database of standard movements of healthy people, includes: Step S1051: Obtain the joint range of motion measurement information sequence of the user to be measured. The joint range of motion measurement information sequence includes the joint three-dimensional coordinate information, joint range of motion angle and joint range of motion angular velocity corresponding to multiple frames of images.

[0067] Step S1052: Extract dynamic motion feature parameters from the joint range of motion measurement information sequence. The dynamic motion feature parameters include: joint angle and time series relationship features, joint angular velocity and time series relationship features, motion trajectory smoothness, and motion period symmetry.

[0068] Specifically, key feature parameters can be extracted from a continuous sequence of joint range of motion measurements to form a feature vector. Mainly includes: Angle-Time Series In this embodiment, the joint angle and time sequence relationship feature refers to the joint angle value sequence that changes over time when the user completes a specified measurement action.

[0069] Angular velocity-time series In this embodiment, the characteristic of the relationship between joint angular velocity and time sequence refers to the sequence of joint angular velocity values ​​that change over time when the user completes a specified measurement action.

[0070] The smoothness of the motion trajectory is evaluated by calculating the variance of the second derivative (acceleration) of the motion path of the joints. The smaller the variance, the smoother the motion.

[0071] Movement cycle symmetry: For periodic movements (such as fist clenching-extension), calculate the symmetry error of the angle curves of the left half cycle and the right half cycle, or compare the functional symmetry of the two sides of the joint by calculating the abnormality score.

[0072] Step S1053: Based on a pre-established database of standard movements of healthy individuals, determine whether there are any abnormality indicators in the dynamic motion characteristic parameters. The database of standard movements of healthy individuals is a dynamic database that can characterize the normal activity patterns of various joints in healthy individuals, established through standardized data collection and data cleaning. Among them, reference Figure 3 As shown, the specific construction process of the database of standard movements for healthy individuals is as follows: 1. Standardized data collection: (1) Subject Recruitment and Ethical Guidelines: It is recommended to have no fewer than 100 healthy subjects, stratified according to gender (male / female) and age group (6-17, 18-30, 31-50, 51 and above) to establish more refined subgroup norms. Subjects must meet the following criteria: no history of upper limb skeletal, muscular, or neurological diseases, no pain, and a clearly defined dominant hand. All subjects must sign an informed consent form, and the protocol must be reviewed by the ethics committee.

[0073] (2) Standardization of movement paradigms: To ensure data comparability, it is essential to strictly standardize the starting posture and completion time of movements, and to unify the rhythm of movement using visually guided animations. The subject movement paradigm guidance table is shown in Table 1. Table 1 Subject Action Paradigms

[0074] (3) Data Acquisition: To collect data from subjects in groups using the joint range of motion detection system designed above, the system will output a time series for each movement of each subject. Where n is the sequence length determined by the fixed acquisition duration and acquisition frequency. For collected values (Including but not limited to angle values ​​and velocity values). Among them, when hour Considered as a valid start point of the action sequence, when hour Considered as the valid end of the action sequence, where It sets a threshold.

[0075] 2. Data cleaning and filtering: (1) Motion smoothness screening: This aims to calculate the mean absolute acceleration (MAA) of the collected sequences. A healthy individual should have good control over their movements, resulting in relatively smooth motion with small and stable accelerations during angular changes. The calculation method is as follows: The first-order difference (angular velocity) is calculated from the collected sequence: ; Further calculation of the second-order difference (angular acceleration): ; Further calculate the average acceleration of the entire sequence: ; The MAA of all subjects performing the same action was calculated above. The sequences of the top 5% of subjects with the highest MAA values ​​were removed. These sequences usually correspond to situations of motion stuttering, shaking, or abnormal acquisition.

[0076] (2) Range of motion rationality screening: This aims to identify the maximum and minimum angle values ​​(peak and trough) for each movement sequence. To ensure the robustness and accuracy of the constructed database of standard movements in healthy individuals, for each movement, the peak and trough values ​​of all subjects' sequences are summarized, and their distributions are calculated separately. Since the trends of the sequences collected from each subject are roughly the same, peaks and troughs are bound to exist. The interquartile range (IQR) method is used to identify extreme outliers. Calculate the upper and lower quartiles of the peak / valley values. and ,but ; Set normal range: [ .

[0077] The above calculations show that by removing some sequences whose peaks or troughs fall outside this range, it is possible to effectively eliminate non-representative abnormal data caused by improper movements or excessive overextension.

[0078] 3. Standard sequence modeling and dynamic updating: (1) Time axis normalization: Since sequence lengths can vary slightly even at the same rhythm, directly averaging by time points is inaccurate. The solution is to use DTW or interpolation for sequence alignment: From all the selected sequences, choose the most representative one (e.g., the one whose MAA is closest to the median) as the reference sequence. Use the DTW algorithm to align all other sequences with the reference sequence. After alignment, all sequences are normalized to the same length, and the corresponding time points represent the same phase of the motion.

[0079] (2) Mean fitting to generate standard sequences: On the aligned sequence set, for each time point Calculate the mean of its angle values and standard deviation .

[0080] = ; ; in, It is determined by the number of subjects screened. It is the first Each subject at time point The angle after alignment.

[0081] Ultimately, the standard norm for this action is a rule composed of... The mean sequence formed And attached a clause The standard deviation sequence constitutes the standard deviation. The standard deviation defines the "boundary" of normal activity in healthy individuals under this action.

[0082] (3) Dynamic update mechanism of the database: When data from a new subject k is added, there is no need to recalculate all the data; an incremental update algorithm is used. ; The incremental update algorithm described above can be used to ensure that the database can be continuously optimized and expanded.

[0083] In one optional implementation, when determining whether there are abnormalities in the relationship between joint angles and time series characteristics or the relationship between joint angular velocity and time series characteristics based on a pre-established database of standard movements of healthy individuals, the specific steps include the following: Step b1: Obtain the joint-time features corresponding to the test sequence. The test sequence is any time sequence in the joint range of motion measurement information sequence. The joint-time features are the relationship between joint angle and time sequence or the relationship between joint angular velocity and time sequence.

[0084] Step b2: Obtain standard norm data from the healthy population standard movement database. In this embodiment, the healthy population standard movement database pre-stores the characteristic parameter ranges of healthy individuals of different ages, genders, and body types performing standard movements. and and standard norm sequences .

[0085] Step b3: Determine the distance information between each node in the joint-time feature to be tested and each node in the standard norm data, and determine the distance matrix.

[0086] In this embodiment, the Dynamic Time Warping (DTW) algorithm is used to calculate the standard curve. Curve of the user under test The minimum path distance between (t represents the discrete time series) That is, to calculate the minimum path distance between the joint-time feature to be tested and the standard norm data, the specific steps are as follows: In this embodiment, the relationship between joint angle and time series is used as a feature, namely angle-time series. For example, the relationship between joint angular velocity and time series, i.e., angular velocity-time series. Similarly, specifically, based on the characteristics of the relationship between the angle of the joint to be measured and the time series obtained by the system... (of length n), and standard norm data from a pre-stored database of standard movements of healthy individuals. Construct a distance matrix of length m. Distance matrix Each element in the matrix express The Middle points and The Middle Euclidean distance (or other distance metric) between points: This matrix visually represents the local distance between all pairs of points in two sequences.

[0087] Step b4: Based on the distance matrix, determine the minimum cost path between each node in the joint-time feature to be tested and each node in the standard norm data; the minimum cost path is used to align each node in the joint-time feature to be tested with each node in the standard norm data.

[0088] To determine similarity, a path needs to be found through the distance matrix. path ,in This path defines and Alignment between paths. The following conditions must be met: Boundary condition: The path must start from Beginning, to End. This ensures that the start and end points of the sequence are aligned; Continuity condition: The path must be continuous. If ,So It can only be , or ; Monotonicity condition: on the path and The index must be monotonically increasing. This ensures that the chronological order is not reversed.

[0089] Then, the cumulative cost is calculated (dynamic programming), with the goal of finding a path that minimizes the total cumulative cost. This can be solved efficiently using dynamic programming. Define a cumulative cost matrix. Its size is same. Indicates from arrive The minimum cumulative cost. The recursive formula is: ; Start counting from the top left corner to the bottom right corner. Finally... It refers to the optimal alignment of two sequences. The node-aligned sequence, refer to... Figure 4 As shown.

[0090] Step b5: Based on the minimum cost path, determine the similarity between the joint-time features to be tested and the standard norm data.

[0091] Specifically, Distance is an absolute value; to obtain a more intuitive similarity score (between 0 and 1), normalization is needed. After obtaining the minimum path distance, the similarity... The calculation formula is: (Simplified) or (whole); in, It is a normalization factor, which can be the path length. Or Thus, when the distance is 0, the similarity is 1; the greater the distance, the closer the similarity is to 0.

[0092] Here's an example: Suppose the norm's "shoulder abduction" movement angle sequence is a gentle 90-degree arc, while the patient's movement sequence has pauses in the middle (making the sequence longer). In this case, aligning the patient's "pause points" with multiple points of the norm, we can finally calculate a similarity score unaffected by the pauses, thus accurately reflecting the similarity of the movement trajectory shape.

[0093] Step b6: Based on similarity, determine whether there are any abnormalities in the joint-time features to be tested.

[0094] Taking the above example again, a similarity threshold can be preset. When the calculated similarity exceeds the similarity threshold, it is determined that there is an anomaly in the relationship between the joint angle and the time series.

[0095] Furthermore, a weighted average anomaly score can be calculated by combining the similarity of multiple features such as angle and angular velocity. ; in, It is the first The similarity of each feature It is its corresponding weight ( (This can be adjusted according to clinical importance (e.g., angle weight is higher than speed).

[0096] Then, the weighted average abnormality score is used to determine whether the action is abnormal.

[0097] Step S1054: If at least one anomaly index is found among the dynamic motion characteristic parameters, it is determined that the user to be measured has an abnormal posture or movement. Finally, an anomaly analysis report is generated, as detailed in step S106, and will not be repeated here.

[0098] In this embodiment, by comprehensively analyzing the user's joint range of motion measurement information sequence, dynamic motion feature parameters are extracted and compared with a database of standard movements of healthy people. By analyzing features such as joint angle, angular velocity, smoothness of motion trajectory, and symmetry of motion cycle, abnormal situations in the user's movement can be effectively identified, and an abnormality analysis report can be generated in a timely manner. This can help the user carry out targeted rehabilitation training and improve the effect of motor function recovery.

[0099] This embodiment also provides a joint range of motion measurement system, which is used to implement the above embodiments and preferred embodiments, and will not be repeated hereafter.

[0100] This embodiment provides a joint range of motion measurement system, the system comprising: The depth camera module is used to acquire video data of the joint movements of the user being measured.

[0101] The depth camera module in this embodiment can be an Intel RealSense D455 (hereinafter referred to as RS-D455), a binocular depth camera that is flexible, easy to use, and supports multiple operating systems and development platforms. The RS-D455 device integrates an infrared emission module to assist in acquiring depth information, especially in low-light environments, where this sensor provides more accurate data compared to traditional cameras. The RS-D455 device uses an infrared emitter to project an infrared spot, which is captured by the camera unit and then used to perform depth calculations. The RS-D455 device also employs left and right binocular depth imagers, including two camera units configured to capture images from different perspectives. By analyzing the parallax between the two images, the system can calculate the distance and depth of objects. The RS-D455 device also includes a high-resolution RGB camera (with a maximum resolution of 1920×1080 pixels, i.e., 1080p), which can not only capture depth information but also obtain high-quality color images. By combining RGB images with depth images, this device can achieve excellent spatial understanding and object recognition capabilities. Especially in application scenarios that rely heavily on color information, this combination offers significant advantages, thereby greatly enhancing the overall image processing and application performance.

[0102] The information processing module is used to execute the joint movement detection method in any of the above embodiments.

[0103] The information processing module can be integrated into the main control device (computer, etc.). The main control device uses the RS-D455 driver as its core, connecting the camera via Type-C and the host computer via USB 3.1. The RS-D455 can output various data formats, including depth maps (each pixel represents a depth value) and point cloud data (each point represents a 3D coordinate). Intel provides a related software development kit, which developers can use to access raw depth data, perform calculations, and visualize it. The depth image data stream and RGB image data stream captured by the RS-D455 are transmitted to the host computer for further processing.

[0104] In addition, the information processing module provided in this embodiment is implemented based on Python 3.8 programming and uses a special toolkit, which includes pyrealsense2 (2.55.1) for driving RS-D455, mediapipe (0.10) for upper limb joint recognition, OpenCV (4.7.0) for color image processing and PyQt6 (6.4.2) for program interface design.

[0105] Reference Figure 5 As shown, the system provided in this embodiment also employs dual-thread processing. One thread is used to initiate the operation of the depth camera and upper limb joint recognition module to acquire color image data streams and depth image data streams, thereby obtaining the target's 3D coordinate data. The other thread is dedicated to handling 3D coordinate data calculation and data display in the interface design. Inter-thread communication technology is used to transmit 3D coordinate data and ensure data synchronization and thread safety. Experiments demonstrate that in terms of subsequent video playback frame rate performance, dual-thread processing is approximately 10 frames higher than single-thread processing, indicating higher program efficiency.

[0106] This embodiment also utilizes functions from the pyrealsense2 toolkit to launch camera video streams, including depth and color image streams—that is, video data streams combining 2D image data with time series data. Parameter settings include: creating and configuring a time-domain filter to reduce noise and motion artifacts while preserving details of depth changes when processing depth data captured by the RealSense depth camera; setting the depth stream format to 16-bit depth values; setting the color stream format to BGR8; setting the video stream pixel size to 640×360; and setting the frame rate to 30. By continuously calling the video data stream in an infinite loop, a matrix of 30 values ​​per second (640 pixels wide and 360 pixels high) [I0, I1…] and an image depth matrix [D0, D1…] are obtained. This data is provided to the upper limb joint recognition module for subsequent processing.

[0107] The upper limb joint recognition module is a component of the information processing module, specifically designed for computational processing related to joint motion detection methods. Specifically, it utilizes toolkits within Mediapipe, an open-source, cross-platform multimedia processing framework developed and proposed by Google in 2019. Mediapipe is used to build machine learning-based applications, particularly those involving computer vision. In essence, Mediapipe is an integrated library of machine learning and vision algorithms (including various models for face detection, facial landmarks, gesture recognition, avatar segmentation, and pose recognition). Mediapipe provides a series of pre-built models and functionalities for processing various media types, including video, audio, and images, and supports a wide range of computer vision and audio processing tasks, including machine learning.

[0108] In this embodiment, the new module calls the Pose and Hand modules of Mediapipe, excluding other joints in the upper limbs and duplicate points in the wrist. It targets 42 joints in the left and right hands, and 4 joints in the shoulder and elbow (the hand includes the metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), distal interphalangeal joint (DIP), fingertip, and wrist joint). The module input is a pixel value matrix In. The Mediapipe machine learning pre-trained model predicts 46 joints from the matrix (in macroscopic terms, the image). The model is pre-trained on a large dataset, which needs to be labeled, i.e., classified, labeled with key points, or other relevant information. The module output is 46 joint pixel coordinate vectors of length 3, where the vectors represent [x-coordinate (0-640), y-coordinate (0-360), pixel value]. The known joint pixel coordinates are then mapped and connected on the image based on physiological structure using the OpenCV toolkit, making the results more intuitive. Then, the third pixel value of the pixel coordinate vector is replaced with the depth value of the corresponding coordinate in the image depth value matrix (the distance from the coordinate point to the lens in the real world, in mm), to obtain the three-dimensional coordinates [T0, T1...], which means [horizontal coordinate (0-640), vertical coordinate (0-360), depth value].

[0109] Then, a temporal nearest neighbor compensation algorithm is adopted. That is, if all the values ​​in the Ti vector are 0, it is determined as a breakpoint, and the vector with Ti-1 that is not 0 is used to replace it to ensure the continuity of the joint motion temporal sequence and ensure the real-time measurement of this system.

[0110] Finally, the angles of the joints formed by three specific bony landmarks in space and the angular velocities of joint movement are calculated.

[0111] The user interface module is used to display joint activity video data and joint range of motion measurement information of the user being measured.

[0112] Reference Figure 6 As shown, the range of motion of the user's upper limb is measured under the condition that the background environment (such as a desktop) is clean and simple; the depth sensor is set directly above and in front of the user using a sensor bracket, which is conducive to the user's upper limb being completely within the field of view of the depth camera; the main control device is equipped with a machine learning model for depth sensor driving, upper limb joint point recognition, and goniometer usage system for use by clinicians and others.

[0113] This joint range of motion measurement system automates the detection of all hand joints, filling a technological gap in refined clinical assessment. This invention proposes an automated joint range of motion measurement system that can assist clinicians in assessing upper limb function. Real-time video capture ensures the immediacy of the detection and enables dynamic measurement. Furthermore, the system is equipped with a user-friendly and easy-to-use interface for outputting and storing measurement results for clinicians' reference.

[0114] Compared with related technologies, this invention has the following advantages: 1. High degree of automation; This system adopts advanced technology, combining depth sensors and machine learning models, and designs a program to automatically acquire three-dimensional coordinate information, achieving fully automated measurement without manual intervention by doctors. This not only significantly improves measurement efficiency but also reduces human error, ensuring the accuracy of measurement results. 2. Improved measurement accuracy; Through intelligent algorithms and precision sensors, the system can capture joint angle changes in real time, thereby improving measurement accuracy and providing reliable data support for clinical diagnosis and treatment. 3. No additional weight required; This system adopts a non-wearable design, so patients do not need to wear any devices during the measurement process. They only need to rely on their own upper limb joint movements to collect data, effectively reducing the burden on patients and making the measurement results closer to the patient's normal independent activity ability. 4. Three-dimensional spatial measurement; The upper limb joint intelligent angle measuring instrument system adopts three-dimensional measurement technology, avoiding projection errors caused by two-dimensional planar measurement, and can more accurately reflect the actual range of motion of the joint in three-dimensional space. 5. Easy to operate; The system interface is user-friendly and easy to operate. Medical staff can get started after simple training, lowering the threshold for use. 6. Real-time data feedback: Measurement results can be displayed on the screen in real time, facilitating timely adjustments to treatment plans by medical staff and improving treatment outcomes. 7. High adaptability: This system is suitable for patients of different ages and with varying joint ranges of motion. It is easy to operate and has a simple layout, making it highly valuable for clinical applications.

[0115] The present invention proposes an automated upper limb joint range of motion measurement system, which provides clinicians with an efficient, accurate and easy-to-use tool to effectively assist them in completing functional assessments of patients with upper limb dysfunction.

[0116] In some alternative implementations, the user interface module includes a graphical user interface; the graphical user interface includes: The video playback window is used to display joint activity video data with mapping node information.

[0117] The dynamic change display window is used to show a line graph of the dynamic changes in joint range of motion measurement information.

[0118] The measurement results display window shows the hand position map and hand position information.

[0119] Reference Figure 7 As shown, the main window adopts a split-column design, with the upper left corner serving as the main display area, i.e., the video playback window. This window displays the color video stream captured by the RS-D455 camera, as well as joint activity video data displaying mapping node information. The content of a single frame is shown in the image below. Figure 8 As shown, this window allows you to observe the user's movements and the results of joint recognition.

[0120] The bottom left side displays the measurement results, including a sub-interface switching bar. This bar lists several tabs, such as "Left Hand Joints," "Right Hand Joints," "Wrist Joints," and "Hand Point Map." Clicking these buttons allows you to quickly switch to the corresponding sub-interface for more detailed settings and data viewing.

[0121] In the small window located in the upper right corner, clicking the "Dynamic Value Change Line Graph" checkbox displays a trend graph of the angle of a joint of interest changing over time. The horizontal axis represents time (in seconds / s), and the left vertical axis represents the angle value (ranging from 0° to 180°). The blue curve shows the fluctuation of the angle over time (within 10 seconds), helping users intuitively understand the joint's movement pattern and including a record of the joint's movement status throughout the measurement process.

[0122] Located in the middle section below the main display area is the camera on / off switch bar, which includes two rectangular buttons: "Lens on" and "Lens off". These two buttons allow users to manually control the camera's on / off state.

[0123] In addition to the main elements mentioned above, the system may also include other auxiliary functions and settings options, including some recognition details and the ability to record data required by doctors.

[0124] To improve system usability, this embodiment also provides a user-friendly and easy-to-use interface. This interface not only displays real-time measurement data and video feedback but also provides functions for outputting and storing measurement results. This graphical user interface can completely record hand movement trajectories and generate dynamic angle change curves to assist doctors in assessing rehabilitation progress, enabling clinicians to conduct detailed analysis and reference. The software interface provided in this embodiment has a simple and clear design, facilitating quick learning and use by operators. Simultaneously, its rich data display and analysis capabilities provide strong support for clinical applications.

[0125] In some optional implementations, the measurement result display window includes: The sub-interface switching bar includes a hand position map button and a joint switching button corresponding to different parts of the body.

[0126] When the hand point map button or joint switching button in the sub-interface switching bar is triggered, the measurement result display area displays the corresponding hand point map or joint measurement result. Each measurement result also has a corresponding line graph selection button. When the line graph selection button is triggered, the dynamic change display window displays the corresponding dynamic change line graph.

[0127] The sub-interface is used to display the real-time calculation results of the range of motion and angular velocity of each upper limb joint, for reference. Figure 6 As shown, the sub-interface contains five interfaces: the hand position map of the initial interface, the measurement results of the left hand joints, the measurement results of the right hand joints, the measurement results of the wrist joints, and the measurement results of the shoulder / elbow joints. The sub-interface switching button allows users to select the upper limb joints of interest for observation.

[0128] Below the main display area, there is a hand point map showing the distribution and connections of various key points on the hand. Each node has a corresponding number (e.g., point 0 represents the left and right metacarpophalangeal joints, point 5 represents the left and right metacarpophalangeal joints), indicating different joints or locations. Additionally, there is a button group to start / stop the dynamic measurement program. The start button activates the joint measurement program to display real-time measurement results. The stop button resets the measurement data to zero and closes the joint measurement program. The real-time measurement results for each joint will be displayed as follows: Figure 9 The results are displayed in the form of "Joint Location: ---r---r / s". For hand joints, the results need to be read in conjunction with a hand position diagram. A specific hand joint is generally represented in the form of "Adjacent Joint Position - Target Joint Position - Adjacent Joint Position", that is, the three adjacent joint positions that constitute the joint angle. The measurement results are the joint range of motion, in radians / rad (abbreviated r), and the joint angular velocity, in radians per second / rad / s (abbreviated r / s).

[0129] Reference Figure 10 As shown, the depth sensor is positioned directly above and in front of the user using a sensor bracket. Clicking the "Start RS-D455 Camera Lens" button initiates the joint range of motion measurement program. Then, the measurement data for each joint can be viewed using the buttons in the single-step sub-interface switch bar. Simultaneously, clicking the "Line Graph Selection" button displays a dynamic line graph of the corresponding changes in the dynamic display window.

[0130] In the early stages of system development, this invention underwent a comparative experiment with clinical manual measurement methods. Specifically, the accuracy of measurements of upper limb joints, such as the metacarpophalangeal joint of the index finger, ulnar deviation of the wrist, wrist flexion, and wrist ulnar deviation, was compared between the intelligent goniometer of this invention and the manual measurement using a specific clinical mechanical goniometer. Because manual measurement also has significant errors, leading to inaccuracies in the actual values, and because not all joints were measured, this experiment was a preliminary comparative measurement experiment.

[0131] Comparative experimental results refer to Figure 2 As shown, the metacarpophalangeal joint (index finger) is present. A measurement error of 1.78° indicates ulnar deviation of the wrist joint. A measurement error of 2.25° indicates wrist flexion is present. A measurement error of 2.65° indicates wrist joint deviation. The measurement error was 1.73°, which is within an acceptable range.

[0132] Table 2 Comparative Experimental Data Table

[0133] This application utilizes an independently developed user interface for more accurate angle calculations, making it particularly suitable for complex movements (such as wrist flexion and ulnar deviation), effectively avoiding projection errors. Furthermore, the interface is user-friendly, with dynamic line graphs and multi-sub-interface switching (such as independent display of left and right hands) significantly improving doctors' operational efficiency.

[0134] This invention is mainly used in clinical applications for measuring the joint range of motion of the human upper limb. It can also be used for upper limb activity data acquisition, as well as for data acquisition of human hand operation data and motion control data input for non-wearable robotic arms.

[0135] In some optional implementations, as the user performs an action (such as clenching and extending a fist), the system displays a real-time line graph of wrist joint angle changes and triggers a voice prompt (such as "insufficient angle"). Combined with historical data, a rehabilitation progress report is generated, supporting doctors in adjusting training plans. This improves patient compliance and optimizes rehabilitation efficiency.

[0136] In some alternative implementations, two D455 cameras are deployed to capture motion from different angles, and the data is fused using a calibration algorithm to eliminate single-view occlusion issues. This can effectively solve the measurement blind spots for complex movements such as internal rotation of the shoulder joint.

[0137] In some optional implementations, historical data analysis based on LSTM models is added to detect abnormal postures or movements, identify compensatory movements (such as shrugging to compensate for shoulder joint movement), and automatically label wind. This can assist doctors in early intervention and avoid secondary injuries caused by incorrect training.

[0138] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0139] This invention also provides a computer device having an information processing module and a user interface module in the above-described joint range of motion measurement system.

[0140] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 11 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 11 Take a processor 10 as an example.

[0141] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0142] The memory 20 stores instructions executable by at least one processor 10 to cause the at least one processor 10 to perform the method shown in the above embodiments.

[0143] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0144] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0145] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0146] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0147] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A method for detecting joint movement, characterized in that, The method includes: Acquire joint activity video data of the user to be measured, wherein the joint activity video data is continuously acquired by a depth camera; The joint activity video data is input into a pre-established machine learning model to identify the joint points corresponding to each frame of the image, and the joint points corresponding to each frame of the image are calibrated to obtain the joint two-dimensional coordinate information corresponding to each joint point. Based on the depth information in the joint activity video data, the two-dimensional coordinate information of the joint is converted into three-dimensional coordinate information of the joint; Based on the joint three-dimensional coordinate information corresponding to each frame of the image, the joint range of motion measurement information of the user to be measured is determined; wherein, the determination includes: determining a first frame image and a second frame image adjacent to the first frame image, wherein the first frame image is any frame image in the joint activity video data; and determining the joint range of motion measurement information of the user to be measured based on the joint three-dimensional coordinate information of the first frame image and the joint three-dimensional coordinate information of the second frame image; wherein, the joint range of motion measurement information includes: joint movement angle and joint movement angular velocity; Based on the joint range of motion measurement information, it is determined whether the user being measured has any abnormal posture or movement; If the user being measured exhibits abnormal postures or movements, an anomaly analysis report is generated, which includes the anomaly type identification results and rehabilitation training suggestions.

2. The method according to claim 1, characterized in that, The determination of the joint range of motion measurement information of the user to be measured based on the joint three-dimensional coordinate information of the first frame image and the joint three-dimensional coordinate information of the second frame image includes: A first target joint point is determined from the joint three-dimensional coordinate information of the first frame image, wherein the first target joint point is any one of the joint points corresponding to the first frame image; Identify two first adjacent joints that are adjacent to the first target joint; Based on the first target joint and the first adjacent joint, a first bone vector is determined; Based on the first bone vector, determine the first bone joint angle formed by the first target joint point and the first adjacent joint point; Determine the second target joint point corresponding to the first target joint point and the two second adjacent joint points corresponding to the two first adjacent joint points from the joint three-dimensional coordinate information of the second frame image; Based on the second target joint and the second adjacent joint, determine the second bone vector; Based on the second bone vector, determine the second bone joint angle formed by the second target joint point and the second adjacent joint point; Based on the first joint angle and the second joint angle, the joint movement angle and the joint movement angular velocity are determined.

3. The method according to claim 1, characterized in that, After acquiring video data of the joint movements of the user to be measured, the process also includes: Analyze the motion trajectories of adjacent frames in the joint activity video data to determine if there is any missing data; Perform time-series interpolation in the presence of missing data; The joint movement video data after time-series interpolation is input into the machine learning model.

4. The method according to claim 1, characterized in that, The step of determining whether the user under test has abnormal postures or movements based on the joint range of motion measurement information includes: Obtain the joint range of motion measurement information sequence of the user to be measured. The joint range of motion measurement information sequence includes the three-dimensional coordinate information of the joint, the joint range of motion angle, and the joint range of motion angular velocity corresponding to multiple frames of images. Extract dynamic motion feature parameters from the joint range of motion measurement information sequence. The dynamic motion feature parameters include: joint angle and time series relationship features, joint angular velocity and time series relationship features, motion trajectory smoothness, and motion period symmetry. Based on a pre-established database of standard movements of healthy individuals, it is determined whether there are any abnormality indicators in the dynamic movement characteristic parameters; the database of standard movements of healthy individuals is a dynamic database that can characterize the normal activity patterns of various joints of healthy individuals after standardized data collection and data cleaning. If at least one of the abnormality indices is found among the dynamic motion characteristic parameters, it is determined that the user to be measured has an abnormal posture or action.

5. The method according to claim 4, characterized in that, Based on a pre-established database of standard movements of healthy individuals, determine whether there are any abnormalities in the relationship between joint angles and time series or the relationship between joint angular velocity and time series, including: Obtain the joint-time feature corresponding to the test sequence, wherein the test sequence is any time sequence in the joint range of motion measurement information sequence; the joint-time feature is the relationship feature between the joint angle and the time sequence or the relationship feature between the joint angular velocity and the time sequence. Obtain standard norm data from the database of standard movements of healthy individuals; Determine the distance information between each node in the joint-time feature to be tested and each node in the standard norm data, and determine the distance matrix; Based on the distance matrix, a minimum cost path is determined between each node in the joint-time feature to be tested and each node in the standard norm data; the minimum cost path is used to align each node in the joint-time feature to be tested with each node in the standard norm data. Based on the minimum cost path, the similarity between the joint-time feature to be tested and the standard norm data is determined; Based on the similarity, it is determined whether the joint-time feature to be tested is abnormal.

6. A joint range of motion measurement system, characterized in that, The system includes: The depth camera module is used to acquire video data of the joint movements of the user being measured. The information processing module is used to execute the joint activity detection method according to any one of claims 1-5; The user interface module is used to display the joint activity video data and the joint range of motion measurement information of the user to be measured.

7. The system according to claim 6, characterized in that, The user interface module includes a graphical user interface; the graphical user interface includes: The video playback window is used to display joint activity video data with mapping node information; A dynamic change display window is used to display a line graph showing the dynamic changes of the joint range of motion measurement information; The measurement results display window shows the hand position map and hand position information.

8. The system according to claim 7, characterized in that, The measurement result display window includes: The sub-interface switching bar includes a hand position map button and a joint switching button corresponding to different parts of the body. When the hand point map button or joint switching button in the sub-interface switching bar is triggered, the measurement result display area displays the corresponding hand point map or joint measurement result; each measurement result also has a corresponding line graph selection button, and when the line graph selection button is triggered, the dynamic change display window displays the corresponding dynamic change line graph.

9. A computer device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the computer instructions to perform the joint movement detection method according to any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the joint movement detection method according to any one of claims 1-5.

Images