A method for remotely intelligent assessment of knee angle through photographs

Abstract

The present application relates to a kind of methods for remote intelligent evaluation of knee joint angle by photo, belong to medical information processing technical field, solve the problem that accurate remote knee joint angle measurement is difficult in prior art.Method includes: obtaining the knee joint side photo of patient;Wherein, the outside of the leg of patient is pasted with flexible marking band;Marking point is arranged on the flexible marking band with equal interval;Hip joint point, knee joint point, ankle joint point in the knee joint side photo of patient and the marking point on the flexible marking band are detected based on trained detection model;Big thigh direction vector and small leg direction vector are fitted based on the hip joint point, knee joint point, ankle joint point and marking point of detection;Knee joint angle is calculated according to big thigh direction vector and small leg direction vector.It is realized to quickly and accurately remote knee joint angle evaluation.

Description

Technical Field

[0001] This invention relates to the field of medical information processing technology, and in particular to a method for remotely and intelligently assessing knee joint angles using photographs. Background Technology

[0002] In the functional rehabilitation assessment of knee surgery (such as anterior cruciate ligament reconstruction, total knee replacement, etc.), the range of motion (ROM, especially the flexion and extension angles of the knee joint) is one of the core indicators. Accurate and repeatable joint angle measurements are of great clinical significance for judging rehabilitation progress, preventing complications (such as joint fibrosis, Cyclops' disease), and developing personalized rehabilitation plans.

[0003] Currently, common methods for measuring knee angles in clinical and home settings can be categorized as follows, but each has significant limitations. The universal goniometer is currently the most commonly used measuring tool in clinical practice. During measurement, the operator manually aligns the goniometer's fulcrum with the bony landmark on the knee joint (lateral femoral condyle), with both arms aligned with the greater trochanter of the hip and the lateral malleolus of the ankle, respectively, and reads the scale visually. This method relies heavily on the operator's experience and judgment, resulting in significant errors between different measurers. Patients cannot perform the measurement independently, and it is difficult to use in home settings without professional guidance.

[0004] Current mobile apps typically utilize built-in gyroscopes or accelerometers, requiring users to place the edge of their phone against the side of their thigh and calf, and then calculate the angle using sensors. This method necessitates manual positioning of bony landmarks and maintaining the phone parallel to the limb, making the process complex. Patients are prone to significant errors due to incorrect phone placement or limb misalignment. Furthermore, differences in sensor accuracy between different phone models can also affect the results.

[0005] Some methods allow users to take lateral photos of the knee joint and then manually mark three points—hip, knee, and ankle—on the photos, with software automatically calculating the angle. While this is an improvement over a simple goniometer, it still relies on the user accurately identifying bony landmarks. Ordinary patients often struggle to precisely locate structures like the lateral femoral condyle and greater trochanter in photos (especially during postoperative swelling or muscle atrophy), leading to inconsistent measurement results. Furthermore, the manual marking process is cumbersome and cannot be automated.

[0006] Whether it's a traditional goniometer or a mobile app, users or professionals need to manually locate bony landmarks. Patients cannot operate it independently and accurately at home, and therefore cannot accurately measure the knee joint angle. Summary of the Invention

[0007] Based on the above analysis, the present invention aims to provide a method for remote intelligent assessment of knee joint angles using photographs, in order to solve the problem of the difficulty in accurately measuring remote knee joint angles in existing methods.

[0008] On one hand, embodiments of the present invention provide a method for remote intelligent assessment of knee joint angles using photographs, comprising the following steps: Obtain a lateral photograph of the patient's knee joint; a flexible marking strip is affixed to the outer side of the patient's leg; the flexible marking strip has marking points at equal intervals; The trained detection model was used to detect the hip joint, knee joint, ankle joint, and marker points on the flexible marker band in the lateral images of the patient's knee joint. Based on the detected hip joint points, knee joint points, ankle joint points, and marker points, fit the thigh direction vector and the lower leg direction vector; The knee joint angle is calculated based on the thigh direction vector and the calf direction vector.

[0009] Based on further improvements to the above method, the trained detection model is obtained in the following way: Multiple patients' lateral knee joint photographs were collected; flexible marking strips were affixed to the outer side of the patients' legs, and marking points were evenly spaced on the flexible marking strips; a training sample set was constructed based on the lateral knee joint photographs and the corresponding annotation information; the annotation information included the coordinates of the hip joint, knee joint, ankle joint, and marking points; A multi-task detection model is constructed, which is used to predict the coordinates of key points and marker points; The multi-task detection model is trained based on the constructed training samples to obtain a trained detection model.

[0010] Based on further improvements to the above method, the multi-task detection model includes: The backbone encoder is used to extract feature maps from input photos at multiple scales. The joint heatmap branch is used to predict the joint heatmap based on the feature map. The joint offset branch is used to predict the offset field of joints based on the feature map. The marker point heatmap branch is used to predict the heatmap distribution of marker points based on the feature map. The marker offset branch is used to predict the offset field of marker points based on the feature map; The coordinate calculation module is used to calculate the coordinates of joints based on the joint heat map and the offset field of the joints; and to calculate the coordinates based on the marker point heat map and the marker point offset field calculation table.

[0011] Based on the further improvements to the above method, the training loss of the multi-task detection model is calculated using the following formula:

[0012] in, The heatmap loss at the joint points, The heatmap loss represents the value of the marked point. This represents the offset loss of the joints. This represents the offset loss of the marker point.

[0013] Based on the further improvement of the above method, the thermal map loss of the joint is calculated in the following way. :

[0014] in, Indicates the height and width of the heatmap. This represents the heatmap corresponding to the k-th joint predicted by the model. The pixel value of the location, Representing the true heat map The pixel value of the location.

[0015] Based on further improvements to the above method, the thigh direction vector and the lower leg direction vector are fitted using the following method, based on the detected hip joint point, knee joint point, ankle joint point, and marker points on the flexible marker strip: The direction vector of the thigh marker points is calculated based on the marker points on the thigh. The direction vector of the lower leg marker points is calculated based on the marker points on the lower leg. The direction vectors of the thigh anatomical points and the lower leg anatomical points are calculated based on the hip joint, knee joint, and ankle joint points. The thigh direction vector is obtained by weighted fusion of the direction vectors of the thigh marker points and the direction vectors of the thigh anatomical points. The lower leg direction vector is obtained by weighted fusion of the direction vectors of the lower leg marker points and the direction vectors of the lower leg anatomical points.

[0016] Based on a further improvement to the above method, the direction vector of the thigh marker points is calculated based on the marker points on the thigh, including: Three-dimensional markers are obtained by performing three-dimensional reconstruction on the marker points in the thigh area; The direction vector of the thigh marker point is obtained by calculating the main direction based on the three-dimensional marker point.

[0017] Based on further improvements to the above method, three-dimensional reconstruction of the marker points on the thigh area is performed to obtain three-dimensional marker points, including: Construct an objective function based on reprojection error and distance constraints; The objective function is solved based on the marker points on the thigh area to obtain the three-dimensional marker points.

[0018] Based on a further improvement of the above method, the objective function is expressed as:

[0019] in, This indicates the number of marked points on the thigh area. This represents the pixel coordinates of the i-th marker point in the image. Indicates the camera's focal length. Indicates the coordinates of the center point of the photo. Indicates the calibration interval between standard points. This represents common sense or experience. This represents the three-dimensional coordinates of the (i+1)th marker point. Represents the three-dimensional coordinates of the i-th marker point. This indicates that the distance is being calculated.

[0020] Based on the above method, the direction vector of the thigh marker point is calculated using the following method based on the three-dimensional marker point:

[0021]

[0022] in, Represents the three-dimensional coordinates of the i-th marker point. This indicates the number of marked points on the thigh area. Indicates the centroid of the marked points on the thigh. This represents the covariance matrix, with the superscript T indicating transpose. Perform eigenvalue decomposition on the covariance matrix; the eigenvector corresponding to the largest eigenvalue is the direction vector of the thigh marker point.

[0023] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects: 1. Patients only need to use a regular smartphone to take a picture of the knee joint from the side with a flexible marker tape attached, and the angle assessment can be completed automatically. No on-site guidance from a doctor or rehabilitation therapist is required throughout the process, and patients do not need to manually locate bony landmarks. This is especially suitable for patients with limited mobility after surgery, those living in remote areas, or those who cannot frequently visit the hospital for follow-up examinations, significantly reducing the time and economic costs of rehabilitation monitoring.

[0024] 2. This invention utilizes known, equally spaced flexible marker strips as an absolute scale reference. By detecting the position of the marker points, the true physical length corresponding to each pixel in the image can be recovered, and projection distortion caused by camera pitch and yaw angles can be corrected. Compared to pure image methods that rely solely on anatomical key points, this invention can improve the accuracy of knee joint angle measurement.

[0025] 3. Fully automated, deep learning-based model simultaneously detects three key anatomical points (hip, knee, and ankle) and multiple markers on a marker band, requiring no manual user clicks or intervention. The model incorporates anatomical priors (such as the thigh / calf length ratio) and equal-spaced marker constraints during training, exhibiting natural robustness to postoperative swelling, muscle atrophy, and changes in lighting conditions during photography. Its detection success rate surpasses that of general pose estimation algorithms.

[0026] 4. The marking tape is made of flexible material and can be naturally adhered to the outer side of the thigh and calf, bending with the skin during knee flexion and extension without falling off or twisting. Patients only need to apply it according to the diagram for the first use, and it can be reused for subsequent follow-up examinations without the need for repositioning each time. The marking tape is extremely low-cost (it can be printed on medical tape), meeting the hygiene and economic requirements of home rehabilitation.

[0027] 5. Each measurement result (knee joint angle, coordinates of detected key points, distribution of marker points) and photographs are automatically archived, forming continuous time-series data. Doctors can dynamically track the changing trends of the patient's flexion and extension angles through a remote platform, and promptly warn of complications such as joint fibrosis and Cyclops disease.

[0028] 6. This invention only requires a regular smartphone camera and basic computing power, without the need for a depth sensor, gyroscope, or professional goniometer. It is suitable for integration into mobile apps or WeChat mini-programs, facilitating large-scale deployment and application.

[0029] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0030] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. Figure 1 This is a flowchart illustrating a method for remotely and intelligently assessing knee joint angles using photographs, as described in an embodiment of the present invention.

[0031] Figure 2 This is a schematic diagram of the flexible marking strip according to an embodiment of the present invention. Detailed Implementation

[0032] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0033] One specific embodiment of the present invention discloses a method for remote intelligent assessment of knee joint angles using photographs, such as... Figure 1 As shown, it includes the following steps: S1. Obtain a side view photograph of the patient's knee joint; wherein, a flexible marking strip is affixed to the outer side of the patient's leg; the flexible marking strip has marking points at equal intervals; S2. Based on the trained detection model, detect the hip joint, knee joint, ankle joint, and marker points on the flexible marker band in the side view photograph of the patient's knee joint; S3. Fit thigh and calf direction vectors based on detected hip, knee, and ankle joint points and marker points; S4. Calculate the knee joint angle based on the thigh direction vector and the calf direction vector.

[0034] Compared with existing technologies, the method for remote intelligent assessment of knee joint angles via photographs provided in this embodiment acquires a side view photograph of the patient's knee joint with flexible marker tape affixed. A trained detection model is used to detect the joint points and marker points within the photograph. Based on the detected hip joint points, knee joint points, ankle joint points, and marker points, thigh direction vectors and calf direction vectors are fitted. The knee joint angle is then calculated. This method achieves rapid and accurate remote knee joint angle assessment without the need for on-site guidance from doctors or rehabilitation therapists, or for patients to manually locate bony landmarks. It only requires a single side view photograph of the knee joint with flexible marker tape taken with a regular smartphone.

[0035] It should be noted that the method for remotely and intelligently assessing the knee joint angle through photos provided in this embodiment is an information processing method in which all steps are performed by a computer or other device. The prediction result obtained is used as intermediate information to assist in the judgment and is not the final diagnosis result.

[0036] In practice, the flexible marking tape is made of flexible material, such as medical tape, with marking dots printed on it at equal intervals.

[0037] During implementation, to ensure accurate orientation fitting, flexible marker strips can be attached to the outer sides of the thigh and calf, respectively. Figure 2 As shown in the diagram, the flexible marking strip adheres closely to the skin, naturally attaching along the outer side of the thigh and calf, and bends with the skin during knee flexion and extension without falling off or twisting. Patients only need to apply the strip as shown in the diagram for initial use; it can be reused for subsequent checkups without needing to be repositioned each time. Each marking strip has at least three marking points.

[0038] During implementation, to distinguish between the flexible marking strips on the thigh and the calf, the marking points can be set to different colors.

[0039] After obtaining a side view photo of the patient's knee joint, the hip joint, knee joint, ankle joint, and marker points on the flexible marker band in the photo are detected based on the trained detection model.

[0040] The lateral view of the patient's knee joint is a photograph taken when the patient's knee joint is in its maximum flexion position.

[0041] Specifically, the trained detection model is obtained using the following method: S01. Collect lateral photos of the knee joints of multiple patients; flexible marking strips are affixed to the thigh and lower leg of the patients, and marking points are evenly spaced on the flexible marking strips; construct a training sample set based on the lateral photos of the knee joints and the corresponding annotation information; the annotation information includes the coordinates of the hip joint, knee joint, ankle joint and the marking points; S02. Construct a multi-task detection model, which is used to predict the coordinates of key points and marker points; S03. The multi-task detection model is trained based on the constructed training samples to obtain a trained detection model.

[0042] During implementation, multiple lateral knee photographs were collected from patients, with the aforementioned flexible marking strips affixed to the lateral side of the patient's leg. The photographs maintained a uniform size. The photographs should clearly show the hip, knee, and ankle joints, as well as multiple marker points on the two marking strips.

[0043] The collected photos were labeled with information including key anatomical points such as the patient's hip, knee, and ankle joints, as well as the coordinates of the marked points on the thigh and lower leg.

[0044] A training sample set was constructed based on lateral images of the knee joint and corresponding annotation information to train a multi-task detection model. The multi-tasks included joint detection and marker detection.

[0045] Specifically, the constructed multi-task detection model includes: The backbone encoder is used to extract feature maps from input photos at multiple scales. The joint heatmap branch is used to predict the joint heatmap based on the feature map. The joint offset branch is used to predict the offset field of joints based on the feature map. The marker point heatmap branch is used to predict the heatmap distribution of marker points based on the feature map. The marker offset branch is used to predict the offset field of marker points based on the feature map; The coordinate calculation module is used to calculate the coordinates of joints based on the joint heat map and the offset field of the joints; and to calculate the coordinates based on the marker point heat map and the marker point offset field calculation table.

[0046] In implementation, the backbone encoder can use HRNet-W32 or High-Resolution Net, and perform multi-scale feature extraction and feature fusion on the input photo through multiple downsampling branches to obtain feature maps.

[0047] In practice, the size of the feature map can be C×H / 4×W / 4, where C represents the number of channels and (H,W) represents the height and width of the input image.

[0048] To achieve more robust and accurate detection of occlusion and blur, this invention uses a heatmap + offset approach for image detection of joints and markers. Therefore, the multi-task detection model includes a joint heatmap branch, a joint offset branch, a marker heatmap branch, and a marker offset branch.

[0049] In practice, the joint heatmap branch can adopt a convolutional layer + sigmoid activation function structure to output 3 heatmaps, each corresponding to a joint (hip, knee, ankle), and the size of the heatmap is H / 4 × W / 4 of the original image.

[0050] The joint offset branch can use a convolutional layer + Tanh intercept function structure to output 6 offset field maps, which represent the x and y offsets of each joint.

[0051] The heatmap branch also uses a convolutional layer + sigmoid activation function structure, outputting M heatmaps, each corresponding to one marker point. The size of the heatmap is H / 4 × W / 4 of the original image. M represents the total number of marker points.

[0052] The marker offset branch also adopts a convolutional layer + Tanh intercept function structure, outputting 2M offset field maps, which represent the x and y offsets of each joint point.

[0053] Since the resolution of the heatmap branch and the offset branch is different from that of the photo, the coordinates of the joints and markers in the photo are calculated by the coordinate calculation module based on the joint heatmap and the offset field of the joints.

[0054] During implementation, for each joint point, the location of the maximum value is found on the corresponding heatmap. (Integer coordinates, resolution H / 4).

[0055] Extract the offset vector at that position from the offset branch. .in, This represents the x-direction offset field predicted by the model for that joint. The pixel value of the location, This represents the y-direction offset field predicted by the model at this joint point. The pixel value of the location.

[0056] Sub-pixel coordinates of the joints (heatmap resolution): .

[0057] Map back to original image resolution: multiply by the following sampling factor (Usually s=4) to obtain the coordinates of the joints in the photo.

[0058] For each marker point, the same steps are used to calculate its coordinates in the original photo based on the corresponding heatmap and offset.

[0059] Specifically, the training loss of the multi-task detection model is calculated using the following formula:

[0060] in, The heatmap loss at the joint points, The heatmap loss represents the value of the marked point. This represents the offset loss of the joints. This represents the offset loss of the marker point.

[0061] During implementation, the thermal map losses at the joints can be calculated using the following formula. :

[0062] in, Indicates the height and width of the heatmap. This represents the heatmap corresponding to the k-th joint predicted by the model. The pixel value of the location, Representing the true heat map The pixel value of the location.

[0063] The heatmap loss of the marked points is calculated in the same way. :

[0064] Where M represents the number of markers, Indicates the height and width of the heatmap. This represents the heatmap corresponding to the m-th marker predicted by the model. The pixel value of the location, This represents the actual heat map corresponding to the m-th marker point. The pixel value of the location.

[0065] In practice, the actual heatmap can be obtained based on a Gaussian distribution using the coordinates of the marked joints:

[0066] in, This represents the actual heatmap generated for the k-th joint. This represents the coordinate value of the k-th joint after downsampling from the actual coordinates to the heatmap resolution. This represents the standard deviation of the Gaussian kernel; in practice, it can be taken as 2 pixels.

[0067] In the same way, the actual heat map corresponding to the marked point is obtained based on the Gaussian distribution according to the coordinates of the marked point.

[0068] In detection models, the heatmap output by the network is typically smaller than the original image (e.g., original image 512×512, heatmap 128×128). Each pixel in the heatmap corresponds to a 4×4 region in the original image (downsampling factor s=4). Even if the heatmap accurately predicts the integer coordinates (x_int, y_int) of the peak, mapping these coordinates back to the original image can result in a maximum error of s / 2 = 2 pixels (because the true point could fall anywhere within that 4×4 region). If integer coordinates are used directly, the angle calculation error can reach 1~2°. Therefore, an offset branch is used to have the network predict an additional small amount (Δx, Δy) added to the integer coordinates, thereby recovering the sub-pixel level of accurate location and compensating for the quantization error caused by downsampling.

[0069] Specifically, the offset loss of the joints is calculated using the following formula. :

[0070]

[0071]

[0072] in, This represents the coordinate value of the k-th joint after downsampling from the actual coordinates to the heatmap resolution. This represents the coordinates of the k-th joint point obtained from the heatmap peak and offset predicted by the model. This represents the peak coordinate of the heatmap predicted by the model for the k-th joint. This represents the x-direction offset field of the k-th joint predicted by the model. The pixel value of the location, This represents the y-direction offset field of the model-predicted k-th joint. The pixel value of the location, This represents smoothed L1 loss.

[0073] The offset loss of the marker points is calculated in the same way. :

[0074]

[0075] Where M represents the number of markers, This represents the coordinate value of the m-th marker point after downsampling from the actual coordinates to the heatmap resolution. This represents the coordinates of the m-th marker point obtained based on the heatmap peak and offset predicted by the model. This represents the peak coordinate of the heatmap of the m-th marker predicted by the model. This represents the x-direction offset field of the m-th marker predicted by the model. The pixel value of the location, In the y-direction offset field of the m-th marker predicted by the model The pixel value of the location.

[0076] Based on the calculated total loss Backpropagation updates the parameters of the detection model. Training stops when the model converges, resulting in a well-trained detection model. Inputting a side-view photograph of the patient's knee joint into the trained model yields the coordinates of each joint point and marker point.

[0077] Then, based on the detected hip joint, knee joint, and ankle joint points, as well as the marked points, thigh and lower leg direction vectors are fitted. Specifically, the thigh and lower leg direction vectors are fitted using the following method: S31. Calculate the direction vector of the thigh marker points based on the marker points in the thigh area; calculate the direction vector of the calf marker points based on the marker points in the calf area. S32. Calculate the direction vectors of the thigh anatomical points and the lower leg anatomical points based on the hip joint, knee joint, and ankle joint points. S33. The thigh direction vector is obtained by weighted fusion of the thigh marker direction vector and the thigh anatomical point direction vector; the lower leg direction vector is obtained by weighted fusion of the lower leg marker direction vector and the lower leg anatomical point direction vector.

[0078] Specifically, the direction vector of the thigh marker points is calculated based on the marker points on the thigh, including: S311. Perform three-dimensional reconstruction on the marker points in the thigh area to obtain three-dimensional marker points; S312. Calculate the main direction based on the three-dimensional marker points to obtain the direction vector of the thigh marker points.

[0079] Since the actual distance between adjacent marker points is known, the three-dimensional coordinates of each marker point in the camera coordinate system can be recovered using a chain-reconstruction method. The marker points are then reconstructed in three dimensions, and the direction vectors of the thigh and calf are calculated based on the reconstructed three-dimensional points. This invention uses the "chain-reconstruction local rigidity assumption": the true distance between adjacent marker points is known as d (e.g., 2 cm), and it is assumed that within a small local segment, the skin surface where the marker points are located is approximately planar or the length error caused by curvature is negligible.

[0080] Specifically, the following method was used to perform 3D reconstruction of the marker points on the thigh area to obtain the 3D marker points: Construct an objective function based on reprojection error and distance constraints; The objective function is solved based on the marker points on the thigh area to obtain the three-dimensional marker points.

[0081] Specifically, the objective function is expressed as:

[0082] in, This indicates the number of marked points on the thigh area. This represents the pixel coordinates of the i-th marker point in the image. Indicates the camera's focal length. Indicates the coordinates of the center point of the photo. Indicates the calibration interval between standard points. This represents common sense or experience. This represents the three-dimensional coordinates of the (i+1)th marker point. Represents the three-dimensional coordinates of the i-th marker point. This indicates that the distance is being calculated.

[0083] During implementation, focal length (in pixels), image center coordinates It can be approximated from the focal length, sensor size, and image size in the phone's EXIF ​​data. (Common sense / experience) For example, a value of 0.1 can be used to balance the dimensional differences between the two errors.

[0084] Any pixel in the photo corresponding spatial points in the camera coordinate system satisfy: .

[0085] The first term of the objective function guarantees that the reconstructed 3D point projection matches the image detection position; the second term forces the true distance between adjacent marker points to be equal to the known spacing, thereby restoring the absolute scale.

[0086] During implementation, the coordinates of the marker points on the detected thigh area can be used to solve the problem through nonlinear optimization (such as Levenberg-Marquardt). The initial values ​​can be assumed to be the same depth for all points (such as Z=1 meter), and then iterative optimization can be performed.

[0087] Through iterative optimization, the 3D distance between adjacent marker points is made equal to d, and the reprojection error of all marker points is minimized. Finally, the 3D coordinates of each marker point relative to the camera's optical center are obtained. ,in For depth.

[0088] Then, by fitting a straight line in space based on the three-dimensional marker points, the direction vector of the thigh is obtained.

[0089] During implementation, take all the marked points on the thigh marking band (or, to avoid the influence of bending near the knee joint, only take 3-5 points near the hip), and assign the three-dimensional coordinates of these points (arranged in order from hip to knee) as follows: .

[0090] Specifically, the thigh direction vector is calculated based on the 3D marker points using the following method:

[0091]

[0092] in, Represents the three-dimensional coordinates of the i-th marker point. This indicates the number of marked points on the thigh area. Indicates the centroid of the marked points on the thigh. This represents the covariance matrix, with the superscript T indicating transpose. Perform eigenvalue decomposition on the covariance matrix; the eigenvector corresponding to the largest eigenvalue is the direction vector of the thigh marker point. (From hip to knee). The eigenvector corresponding to the largest eigenvalue represents the extension direction of the points on the marker band. Since the marker band is pasted along the direction of the femur, it is approximately equal to the direction of the femur in space.

[0093] Following the same steps S311~S312, the marker points on the lower leg are reconstructed in three dimensions to obtain three-dimensional marker points; the principal direction is calculated based on the three-dimensional marker points to obtain the direction vector of the lower leg marker points. (From knee to ankle).

[0094] In practice, the direction vectors of the thigh marker point and the calf marker point can be directly used as the thigh direction vector and the calf direction vector.

[0095] The knee joint angle is calculated based on the thigh direction vector and the calf direction vector:

[0096] in, Represents the vector dot product. This represents the inverse cosine function.

[0097] When fully extended, the thigh and calf form a straight line, and the knee angle is [value missing]. The angle increases when bending.

[0098] To improve biomechanical relevance, the pixel coordinates of hip, knee, and ankle joints detected by deep learning can be used to roughly reconstruct their 3D coordinates using a camera model (assuming the depth is similar to adjacent marker points). Then, direction vectors based on anatomical points are obtained. The thigh direction vector is obtained by weighted fusion of the thigh marker direction vector and the thigh anatomical point direction vector; similarly, the lower leg direction vector is obtained by weighted fusion of the lower leg marker direction vector and the lower leg anatomical point direction vector.

[0099] During implementation, the three-dimensional coordinates of the hip, knee, and ankle are first obtained using a camera model and depth approximation of marker points. (For example, the knee depth is obtained by taking the average depth of the thigh markers and then calculating it by projection).

[0100] Thigh Anatomical Point Direction Vector Path vector of anatomical point of lower leg , This indicates taking the modulus.

[0101] Weighted fusion yields the thigh direction vector ; Lower leg direction vector .

[0102] Then, re-normalize and substitute the values ​​into the knee joint angle calculation formula to calculate the angle. During implementation, the weighting coefficients... 0.7 is acceptable (more trustworthy marker band).

[0103] One specific embodiment of the present invention discloses a system for remote intelligent assessment of knee joint angles via photographs, comprising: The photo acquisition module is used to acquire a side view photo of the patient's knee joint; a flexible marking strip is attached to the outside of the patient's leg; the flexible marking strip has marking points at equal intervals; The detection module is used to detect the hip joint, knee joint, ankle joint, and marker points on the flexible marker band in the lateral photograph of the patient's knee joint based on a trained detection model. The orientation fitting module is used to fit thigh and lower leg orientation vectors based on detected hip, knee, and ankle joint points and marker points. The angle calculation module is used to calculate the knee joint angle based on the thigh direction vector and the calf direction vector.

[0104] The above-described method and system embodiments are based on the same principles, and their related aspects can be referenced from each other to achieve the same technical effects. For specific implementation processes, please refer to the foregoing embodiments, which will not be repeated here.

[0105] Electronic device example: One specific embodiment of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the method in the method embodiment for remote intelligent assessment of knee joint angles via photographs.

[0106] Examples of readable storage media: One specific implementation of this application discloses a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method in the method embodiment for remote intelligent assessment of knee joint angles via photographs.

[0107] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0108] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for remote intelligent assessment of knee joint angles using photographs, characterized in that, Includes the following steps: Obtain a lateral photograph of the patient's knee joint; a flexible marking strip is affixed to the outer side of the patient's leg; the flexible marking strip has marking points at equal intervals; The trained detection model was used to detect the hip joint, knee joint, ankle joint, and marker points on the flexible marker band in the lateral images of the patient's knee joint. Based on the detected hip joint points, knee joint points, ankle joint points, and marker points, fit the thigh direction vector and the lower leg direction vector; The knee joint angle is calculated based on the thigh direction vector and the calf direction vector.

2. The method for remote intelligent assessment of knee joint angle via photographs according to claim 1, characterized in that, The trained detection model is obtained using the following method: Multiple patients' lateral knee joint photographs were collected; flexible marking strips were affixed to the outer side of the patients' legs, and marking points were evenly spaced on the flexible marking strips; a training sample set was constructed based on the lateral knee joint photographs and the corresponding annotation information; the annotation information included the coordinates of the hip joint, knee joint, ankle joint, and marking points; A multi-task detection model is constructed, which is used to predict the coordinates of key points and marker points; The multi-task detection model is trained based on the constructed training samples to obtain a trained detection model.

3. The method for remote intelligent assessment of knee joint angle via photographs according to claim 2, characterized in that, The multi-task detection model includes: The backbone encoder is used to extract feature maps from input photos at multiple scales. The joint heatmap branch is used to predict the joint heatmap based on the feature map. The joint offset branch is used to predict the offset field of joints based on the feature map. The marker point heatmap branch is used to predict the heatmap distribution of marker points based on the feature map. The marker offset branch is used to predict the offset field of marker points based on the feature map; The coordinate calculation module is used to calculate the coordinates of joints based on the joint heat map and the offset field of the joints; and to calculate the coordinates based on the marker point heat map and the marker point offset field calculation table.

4. The method for remote intelligent assessment of knee joint angle via photographs according to claim 2, characterized in that, The training loss of the multi-task detection model is calculated using the following formula: in, The heatmap loss at the joint points, The heatmap loss represents the value of the marked point. This represents the offset loss of the joints. This represents the offset loss of the marker point.

5. The method for remote intelligent assessment of knee joint angle via photographs according to claim 4, characterized in that, The thermal map loss at the joints is calculated using the following method. : in, Indicates the height and width of the heatmap. This represents the heatmap corresponding to the k-th joint predicted by the model. The pixel value of the location, Representing the true heat map The pixel value of the location.

6. The method for remote intelligent assessment of knee joint angle via photographs according to claim 1, characterized in that, Based on the detected hip joint, knee joint, ankle joint, and marker points on the flexible marker strip, the thigh direction vector and lower leg direction vector are fitted in the following manner: The direction vector of the thigh marker points is calculated based on the marker points on the thigh. The direction vector of the lower leg marker points is calculated based on the marker points on the lower leg. The direction vectors of the thigh anatomical points and the lower leg anatomical points are calculated based on the hip joint, knee joint, and ankle joint points. The thigh direction vector is obtained by weighted fusion of the direction vectors of the thigh marker points and the direction vectors of the thigh anatomical points. The lower leg direction vector is obtained by weighted fusion of the direction vectors of the lower leg marker points and the direction vectors of the lower leg anatomical points.

7. The method for remote intelligent assessment of knee joint angle via photographs according to claim 6, characterized in that, The direction vector of the thigh marker points is calculated based on the marker points on the thigh, including: Three-dimensional markers are obtained by performing three-dimensional reconstruction on the marker points in the thigh area; The direction vector of the thigh marker point is obtained by calculating the main direction based on the three-dimensional marker point.

8. The method for remote intelligent assessment of knee joint angle via photographs according to claim 7, characterized in that, The 3D markers on the thigh area are reconstructed to obtain 3D markers, including: Construct an objective function based on reprojection error and distance constraints; The objective function is solved based on the marker points on the thigh area to obtain the three-dimensional marker points.

9. The method for remote intelligent assessment of knee joint angle via photographs according to claim 8, characterized in that, The objective function is expressed as: in, This indicates the number of marked points on the thigh area. This represents the pixel coordinates of the i-th marker point in the image. Indicates the camera's focal length. Indicates the coordinates of the center point of the photo. Indicates the calibration interval between standard points. This represents common sense or experience. This represents the three-dimensional coordinates of the (i+1)th marker point. Represents the three-dimensional coordinates of the i-th marker point. This indicates that the distance is being calculated.

10. The method for remote intelligent assessment of knee joint angle via photographs according to claim 3, characterized in that, The direction vector of the thigh marker point is calculated based on the three-dimensional marker point using the following method: in, Represents the three-dimensional coordinates of the i-th marker point. This indicates the number of marked points on the thigh area. Indicates the centroid of the marked points on the thigh. This represents the covariance matrix, with the superscript T indicating transpose. Perform eigenvalue decomposition on the covariance matrix; the eigenvector corresponding to the largest eigenvalue is the direction vector of the thigh marker point.

Images