Joint motion range measuring system based on deep learning and machine vision

Abstract

The invention discloses a joint motion range measuring system based on deep learning and machine vision, and relates to the technical field of rehabilitation medicine, and the system comprises a depth camera module which is used for capturing RGB and depth images of human body motions; the image processing module is used for preprocessing the captured image, including noise reduction and zooming; the posture estimation module is used for carrying out human body posture estimation on the preprocessed image by adopting an OpenPose deep learning network model and identifying space coordinates of human body joint points; human body joint points can be accurately tracked and positioned by using an OpenPose deep learning network model and an RGBD depth camera; compared with a traditional protractor method and a visual estimation method, the system can reduce errors caused by subjective judgment of an operator, and a more accurate joint motion range measurement result is provided; in addition, the system can continuously optimize the algorithm by learning a large number of joint activity samples, and the accuracy and reliability of measurement are further improved.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation medicine, and in particular to a joint range of motion measurement system based on deep learning and machine vision. Background Art

[0002] The assessment of joint range of motion (ROM) is one of the necessary steps before therapists carry out rehabilitation treatment. It can be used to diagnose and monitor various joint-related diseases, such as osteoarthritis and joint dislocation. In the fields of rehabilitation medicine, sports science and clinical diagnosis, accurate measurement of human joint range of motion (ROM) is crucial for assessing individual health status and formulating treatment plans. Although traditional ROM assessment methods, such as the protractor method and visual estimation method, can provide the required data to a certain extent, they have the problems of complex operation, strong subjectivity and limited accuracy.

[0003] To this end, the present invention proposes a joint range of motion measurement system based on deep learning and machine vision. Based on the OpenPose deep learning network model and deep camera three-dimensional (3D) capture technology, it provides an efficient, accurate and objective new method for human ROM assessment, which helps to improve the effect of rehabilitation treatment, optimize sports training plans, and provide more reliable data support for clinical diagnosis. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a joint range of motion measurement system based on deep learning and machine vision, which solves the problems of the prior art mentioned in the background technology, such as complex operation, strong subjectivity and limited accuracy.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a joint range of motion measurement system based on deep learning and machine vision, the system comprising:

[0006] Depth camera module: used to capture RGB and depth images of human motion;

[0007] Image processing module: pre-process the captured images, including noise reduction and scaling;

[0008] Posture estimation module: uses the OpenPose deep learning network model to estimate the human posture of the preprocessed images and identify the spatial coordinates of the human joints;

[0009] Action recognition module: By comparing the changes in the 2D Euclidean distance of the joint points, and then comparing the changes in the 2D coordinates of the joint points in consecutive frames, the complete action of the joint activity is identified;

[0010] Data analysis module: calculates the 3D coordinates of the joint points and calculates the joint range of motion based on these coordinates;

[0011] User interface module: includes a graphical user interface (GUI) for displaying real-time video streams, measurement results, and allowing users to configure measurement parameters, and supports a history viewing function for viewing and analyzing past measurement data;

[0012] Data management module: stores and manages measurement data, and supports data export function to export measurement results into CSV or Excel format.

[0013] Preferably, the camera types of the depth camera module are RGB camera and infrared camera, and the capture rate of the RGB camera and infrared camera is 30 frames per second; the image types captured synchronously by the depth camera module are RGB image and depth image, and the field of view range is sufficient to cover the main area of ​​human joint movement.

[0014] Preferably, the image processing module reduces image noise and improves the accuracy of subsequent processing through a real-time image denoising algorithm, and the image processing module also includes an image scaling unit for adjusting the image resolution to adapt to the input requirements of the network model, wherein the image denoising algorithm uses a bilateral filter, which can maintain edge information while denoising, and its algorithm is as follows:

[0015]

[0016] Among them, I filtered (p) is the pixel value after filtering; is a spatial Gaussian function that takes into account the spatial proximity between pixels; is the grayscale Gaussian function, which is used to consider the similarity of pixel values; W p is the normalization factor to ensure that the sum of pixel values ​​is 1; S is the input pixel domain, p is the center pixel, and q is the input pixel.

[0017] Preferably, the posture estimation module is based on the OpenPose network model of deep learning to identify human joints from the image. The input of the posture estimation module model is a preprocessed RGB image, and the output is the 2D coordinates of the joints, and the number of joints is 18 standard human joints.

[0018] Preferably, the OpenPose model includes:

[0019] Input layer: receives input image;

[0020] Convolutional layer: A series of convolutional layers used to extract image features, including multiple convolution operations and non-linear activation functions;

[0021] Pooling layer: pooling operation used to reduce feature dimensions and enhance feature invariance;

[0022] Fully connected layer: maps the learned features to joint positions and key points;

[0023] Regression layer: predicts the location and visibility of key points;

[0024] Posture estimation layer: integrates the outputs of the above layers to estimate the posture of the human body;

[0025] Output layer: Output the 2D coordinates of human joints.

[0026] Preferably, the RGB image and the depth image are aligned in time and space so that each RGB pixel can accurately correspond to its depth value, the color information of the RGB image and the depth information of the depth image are fused, and the depth information is used to convert the two-dimensional image coordinates into three-dimensional space coordinates, calculate the position of the joint point in the three-dimensional space, and calculate the joint activity, such as angle or range of motion, according to the 3D coordinates;

[0027] Preferably, in the 2D to 3D coordinate conversion, the depth image provides the depth value d of each pixel point (u, v) on the Z axis (depth axis). Combined with the 2D coordinates (u, v) of the RGB image, the 3D coordinates (X, Y, Z) of the image in the camera coordinate system can be calculated:

[0028]

[0029] Z=d

[0030] Among them, d is the depth value of the pixel, f x and f y is the focal length of the camera in the x-axis and y-axis directions, c x and c y The camera's principal point coordinates are converted to three-dimensional coordinates through the two-dimensional coordinates (u, v) and the corresponding depth value d, and the joint points are located in the three-dimensional space. The joint angle is calculated using the coordinates of the two joint points in the three-dimensional space using the distance formula or the angle between the vectors:

[0031]

[0032] in, and is the vector representing the position of the two joint points, and θ is the angle between the vectors.

[0033] Preferably, the posture estimation module is also used for model training and testing, and the posture estimation module uses a lightweight network MobileNet or HRNet as a feature extractor to improve computational efficiency with only a small loss of accuracy; the features include: edges, color distribution, texture information, shape contours, and spatial relationships in the image;

[0034] Preferably, the pose estimation module uses the public COCO dataset for model training, adjusts the learning rate, number of cycles, training step size, and gradient optimizer parameters, and trains and verifies the OpenPose network model;

[0035] The initial value of the learning rate is set between 0.001 and 0.0001, and a learning rate decay strategy is adopted; the total number of cycles is set to 50 to 100 cycles, and an early stopping rule is implemented to avoid overfitting; the training step size is set to 32 or 64 to balance memory usage and training efficiency;

[0036] The gradient optimizer selects Adam or SGD and sets β for Adam 1 =0.9,β 2 =0.999,∈=10 -8 ; Set momentum parameters for SGD; the model training evaluation indicators use accuracy and mAP to measure model performance.

[0037] Preferably, the test data set for the OpenPose network model test is a ROM assessment video shot by project members. The average error of the joint positions predicted by the OpenPose network model is calculated to evaluate the inference accuracy and the generalization ability of the model on unseen data, ensuring that the test accuracy and inference speed of the lightweight OpenPose are close to those of the original OpenPose network.

[0038] Preferably, in joint identification and activity calculation, the corresponding type of joint activity to be measured is first selected in the GUI interface, and the openpose network model only captures specific human joint points according to the selected joints to reduce the amount of calculation and improve the efficiency of joint identification and activity calculation; wherein, the posture estimation module only selects the minimum joint point set related to the measurement of specific joint activity in the key point selection;

[0039] When measuring elbow range of motion, only three joint points of the wrist, elbow, and shoulder are processed;

[0040] When measuring knee range of motion, only three joint points at the ankle, knee, and hip are processed;

[0041] When measuring the range of motion of the neck joint, only the critical points of the neck and head are processed;

[0042] When measuring hip range of motion, only the joint points directly involved in hip movement are processed.

[0043] The present invention provides a joint range of motion measurement system based on deep learning and machine vision. It has the following beneficial effects:

[0044] (1) Improve measurement accuracy: By using the OpenPose deep learning network model and RGBD depth camera, the present invention can accurately track and locate human joints. Compared with the traditional protractor method and visual estimation method, the system can reduce the error caused by the operator's subjective judgment and provide more accurate joint range of motion measurement results. In addition, the system can continuously optimize the algorithm by learning a large number of joint activity samples, thereby further improving the accuracy and reliability of the measurement.

[0045] (2) Simplified operation process: The system is designed with an intuitive user interface, allowing operators to complete measurement tasks by simply clicking and dragging; the automated measurement process reduces the need for manual data input and reduces the complexity of operation;

[0046] (3) Improve measurement efficiency: Using deep learning and machine vision technology, the system can process video stream data in real time and achieve simultaneous tracking and measurement of multiple joints. This means that more patients’ joint range of motion can be measured in the same amount of time, significantly improving work efficiency. In addition, the system can automatically record and store measurement data, reducing the time and labor intensity of subsequent data processing.

[0047] Thereby, the problems of complex operation, strong subjectivity and limited accuracy in the prior art are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a schematic diagram of the module of the present invention.

[0049] Figure 2 It is a schematic diagram of the technical route of the present invention.

[0050] Figure 3 This is a schematic diagram of posture detection based on OpenPose and a depth camera in the present invention.

[0051] Figure 4 This is a partial UI interface diagram of the software system to be developed by the present invention.

[0052] Figure 5 Schematic diagram of 18 joint points detected by the OpenPose network of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] See also Figure 1-2 The embodiment of the present invention provides a technical solution: a joint range of motion measurement system based on deep learning and machine vision, the system comprising:

[0055] Depth camera module: used to capture RGB and depth images of human motion;

[0056] Image processing module: pre-process the captured images, including noise reduction and scaling;

[0057] Posture estimation module: uses the OpenPose deep learning network model to estimate the human posture of the preprocessed image and identify the spatial coordinates of the human joints; Figure 3 As shown;

[0058] Action recognition module: By comparing the changes in the 2D Euclidean distance of the joint points, and then comparing the changes in the 2D coordinates of the joint points in consecutive frames, the complete action of the joint activity is identified;

[0059] Data analysis module: calculates the 3D coordinates of the joint points and calculates the joint range of motion based on these coordinates;

[0060] User interface module: includes a graphical user interface (GUI) for displaying real-time video streams, measurement results, and allowing users to configure measurement parameters, and supports a history viewing function for viewing and analyzing past measurement data, such as Figure 4 As shown;

[0061] Data management module: stores and manages measurement data, and supports data export function to export measurement results into CSV or Excel format.

[0062] The camera types of the depth camera module are RGB camera and infrared camera, and the capture rate of the RGB camera and infrared camera is 30 frames per second; the image types captured synchronously by the depth camera module are RGB image and depth image, and the field of view range is sufficient to cover the main area of ​​human joint movement.

[0063] The image processing module reduces image noise and improves the accuracy of subsequent processing through a real-time image denoising algorithm, and the image processing module also includes an image scaling unit for adjusting the image resolution to adapt to the input requirements of the network model, wherein the image denoising algorithm uses a bilateral filter, which can maintain edge information while denoising, and its algorithm is as follows:

[0064]

[0065] Among them, I filtered (p) is the pixel value after filtering; is a spatial Gaussian function that takes into account the spatial proximity between pixels; is the grayscale Gaussian function, which is used to consider the similarity of pixel values; W p is the normalization factor to ensure that the sum of pixel values ​​is 1; S is the input pixel domain, p is the center pixel, and q is the input pixel.

[0066] The posture estimation module is based on the OpenPose network model of deep learning to identify human joints from images. The input of the posture estimation module model is a preprocessed RGB image, and the output is the 2D coordinates of the joints, and the number of joints is 18 standard human joints.

[0067] The OpenPose model includes:

[0068] Input layer: receives input image;

[0069] Convolutional layer: A series of convolutional layers used to extract image features, including multiple convolution operations and non-linear activation functions;

[0070] Pooling layer: pooling operation used to reduce feature dimensions and enhance feature invariance;

[0071] Fully connected layer: maps the learned features to joint positions and key points;

[0072] Regression layer: predicts the location and visibility of key points;

[0073] Pose estimation layer: integrates the outputs of the above layers to estimate the posture of the human body;

[0074] Output layer: Output the 2D coordinates of human joints.

[0075] The RGB image and the depth image are aligned in time and space so that each RGB pixel can accurately correspond to its depth value, the color information of the RGB image and the depth information of the depth image are fused, and the depth information is used to convert the two-dimensional image coordinates into three-dimensional space coordinates, calculate the position of the joint point in the three-dimensional space, and calculate the joint activity, such as angle or range of motion, according to the 3D coordinates;

[0076] In the conversion from 2D to 3D coordinates, the depth image provides the depth value d of each pixel (u, v) on the Z axis (depth axis). Combined with the 2D coordinates (u, v) of the RGB image, the 3D coordinates (X, Y, Z) of the image in the camera coordinate system can be calculated:

[0077]

[0078] Z=d

[0079] Among them, d is the depth value of the pixel, f x and f yis the focal length of the camera in the x-axis and y-axis directions, c x and c y The camera's principal point coordinates are converted to three-dimensional coordinates through the two-dimensional coordinates (u, v) and the corresponding depth value d, and the joint points are located in the three-dimensional space. The joint angle is calculated using the coordinates of the two joint points in the three-dimensional space using the distance formula or the angle between the vectors:

[0080]

[0081] in, and is the vector representing the position of the two joint points, and θ is the angle between the vectors.

[0082] The posture estimation module is also used for model training and testing. The posture estimation module uses a lightweight network MobileNet or HRNet as a feature extractor to improve computational efficiency with only a small loss of accuracy; the features include: edges, color distribution, texture information, shape contours, and spatial relationships in the image;

[0083] The pose estimation module uses the public COCO dataset for model training, adjusts the learning rate, number of cycles, training step size, and gradient optimizer parameters, and trains and verifies the OpenPose network model;

[0084] The initial value of the learning rate is set between 0.001 and 0.0001, and a learning rate decay strategy is adopted; the total number of cycles is set to 50 to 100 cycles, and an early stopping rule is implemented to avoid overfitting; the training step size is set to 32 or 64 to balance memory usage and training efficiency;

[0085] The gradient optimizer selects Adam or SGD and sets β for Adam 1 =0.9,β 2 =0.999,∈=10 -8 ; Set momentum parameters for SGD; the model training evaluation indicators use accuracy and mAP to measure model performance.

[0086] The test dataset for the OpenPose network model test is the ROM assessment videos shot by project members. The average error of the joint positions predicted by the OpenPose network model is calculated to evaluate the inference accuracy and the generalization ability of the model on unseen data, ensuring that the test accuracy and inference speed of the lightweight OpenPose are close to those of the original OpenPose network.

[0087] The posture estimation module selects only the minimum set of joint points related to the specific joint activity measurement in the key point selection, such as Figure 5 As shown:

[0088] When measuring elbow range of motion, only three joint points of the wrist, elbow, and shoulder are processed;

[0089] When measuring knee range of motion, only three joint points at the ankle, knee, and hip are processed;

[0090] When measuring the range of motion of the neck joint, only the critical points of the neck and head are processed;

[0091] When measuring hip range of motion, only the joint points directly involved in hip movement are processed.

[0092] In this embodiment, the complete action counting strategy of the elbow joint range of motion is taken as an example, as shown below:

[0093] By identifying the flexed state of the joint as the initial state; when measuring the range of motion of the elbow joint, the elbow from the flexed state to the extension limit is a complete set of joint range of motion movements, and each set of movements starts with the elbow flexion movement; the following takes the elbow ROM assessment as an example: since each set of joint range of motion movements is completed, the spatial distance between the wrist point and the shoulder point presents a periodic fluctuation similar to a sine function; therefore, it is only necessary to compare the spatial distance of the two key points in the current video frame with the spatial distance of the previous frame. If it is smaller than the spatial distance of the previous frame, it means that the wrist begins to return (i.e., elbow flexion), and a set of measurement movements has been completed, triggering the calculation of joint range of motion; next, if the spatial distance between the two key points of the current frame is larger than the spatial distance of the previous frame, it means that the wrist has returned to the initial point position, and the next set of movements begins; this cycle is repeated to calculate the average value of multiple sets of movements and use it as the final joint range of motion assessment value.

[0094] The above shows and describes the basic principles and main features of the present invention and the advantages of the present invention. It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and the present invention can be implemented in other specific forms without departing from the spirit or basic features of the present invention. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved.

[0095] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A joint range of motion measurement system based on deep learning and machine vision, characterized by: The system includes: Depth camera module: used to capture RGB and depth images of human motion; Image processing module: pre-process the captured images, including noise reduction and scaling; Posture estimation module: uses the OpenPose deep learning network model to estimate the human posture of the preprocessed images and identify the spatial coordinates of the human joints; Action recognition module: By comparing the changes in the 2D Euclidean distance of the joint points, and then comparing the changes in the 2D coordinates of the joint points in consecutive frames, the complete action of the joint activity is identified; Data analysis module: calculates the 3D coordinates of the joint points and calculates the joint range of motion based on these coordinates; User interface module: includes a graphical user interface (GUI) for displaying real-time video streams, measurement results, and allowing users to configure measurement parameters, and supports a history viewing function for viewing and analyzing past measurement data; Data management module: stores and manages measurement data, and supports data export function to export measurement results into CSV or Excel format.

2. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The camera types of the depth camera module are RGB camera and infrared camera, and the capture rate of the RGB camera and infrared camera is 30 frames per second; the image types captured synchronously by the depth camera module are RGB image and depth image, and the field of view range is sufficient to cover the main area of ​​human joint movement.

3. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The image processing module reduces image noise and improves the accuracy of subsequent processing through a real-time image denoising algorithm, and the image processing module also includes an image scaling unit for adjusting the image resolution to adapt to the input requirements of the network model, wherein the image denoising algorithm uses a bilateral filter, which can maintain edge information while denoising, and its algorithm is as follows: Among them, I filtered (p) is the pixel value after filtering; is a spatial Gaussian function that takes into account the spatial proximity between pixels; is the grayscale Gaussian function, which is used to consider the similarity of pixel values; W p is the normalization factor to ensure that the sum of pixel values ​​is 1; S is the input pixel domain, p is the center pixel, and q is the input pixel.

4. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The posture estimation module is based on the OpenPose network model of deep learning to identify human joints from images. The input of the posture estimation module model is a preprocessed RGB image, and the output is the 2D coordinates of the joints, and the number of joints is 18 standard human joints.

5. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The OpenPose model includes: Input layer: receives input image; Convolutional layer: A series of convolutional layers used to extract image features, including multiple convolution operations and non-linear activation functions; Pooling layer: pooling operation used to reduce feature dimensions and enhance feature invariance; Fully connected layer: maps the learned features to joint positions and key points; Regression layer: predicts the location and visibility of key points; Pose estimation layer: integrates the outputs of the above layers to estimate the posture of the human body; Output layer: Output the 2D coordinates of human joints.

6. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The RGB image and the depth image are aligned in time and space so that each RGB pixel can accurately correspond to its depth value, the color information of the RGB image and the depth information of the depth image are fused, and the depth information is used to convert the two-dimensional image coordinates into three-dimensional space coordinates, calculate the position of the joint point in the three-dimensional space, and calculate the joint activity, such as angle or range of motion, according to the 3D coordinates; In the conversion from 2D to 3D coordinates, the depth image provides the depth value d of each pixel (u, v) on the Z axis (depth axis). Combined with the 2D coordinates (u, v) of the RGB image, the 3D coordinates (X, Y, Z) of the image in the camera coordinate system can be calculated: Z=d Among them, d is the depth value of the pixel, f x and f y is the focal length of the camera in the x-axis and y-axis directions, c x and c y The camera's principal point coordinates are converted to three-dimensional coordinates through the two-dimensional coordinates (u, v) and the corresponding depth value d, and the joint points are located in the three-dimensional space. The joint angle is calculated using the coordinates of the two joint points in the three-dimensional space using the distance formula or the angle between the vectors: in, and is the vector representing the position of the two joint points, and θ is the angle between the vectors.

7. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The posture estimation module is also used for model training and testing. The posture estimation module uses a lightweight network MobileNet or HRNet as a feature extractor to improve computational efficiency with only a small loss of accuracy. Features include: edges, color distribution, texture information, shape contours, and spatial relationships in the image.

8. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The pose estimation module uses the public COCO dataset for model training, adjusts the learning rate, number of cycles, training step size, and gradient optimizer parameters, and trains and verifies the OpenPose network model; The initial value of the learning rate is set between 0.001 and 0.0001, and a learning rate decay strategy is adopted; the total number of cycles is set to 50 to 100 cycles, and an early stopping rule is implemented to avoid overfitting; the training step size is set to 32 or 64 to balance memory usage and training efficiency; The gradient optimizer selects Adam or SGD, and sets β1=0.9, β2=0.999, ∈=10 for Adam. -8 ; Set momentum parameters for SGD; the model training evaluation indicators use accuracy and mAP to measure model performance.

9. The joint range of motion measurement system based on deep learning and machine vision according to claim 8, characterized in that: The test dataset for the OpenPose network model test is the ROM assessment videos shot by project members. The average error of the joint positions predicted by the OpenPose network model is calculated to evaluate the inference accuracy and the generalization ability of the model on unseen data, ensuring that the test accuracy and inference speed of the lightweight OpenPose are close to those of the original OpenPose network.

10. The joint range of motion measurement system based on deep learning and machine vision according to claim 1, characterized in that: The posture estimation module selects only the minimum set of joint points related to the specific joint activity measurement in the key point selection; When measuring elbow range of motion, only three joint points of the wrist, elbow, and shoulder are processed; When measuring knee range of motion, only three joint points at the ankle, knee, and hip are processed; When measuring the range of motion of the neck joint, only the critical points of the neck and head are processed; When measuring hip range of motion, only the joint points directly involved in hip movement are processed.

Images