Patient rehabilitation training data acquisition method and system based on visual identification

Abstract

The invention discloses a patient rehabilitation training data acquisition method and system based on visual identification, and relates to the technical field of rehabilitation training data acquisition, and the method comprises the steps: carrying out the image segmentation of a patient rehabilitation training video through a deep learning model, and separating a patient motion feature region from the patient rehabilitation training video; marking key skeleton nodes in the patient motion feature region, and establishing a three-dimensional coordinate system of the key skeleton nodes; calculating a motion angle and displacement data between adjacent skeleton nodes according to the three-dimensional coordinates of the key skeleton nodes; and performing matching degree analysis on the motion angle and displacement data and a preset rehabilitation standard to generate a rehabilitation training score. According to the method, the deep learning model is adopted to perform image segmentation on the rehabilitation training video, the motion feature region of the patient is accurately extracted from the image, and key skeleton nodes are marked and analyzed in combination with the three-dimensional coordinate system, so that accurate acquisition of rehabilitation training data of the patient is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation training data collection, in particular to a method and system for collecting patient rehabilitation training data based on visual recognition. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and computer vision technology, medical image analysis based on visual recognition has gradually become an important research direction in the field of rehabilitation medicine. Especially in the process of patient rehabilitation training, the use of deep learning models for motion analysis and evaluation has become an important means to improve the rehabilitation effect. The application of visual recognition technology in rehabilitation training can accurately evaluate the patient's motor function recovery by capturing the patient's posture changes, motion trajectory and movement details during the movement. With the continuous advancement of deep learning and image processing technology, motion evaluation based on video analysis has gradually shifted from traditional sensor-based evaluation methods to non-invasive, contactless visual evaluation methods, thereby reducing the burden on patients and improving the convenience and accuracy of rehabilitation training.

[0003] However, the existing technology still has some obvious deficiencies in the process of patient rehabilitation training data collection and evaluation. Although the traditional sensor-based data collection method can provide relatively accurate motion data, it relies on expensive hardware equipment and has high requirements for the patient's range of motion and comfort. Relatively speaking, although the technology based on visual recognition can avoid these problems, the existing visual recognition system still faces challenges in motion feature extraction, motion analysis and scoring accuracy. Most current visual recognition technologies mainly rely on two-dimensional images for posture recognition, and it is difficult to effectively obtain the three-dimensional spatial information of the patient's movements, thereby limiting the accurate calculation of data such as motion angles and displacements during rehabilitation training. In addition, the existing deep learning models are also relatively lacking in recognition accuracy and real-time performance in complex backgrounds, and are easily affected by environmental interference or background complexity, resulting in instability and errors in the analysis results. Summary of the invention

[0004] In view of the problems existing in the prior art, the present invention proposes a method and system for collecting patient rehabilitation training data based on visual recognition.

[0005] Therefore, the present invention provides a patient rehabilitation training data collection method based on visual recognition, which can solve the problems mentioned in the background technology.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides a method for collecting patient rehabilitation training data based on visual recognition, which comprises: Using a deep learning model to perform image segmentation on the patient rehabilitation training video, and separating the patient's motion feature area from the patient rehabilitation training video; Marking key bone nodes in the patient's motion feature area and establishing a three-dimensional coordinate system of the key bone nodes; Calculate the motion angle and displacement data between adjacent bone nodes according to the three-dimensional coordinates of the key bone nodes; The motion angle and displacement data are analyzed for matching with preset rehabilitation standards to generate a rehabilitation training score.

[0007] As a preferred solution of the patient rehabilitation training data collection method based on visual recognition of the present invention, wherein: the patient motion feature area includes spatial features and temporal features; The spatial features include human body contour, limb joints and motion posture information; The temporal features include motion displacement and speed information between consecutive frames; Separating the patient motion characteristic area includes: Performing feature fusion on the spatial features and the temporal features to obtain fused features; Calculating the generation probability of the segmentation mask according to the fusion feature; Generating a segmentation mask of the patient's motion features according to the generation probability of the segmentation mask; The motion feature region is separated from the patient rehabilitation training video according to the segmentation mask.

[0008] As a preferred solution of the patient rehabilitation training data collection method based on visual recognition of the present invention, wherein: the fusion feature It is expressed as follows: ; in, , , , is the learnable weight matrix, represents the sigmoid function, represents element-wise multiplication, is the spatial feature, It is the time series feature; The generation probability of the segmentation mask The calculation is expressed as follows: ; in, represents the input pixel, Represents the category label, and C represents the set of all possible categories.

[0009] As a preferred solution of the patient rehabilitation training data collection method based on visual recognition of the present invention, wherein: the key bone nodes include the center point of the head, the center point of the neck, the left and right shoulder joints, the left and right elbow joints, the left and right wrist joints, the center point of the trunk, the left and right hip joints, the left and right knee joints and the left and right ankle joints; Establishing the three-dimensional coordinate system of the key skeletal nodes includes: Obtain the intrinsic parameter matrix and distortion coefficient through camera calibration; Calculate the disparity map of the corresponding key bone nodes in the left and right views; Perform triangulation based on the disparity map and camera parameters to calculate the 3D coordinates (X, Y, Z) of each key bone node; The human body local coordinate system is established through analytical geometry method, with the center point of the torso as the origin, the vertical direction as the Y axis, the forward direction as the Z axis, and the right direction as the X axis.

[0010] As a preferred solution of the patient rehabilitation training data collection method based on visual recognition of the present invention, wherein: calculating the disparity map includes cost calculation and cost aggregation; The cost calculation is expressed by the following formula: ; Among them, the mutual information cost The calculation is: ; Among them, the zero-mean normalized cross-correlation cost The calculation is: ; Among them, the structural similarity cost The calculation is: ; in, For adaptive weights, it is calculated based on the consistency of the gradient direction: ; in, is the overall cost function, The two-dimensional coordinates of the current pixel , is the disparity value, which indicates the horizontal displacement of corresponding points in the left and right images. , , are the weight coefficients of the three cost items, is the gray value at point p in the left image, is the gray value of the corresponding point in the right image, is the probability distribution of the grayscale value of the left image, is the probability distribution of the grayscale value of the right image, is the joint probability distribution, For The pixel in the local window centered at is the mean value in the left image window, is the mean value in the right image window, is the standard deviation within the left image window, is the standard deviation in the right image window, is the number of local feature points, is the weight of the kth feature point, is the gradient of the left image, is the gradient of the right image, is the gradient direction of the left image, is the gradient direction of the right image, is the standard deviation parameter of the directional difference.

[0011] As a preferred solution of the patient rehabilitation training data collection method based on visual recognition of the present invention, the cost aggregation is expressed by the following formula: ; in, is the overall energy function, is the disparity map of the entire image, containing the disparity values ​​of all pixels. Pixel The disparity value at Pixel Neighborhood points The disparity value at is an indicator function used to impose appropriate penalties based on the disparity differences between adjacent pixels. is the matching cost at pixel p, is the neighborhood set of pixel p, is the penalty coefficient when the disparity values ​​of adjacent pixels differ by 1, It is the penalty coefficient when the difference between the disparity values ​​of adjacent pixels is greater than 1.

[0012] As a preferred solution of the patient rehabilitation training data collection method based on visual recognition of the present invention, the calculation of the motion angle and displacement data between adjacent bone nodes includes: Establish a local coordinate system based on the three-dimensional coordinates of adjacent bone nodes; Calculating the movement angle of the joint, wherein the movement angle includes flexion and extension angle, abduction angle and rotation angle; According to the calculated motion angle, a complete motion chain transfer matrix is ​​constructed; Calculate the angular velocity, angular acceleration, linear velocity and linear acceleration of joint motion; Calculate the range of motion of each joint, that is, the difference between the maximum and minimum angles during movement; The determination of the local coordinate system includes: The line connecting the distal bone point and the proximal bone point is used as the Z axis, a suitable reference vector is selected, the Y axis is determined by the cross product operation, and finally the X axis is obtained by the cross product of the Y axis and the Z axis, thereby constructing a complete local coordinate system.

[0013] As a preferred solution of the patient rehabilitation training data collection method based on visual recognition of the present invention, wherein: the preset rehabilitation standards include a standard range of joint activity, a standard template of motion trajectory, a standard range of motion speed, motion symmetry requirements and a posture stability index; The matching degree analysis includes motion range matching degree, motion trajectory matching degree, motion speed matching degree, motion symmetry matching degree and posture stability matching degree.

[0014] In a second aspect, the present invention provides a patient rehabilitation training data acquisition system based on visual recognition, which comprises: An image segmentation module, used to perform image segmentation on the patient rehabilitation training video using a deep learning model, and separate the patient's motion feature area from the patient rehabilitation training video; A human body coordinate system establishment module, used for marking key bone nodes in the patient's motion feature area and establishing a three-dimensional coordinate system of the key bone nodes; A motion recognition module, used to calculate the motion angle and displacement data between adjacent bone nodes according to the three-dimensional coordinates of the key bone nodes; The matching module is used to perform matching analysis on the movement angle and displacement data with the preset rehabilitation standard to generate a rehabilitation training score.

[0015] The beneficial effects of the present invention are as follows: by adopting a deep learning model to perform image segmentation on the rehabilitation training video, the patient's motion feature area is accurately extracted therefrom, and the key bone nodes are marked and analyzed in combination with the three-dimensional coordinate system, thereby realizing the accurate calculation of the patient's motion angle and displacement data. Unlike traditional two-dimensional image recognition technology, our invention can comprehensively capture the patient's motion state through three-dimensional spatial information and provide more accurate rehabilitation training scores. In addition, the combination of deep learning image segmentation and bone node calibration technology can improve the system's recognition accuracy of complex motion processes in a dynamic environment, thereby providing patients with more accurate rehabilitation assessments. By analyzing the matching degree of motion data with preset rehabilitation standards, the patient's training effect can be fed back in real time, providing strong support for the formulation of personalized rehabilitation plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 Flowchart of the patient rehabilitation training data collection method based on visual recognition. DETAILED DESCRIPTION

[0017] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0018] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0019] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0020] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0021] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0022] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0023] Example 1: Reference Figure 1 , which is the first embodiment of the present invention, provides a patient rehabilitation training data collection method based on visual recognition, comprising: S1: Using a deep learning model to perform image segmentation on a patient rehabilitation training video, and separating the patient's motion feature area from the video; Video data of patient rehabilitation training is collected by a multi-view camera device, wherein the multi-view camera device includes a forward camera, a left camera and a right camera, the video data collection frame rate is 60 frames per second, and the resolution is 1920×1080 pixels; the video data includes an RGB color image sequence and a depth image sequence, wherein the RGB color image sequence is stored in an H.264 encoding format, and the depth image sequence is stored in a 16-bit grayscale format.

[0024] The RGB color image sequence is segmented in real time based on the improved YOLOv5 deep learning model, wherein the loss function of the improved YOLOv5 model is Defined as: ; in, represents the target confidence loss, represents the position regression loss, represents the size regression loss, represents the timing consistency loss, , , , is the weight coefficient.

[0025] The timing consistency loss The calculation formula is: ; in, Indicates the length of the time series, represents the number of local feature points, is the adaptive weight coefficient, Indicates Frame No. The feature vector of the feature points.

[0026] The improved YOLOv5 deep learning model adopts a transfer learning method to fine-tune the model by combining the pre-trained human body detection weight parameters with a custom 4000 labeled rehabilitation training data set. The weight update formula for the model fine-tuning is: ; in, Indicates The model weights for the iterations, Indicates The model weights for the iterations, is the learning rate, is the balance coefficient, represents the pre-training task loss, represents the fine-tuning task loss.

[0027] The spatial features and temporal features in the RGB color image sequence are further extracted through an improved multi-scale feature pyramid network, wherein the spatial features include human body contours, limb joints and motion posture information, and the temporal features include motion displacement and speed information between consecutive frames.

[0028] A dual-branch attention mechanism is used to fuse the spatial features and temporal features to generate a pixel-level segmentation mask of the patient's motion features, wherein the resolution of the pixel-level segmentation mask is the same as that of the original image sequence; a motion feature region including the patient's trunk and limbs is separated from the RGB color image sequence according to the segmentation mask, wherein the segmentation accuracy of the motion feature region is not less than 95%, and the segmentation speed reaches 30 frames per second; meanwhile, the depth image sequence is aligned with the separated motion feature region to obtain depth information of the motion feature region.

[0029] More specifically, the feature fusion adopts the attention mechanism, and the attention weight It can be expressed as follows: ; in, denote query, key, and value matrices respectively, is the feature dimension, is the feature map, is a learnable parameter, It is a feed-forward neural network.

[0030] Furthermore, according to the above attention mechanism fusion feature, the fusion feature of the spatial feature and the temporal feature is obtained It can be expressed as follows: ; in, , , , is the learnable weight matrix, represents the sigmoid function, represents element-wise multiplication, is the spatial feature, It is a timing feature.

[0031] The probability of generating a segmentation mask is calculated based on the fusion features, and then a pixel-level segmentation mask of the patient's motion features is generated based on the probability, wherein the probability of generating the segmentation mask is The calculation can be expressed as follows: ; in, represents the input pixel, Represents the category label, and C represents the set of all possible categories.

[0032] S2: marking key bone nodes in the patient's motion feature area and establishing a three-dimensional coordinate system of the key bone nodes; The key bone nodes of the motion feature area are marked through a human skeleton feature extraction network, wherein the key bone nodes include 17 human key points, namely the center point of the head, the center point of the neck, the left and right shoulder joints, the left and right elbow joints, the left and right wrist joints, the center point of the trunk, the left and right hip joints, the left and right knee joints and the left and right ankle joints; the human skeleton feature extraction network adopts an improved HRNet model structure, including a fourth-order parallel sub-network, and the feature map resolution ratio of each sub-network is 1:2:4:8, and information interaction of features with different resolutions is realized through a multi-scale fusion module.

[0033] The detection confidence threshold of the key skeleton nodes is set to 0.85. For key points with confidence lower than the threshold, a time series Kalman filter is used for trajectory prediction compensation. The state vector of the Kalman filter includes the two-dimensional coordinates and velocity components of the key points.

[0034] The three-dimensional coordinate system of the key skeletal nodes is established based on the binocular vision principle, wherein the process of establishing the three-dimensional coordinate system includes: Firstly, the intrinsic parameter matrix and distortion coefficient are obtained through camera calibration. The dimension of the intrinsic parameter matrix is ​​3×3, which contains the focal length and principal point coordinate information. Secondly, the disparity map corresponding to the key bone nodes in the left and right views is calculated by the stereo matching algorithm. Thirdly, triangulation is performed based on the disparity map and camera parameters to calculate the three-dimensional spatial coordinates (X, Y, Z) of each key bone node. Finally, the local human body coordinate system is established with the center point of the torso as the origin, the vertical direction as the Y axis, the forward direction as the Z axis, and the right direction as the X axis through the analytical geometry method, and the three-dimensional coordinates of all key bone nodes are converted to this coordinate system. The coordinate conversion accuracy is better than ±5mm.

[0035] For example, the disparity calculation uses an improved multi-scale cost calculation function: ; Among them, the mutual information cost The calculation is: ; Among them, the zero-mean normalized cross-correlation cost The calculation is: ; Among them, the structural similarity cost The calculation is: ; in, For adaptive weights, it is calculated based on the consistency of the gradient direction: ; in, is the overall cost function, The two-dimensional coordinates of the current pixel , is the disparity value, which indicates the horizontal displacement of corresponding points in the left and right images. , , are the weight coefficients of the three cost items, is the gray value at point p in the left image, is the gray value of the corresponding point in the right image, is the probability distribution of the grayscale value of the left image, is the probability distribution of the grayscale value of the right image, is the joint probability distribution, For The pixel in the local window centered at is the mean value in the left image window, is the mean value in the right image window, is the standard deviation within the left image window, is the standard deviation in the right image window, is the number of local feature points, is the weight of the kth feature point, is the gradient of the left image, is the gradient of the right image, is the gradient direction of the left image, is the gradient direction of the right image, is the standard deviation parameter of the directional difference.

[0036] Then, cost aggregation is performed to obtain the disparity map corresponding to the key bone nodes in the left and right views, which can be expressed as follows: ; in, is the overall energy function, is the disparity map of the entire image, containing the disparity values ​​of all pixels. Pixel The disparity value at Pixel Neighborhood points The disparity value at is an indicator function used to impose appropriate penalties based on the disparity differences between adjacent pixels. is the matching cost at pixel p, is the neighborhood set of pixel p, is the penalty coefficient when the disparity values ​​of adjacent pixels differ by 1, is the penalty coefficient when the difference between the disparity values ​​of adjacent pixels is greater than 1. The disparity map that minimizes the energy function is found as the optimal disparity map, and the three-dimensional coordinate system is subsequently established.

[0037] Preferably, the disparity calculation scheme adopted in this embodiment utilizes a structural description of adaptive weights instead of fixed window aggregation, and uses computationally efficient semi-global optimization to replace complex global optimization or simple local optimization. While ensuring a real-time processing speed of 50 Hz, the key point positioning error is controlled within 5 mm, which significantly improves the matching accuracy of the algorithm in weak texture areas (increased by 40%) and the ability to resist mismatching (increased by 60%), achieving an overall breakthrough in accuracy, efficiency, and robustness.

[0038] S3: Calculating the motion angle and displacement data between adjacent bone nodes according to the three-dimensional coordinates of the key bone nodes; According to the 3D coordinates of adjacent bone nodes, a local coordinate system is first established. The line connecting the distal bone point and the proximal bone point is used as the Z axis, a suitable reference vector is selected, the Y axis is determined by the cross multiplication operation, and finally the X axis is obtained by the cross multiplication of the Y axis and the Z axis, thereby constructing a complete local coordinate system.

[0039] The ZXY Euler angle sequence is used to calculate the three main motion angles of the joint: flexion and extension angle, abduction angle and rotation angle. The flexion and extension angle is calculated by the projection of the bone vector on the YZ plane, the abduction angle is calculated by the angle between the bone vector and the X axis, and the rotation angle is calculated by the projection of the reference vector in the local coordinate system.

[0040] According to the three calculated motion angles, a complete kinematic chain transfer matrix is ​​constructed. This matrix contains a rotation part and a translation part. The rotation part is composed of the combination of three basic rotation matrices, and the translation part is determined by the position difference of adjacent bone points.

[0041] Calculate the angular velocity, angular acceleration, linear velocity and linear acceleration of joint motion. The velocity parameter is obtained by dividing the angle difference and position difference at adjacent moments by the sampling time interval; the acceleration parameter is obtained by dividing the velocity difference at adjacent moments by the sampling time interval.

[0042] Specific motion parameters are calculated based on the characteristics of different joints. For example, calculating the flexion and extension angles of the knee joint requires considering the bone vectors of the thigh and calf, calculating the abduction angle of the hip joint requires considering the relationship between the thigh vector and the pelvic coordinate system, and calculating the dorsiflexion angle of the ankle joint requires considering the relationship between the foot vector and the calf coordinate system.

[0043] Calculate the range of motion of each joint, that is, the difference between the maximum angle and the minimum angle during the movement. At the same time, calculate the movement symmetry index of the corresponding joints on the left and right sides to evaluate the balance of the movement.

[0044] The entire calculation process uses a sampling frequency of 100Hz to ensure that the angle calculation accuracy reaches ±1 degree and the displacement calculation accuracy reaches ±1 mm. The final output data includes: the change data of the three-dimensional angle of each joint over time, the three-dimensional component data of the joint displacement, the movement speed and acceleration data, the joint range of motion and symmetry index, and other complete kinematic parameter sets.

[0045] S4: performing matching analysis on the movement angle and displacement data and the preset rehabilitation standard to generate a rehabilitation training score; The motion angle and displacement data are analyzed for matching with the preset rehabilitation standards to generate a rehabilitation training score, wherein the preset rehabilitation standards include the standard range of joint range of motion, the standard template of motion trajectory, the standard interval of motion speed, the motion symmetry requirement and the index of posture stability; the matching analysis first performs preprocessing on the motion data such as time series alignment, data normalization, outlier detection, data smoothing and key feature extraction; then the matching degrees of multiple dimensions such as motion range matching, motion trajectory matching, motion speed matching, motion symmetry matching and posture stability matching are calculated respectively; wherein the motion range matching evaluates the overlap between the actual joint range of motion and the standard range, and the motion The trajectory matching degree calculates the dynamic time regularization distance between the actual trajectory and the standard trajectory, the movement speed matching degree compares the degree of conformity between the actual speed and the standard speed range, the movement symmetry matching degree calculates the similarity of the movement parameters of the left and right limbs, and the posture stability matching degree evaluates the degree of deviation of the trunk posture; the final score is generated by weighted fusion of the matching degrees of each dimension, and the scoring result adopts a 0-100 point system and is graded according to the score; at the same time, a detailed scoring report including the total score, sub-item score, problem analysis and improvement suggestions is generated to guide the adjustment of subsequent rehabilitation training programs; the evaluation accuracy of the scoring system is not less than 90%, and the correlation coefficient between the scoring result and the subjective score of professional physicians is above 0.85.

[0046] A patient rehabilitation progress curve is constructed based on the rehabilitation training score, and a personalized rehabilitation recommendation report is output.

[0047] Furthermore, this embodiment also provides a patient rehabilitation training data collection system based on visual recognition, based on the patient rehabilitation training data collection method based on visual recognition proposed in the above embodiment, including: An image segmentation module, used to perform image segmentation on the patient rehabilitation training video using a deep learning model, and separate the patient's motion feature area from the patient rehabilitation training video; A human body coordinate system establishment module, used for marking key bone nodes in the patient's motion feature area and establishing a three-dimensional coordinate system of the key bone nodes; A motion recognition module, used to calculate the motion angle and displacement data between adjacent bone nodes according to the three-dimensional coordinates of the key bone nodes; The matching module is used to perform matching analysis on the movement angle and displacement data with the preset rehabilitation standard to generate a rehabilitation training score.

[0048] This embodiment also provides a computer device, which is suitable for the patient rehabilitation training data collection method based on visual recognition, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the patient rehabilitation training data collection method based on visual recognition proposed in the above embodiment.

[0049] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0050] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for collecting patient rehabilitation training data based on visual recognition as proposed in the above embodiment is implemented.

[0051] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0052] Embodiment 2: This is the second embodiment of the present invention. This embodiment provides a patient rehabilitation training data collection method based on visual recognition. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0053] This experiment selected 30 patients undergoing rehabilitation training after knee replacement as research subjects, with an age range of 45-70 years old, an average age of 58.5 years old, including 16 males and 14 females. The experimental site was set up in a standardized training room of the rehabilitation department of a tertiary hospital, with sufficient and uniform ambient light and a room temperature of 24±1℃.

[0054] The experiment used three high-speed industrial cameras (model: MV-CA060-10GC) installed on tripods at a height of 1.5 meters above the ground, arranged in an equilateral triangle, with a camera spacing of 2 meters to ensure that the research object is within the common field of view of the three cameras. The camera acquisition parameters are set to: frame rate 60fps, resolution 1920×1080 pixels, exposure time set to 1 / 120 seconds. Depth information acquisition uses a matching structured light depth sensor with a sampling accuracy of up to 0.1mm.

[0055] Before the experiment, the three cameras were calibrated, and 20 sets of calibration acquisitions at different positions and angles were performed using a 9×6 chessboard calibration plate to obtain the camera intrinsic parameter matrix and distortion coefficient. Subsequently, dual-target calibration was performed to establish the geometric relationship between the cameras to ensure the accuracy of 3D reconstruction.

[0056] The improved YOLOv5 model was trained using a dataset containing 4,000 labeled rehabilitation training images, of which 3,200 were used for training and 800 for validation. The training process adopted a transfer learning strategy and fine-tuned the model weights based on the pre-trained COCO dataset. The training parameters were set as follows: batch size 16, initial learning rate 0.001, weight decay coefficient 0.0005, momentum coefficient 0.937, and 100 iterations.

[0057] During the experiment, each patient performed standardized knee rehabilitation training, including knee flexion and extension, straight leg raising, and tiptoeing. Each movement was repeated 10 times, and the acquisition time was 60 seconds. The system processed the video stream in real time, extracted spatial features and temporal features through an improved multi-scale feature pyramid network, and applied a dual-branch attention mechanism for feature fusion to generate accurate motion feature region segmentation masks.

[0058] Based on the human skeleton feature extraction network, 17 key bone nodes are marked in the segmented motion feature area. Through the designed parallax calculation scheme, the three-dimensional coordinate system of these key points is established to achieve high-precision motion parameter measurement. The system records the complete set of kinematic parameters such as the motion angle, displacement, velocity and acceleration of each joint, and performs real-time matching analysis with the preset rehabilitation standards.

[0059] To ensure the reliability of the data, each patient's training process was completed under the guidance of a professional rehabilitation therapist, and another rehabilitation therapist performed subjective scoring for comparison and verification with the system scoring results. After all data were collected, they were analyzed and evaluated using a specially developed data processing program. The following experimental data were obtained through the above process, as shown in Table 1: Table 1 Rehabilitation training evaluation data record table Patient number Knee joint range of motion matching (%) Motion trajectory matching degree (%) Movement speed matching degree (%) Postural stability (%) System score (points) Physician rating (points) Evaluation duration (seconds) P001 92.5 88.7 90.2 94.3 91.4 90.0 58.3 P002 89.8 91.5 87.6 92.1 90.2 89.5 59.1 P003 94.2 93.8 92.4 95.6 94.0 93.5 57.8 P004 87.6 85.9 88.3 90.2 88.0 87.0 60.2 P005 91.3 90.4 89.7 93.5 91.2 90.5 58.7 P006 93.7 92.8 91.5 94.8 93.2 92.5 59.4 Through the analysis of experimental data, we can conclude that: By comparing the system score and the physician score, it can be found that the average error between the system score and the physician score is within 1 point, and the correlation coefficient reaches 0.96, which shows that the system can accurately simulate the evaluation criteria of professional physicians. In particular, in the evaluation of patient P003, the system gave 94.0 points and the physician gave 93.5 points, with an error of only 0.5 points, which fully proves the reliability of the system evaluation.

[0060] The system conducts a comprehensive assessment from four dimensions: knee range of motion, motion trajectory, motion speed, and posture stability. The data shows that there are certain differences in the scores of different dimensions, which reflects that the system can carefully capture the patient's rehabilitation in different aspects. For example, patient P004 performed well in posture stability (90.2%), but the motion trajectory matching was relatively low (85.9%). This refined evaluation result helps to adjust the rehabilitation training plan in a targeted manner.

[0061] From the standard deviation analysis of the six sets of data, it can be seen that the system score has a small degree of dispersion (the standard deviation is about 2.1 points), indicating that the evaluation results have good stability and repeatability. This stability is due to the improved YOLOv5 model and accurate 3D reconstruction algorithm used by the system.

[0062] The data showed that the system generally achieved a match of more than 90% in postural stability assessment, thanks to the dual-branch attention mechanism and multi-scale feature fusion strategy adopted. In particular, in the assessment of patient P003, the match of the four dimensions exceeded 92%, fully demonstrating the high-precision characteristics of the system.

[0063] The experimental results show that the invention not only realizes the automation and standardization of rehabilitation training data collection, but also shows obvious advantages in terms of evaluation accuracy, efficiency and stability. The system's multi-dimensional collection and evaluation method provides a scientific basis for formulating personalized rehabilitation plans. At the same time, the highly quantitative nature of the evaluation results also provides a reliable guarantee for tracking the patient's rehabilitation progress.

[0064] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A patient rehabilitation training data collection method based on visual recognition, characterized in that: include: Using a deep learning model to perform image segmentation on the patient rehabilitation training video, and separating the patient's motion feature area from the patient rehabilitation training video; Marking key bone nodes in the patient's motion feature area and establishing a three-dimensional coordinate system of the key bone nodes; Calculate the motion angle and displacement data between adjacent bone nodes according to the three-dimensional coordinates of the key bone nodes; Performing matching analysis on the movement angle and displacement data and the preset rehabilitation standard to generate a rehabilitation training score; The patient motion feature area includes spatial features and temporal features; The spatial features include human body contour, limb joints and motion posture information; The temporal features include motion displacement and speed information between consecutive frames; Separating the patient motion characteristic area includes: Performing feature fusion on the spatial features and the temporal features to obtain fused features; Calculating the generation probability of the segmentation mask according to the fusion feature; Generating a segmentation mask of the patient's motion features according to the generation probability of the segmentation mask; The motion feature region is separated from the patient rehabilitation training video according to the segmentation mask.

2. The method for collecting patient rehabilitation training data based on visual recognition according to claim 1, characterized in that: The fusion feature It is expressed by the following formula: ； in, , , , is the learnable weight matrix, represents the sigmoid function, represents element-wise multiplication, is the spatial feature, It is the time series feature; The generation probability of the segmentation mask The calculation is expressed as follows: ； in, represents the input pixel, Represents the category label, and C represents the set of all possible categories.

3. The method for collecting patient rehabilitation training data based on visual recognition according to claim 2, characterized in that: The key bone nodes include the center point of the head, the center point of the neck, the left and right shoulder joints, the left and right elbow joints, the left and right wrist joints, the center point of the torso, the left and right hip joints, the left and right knee joints, and the left and right ankle joints; Establishing the three-dimensional coordinate system of the key skeletal nodes includes: Obtain the intrinsic parameter matrix and distortion coefficient through camera calibration; Calculate the disparity map of the corresponding key bone nodes in the left and right views; Perform triangulation based on the disparity map and camera parameters to calculate the 3D coordinates (X, Y, Z) of each key bone node; The human body local coordinate system is established through analytical geometry method, with the center point of the torso as the origin, the vertical direction as the Y axis, the forward direction as the Z axis, and the right direction as the X axis.

4. The method for collecting patient rehabilitation training data based on visual recognition according to claim 3, characterized in that: Calculating the disparity map includes cost calculation and cost aggregation; The cost calculation is expressed by the following formula: ； Among them, the mutual information cost The calculation is: ； Among them, the zero-mean normalized cross-correlation cost The calculation is: ； Among them, the structural similarity cost The calculation is: ； in, For adaptive weights, it is calculated based on the consistency of the gradient direction: ； in, is the overall cost function, The two-dimensional coordinates of the current pixel , is the disparity value, which indicates the horizontal displacement of corresponding points in the left and right images. , , are the weight coefficients of the three cost items, is the gray value at point p in the left image, is the gray value of the corresponding point in the right image, is the probability distribution of the grayscale value of the left image, is the probability distribution of the grayscale value of the right image, is the joint probability distribution, For The pixel in the local window centered at is the mean value in the left image window, is the mean value in the right image window, is the standard deviation within the left image window, is the standard deviation in the right image window, is the number of local feature points, is the weight of the kth feature point, is the gradient of the left image, is the gradient of the right image, is the gradient direction of the left image, is the gradient direction of the right image, is the standard deviation parameter of the directional difference.

5. The method for collecting patient rehabilitation training data based on visual recognition according to claim 4, characterized in that: The cost aggregation is expressed as follows: ； in, is the overall energy function, is the disparity map of the entire image, containing the disparity values ​​of all pixels. Pixel The disparity value at Pixel Neighborhood points The disparity value at is an indicator function used to impose appropriate penalties based on the disparity differences between adjacent pixels. is the matching cost at pixel p, is the neighborhood set of pixel p, is the penalty coefficient when the disparity values ​​of adjacent pixels differ by 1, It is the penalty coefficient when the difference between the disparity values ​​of adjacent pixels is greater than 1.

6. The method for collecting patient rehabilitation training data based on visual recognition according to claim 5, characterized in that: The calculation of the motion angle and displacement data between adjacent bone nodes includes: Establish a local coordinate system based on the three-dimensional coordinates of adjacent bone nodes; Calculating the movement angle of the joint, wherein the movement angle includes flexion and extension angle, abduction angle and rotation angle; According to the calculated motion angle, a complete motion chain transfer matrix is ​​constructed; Calculate the angular velocity, angular acceleration, linear velocity and linear acceleration of joint motion; Calculate the range of motion of each joint, that is, the difference between the maximum and minimum angles during movement; The determination of the local coordinate system includes: The line connecting the distal bone point and the proximal bone point is used as the Z axis, a suitable reference vector is selected, the Y axis is determined by the cross product operation, and finally the X axis is obtained by the cross product of the Y axis and the Z axis, thereby constructing a complete local coordinate system.

7. The method for collecting patient rehabilitation training data based on visual recognition according to claim 6, characterized in that: The preset rehabilitation standards include standard range of joint mobility, standard template of motion trajectory, standard range of motion speed, motion symmetry requirements and posture stability index; The matching degree analysis includes motion range matching degree, motion trajectory matching degree, motion speed matching degree, motion symmetry matching degree and posture stability matching degree.

8. A patient rehabilitation training data acquisition system based on visual recognition, based on the patient rehabilitation training data acquisition method based on visual recognition according to any one of claims 1 to 7, characterized in that: include: An image segmentation module, used to perform image segmentation on the patient rehabilitation training video using a deep learning model, and separate the patient's motion feature area from the patient rehabilitation training video; A human body coordinate system establishment module, used for marking key bone nodes in the patient's motion feature area and establishing a three-dimensional coordinate system of the key bone nodes; A motion recognition module, used to calculate the motion angle and displacement data between adjacent bone nodes according to the three-dimensional coordinates of the key bone nodes; The matching module is used to perform matching analysis on the movement angle and displacement data with the preset rehabilitation standard to generate a rehabilitation training score.

Images