A joint point tracking method, Parkinson's disease auxiliary diagnosis method and device

Through the joint node tracking method and the human skeleton extraction model, combined with the Lucas–Kanade optical flow method, the detection accuracy and speed problems in Parkinson's diagnosis were solved, and more accurate joint node detection and Parkinson's rating were achieved, reducing the calculation volume and subjective error.

CN114612526BActive Publication Date: 2024-11-15NANJING UNIV OF POSTS & TELECOMM
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202210203296.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-02
Publication Date
2024-11-15
Estimated Expiration
2042-03-02

AI Technical Summary

Technical Problem

The prior art has low detection accuracy, low speed, and large calculation and calculation operations in Parkinson's diagnosis, which leads to difficulties in early diagnosis and has subjective errors depending on naked eyes.

Method used

The joint node tracking method is adopted to obtain joint confidence maps and limb affinity vector field through the human skeleton extraction model, and the joint node tracking is combined with the Lucas–Kanade optical flow method, and the detection accuracy and speed are improved by using multi-scale fusion and front-to-back frame intersection method. , reduce the amount of calculation and generate Parkinson's rating information.

Benefits of technology

It improves the prediction accuracy of the joint node sequence, reduces the amount of calculation, enhances the detection speed, reduces the need for manual correction after tracking failure, avoids the subjectivity of naked eyes, and provides a more accurate Parkinson's tremor rating.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114612526B_ABST
    Figure CN114612526B_ABST
Patent Text Reader

Abstract

The present invention discloses a joint point tracking method, a Parkinson's auxiliary diagnosis method and a device, wherein the Parkinson's auxiliary diagnosis method comprises the following steps: placing a human body video to be tested into a human skeleton extraction model and adopting multi-scale prediction to obtain two-dimensional joint points; adopting a tracking method to obtain a target two-dimensional joint point in the next frame of the human body video; adopting a front and back frame intersection method to obtain a corrected two-dimensional joint point in a frame where the two-dimensional joint point tracking fails, and then continuing to track the corrected two-dimensional joint point as a new two-dimensional joint point; obtaining an accurate joint point sequence according to the tracked two-dimensional joint point and the corrected two-dimensional joint point; and inputting the joint point sequence into a Parkinson's rating model to obtain Parkinson's auxiliary diagnosis information. The present invention has simple steps, further improves the prediction accuracy of the joint point sequence, and obtains effective Parkinson's auxiliary diagnosis information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a joint point tracking method, a Parkinson's disease auxiliary diagnosis method and a device, and belongs to the technical field of artificial intelligence video images. Background Art

[0002] Parkinson's disease is a common neurodegenerative disease in the elderly. 50% to 80% of Parkinson's disease cases are not discovered until the disease progresses to the point where the symptoms are very obvious and even affect the quality of life, thus missing the best time for early treatment. Treatment for people with moderate to severe Parkinson's disease is more difficult and the medical burden is heavier.

[0003] Parkinson's disease has an insidious onset, and most of the indicators of Parkinson's patients do not change abnormally when using existing auxiliary examination technologies. Therefore, Parkinson's disease is currently usually diagnosed by experienced neurologists based on medical history, clinical symptoms, and tedious and complex physical examinations. Due to the uneven professional experience of doctors and the uneven medical technology levels of medical institutions, Parkinson's disease is often not discovered and correctly diagnosed early.

[0004] Doctors often make clinical diagnoses by observing the tremor of a patient's joint with the naked eye. Since the naked eye cannot accurately estimate the movement distance of the joint, clinical diagnosis often produces serious subjectivity and large errors. The application of human posture estimation technology can accurately extract the position of the joint points of Parkinson's patients. Compared with the naked eye estimation during clinical diagnosis, the tremor of the patient's joint can be analyzed more accurately.

[0005] Most of the current human posture detection algorithms use the confidence map method to predict the probability of joint points. The accuracy of this type of method depends largely on the relative size of the confidence map and the human body in the image, which limits the prediction accuracy. The current human posture detection algorithm based on a single image has obvious jitter in the detection of video streams, while the current human posture detection algorithm based on multiple images has a huge amount of computation and requires a large amount of video memory, which is not conducive to the accurate analysis of joint points.

[0006] CPM stands for Convolutional Pose Machines, which is a top-down human pose detection method. CPM is a serialized structure composed of fully convolutional grids. The convolutional network directly operates on the confidence map of the previous stage to output increasingly refined joint points. CPM can learn the feature representation of image and spatial information at the same time, and does not need to build any explicit relationship model between joint points. However, since it is a top-down detection method, when detecting the poses of multiple people, the detection performance will decrease as the number of people increases.

[0007] The Openpose human posture recognition method is an important method in the field of human posture estimation. It can realize posture estimation of human movements, facial expressions, finger movements, etc., and has very good robustness. The predecessor of Openpose is the CPM algorithm. In the scenario of multi-person recognition, the CPM algorithm can also recognize multiple people through heat maps. Openpose is a bottom-up human posture estimation method. It uses a confidence map (Part Confidence Map) to represent the joints of the human body. It is suitable for single-person human posture estimation. It uses a limb affinity vector field (Part Affinity Field) to represent the skeleton connection of the human body, which better solves the problem of multi-person human posture estimation. However, Openpose only detects single-frame images. For video detection, the detection results of Openpose will have obvious jitter problems. If only the native Openpose is used as a human posture estimation method to extract joints, it will also produce large errors in the diagnosis of Parkinson's disease. Therefore, in order to accurately diagnose Parkinson's disease, a set of accurate joint point extraction methods and a set of tremor analysis methods based on the proposed joints are very important. Summary of the invention

[0008] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a joint point tracking method, a Parkinson's auxiliary diagnosis method and device, which can solve the defects of the prior art such as low detection accuracy, slow detection speed and large amount of calculation. Through accurate joint point tracking detection, the Parkinson's tremor rating is calculated and effective auxiliary information is provided for Parkinson's diagnosis.

[0009] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0010] In a first aspect, the present invention provides a joint point tracking method, the method comprising the following steps:

[0011] Step A: Obtain a frame sequence of the video to be tested;

[0012] Step B: inputting the current frame of the video to be tested into the human skeleton extraction model to obtain a joint confidence map and a limb affinity vector field, and obtaining the two-dimensional joint points of the current frame according to the joint confidence map and the limb affinity vector field;

[0013] Step C: For the next frame of the current frame, select the required joint points as the joint points to be tracked, and track the target two-dimensional joint points of the next frame;

[0014] Step D: Using the two-dimensional joint points obtained by tracking the next frame of the video as the joint points to be tracked, and reversely tracking the previous frame of the video to obtain the two-dimensional joint points obtained by reverse tracking of the previous frame;

[0015] Step E: Calculate the Euclidean distance between the two-dimensional joint point tracked in the previous frame and the actual two-dimensional joint point in the previous frame;

[0016] Step R: If the Euclidean distance is too large, the joint points tracked in the next frame cannot be tracked back to the joint points of the previous frame, and the tracking of the joint points tracked in the next frame is judged to have failed. Otherwise, the tracking is successful, and the joint points of the next frame tracked are used as the actual joint points of the next frame.

[0017] Step G: If the joint point tracking of the next frame fails, the previous and next frame intersection method is used to superimpose the joint confidence maps of the previous frame and the next frame, reduce the detection area of ​​the joint point, obtain the corrected target joint point of the next frame, and use the corrected joint point of the next frame as the actual joint point of the next frame;

[0018] Step H: Jump the sequence number of the current frame to the next frame;

[0019] Step I: Repeat steps B to H to extract a complete two-dimensional joint point sequence of the video to be tested.

[0020] Furthermore, in the Parkinson's auxiliary diagnosis method based on joint precision tracking of the present invention, the human skeleton extraction model is based on Openpose, and its structure includes: a feature extraction module, a limb affinity vector field estimation module, and a joint confidence map estimation module; the feature extraction module includes the first ten layers and two convolutional layers of the VGG-19 network;

[0021] The method for obtaining the human skeleton extraction model comprises:

[0022] The training dataset is calibrated based on the MSCOCO dataset and the Parkinson's rating video data;

[0023] The training data set is input into the human skeleton extraction model, and the final human skeleton extraction model is obtained after multiple iterations and training.

[0024] Furthermore, the method of inputting the current frame of the video to be tested into the human skeleton extraction model to obtain the joint confidence map and the limb affinity vector field also includes:

[0025] Use multi-scale fusion estimation to scale the current frame proportionally to obtain images of different sizes;

[0026] Input images of different sizes into the human skeleton extraction model to obtain joint confidence maps of images of different sizes;

[0027] The joint confidence maps of images of different sizes are superimposed to obtain the average and the output joint confidence map is obtained.

[0028] Furthermore, the method of inputting the image into the human skeleton extraction model to obtain the joint confidence map and the limb affinity vector field includes:

[0029] Step 1: Pass a frame of image into the first 10 layers of the VGG-19 network to obtain the feature map of the first stage;

[0030] Step 2: The feature map is passed through a convolutional layer to obtain the limb affinity vector field of the first stage;

[0031] Step 3: Input the limb affinity vector field and feature map of the first stage into the network, and obtain the limb affinity vector field of the second stage through the convolution layer;

[0032] Step 4: Repeat steps 2 and 3 to a specified number of times to obtain the final limb affinity vector field;

[0033] Step 5: Input the final limb affinity vector field and feature map into the network, and obtain the joint confidence map of the first stage through the convolution layer;

[0034] Step 6: Input the joint confidence map and feature map of the first stage into the network, and obtain the joint confidence map of the second stage through the convolution layer;

[0035] Step 7: Repeat steps 5 and 6 for a predetermined number of times to obtain the output joint confidence map.

[0036] Furthermore, the method for tracking the target two-dimensional joint points of the next frame is the Lucas–Kanade optical flow method; the method for reverse tracking the previous frame of the video is the Lucas–Kanade optical flow method.

[0037] Furthermore, the method of using the previous and next frame intersection method to superimpose the joint confidence maps of the previous frame and the next frame, narrowing the detection area of ​​the joint point, and obtaining the corrected target joint point of the next frame includes:

[0038] By superimposing the confidence maps of adjacent frames and finding their intersection, the corrected target joint points of the next frame are obtained.

[0039] In a second aspect, the present invention provides a method for assisting diagnosis of Parkinson's disease based on precise joint tracking, based on the joint tracking method described in the first aspect, comprising the following steps:

[0040] Obtain a complete two-dimensional joint point sequence of the extracted video to be tested;

[0041] The complete two-dimensional joint point sequence of the extracted video to be tested is input into the Parkinson's rating model to obtain the Parkinson's rating information of the video to be tested, and generate Parkinson's auxiliary diagnosis information.

[0042] Furthermore, the input of the Parkinson's rating model is a two-dimensional joint point sequence, and the output is the Parkinson's tremor grade;

[0043] The structure of the Parkinson's rating model is based on the action classification network DD-Net, and its network structure includes:

[0044] The first branch: includes three convolutional layers, which are used to predict the relationship between the skeleton points in the current frame and capture the spatial domain features of the image;

[0045] The second branch consists of three convolutional layers, which are used to calculate the differential data between joint points at intervals of 1 frame and learn the fast temporal information of different frames.

[0046] The third branch consists of three convolutional layers, which are used to calculate the differential data between joint points at an interval of 2 frames and learn the slow timing information of different frames;

[0047] Classification module: It consists of three convolutional layers and two fully connected layers, which are used to fuse the features extracted by the three branches to obtain the tremor grade of Parkinson's disease.

[0048] Furthermore, the Parkinson's rating model is obtained by:

[0049] Using a human skeleton extraction model to extract the Parkinson's rating video to obtain a human joint point sequence, and calibrating the human joint point sequence to obtain a rating training data set;

[0050] The rating training data set is input into the Parkinson's rating model, and after multiple iterations of training, a trained Parkinson's rating model is obtained.

[0051] In a third aspect, the present invention provides a joint point tracking device, comprising:

[0052] Acquisition module: used to acquire the frame sequence of the video to be tested;

[0053] Extraction module: used to input the current frame of the video to be tested into the human skeleton extraction model, obtain the joint confidence map and the limb affinity vector field, and obtain the two-dimensional joint points of the current frame according to the joint confidence map and the limb affinity vector field;

[0054] Tracking module: for the next frame of the current frame, selects the required joint points as the joint points to be tracked, and tracks the target two-dimensional joint points of the next frame;

[0055] Reverse tracking module: used to use the two-dimensional joint points obtained by tracking the next frame of the video as the joint points to be tracked, reversely track the previous frame of the video, and obtain the two-dimensional joint points obtained by reverse tracking of the previous frame;

[0056] Joint point calculation module: used to calculate the Euclidean distance between the two-dimensional joint points tracked in the previous frame and the actual two-dimensional joint points in the previous frame; if the Euclidean distance is too large, the joint points tracked in the next frame cannot be tracked in reverse to obtain the joint points of the previous frame, and the tracking of the joint points tracked in the next frame is judged to have failed; otherwise, the tracking is successful, and the joint points of the next frame obtained by tracking are used as the actual joint points of the next frame; if the tracking of the joint points of the next frame fails, the intersection method of the previous and next frames is used to superimpose the joint confidence maps of the previous frame and the next frame, and the detection area of ​​the joint points is reduced to obtain the corrected target joint points of the next frame, and the corrected joint points of the next frame are used as the actual joint points of the next frame;

[0057] Jump module: used to jump the sequence number of the current frame to the next frame.

[0058] In a fourth aspect, the present invention provides a Parkinson's disease auxiliary diagnosis device based on precise joint tracking, comprising:

[0059] The device according to the third aspect is used to extract a complete two-dimensional joint point sequence of the video to be tested;

[0060] Joint point sequence module: used to obtain the complete two-dimensional joint point sequence of the extracted video to be tested;

[0061] Output module: used to input the complete two-dimensional joint point sequence extracted from the video to be tested into the Parkinson's rating model, obtain the Parkinson's rating information of the video to be tested, and generate Parkinson's auxiliary diagnosis information.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] 1. The present invention uses a tracking method to obtain a target two-dimensional joint point in the next frame of a human body video; uses a previous and next frame intersection method to obtain a corrected two-dimensional joint point in a frame where the two-dimensional joint point tracking fails, and then uses the corrected two-dimensional joint point as a new two-dimensional joint point to continue tracking; obtains an accurate joint point sequence based on the tracked two-dimensional joint point and the corrected two-dimensional joint point, further improving the prediction accuracy of the joint point sequence, while reducing the amount of calculation and improving the detection speed;

[0064] 2. The present invention improves the detection accuracy of the human skeleton extraction model by applying multi-scale estimation to the human skeleton extraction model without retraining the model;

[0065] 3. The present invention improves the extraction accuracy of joint estimation on the video stream by applying the Lucas–Kanade optical flow method to track the joint points extracted by the human skeleton extraction model;

[0066] 4. The present invention obtains the intersection of the joint confidence map and the limb affinity vector field of two consecutive frames to narrow the estimation range, and corrects the joint points that fail to be tracked by the Lucas–Kanade optical flow method, thereby avoiding the problem of manual correction after tracking failure;

[0067] 5. The present invention uses a Parkinson's rating model that can extract spatial and temporal domain information to rate Parkinson's tremor according to input joint points, thereby avoiding the subjectivity of clinical diagnosis by naked eye observation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 It is a flow chart of the Parkinson's disease auxiliary diagnosis method based on accurate joint tracking of the present invention;

[0069] Figure 2 It is a network structure diagram of a human skeleton extraction model of a Parkinson's disease auxiliary diagnosis method based on accurate joint tracking of the present invention;

[0070] Figure 3 It is a network structure diagram of the Parkinson's rating model of the Parkinson's auxiliary diagnosis method based on precise joint tracking of the present invention. DETAILED DESCRIPTION

[0071] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0072] Embodiment 1:

[0073] This embodiment provides a method for tracking human joints, the method comprising the following steps:

[0074] Step 1: Use the MSCOCO public dataset and the Parkinson's rating video calibration dataset to train the human skeleton extraction model;

[0075] Step 2: Input the first frame of the video to be tested into the human skeleton extraction model, and use the multi-scale fusion confidence map and limb affinity vector field to obtain the precise two-dimensional joint points of the first frame of the video;

[0076] Step 3: For the next frame of the video, the joint points that need to be evaluated for Parkinson's rating are taken as the joint points that need to be tracked, and the Lucas–Kanade optical flow method is used to track the target two-dimensional joint points of the next frame;

[0077] Step 4: Use the two-dimensional joint points obtained by tracking the next frame of the video as the joint points to be tracked, and use the Lucas–Kanade optical flow method to reversely track the previous frame of the video to obtain the two-dimensional joint points obtained by reverse tracking of the previous frame;

[0078] Step 5: Calculate the Euclidean distance between the two-dimensional joint point tracked in the previous frame and the actual two-dimensional joint point in the previous frame;

[0079] Step 6: If the Euclidean distance is too large, the joint points tracked in the next frame cannot be tracked back to the joint points of the previous frame, and the joint points tracked in the next frame are judged to have failed to be tracked, otherwise the tracking is successful;

[0080] Step 7: If the joint point tracking of the next frame fails, the previous and next frame intersection method is used to superimpose the confidence maps of the previous frame and the next frame, narrow the detection area of ​​the joint point, and obtain the corrected target joint point of the next frame;

[0081] Step 8: Use the joint points of the next frame obtained by tracking or the joint points of the next frame obtained by correction as the actual joint points of the next frame;

[0082] Step 9: Repeat steps 2 to 8 to extract a complete two-dimensional joint point sequence of the video to be tested.

[0083] Specifically, the human skeleton extraction model is a human skeleton extraction model based on openpose. First, the human skeleton extraction model is pre-trained based on the COCO dataset to obtain its pre-trained model, wherein the purpose of pre-training is to improve the generalization performance of the human skeleton extraction model;

[0084] The training method of the model is: collect videos of Parkinson's patients, mark the human joints in the videos, obtain a new data set, input the obtained data set into the pre-trained model of the human skeleton extraction model, and train it again, so that the human skeleton extraction model is more suitable for the application scenario of Parkinson's skeleton extraction.

[0085] The specific steps for joint extraction, tracking and correction are:

[0086] Scale the first frame of the video into scales of 0.5, 1, 1.5, and 2, and input them into the trained human skeleton extraction model to obtain multi-scale joint confidence maps and limb affinity vector fields;

[0087] Then, the multi-scale joint confidence map and limb affinity vector field are superimposed and averaged to obtain the multi-scale fused joint confidence map and limb affinity vector field, and obtain a more accurate two-dimensional joint point in the first frame;

[0088] For the second frame of the video, the joint points that need to be evaluated for Parkinson's rating in the previous step are taken as the joint points that need to be tracked, and the Lucas–Kanade optical flow method is used to track the two-dimensional joint points in the second frame of the video;

[0089] In order to verify whether the joint points tracked in the previous step are accurate, the joint points that need to be evaluated for Parkinson's rating in the second frame are taken as the joint points that need to be tracked. The Lucas–Kanade optical flow method is used to reversely track the first frame of the video to obtain a frame of reversely tracked two-dimensional joint points, and then the Euclidean distance between the obtained joint points and the joint points estimated by the model in the first frame is calculated;

[0090] If the Euclidean distance in the previous step is less than the set threshold, the joint points obtained by tracking in the next frame can be reversely tracked to obtain the joint points of the previous frame, and it is determined that the joint points of the second frame are tracked accurately, otherwise the joint points need to be corrected;

[0091] For videos with continuous frames, the joint points of characters are always located in the intersection of the joint confidence maps of two consecutive frames. Therefore, the intersection method of previous and next frames is designed to narrow the detection range of joint points in consecutive frames. To correct the joint points of the second frame, the human skeleton extraction model is first used to extract the joint confidence maps of the first frame and the second frame respectively, and then they are superimposed and averaged to narrow the detection area of ​​the joint points to correct the joint points of the second frame;

[0092] Repeat the above steps to finally obtain the two-dimensional joint point sequence of the video.

[0093] Multi-scale testing is an important idea in the field of object detection, because the size of the input image has a significant impact on the performance of the detection model. In deep learning, a larger convolution window can see larger objects after convolution, otherwise only smaller images can be seen. Taking human pose estimation as an example, when the size of the person to be detected is small, the area of ​​the extracted confidence map and affinity vector field is also small, which will reduce the accuracy of the model prediction to a certain extent. In the field of object detection, the detection of the last layer of the YOLO v5 network uses feature map fusion of multiple scales to improve the detection accuracy of extremely large objects and extremely small objects.

[0094] like Figure 2 As shown, Figure 2It is a network structure diagram of the human skeleton recognition model. The human skeleton recognition model mainly predicts the joint confidence map and the limb affinity vector field. The joint confidence map represents the probability value of a certain joint point appearing at each pixel in the image. It takes a certain joint point as the origin coordinate. The generated joint confidence map will generate a probability area with a Gaussian distribution at the origin. The maximum probability value of the origin is 1. With the origin, that is, the joint point, as the center, the probability value decreases as it moves around. The limb affinity vector field represents the position and direction of the limbs in the image. Its generation method is similar to the joint confidence map. The human skeleton recognition model first uses the first ten layers of the VGG-19 network plus two convolutional layers to extract the feature map F of the image. Then VGG sends the feature map F to the first part of the network to predict the limb affinity vector field. The network module of the first stage of the first part consists of 7 convolution blocks. The first 5 convolution blocks are 3 layers of 3*3 convolutions, each 3*3 convolution is connected in series, and the last two convolution blocks are 1 layer of 1*1 convolutions, and finally the limb affinity vector field L1 of the first stage is obtained. The feature map F of the image and the limb affinity vector field L1 predicted in the first stage constitute the input of the second part of the first stage. After repeating t times, the limb affinity vector field Lt of the first part is finally obtained. Then VGG sends the feature map F and the limb affinity vector field Lt to the second part of the network to predict the joint confidence map. The network structure of each stage of the second part is the same as that of the first part. After repeating t times, the joint confidence map St of the second part is finally obtained. Finally, according to the joint confidence map and the limb affinity vector field, the network outputs a joint point coordinate of the current frame.

[0095] Embodiment 2:

[0096] As attached Figure 1 As shown, the steps of a Parkinson's disease auxiliary diagnosis method based on precise joint tracking provided in this embodiment are as follows:

[0097] The human skeleton extraction model is a human skeleton extraction model based on Openpose. First, the human skeleton extraction model is pre-trained based on the COCO dataset to obtain its pre-trained model, wherein the purpose of pre-training is to improve the generalization performance of the human skeleton extraction model;

[0098] Then, videos of Parkinson's patients are collected, and the human joints in the videos are annotated to obtain a new data set. The obtained data set is input into the pre-trained model of the human skeleton extraction model for re-training, so that the human skeleton extraction model is more suitable for the application scenario of Parkinson's skeleton extraction;

[0099] Next, the first frame of the video is scaled to 0.5, 1, 1.5, and 2, and input into the trained human skeleton extraction model to obtain a multi-scale joint confidence map and limb affinity vector field.

[0100] Then, the multi-scale joint confidence map and limb affinity vector field are superimposed and averaged to obtain the multi-scale fused joint confidence map and limb affinity vector field, and obtain a more accurate two-dimensional joint point in the first frame;

[0101] For the second frame of the video, the joint points that need to be evaluated for Parkinson's rating in the previous step are taken as the joint points that need to be tracked, and the Lucas–Kanade optical flow method is used to track the two-dimensional joint points in the second frame of the video;

[0102] In order to verify whether the joint points tracked in the previous step are accurate, the joint points that need to be evaluated for Parkinson's rating in the second frame are taken as the joint points that need to be tracked. The Lucas–Kanade optical flow method is used to reversely track the first frame of the video to obtain a frame of reversely tracked two-dimensional joint points, and then the Euclidean distance between the obtained joint points and the joint points estimated by the model in the first frame is calculated;

[0103] If the Euclidean distance in the previous step is less than the set threshold, the joint points obtained by tracking in the next frame can be reversely tracked to obtain the joint points of the previous frame, and it is determined that the joint points of the second frame are tracked accurately, otherwise the joint points need to be corrected;

[0104] For videos with continuous frames, the joint points of characters are always located in the intersection of the joint confidence maps of two consecutive frames. Therefore, the intersection method of previous and next frames is designed to narrow the detection range of joint points in consecutive frames. To correct the joint points of the second frame, the human skeleton extraction model is first used to extract the joint confidence maps of the first frame and the second frame respectively, and then they are superimposed and averaged to narrow the detection area of ​​the joint points to correct the joint points of the second frame;

[0105] Repeat the above steps to finally obtain the two-dimensional joint point sequence of the video;

[0106] Use the human skeleton extraction model to extract the human joint sequence from the Parkinson's rating video and calibrate the rating training data set;

[0107] A set of joint point sequences in the rating training data set is input into the convolution layer to extract the spatial domain features of the current frame and the temporal domain features of several consecutive frames. The spatial features and temporal features are input into the convolution layer to train and obtain the Parkinson's rating model.

[0108] The video of the Parkinson's patient to be tested is input into the human skeleton extraction model to obtain a two-dimensional joint point sequence, and the two-dimensional joint point sequence is input into the Parkinson's rating model to finally obtain the rating of the Parkinson's patient.

[0109] Specifically, the input of the Parkinson's rating model is a two-dimensional joint point sequence, and the output is the Parkinson's tremor grade;

[0110] The structure of the Parkinson's rating model is based on the action classification network DD-Net, and its network structure includes:

[0111] The first branch: includes three convolutional layers, which are used to predict the relationship between the skeleton points in the current frame and capture the spatial domain features of the image;

[0112] The second branch consists of three convolutional layers, which are used to calculate the differential data between joint points at intervals of 1 frame and learn the fast temporal information of different frames.

[0113] The third branch consists of three convolutional layers, which are used to calculate the differential data between joint points at an interval of 2 frames and learn the slow timing information of different frames;

[0114] Classification module: It consists of three convolutional layers and two fully connected layers, which are used to fuse the features extracted by the three branches to obtain the tremor grade of Parkinson's disease.

[0115] The training method of the Parkinson's rating model is:

[0116] Using a human skeleton extraction model to extract the Parkinson's rating video to obtain a human joint point sequence, and calibrating the human joint point sequence to obtain a rating training data set;

[0117] A set of joint point sequences in the training data set is input into the Parkinson's rating model, and the spatial domain features of the current frame and the temporal domain features of several consecutive frames are extracted through the three branches in the Parkinson's rating model;

[0118] The above spatial features and temporal features are input into the classification module to obtain the Parkinson's tremor rating;

[0119] After multiple iterations of training, a trained Parkinson's rating model is obtained.

[0120] In computer vision, optical flow is a motion pattern that refers to the apparent movement of an object, surface, or edge between an observer (such as the eye, camera, etc.) and the background from a single perspective. The ideal output of an optical flow algorithm is an estimated correlation of the velocity of each pixel in the two frames, or equivalently, a displacement vector for each pixel in one image, indicating the relative position of the pixel in the other image. If this method is used for every pixel in the image, it is usually called "dense optical flow". There is also an algorithm called "sparse optical flow" that only tracks a subset of certain points in the image. The Lucas–Kanade optical flow algorithm is a sparse optical flow estimation algorithm for two-frame difference, commonly used for point tracking.

[0121] Multi-scale testing is an important idea in the field of object detection, because the size of the input image has a significant impact on the performance of the detection model. In deep learning, a larger convolution window can see larger objects after convolution, otherwise only smaller images can be seen. Taking human pose estimation as an example, when the size of the person to be detected is small, the area of ​​the extracted confidence map and affinity vector field is also small, which will reduce the accuracy of the model prediction to a certain extent. In the field of object detection, the detection of the last layer of the YOLO v5 network uses feature map fusion of multiple scales to improve the detection accuracy of extremely large objects and extremely small objects.

[0122] like Figure 2 As shown, Figure 2 It is a network structure diagram of the human skeleton recognition model. The human skeleton recognition model mainly predicts the joint confidence map and the limb affinity vector field. The joint confidence map represents the probability value of a certain joint point appearing at each pixel in the image. It takes a certain joint point as the origin coordinate. The generated joint confidence map will generate a probability area with a Gaussian distribution at the origin. The maximum probability value of the origin is 1. With the origin, that is, the joint point, as the center, the probability value decreases as it moves around. The limb affinity vector field represents the position and direction of the limbs in the image. Its generation method is similar to the joint confidence map. The human skeleton recognition model first uses the first ten layers of the VGG-19 network plus two convolutional layers to extract the feature map F of the image. Then VGG sends the feature map F to the first part of the network to predict the limb affinity vector field. The network module of the first stage of the first part consists of 7 convolution blocks. The first 5 convolution blocks are 3 layers of 3*3 convolutions, each 3*3 convolution is connected in series, and the last two convolution blocks are 1 layer of 1*1 convolutions, and finally the limb affinity vector field L1 of the first stage is obtained. The feature map F of the image and the limb affinity vector field L1 predicted in the first stage constitute the input of the second part of the first stage. After repeating t times, the limb affinity vector field Lt of the first part is finally obtained. Then VGG sends the feature map F and the limb affinity vector field Lt to the second part of the network to predict the joint confidence map. The network structure of each stage of the second part is the same as that of the first part. After repeating t times, the joint confidence map St of the second part is finally obtained. Finally, according to the joint confidence map and the limb affinity vector field, the network outputs a joint point coordinate of the current frame.

[0123] The image is then scaled to scales of 0.5, 1, 1.5, and 2, and the images of different scales are input into the human skeleton recognition model to obtain joint confidence maps and limb affinity vector fields of corresponding scales. The fusion of these joint confidence maps and limb affinity vector fields can improve the prediction accuracy of joint points and extract more accurate joint point coordinates.

[0124] In order to effectively track the coordinates of joint points, a joint tracking module is designed. The tracking module mainly uses the Lucas–Kanade optical flow method, a sparse tracking method based on feature points. It is an optical flow estimation algorithm based on two-frame differences, and its basic idea is based on the following three assumptions.

[0125] Brightness constancy: Pixels of the target image in the scene appear to remain unchanged from frame to frame.

[0126] Temporal persistence (small movement): The movement of the camera on the image changes slowly over time.

[0127] Spatial consistency: neighboring points on the same surface in the scene have similar motions and their projections onto the image plane are close.

[0128] The Lucas–Kanade optical flow method can be used to effectively track feature points such as joints, but when the features of two consecutive frames do not meet the above assumptions, tracking will fail. To determine whether the tracking of joints has failed, reverse tracking can be used to reversely track the joints of the previous frame using the joints tracked in the next frame. The joints tracked in the previous frame are compared with the joints of the previous frame itself to determine whether the tracking of the joints of the next frame is successful. Therefore, it is only necessary to calculate the Euclidean distance between the joints of the previous frame obtained by reverse tracking and the actual coordinates of the joints of the previous frame. If the Euclidean distance is too large, the joints tracked in the next frame cannot be reversely tracked to obtain the joints of the previous frame, and it is determined that the tracking of the joints of the next frame has failed.

[0129] In order to compensate for occasional tracking failures, a joint correction module was designed, which uses the previous and next frame intersection method to promptly correct the joint points that failed to be tracked. Since the tremor of Parkinson's patients does not have a large range of motion, the joint points are often located in the intersection of two consecutive frames of joint confidence maps. The previous and next frame intersection method in the joint correction module takes advantage of this feature, narrows the estimated range of the joint points, and thus corrects the joints that failed to be tracked. Finally, through accurate tracking of the joints, the accurate two-dimensional joint point sequence of Parkinson's patients can be extracted from the video of Parkinson's patients.

[0130] like Figure 3 As shown, Figure 3 This is the network structure diagram of the Parkinson's rating model. The network has three input data flow branches. The first branch predicts the relationship between the skeleton points on each frame of the picture. This information is used to capture the static information on the picture. The second and third branches can learn the fast and slow "optical flow" information of different frames by spacing different frames. This information is used to capture the timing information of the action. Finally, the features obtained by these three branches are fused, and the joint point sequence is input into the network to obtain the Parkinson's rating result.

[0131] Embodiment three:

[0132] This embodiment provides a joint point tracking device, including:

[0133] Acquisition module: used to acquire the frame sequence of the video to be tested;

[0134] Extraction module: used for inputting the current frame of the video to be tested into the human skeleton extraction model, obtaining the joint confidence map and the limb affinity vector field, and obtaining the two-dimensional joint points of the current frame according to the joint confidence map and the limb affinity vector field;

[0135] Tracking module: for the next frame of the current frame, selects the required joint points as the joint points to be tracked, and tracks the target two-dimensional joint points of the next frame;

[0136] Reverse tracking module: used to use the two-dimensional joint points obtained by tracking the next frame of the video as the joint points to be tracked, reversely track the previous frame of the video, and obtain the two-dimensional joint points obtained by reverse tracking of the previous frame;

[0137] Joint point calculation module: used to calculate the Euclidean distance between the two-dimensional joint points tracked in the previous frame and the actual two-dimensional joint points in the previous frame; if the Euclidean distance is too large, the joint points tracked in the next frame cannot be tracked in reverse to obtain the joint points of the previous frame, and the tracking of the joint points tracked in the next frame is judged to have failed; otherwise, the tracking is successful, and the joint points of the next frame obtained by tracking are used as the actual joint points of the next frame; if the tracking of the joint points of the next frame fails, the intersection method of the previous and next frames is used to superimpose the joint confidence maps of the previous frame and the next frame, and the detection area of ​​the joint points is reduced to obtain the corrected target joint points of the next frame, and the corrected joint points of the next frame are used as the actual joint points of the next frame;

[0138] Jump module: used to jump the sequence number of the current frame to the next frame;

[0139] Output sequence module: used to output the complete two-dimensional joint point sequence of the video to be tested.

[0140] The device of this embodiment can be used to implement the method of the first embodiment.

[0141] Embodiment 4:

[0142] This embodiment provides a Parkinson's disease auxiliary diagnosis device based on precise joint tracking, comprising:

[0143] Acquisition module: used to acquire the frame sequence of the video to be tested;

[0144] Extraction module: used for inputting the current frame of the video to be tested into the human skeleton extraction model, obtaining the joint confidence map and the limb affinity vector field, and obtaining the two-dimensional joint points of the current frame according to the joint confidence map and the limb affinity vector field;

[0145] Tracking module: for the next frame of the current frame, selects the required joint points as the joint points to be tracked, and tracks the target two-dimensional joint points of the next frame;

[0146] Reverse tracking module: used to use the two-dimensional joint points obtained by tracking the next frame of the video as the joint points to be tracked, reversely track the previous frame of the video, and obtain the two-dimensional joint points obtained by reverse tracking of the previous frame;

[0147] Joint point calculation module: used to calculate the Euclidean distance between the two-dimensional joint points tracked in the previous frame and the actual two-dimensional joint points in the previous frame; if the Euclidean distance is too large, the joint points tracked in the next frame cannot be tracked in reverse to obtain the joint points of the previous frame, and the tracking of the joint points tracked in the next frame is judged to have failed; otherwise, the tracking is successful, and the joint points of the next frame obtained by tracking are used as the actual joint points of the next frame; if the tracking of the joint points of the next frame fails, the intersection method of the previous and next frames is used to superimpose the joint confidence maps of the previous frame and the next frame, and the detection area of ​​the joint points is reduced to obtain the corrected target joint points of the next frame, and the corrected joint points of the next frame are used as the actual joint points of the next frame;

[0148] Jump module: used to jump the sequence number of the current frame to the next frame;

[0149] Output sequence module: used to output the complete two-dimensional joint point sequence of the video to be tested.

[0150] Joint point sequence module: used to obtain the complete two-dimensional joint point sequence of the extracted video to be tested;

[0151] Output module: used to input the complete two-dimensional joint point sequence extracted from the video to be tested into the Parkinson's rating model, obtain the Parkinson's rating information of the video to be tested, and generate Parkinson's auxiliary diagnosis information.

[0152] This embodiment can be used to implement the method described in the second embodiment.

[0153] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0154] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0155] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0156] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0157] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A joint point tracking method, It is characterized in that The method comprises the following steps: Step A: Obtain a frame sequence of the video to be tested; Step B: inputting the current frame of the video to be tested into the human skeleton extraction model to obtain a joint confidence map and a limb affinity vector field, and obtaining the two-dimensional joint points of the current frame according to the joint confidence map and the limb affinity vector field; Step C: For the next frame of the current frame, select the required joint points as the joint points to be tracked, and track the target two-dimensional joint points of the next frame; Step D: Using the two-dimensional joint points obtained by tracking the next frame of the video as the joint points to be tracked, and reversely tracking the previous frame of the video to obtain the two-dimensional joint points obtained by reverse tracking of the previous frame; Step E: Calculate the Euclidean distance between the two-dimensional joint point tracked in the previous frame and the actual two-dimensional joint point in the previous frame; Step R: If the Euclidean distance is too large, the joint points tracked in the next frame cannot be tracked back to the joint points of the previous frame, and the tracking of the joint points tracked in the next frame is judged to have failed. Otherwise, the tracking is successful, and the joint points of the next frame tracked are used as the actual joint points of the next frame. Step G: If the joint point tracking of the next frame fails, the previous and next frame intersection method is used to superimpose the joint confidence maps of the previous frame and the next frame, reduce the detection area of ​​the joint point, obtain the corrected target joint point of the next frame, and use the corrected joint point of the next frame as the actual joint point of the next frame; Step H: Jump the sequence number of the current frame to the next frame; Step I: Repeat steps B to H to extract a complete two-dimensional joint point sequence of the video to be tested.

2. The joint point tracking method according to claim 1, It is characterized in that In the Parkinson's auxiliary diagnosis method based on joint precision tracking of the present invention, the human skeleton extraction model is based on Openpose, and its structure includes: a feature extraction module, a limb affinity vector field estimation module, and a joint confidence map estimation module; the feature extraction module includes the first ten layers and two convolutional layers of the VGG-19 network; The method for obtaining the human skeleton extraction model comprises: The training dataset is calibrated based on the MSCOCO dataset and the Parkinson's rating video data; The training data set is input into the human skeleton extraction model, and the final human skeleton extraction model is obtained after multiple iterations and training.

3. The joint point tracking method according to claim 1, It is characterized in that The method of inputting the current frame of the video to be tested into the human skeleton extraction model to obtain the joint confidence map and the limb affinity vector field includes: Use multi-scale fusion estimation to scale the current frame proportionally to obtain images of different sizes; Input images of different sizes into the human skeleton extraction model to obtain joint confidence maps of images of different sizes; The joint confidence maps of images of different sizes are superimposed to obtain the average and the output joint confidence map is obtained.

4. The joint point tracking method according to claim 3, It is characterized in that The method of inputting the image into the human skeleton extraction model to obtain the joint confidence map and the limb affinity vector field includes: Step 1: Pass a frame of image into the first 10 layers of the VGG-19 network to obtain the feature map of the first stage; Step 2: The feature map is passed through a convolutional layer to obtain the limb affinity vector field of the first stage; Step 3: Input the limb affinity vector field and feature map of the first stage into the network, and obtain the limb affinity vector field of the second stage through the convolution layer; Step 4: Repeat steps 2 and 3 to a specified number of times to obtain the final limb affinity vector field; Step 5: Input the final limb affinity vector field and feature map into the network, and obtain the joint confidence map of the first stage through the convolution layer; Step 6: Input the joint confidence map and feature map of the first stage into the network, and obtain the joint confidence map of the second stage through the convolution layer; Step 7: Repeat steps 5 and 6 for a predetermined number of times to obtain the output joint confidence map.

5. The joint point tracking method according to claim 1, It is characterized in that The method for tracking the target two-dimensional joint points of the next frame is the Lucas–Kanade optical flow method; the method for reverse tracking the previous frame of the video is the Lucas–Kanade optical flow method.

6. The joint point tracking method according to claim 1, It is characterized in that Using the previous and next frame intersection method, the joint confidence maps of the previous frame and the next frame are superimposed to reduce the detection area of ​​the joint point, and the method of obtaining the corrected target joint point of the next frame includes: By superimposing the confidence maps of adjacent frames and finding their intersection, the corrected target joint points of the next frame are obtained.

7. A Parkinson's disease auxiliary diagnosis method based on precise joint tracking, It is characterized in that The joint tracking method according to any one of claims 1 to 6 comprises the following steps: Obtain a complete two-dimensional joint point sequence of the extracted video to be tested; The complete two-dimensional joint point sequence of the extracted video to be tested is input into the Parkinson's rating model to obtain the Parkinson's rating information of the video to be tested, and generate Parkinson's auxiliary diagnosis information.

8. The Parkinson's disease auxiliary diagnosis method according to claim 7, It is characterized in that The input of the Parkinson's rating model is a two-dimensional joint point sequence, and the output is the Parkinson's tremor grade; The structure of the Parkinson's rating model is based on the action classification network DD-Net, and its network structure includes: The first branch: includes three convolutional layers, which are used to predict the relationship between the skeleton points in the current frame and capture the spatial domain features of the image; The second branch includes three convolutional layers, which are used to calculate the differential data between joint points at intervals of 1 frame and learn the fast temporal information of different frames. The third branch: includes three convolutional layers, which are used to calculate the differential data between joint points with an interval of 2 frames and learn the slow timing information of different frames; Classification module: includes three convolutional layers and two fully connected layers, which are used to fuse the features extracted by the three branches to obtain the tremor grade of Parkinson's disease; The Parkinson's rating model is obtained as follows: Using a human skeleton extraction model to extract the Parkinson's rating video to obtain a human joint point sequence, and calibrating the human joint point sequence to obtain a rating training data set; The rating training data set is input into the Parkinson's rating model, and after multiple iterations of training, a trained Parkinson's rating model is obtained.

9. A joint point tracking device, It is characterized in that include: Acquisition module: used to acquire the frame sequence of the video to be tested; Extraction module: used for inputting the current frame of the video to be tested into the human skeleton extraction model, obtaining the joint confidence map and the limb affinity vector field, and obtaining the two-dimensional joint points of the current frame according to the joint confidence map and the limb affinity vector field; Tracking module: for the next frame of the current frame, selects the required joint points as the joint points to be tracked, and tracks the target two-dimensional joint points of the next frame; Reverse tracking module: used to use the two-dimensional joint points obtained by tracking the next frame of the video as the joint points to be tracked, reversely track the previous frame of the video, and obtain the two-dimensional joint points obtained by reverse tracking of the previous frame; Joint point calculation module: used to calculate the Euclidean distance between the two-dimensional joint points tracked in the previous frame and the actual two-dimensional joint points in the previous frame; if the Euclidean distance is too large, the joint points tracked in the next frame cannot be tracked back to get the joint points of the previous frame, and the tracking of the joint points tracked in the next frame is judged to have failed; otherwise, the tracking is successful, and the joint points of the next frame tracked are used as the actual joint points of the next frame; If the joint point tracking of the next frame fails, the previous and next frame intersection method is used to superimpose the joint confidence maps of the previous frame and the next frame, reduce the detection area of ​​the joint point, and obtain the corrected target joint point of the next frame. The corrected joint point of the next frame is used as the actual joint point of the next frame. Jump module: used to jump the sequence number of the current frame to the next frame.

10. A Parkinson's disease auxiliary diagnosis device based on precise joint tracking. It is characterized in that include: The device as claimed in claim 9: used to extract a complete two-dimensional joint point sequence of the video to be tested; Joint point sequence module: used to obtain the complete two-dimensional joint point sequence of the extracted video to be tested; Output module: used to input the complete two-dimensional joint point sequence extracted from the video to be tested into the Parkinson's rating model, obtain the Parkinson's rating information of the video to be tested, and generate Parkinson's auxiliary diagnosis information.

Citation Information

Patent Citations

  • Human motion trail analysis method based on articulation point extraction

    CN110458046A

  • Embedded platform real-time tumble detection method based on improved attitude estimation algorithm

    CN111274954A

  • Complex scene-based human body key point detection system and method

    WO2019232894A1

  • Human body action recognition method, human body action recognition system, and device

    WO2022000420A1