Artificial intelligence system and method for diagnosing parkinson's disease

Abstract

The present invention relates to an artificial intelligence system and a method for diagnosing Parkinson's disease. A method for processing data performed by a processor in an apparatus of the present invention may comprise the steps of: receiving joint point information of a target human body from a joint model; receiving a target image and correcting the joint point information to correspond to the size of the human body in the target image; calculating motion vectors using the corrected joint point information for a plurality of main points on the target image in response to movement in the target image during a plurality of frames, and estimating a Parkinson's disease diagnosis score through a combination of the motion vectors; and evaluating the presence of Parkinson's disease from the estimated diagnosis score.

Description

Artificial intelligence system and method for diagnosing Parkinson's disease

[0001] The present invention relates to an artificial intelligence system for diagnosing Parkinson's disease, which objectively quantifies movement disorder based on the Parkinson's Disease Rating Scale (MDS-UPDRS) through digital processing using image analysis, and a data processing method for the diagnosis.

[0002] Typically, the diagnostic approach for Parkinson's disease relies on the subjective clinical scale, the Parkinson's Disease Rating Scale (MDS-UPDRS), supported by the Movement Disorder Society of America, rather than an objective, quantitative diagnostic method. MDS-UPDRS stands for the MDS-sponsored revision of the UPDRS, MDS for the International Parkinson and Movement Disorder Society, and UPDRS for the Unified Parkinson's Disease Rating Scale.

[0003] In order to determine the severity and progression of Parkinson's disease, the values ​​of this clinical scale are determined as scores through a clinician's questioning and observation, but, for example, in finger tapping, the criteria for determining the score of MDS-UPDRS are not objective, so both the intra-rater reliability of the individual clinician's judgment and the inter-rater reliability of the judgment between clinicians are low, which leads to low accuracy of diagnosis and low efficacy of medication through this, and there is a serious problem that repeated trial and error is required to determine the appropriate treatment for the patient's condition.

[0004] Recent studies have attempted to quantify motor function tests based on the MDS-UPDRS. For example, studies have attempted to quantify finger tapping using a computer-interfaced music keyboard and accelerometers, and studies have attempted to quantify gait using a foot-scan system. However, these methods require additional equipment and still fail to provide objective and quantitative diagnostic results.

[0005] Therefore, there is a need for a method that can provide reliable, objective, and quantifiable diagnostic results for the clinical characteristics of slowness in Parkinson's disease patients without additional devices.

[0006] As prior literature, reference may be made to Patent Application No. 10-2009-0069686 (July 29, 2009), Patent Application No. 10-2011-0092583 (September 14, 2011), etc.

[0007] Accordingly, the present invention has been devised to solve the above-described problem, and the purpose of the present invention is to provide a system and method for objectively and quantitatively evaluating the Parkinson's Disease Rating Scale (UPDRS) through analysis of finger tapping, facial expression, gait, and voice, etc., among movement disorder tests based on the Parkinson's Disease Rating Scale (MDS-UPDRS) supported by the Movement Disorder Society of America, using a posture estimation deep learning model, thereby enabling diagnosis and severity evaluation of Parkinson's disease.

[0008] First, to summarize the features of the present invention, a method for data processing performed by a processor in a device according to one aspect of the present invention for achieving the above object includes the steps of: receiving joint point information of a target body from a joint model; receiving a target image and correcting the joint point information so that it corresponds to the size of the body of the target image; calculating motion vectors using the corrected joint point information for a plurality of key points on the target image in response to movement of the target image for a plurality of frames and estimating a Parkinson's disease diagnostic score by a combination of the motion vectors; and evaluating whether or not Parkinson's disease is present based on the estimated diagnostic score.

[0009] The above joint model is an artificial intelligence deep learning model that operates a library for providing the joint point position vectors, and is a machine-learned model of joint point position vectors for a plurality of human bodies, and can provide the joint point information upon request from the device.

[0010] The step of estimating the Parkinson's disease diagnosis score may include a step of determining the Parkinson's disease diagnosis score for finger tapping by tracking vector changes in the movement of major points of the thumb and index finger of an image including fingers for a predetermined period of time, and determining whether tapping is performed from changes in the distance between the points at the ends of the thumb and index finger, and calculating the number of taps, the time interval between taps, the tapping speed, the consistency of the change in the tapping angle, and the number of tapping hesitations.

[0011] The Parkinson's disease diagnostic score for the finger tapping can be determined by further reflecting the degree of symmetry of the left and right finger tapping.

[0012] The step of estimating the Parkinson's disease diagnosis score may include a step of tracking vector changes in the movement of major points within facial landmarks including the eyes and the area around the eyes and the lips and the area around the lips of an image including a face for a predetermined period of time, calculating the distance between the upper and lower eyelids and the distance between the upper and lower lips, and determining the Parkinson's disease diagnosis score for facial expression from the blinking frequency and changes therein and the lip opening frequency and changes therein.

[0013] The step of estimating the Parkinson's disease diagnosis score may include a step of tracking vector changes in the movement of major points of the feet and arms of a person in an image including a walking person for a predetermined period of time, calculating the stride length and arm rotation angle over time, and determining the Parkinson's disease diagnosis score for the walking posture (Gait) from the stride length change, walking speed, and continuity of walking.

[0014] The step of estimating the above Parkinson's disease diagnosis score may include a step of tracking vector changes in the movement of major points of the arms and legs of a person in an image including a non-walking person for a predetermined period of time, calculating the magnitude of the movement of the arms and the magnitude of the movement of the legs over time, and determining the Parkinson's disease diagnosis score for tremor from the magnitude and duration of arm tremor, the magnitude and duration of leg tremor, the magnitude and duration of lip tremor, and the magnitude and duration of jaw tremor.

[0015] The step of estimating the Parkinson's disease diagnosis score may include a step of tracking changes in the movement of major points of frequency and amplitude in a feature map for a Mel spectrogram image converted from a voice file over a predetermined period of time, calculating changes in frequency and amplitude over time, and comparing the changes with a change profile of frequency and amplitude of a Parkinson's disease patient, thereby determining the Parkinson's disease diagnosis score for speech.

[0016] And, in another aspect of the present invention, a recording medium having recorded thereon a computer-readable code for performing a function for data processing performed by a processor in a device may implement a function of receiving joint point information of a target body from a joint model; a function of receiving a target image and correcting the joint point information so that it corresponds to the size of the body of the target image; a function of calculating motion vectors using the corrected joint point information for a plurality of key points on the target image in response to movement of the target image for a plurality of frames and estimating a Parkinson's disease diagnostic score by a combination of the motion vectors; and a function of evaluating whether or not Parkinson's disease is present based on the estimated diagnostic score.

[0017] According to the artificial intelligence system and method for diagnosing Parkinson's disease according to the present invention, two-dimensional or three-dimensional coordinates are extracted from key points of the human body using various posture estimation deep learning models such as MediaPipe, OpenPose, AlphaPose, DeepCut, DeepPose, and ViTPose, and the Parkinson's disease rating scale (UPDRS) is objectively and quantitatively evaluated through analysis of finger tapping, facial expression, gait, and voice, thereby enabling diagnosis and severity evaluation of Parkinson's disease and can be used to assist diagnosis.

[0018] The accompanying drawings, which are included as part of the detailed description to aid understanding of the present invention, provide examples of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0019] FIG. 1 is a diagram illustrating an artificial intelligence system for diagnosing Parkinson's disease according to one embodiment of the present invention.

[0020] Figure 2a is an example of a target image containing a finger.

[0021] Figure 2b is a diagram for explaining the calculation of motion vectors for joint point information and key points of a target image including fingers.

[0022] Figure 3 shows examples of target images containing faces.

[0023] Figures 4a and 4b are examples of target images including a walking person.

[0024] Figure 5 is an example of a target image and joint point information including a non-walking person.

[0025] Figure 6 is an example of a Mel Spectrogram target image converted into an audio file of the image.

[0026] Figure 7 is an example of a Parkinson's disease diagnosis device of the present invention being implemented in a user terminal and showing the diagnosis process and results on a screen.

[0027] FIG. 8 is a drawing for explaining an example of a method for implementing an artificial intelligence system for diagnosing Parkinson's disease according to one embodiment of the present invention.

[0028] Hereinafter, the present invention will be described in detail with reference to the attached drawings. In this case, the same components are indicated by the same reference numerals in each drawing, where possible. In addition, detailed descriptions of functions and / or configurations already known will be omitted. The content disclosed below focuses on parts necessary for understanding the operation according to various embodiments, and descriptions of elements that may obscure the gist of the description will be omitted. In addition, some components in the drawings may be exaggerated, omitted, or schematically illustrated. The size of each component does not entirely reflect the actual size, and therefore, the contents described herein are not limited by the relative sizes or spacing of components drawn in each drawing.

[0029] In describing embodiments of the present invention, if a detailed description of a known technology related to the present invention is judged to unnecessarily obscure the gist of the present invention, the detailed description will be omitted. In addition, the terms described below are terms defined in consideration of their functions in the present invention, and this may vary depending on the intention or custom of the user or operator. Therefore, the definitions should be made based on the contents throughout this specification. The terminology used in the detailed description is only for the purpose of describing embodiments of the present invention and should never be limited. Unless clearly used otherwise, the singular form includes the plural form. In this description, expressions such as "comprises" or "having" are intended to indicate certain features, numbers, steps, operations, elements, parts or combinations thereof, and should not be construed to exclude the presence or possibility of one or more other features, numbers, steps, operations, elements, parts or combinations thereof other than those described.

[0030] Additionally, although terms such as first, second, etc. may be used to describe various components, the components are not limited by the terms, and the terms are used only for the purpose of distinguishing one component from another.

[0031] FIG. 1 is a drawing for explaining an artificial intelligence system (Parkinson's disease diagnosis device) (100) for diagnosing Parkinson's disease according to one embodiment of the present invention.

[0032] Referring to FIG. 1, a Parkinson's disease diagnosis device (100) according to one embodiment of the present invention may include a joint point providing unit (111), a joint information providing unit (110), a Parkinson's disease estimation unit (120), a voice image obtaining unit (130), and an evaluation unit (140). The estimation unit (120) may include a finger analysis unit (121) for a 2D / 3D camera image, an expression analysis unit (122), a gait analysis unit (123), a tremor analysis unit (124), and a voice analysis unit (125) for a Mel Spectrogram image converted from a voice file.

[0033] Each of the joint point providing unit (111), the joint information providing unit (110), the Parkinson's disease estimation unit (120), the audio image obtaining unit (130), and the evaluation unit (140) constituting the Parkinson's disease diagnosis device (100) according to one embodiment of the present invention may be implemented by hardware such as a semiconductor processor, software such as an application program, or a combination thereof, and may also include a memory for storing data or setting information necessary for performing the image processing process of the present invention (see FIG. 8). Here, the functions of each of the joint point providing unit (111), the joint information providing unit (110), the Parkinson's disease estimation unit (120), the audio image obtaining unit (130), and the evaluation unit (140) will be described, but two or more of these components may also be combined to be implemented as one block.

[0034] Hereinafter, each function of the joint point provision unit (111), joint information provision unit (110), Parkinson's disease estimation unit (120), audio image acquisition unit (130), and evaluation unit (140) performed in the processor of the Parkinson's disease diagnosis device (100) will be described in detail.

[0035] For example, as shown in FIG. 7, the Parkinson's disease diagnosis device (100) may include a display device such as an LCD or LED for displaying the diagnosis process and its results. The Parkinson's disease diagnosis device (100) may be implemented to display the diagnosis process and its results of the Parkinson's disease diagnosis device (100) for a subject on a display screen of a user terminal such as a smart phone, a laptop PC, or a desktop PC as shown in FIG. 7, and as further described in FIG. 8, may be formed of hardware, software, or a combination thereof.

[0036] First, the joint point providing unit (111) provides joint point information of the target body (fingers, face, walking, arms / legs, etc.) from the joint model to the joint information providing unit (110). When the joint information providing unit (110) receives joint point information of the target body (fingers, face, walking, arms / legs, etc.) from the joint point providing unit (111), it receives a target image (fingers image, face image, walking image, arms / legs image, etc.) of the diagnosis subject and can correct the joint point information so that it corresponds to the size of the body of the target image.

[0037] The joint information providing unit (110) can receive joint point information including joint point position vectors of the target body from the joint model of the joint point providing unit (111). The joint information providing unit (110) can make a request to the joint point providing unit (111) whenever necessary, and accordingly, the joint point providing unit (111) can provide joint point position vectors of the target body from the joint model of the human body. The joint point position vectors may be information on vectors on a three-dimensional rectangular coordinate system for display on a display.

[0038] The joint point providing unit (111) is an artificial intelligence deep learning joint model, and can operate an artificial intelligence deep learning model that machine learns joint point position vectors for all or part of a human body, such as a hand, an arm / leg, a face, a body (or a torso). The joint point providing unit (111) can operate a library for providing the joint point position vectors, and can hold the learned artificial intelligence deep learning model by machine learning the artificial intelligence deep learning model for joint point position vectors for a plurality of human bodies. The joint point providing unit (111) can provide joint point position vectors for a target body, such as all or part of a human body, such as a hand (or palm, including an arm), a foot (or including a leg), a face, a body (or a torso), etc., through the learned artificial intelligence deep learning model, at the request of the joint information providing unit (110).

[0039] The joint point providing unit (111) may be operated within the Parkinson's disease diagnosis device (100) or may be a system operating on the Internet. For a system operating on the Internet, joint point position vectors may be provided by requesting and receiving information using a data transmission / reception interface such as an open API (Application Program Interface). For example, in the case where the learned artificial intelligence deep learning model (e.g., Mediapipe hand model, Mediapipe Face model, etc.) is operated using a Python library called Mediapipe on an Internet server (e.g., Google), the joint point providing unit (111) may request and receive joint point position vectors for the entire or a portion of the target body, such as the hand (or palm, including the arm), the foot (or including the leg), the face, the body (or torso), etc.

[0040] Meanwhile, when the joint information provision unit (110) receives a target image (finger image, face image, gait image, arm / leg image, etc.) of a diagnosis subject, it can correct the received joint point information so that it corresponds to the size of the target body of the target image (e.g., finger), as shown in FIG. 2a.

[0041] That is, when the joint point position vectors provided from the joint point providing unit (111) are given as indicating joint points of a hand, for example, as in FIG. 2A, the size of the corresponding target body (e.g., hand) by the joint point position vectors displayed on the display may be different from the size of the target body (e.g., hand) of the target image displayed on the display, and therefore, in order to correct this, the joint information providing unit (110) may transform the joint point position vectors (e.g., transform the movement of coordinates) to fit the size of the target body (fingers, face, walking, arms / legs, etc.) of the target image. Here, the transformation of joint point position vectors for the hand is exemplified, but is not limited thereto, and in a similar manner as above, for joint point position vectors for the entire or a part of the target body (fingers, face, walking, arms / legs, etc.) corresponding to the target image (finger image, face image, walking image, arm / leg image, etc.), the joint point information can be corrected by transforming the joint point position vectors (e.g., translation transformation of coordinates) to fit the size of the target body of the target image.

[0042] Meanwhile, after the joint information providing unit (110) corrects the received joint point information, the estimation unit (120) can, in response to the movement (e.g., thumb / index finger movement, change in facial expression, gait movement, arm / leg movement) of the target image (finger image, face image, gait image, arm / leg image, etc.) for a plurality of frames, calculate motion vectors using the corrected joint point information for a plurality of major (joint) points on the target image, and estimate a Parkinson's disease diagnosis score by combining the motion vectors. The evaluation unit (140) can evaluate whether or not Parkinson's disease is present based on the estimated diagnostic score.

[0043] The key points on the target image may include, as described below, key points of the thumb and index finger of an image containing fingers (see FIGS. 2a and 2b), key points within a facial landmark including the eyes and around the eyes and the lips and around the lips of an image containing a face (see FIG. 3), key points of the feet and arms of a person of an image containing a walking person (see FIGS. 4a and 4b), key points of the arms and legs of a person of an image containing a non-walking person (see FIG. 5), and key points of frequency and amplitude in a feature map for a Mel Spectrogram image converted from a voice file (see FIG. 6).

[0044] Hereinafter, the process of estimating a Parkinson's disease diagnosis score using key points on the target image in the estimation unit (120) will be described in detail with reference to FIGS. 2a to 6.

[0045] Figure 2a is an example of a target image containing a finger.

[0046] Figure 2b is a diagram for explaining the calculation of motion vectors for joint point information and key points of a target image including fingers.

[0047] Referring to FIGS. 2A and 2B, the finger analysis unit (121) of the estimation unit (120) tracks the vector change in the movement of the main (joint) points (e.g., 4 and 8) of the thumb and index finger of the image including the fingers for a predetermined period of time (e.g., 1 minute, 2 minutes, etc.), determines whether tapping occurs from the change in the distance between the tip points (e.g., 4 and 8) of the thumb and index finger, and calculates the number of taps, the time interval between taps, the tapping speed, the amount of change in the tapping angle, and the number of tapping hesitations to determine the Parkinson's disease diagnosis score for finger tapping. The image including the fingers may be a 2D / 3D video captured in real time by a camera, or a 2D / 3D video captured in advance and stored in memory.

[0048] For example, referring to the Parkinson's Disease Assessment Scale (MDS-UPDRS) (see MDS-UPDRS, Section 3.4), the hand of the diagnosis subject / patient is videotaped and the thumb and index finger are spread and tapped 10 times, and the subject is asked to do so as quickly and as broadly as possible. At this time, in order to determine the Parkinson's disease diagnosis score for finger tapping, the finger analysis unit (121) determines whether tapping occurs from the change in the distance between the tip points of the thumb and index finger (e.g., 4 and 8), and calculates the number of taps, the time interval between taps, the tapping speed, the constancy of the tapping angle change (the constancy of the angle change over time), and the number of tapping hesitations (e.g., which serves as the basis for determining whether the movement is continued).

[0049] For example, in calculating the angle between two fingers based on the change in key points (e.g., 4 and 8), the change in the width of the fingers' spread can be calculated consistently regardless of the distance between the camera and the hand or the size of the hand. In order to calculate the angle (θ) formed by key points (e.g., 4, 5, and 8), the vectors for points 4 and 5 and the vectors for points 5 and 8 are calculated, and the angle (θ) between the two vectors can be calculated using the inner product.

[0050] Finger tapping, or tapping, can be determined as a tap when the distance between the two fingertips becomes closer than a certain value (which can be optimized through experiments) or when the angle formed by the two fingers becomes smaller than a certain value (which can be optimized through experiments).

[0051] The number of taps, the time interval between taps, the tapping speed, the constancy of the change in the tapping angle (the constancy of the change in angle over time), and the number of tapping hesitations (e.g., the basis for determining whether the movement continues) produced by the finger analysis unit (121) can be performed on the left and right hands to further determine the degree of symmetry in the left and right finger tapping.

[0052] The finger analysis unit (121) can calculate a Parkinson's disease diagnosis score for finger tapping by applying predetermined weights to the scores of these elements and adding them up based on the number of tappings, the time interval between tappings, the tapping speed, the consistency of the change in the tapping angle (the consistency of the change in angle over time), and the number of tapping hesitations (e.g., which serves as the basis for determining whether the movement is continued). The finger analysis unit (121) can calculate a Parkinson's disease diagnosis score for finger tapping by further reflecting the degree of symmetry of the left and right finger tapping and applying predetermined weights to the above elements and the degree of symmetry of the left and right finger tapping and adding them up. For example, the finger analysis unit (121) can determine a Parkinson's disease diagnosis score such as 0 points for normal, 1 point for slight, 2 points for mild, 3 points for moderate, and 4 points for severe, based on the result of applying and adding the above weights. The evaluation unit (140) can evaluate whether or not Parkinson's disease is present by applying predetermined weights to one or more of the Parkinson's disease diagnostic scores for finger tapping and the Parkinson's disease diagnostic scores for facial expression, gait, tremor, speech, etc. described below and adding them up.

[0053] Figure 3 shows examples of target images containing faces.

[0054] Referring to FIG. 3, the facial expression analysis unit (122) of the estimation unit (120) tracks vector changes in the movement of major (joint) points within the facial landmark (area) (LA) including the eyes and the area around the eyes (e.g., eyebrows, etc.) and the lips and the area around the lips (e.g., nose, cheeks, chin, etc.) of an image including a face for a predetermined period of time (e.g., 10 seconds, 20 seconds, etc.), and in particular, calculates the distance between the upper and lower eyelids (LE) and the distance between the upper and lower lips (LL), and determines the Parkinson's disease diagnosis score for facial expressions from the blinking frequency and its change and the lip opening frequency and its change. The image including the face may be a 2D / 3D video captured in real time by a camera, or a 2D / 3D video captured in advance and stored in memory.

[0055] For example, referring to the Parkinson's Disease Assessment Scale (MDS-UPDRS) (MDS-UPDRS, see Section 3.2), while filming the face of a diagnosis subject / patient, the subject / patient is observed sitting comfortably for 10 seconds whether speaking or not, and the frequency of blinking the eyes, whether the face is stiff or has no facial expression, etc., can be judged, and whether the subject / patient is smiling naturally, whether the mouth is open, etc. At this time, in order to determine the Parkinson's disease diagnosis score for facial expression, the facial expression analysis unit (122) can determine the Parkinson's disease diagnosis score for facial expression from the frequency of blinking and its change and the frequency of lip opening and its change during a conversation with the diagnosis subject / patient with respect to the 2D / 3D facial landmark (LA) as shown in FIG. 3.

[0056] The blink frequency and its changes (the degree of consistency in the blink frequency over time and the degree of consistency in the change in the distance (LE) between the upper and lower eyelids) produced by the facial expression analysis unit (122) can be performed on the left and right eyes to further determine the degree of symmetry in the eye expression among the left and right facial expressions.

[0057] The facial expression analysis unit (122) can calculate a Parkinson's disease diagnosis score for facial expression by applying predetermined weights to the scores of elements such as eye blink frequency and changes thereof and lip opening frequency and changes thereof and adding them up. The facial expression analysis unit (122) can calculate a Parkinson's disease diagnosis score for facial expression by further reflecting the degree of symmetry of eye expression among the left and right facial expressions and applying predetermined weights to the above elements and the degree of symmetry of eye expression among the left and right facial expressions and adding them up. For example, the facial expression analysis unit (122) can determine a Parkinson's disease diagnosis score such as 0 points for normal, 1 point for slight, 2 points for mild, 3 points for moderate, and 4 points for severe, based on the result of applying and adding the above weights. The evaluation unit (140) can evaluate whether or not Parkinson's disease is present by applying predetermined weights to one or more of the Parkinson's disease diagnostic scores for finger tapping, facial expression, gait, tremor, speech, etc. and adding them up.

[0058] Figures 4a and 4b are examples of target images including a walking person.

[0059] Referring to FIGS. 4A and 4B, the gait analysis unit (123) of the estimation unit (120) tracks vector changes in the movement of major points of the feet and arms of a person in an image including a walking person for a predetermined period of time (e.g., 1 minute, 2 minutes, etc.), calculates the stride length and arm rotation angle over time, and can determine the Parkinson's disease diagnostic score for the walking posture (Gait) from the stride length change, walking speed, and gait continuity. The image including the walking person may be a 2D / 3D video captured in real time by a camera, or a 2D / 3D video captured in advance and stored in memory.

[0060] For example, referring to the Parkinson's Disease Assessment Scale (MDS-UPDRS) (see MDS-UPDRS, Section 3.10), while videotaping the face of a diagnostic subject / patient, for example, walking 10 meters (30 feet) and then turning around and returning to the examiner, the walking posture can be quantitatively analyzed while walking a certain distance back and forth using the Human Pose model, one of the pose estimation artificial intelligence models provided by the MediaPipe framework. For example, by tracking the 2D / 3D coordinates of major joint points (e.g., big toe, ankle, heel, elbow, wrist, etc.), it is possible to quantitatively analyze walking posture such as stride length (change in stride length, measured left and right), walking speed, walking with the heel striking the ground, heel height (whether walking with the heel touching the ground, measured left and right), degree of arm swing back and forth, arm rotation angle (degree of arm rotation centered on the shoulder, measured left and right), degree of asymmetry in left and right foot / arm movement, etc., and accordingly, change in stride length, walking speed, and continuity of walking can be determined. At this time, in order to determine the Parkinson's disease diagnosis score for the walking posture (Gait), the gait analysis unit (123) calculates the step length and arm rotation angle over time for the major joint points of the human foot and arm (e.g., big toe, ankle, heel, elbow, wrist, etc.) as indicated by dots in FIGS. 4a and 4b, and can determine the Parkinson's disease diagnosis score for the walking posture (Gait) from the change in step length, walking speed, and continuity of walking.

[0061] The gait analysis unit (123) can calculate a Parkinson's disease diagnosis score for gait by applying predetermined weights to the scores of factors such as stride change, gait speed, and gait continuity and adding them up. For example, the gait analysis unit (123) can determine a Parkinson's disease diagnosis score such as 0 points for normal, 1 point for slight, 2 points for mild, 3 points for moderate, and 4 points for severe, based on the results of applying and adding the weights. The evaluation unit (140) can evaluate whether or not Parkinson's disease is present by applying predetermined weights to one or more of the Parkinson's disease diagnosis scores for finger tapping, facial expression, gait, tremor, and speech and adding them up.

[0062] Figure 5 is an example of a target image and joint point information including a non-walking person.

[0063] Referring to FIG. 5, the tremor analysis unit (124) of the estimation unit (120) tracks the vector change in the movement of major points (points in the drawing) of the arms and legs of a person in an image including a non-walking person (e.g., a sitting person) for a predetermined period of time (e.g., 10 seconds, 20 seconds, etc.), calculates the magnitude of arm movement and the magnitude of leg movement over time, and determines the Parkinson's disease diagnostic score for tremor from the magnitude and duration of arm tremor, the magnitude and duration of leg tremor, the magnitude and duration of lip tremor, and the magnitude and duration of jaw tremor. The image including the non-walking person may be a 2D / 3D video captured in real time by a camera, or a 2D / 3D video captured in advance and stored in memory.

[0064] For example, referring to the Parkinson's Disease Assessment Scale (MDS-UPDRS) (see MDS-UPDRS, Sections 3.17 and 3.18), when recording a video of a non-ambulatory person (e.g., a sitting person) of a diagnosis subject / patient, for example, observing the person sitting quietly in a chair without any instructions for 10 seconds with both hands on the armrests instead of on the knees and both feet on the floor, the tremor can be quantitatively analyzed using the Human Pose model, one of the pose estimation artificial intelligence models provided by the MediaPipe framework, by tracking the 2D / 3D coordinates of the major joints of the arms and legs, and analyzing the magnitude (left-right measurement) of arm / leg tremor, the duration of each tremor, and the cycle of each tremor in Rest Tremor (tremor when at rest), and the same method can be used for Postural Tremor (hand tremor in a posture where the arms are stretched out forward, tremor magnitude, left-right measurement). Accordingly, the size and duration of arm tremor, the size and duration of leg tremor, the size and duration of lip tremor, and the size and duration of jaw tremor can be measured. At this time, in order to determine the Parkinson's disease diagnosis score for tremor (Tremor), the tremor analysis unit (124) tracks the vector change for the movement of major points (points in the drawing) of a person's arms and legs for a predetermined period of time (e.g., 10 seconds, 20 seconds, etc.), as indicated by dots in FIG. 5, and calculates the size of arm movement and leg movement over time, and determines the Parkinson's disease diagnosis score for tremor (Tremor) from the size and duration of arm tremor, the size and duration of leg tremor, the size and duration of lip tremor, and the size and duration of jaw tremor.

[0065] The tremor analysis unit (124) can calculate a Parkinson's disease diagnosis score for tremor by applying predetermined weights to the scores of elements such as the size and duration of arm tremor, the size and duration of leg tremor, the size and duration of lip tremor, and the size and duration of jaw tremor, and adding them up. For example, the tremor analysis unit (124) can determine a Parkinson's disease diagnosis score such as 0 points for normal, 1 point for slight, 2 points for mild, 3 points for moderate, and 4 points for severe, based on the results of applying and adding the above weights. The evaluation unit (140) can evaluate whether or not Parkinson's disease is present by applying predetermined weights to one or more of the Parkinson's disease diagnostic scores for finger tapping, facial expression, gait, tremor, speech, etc. and adding them up.

[0066] Figure 6 is an example of a Mel Spectrogram target image converted into an audio file of the image.

[0067] Referring to FIG. 6, the voice analysis unit (125) of the estimation unit (120) tracks the (vector) changes in the movement of major points of frequency and amplitude in the feature map (vectorized data of the features of image data) of the mel spectrogram image converted from the voice file for a predetermined period of time (e.g., 10 seconds, 20 seconds, etc.), calculates the change in frequency and the change in amplitude over time, and compares it with the change profile of frequency and amplitude of a Parkinson's disease patient, thereby determining the Parkinson's disease diagnosis score for speech. The mel spectrogram image converted from the voice file may be an image that is expressed in frequency and amplitude according to the temporal flow of the voice by converting a predetermined voice file stored in advance into a mel spectrogram image by the voice image acquisition unit (130) as shown in FIG. 6.

[0068] For example, referring to the Parkinson's Disease Rating Scale (MDS-UPDRS) (see MDS-UPDRS, Section 3.1), speech data can be converted into real-time mel-spectrogram images to evaluate voice volume, intonation (modulation), clarity, etc., including slurring or stuttering (palilalia) or speaking too quickly and overlapping pronunciation (tachyphemia) by having a conversation to hear the subject / patient speak naturally, or speech data can be stored in memory and then converted into mel-spectrogram images. Using Mel spectrogram, which converts the frequency and amplitude of the voice into an image, various deep learning techniques for image analysis can be used to identify unique characteristic values ​​for each person, such as a person who speaks slurred, evaluate the correlation between expressions based on the pitch information and the person's emotions (surprise, anger, or joy), and evaluate the speaking characteristics of Parkinson's patients with reduced intonation. Accordingly, in particular, in the feature map (data that vectorizes the features of image data) for the Mel spectrogram image, the (vector) changes in the movement of key points of frequency and amplitude are tracked for a certain period of time (e.g., 10 seconds, 20 seconds), and the changes in frequency and amplitude over time can be calculated, and compared with the change profiles of frequency and amplitude of Parkinson's patients. Here, to reflect the unique characteristic values ​​of each person, the key points of frequency can be a frequency range, such as 80% between the highest frequency and the lowest frequency (can be specified as other values), and also the key points of amplitude can be an amplitude range, such as 80% between the highest amplitude and the lowest amplitude (can be specified as other values), or all amplitudes 10 dB or more above the lowest amplitude (can be specified as other values).In accordance with the setting values, the profile of the frequency change and amplitude change over time of the diagnosis subject / patient can be applied to the voice data of the Parkinson's disease patient in the same manner as the setting values, and can be compared with the change profile of the frequency and amplitude of the Parkinson's disease patient. In this way, in order to determine the Parkinson's disease diagnosis score for speech, the voice analysis unit (125) tracks the (vector) change in the movement of the main points of frequency and amplitude in the feature map (data that vectorizes the features of the image data) for the Mel spectrogram image converted from the voice file for a predetermined period of time (e.g., 10 seconds, 20 seconds, etc.), calculates the change in frequency and amplitude over time, and compares it with the change profile of the frequency and amplitude of the Parkinson's disease patient.

[0069] The voice analysis unit (125) can calculate a similarity value through the above comparison, and for example, can calculate a Parkinson's disease diagnosis score for speech by applying predetermined weights to the scores of each element of the similarity of frequency change and the similarity of amplitude change and adding them up. For example, the voice analysis unit (125) can determine a Parkinson's disease diagnosis score such as 0 points for normal, 1 point for slight, 2 points for mild, 3 points for moderate, and 4 points for severe, based on the result of applying and adding the above weights. The evaluation unit (140) can evaluate whether or not Parkinson's disease is present by applying predetermined weights to one or more of the Parkinson's disease diagnosis scores for finger tapping, facial expression, gait, tremor, speech, etc. and adding them up.

[0070] FIG. 8 is a drawing for explaining an example of a method for implementing a Parkinson's disease diagnosis device (100) according to one embodiment of the present invention.

[0071] Referring to FIG. 8, the Parkinson's disease diagnosis device (100) according to one embodiment of the present invention may be implemented in the form of hardware, software, or a combination thereof. For example, the Parkinson's disease diagnosis device (100) of the present invention may be implemented in the form of a computing system (1000) as shown in FIG. 8, having at least one processor for performing the functions / steps / processes described above, or as a server on the Internet.

[0072] A computing system (1000) may include at least one processor (1100), memory (1300), a user interface input device (1400), a user interface output device (1500), storage (1600), and a network interface (1700) connected via a bus (1200). The processor (1100) may be a central processing unit (CPU) or a semiconductor device that executes processing on instructions stored in the memory (1300) and / or storage (1600). The memory (1300) and storage (1600) may include various types of volatile or non-volatile storage media. For example, the memory (1300) may include a read-only memory (ROM) (1310) and a random access memory (RAM) (1320). In addition, the network interface (1700) may include a communication module such as a modem that supports wired Internet communication, wireless Internet communication such as WiFi or WiBro, mobile communication such as WCDMA or LTE in user terminals such as smartphones, laptop PCs, and desktop PCs, or a communication module such as a modem that supports short-range wireless communication (e.g., Bluetooth, Zigbee, WiFi, etc.).

[0073] Accordingly, the steps of a method or algorithm described in connection with the embodiments disclosed herein may be implemented directly in hardware, a software module, or a combination of the two executed by the processor (1100). The software module may reside in a non-transitory computer-readable storage medium storing computer executable instructions (i.e., memory (1300) and / or storage (1600)), such as RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM. An exemplary storage medium is coupled to the processor (1100) such that the processor (1100) can read information (code) from, and write information (code) to, the storage medium. Alternatively, the storage medium may be integral to the processor (1100). The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside within the user terminal. Alternatively, the processor and storage medium may reside as separate components within the user terminal.

[0074] As described above, according to the Parkinson's disease diagnosis device (100) of the present invention, by using various posture estimation deep learning models such as MediaPipe, OpenPose, AlphaPose, DeepCut, DeepPose, and ViTPose, 2D or 3D coordinates are extracted from key points of the human body, and through analysis of finger tapping, facial expression, gait, and voice, etc., the Parkinson's disease rating scale (UPDRS) is objectively and quantitatively evaluated, thereby enabling diagnosis and severity evaluation of Parkinson's disease, and can be used to assist diagnosis.

[0075] As described above, the present invention has been described with specific details such as specific components and limited examples and drawings, but these are provided only to help a more general understanding of the present invention, and the present invention is not limited to the above-described examples, and those with ordinary skill in the art to which the present invention pertains may make various modifications and variations without departing from the essential characteristics of the present invention. Therefore, the spirit of the present invention should not be limited to the described examples, and all technical ideas that are equivalent or equivalent to the claims described below as well as the claims should be interpreted as being included in the scope of the rights of the present invention.

Claims

1. A method for data processing performed by a processor in a device, A step of providing joint point information of a target body from a joint model; A step of receiving a target image and correcting the joint point information to correspond to the body size of the target image; In response to the movement of the target image for a plurality of frames, a step of calculating motion vectors using the corrected joint point information for a plurality of key points on the target image and estimating a Parkinson's disease diagnosis score by a combination of the motion vectors; and Step for evaluating whether Parkinson's disease is present based on the estimated above diagnostic score A method including:

2. In paragraph 1, The above joint model is a machine-learned model of joint point position vectors for a plurality of human bodies by operating a library for providing the joint point position vectors as an artificial intelligence deep learning model, and a method for providing the joint point information according to a request of the device.

3. In paragraph 1, The steps for estimating the above Parkinson's disease diagnosis score are: A step of determining whether tapping has occurred by tracking vector changes in the movement of major points of the thumb and index finger of an image including fingers for a predetermined period of time, and determining whether tapping has occurred from changes in the distance between the points at the ends of the thumb and index finger, and calculating the number of tappings, the time interval between tappings, the tapping speed, the consistency of tapping angle changes, and the number of tapping hesitations to determine the Parkinson's disease diagnostic score for finger tapping. A method including:

4. In paragraph 3, A method for determining the Parkinson's disease diagnostic score for the finger tapping, further reflecting the degree of symmetry of the left and right finger tapping.

5. In paragraph 1, The steps for estimating the above Parkinson's disease diagnosis score are: A step of tracking vector changes in the movement of key points within the facial landmarks including the eyes and around the eyes and the lips and around the lips of an image including a face for a predetermined period of time, calculating the distance between the upper and lower eyelids and the distance between the upper and lower lips, and determining the Parkinson's disease diagnostic score for facial expression from the blinking frequency and its change and the lip opening frequency and its change. A method including:

6. In paragraph 1, The steps for estimating the above Parkinson's disease diagnosis score are: A step of tracking vector changes in the movement of major points of a person's feet and arms in an image including a walking person for a predetermined period of time, calculating stride length and arm rotation angle over time, and determining a Parkinson's disease diagnostic score for walking posture (Gait) from stride length changes, walking speed, and gait continuity. A method including:

7. In paragraph 1, The steps for estimating the above Parkinson's disease diagnosis score are: A step of tracking vector changes in the movement of major points of the arms and legs of a person in an image including a non-walking person for a predetermined period of time, calculating the magnitude of arm movement and the magnitude of leg movement over time, and determining the Parkinson's disease diagnostic score for tremor from the magnitude and duration of arm tremor, the magnitude and duration of leg tremor, the magnitude and duration of lip tremor, and the magnitude and duration of jaw tremor. A method including:

8. In paragraph 1, The steps for estimating the above Parkinson's disease diagnosis score are: A step of determining a Parkinson's disease diagnosis score for speech by tracking changes in the movement of major points of frequency and amplitude in a feature map of a Mel spectrogram image converted from a voice file for a predetermined period of time, calculating changes in frequency and amplitude over time, and comparing the changes with the frequency and amplitude change profiles of a Parkinson's disease patient. A method including:

9. In a recording medium having recorded thereon a computer-readable code for performing a function for data processing performed by a processor in a device, Ability to obtain joint point information of a target body from a joint model; A function for receiving a target image and correcting the joint point information so that it corresponds to the body size of the target image; A function of generating motion vectors using the corrected joint point information for a plurality of key points on the target image in response to the movement of the target image for a plurality of frames, and estimating a Parkinson's disease diagnostic score by a combination of the motion vectors; and Ability to assess whether Parkinson's disease is present from the estimated above diagnostic scores A recording medium for implementing .

10. In paragraph 9, The above joint model is a machine-learned model of joint point position vectors for a plurality of human bodies by operating a library for providing the joint point position vectors as an artificial intelligence deep learning model, and a recording medium that provides the joint point information according to a request of the device.

11. In paragraph 9, The function to estimate the above Parkinson's disease diagnostic score is: A function of determining whether or not tapping is performed by tracking vector changes in the movement of major points of the thumb and index finger of an image including fingers for a given period of time, and determining the Parkinson's disease diagnostic score for finger tapping by calculating the number of taps, the time interval between taps, the tapping speed, the consistency of tapping angle changes, and the number of tapping hesitations. A recording medium containing:

12. In paragraph 11, A recording medium for determining the Parkinson's disease diagnostic score for the finger tapping, further reflecting the degree of symmetry of the left and right finger tapping.

13. In paragraph 9, The function to estimate the above Parkinson's disease diagnostic score is: A function of tracking vector changes in the movement of key points within the facial landmarks including the eyes and around the eyes and the lips and around the lips of an image containing a face for a given period of time, calculating the distance between the upper and lower eyelids and the distance between the upper and lower lips, and determining the Parkinson's disease diagnostic score for facial expressions from the blinking frequency and its changes and the lip opening frequency and its changes. A recording medium containing:

14. In paragraph 9, The function to estimate the above Parkinson's disease diagnostic score is: A function to track vector changes in the movement of major points of a person's feet and arms in an image including a walking person for a given period of time, calculate stride length and arm rotation angle over time, and determine the Parkinson's disease diagnostic score for the walking posture (Gait) from stride length changes, walking speed, and gait continuity. A recording medium containing:

15. In paragraph 9, The function to estimate the above Parkinson's disease diagnostic score is: A function of tracking vector changes in the movement of major points of the arms and legs of a person in an image including a non-walking person for a given period of time, calculating the magnitude of arm movement and the magnitude of leg movement over time, and determining the Parkinson's disease diagnostic score for tremor from the magnitude and duration of arm tremor, the magnitude and duration of leg tremor, the magnitude and duration of lip tremor, and the magnitude and duration of jaw tremor A recording medium containing:

16. In paragraph 9, The function to estimate the above Parkinson's disease diagnostic score is: A function to determine the Parkinson's disease diagnosis score for speech by tracking changes in the movement of major points of frequency and amplitude in a feature map of a Mel spectrogram image converted from a voice file over a given period of time, calculating changes in frequency and amplitude over time, and comparing them with the change profiles of frequency and amplitude of a Parkinson's disease patient. A recording medium containing:

Images