Electronic device for predicting upper limb function of stroke patient by applying pose estimation model to 2d image and method of predicting upper limb function using the same

Abstract

The electronic device for predicting upper limb function of a stroke patient according to the present disclosure may include a memory for storing at least one instruction and at least one processor for executing said at least one instruction. The at least one processor may collect a two-dimensional image of the stroke patient performing a test operation using an image capturing unit, generate 2D keypoint data from the two-dimensional image using a pose estimation model, and perform at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that learns the correlation between the 2D keypoint data and upper limb function.

Description

Technology Field

[0001] The present disclosure relates to a technology for predicting upper limb function in stroke patients, and more specifically, to an electronic device that predicts upper limb function in stroke patients by applying a posture estimation model to a two-dimensional image, and a method for predicting upper limb function using the same. Background Technology

[0002] Stroke is a disease that causes irreversible neurological damage due to impaired blood flow in a localized area of ​​the brain, and it is one of the chronic diseases whose social and economic importance is increasing in an aging society.

[0003] Stroke has a high mortality rate, and even if patients survive, it is often accompanied by severe physical disabilities, frequently limiting their ability to return to society or restore their functional status after recovery from the acute phase. In particular, stroke occurs at a high rate among the elderly, and in Korea, which is experiencing a rapid aging trend, medical and treatment costs are rising rapidly, resulting in a massive economic burden. Therefore, preventing stroke while ensuring appropriate treatment and management remains a critical task at both the individual and national levels.

[0004] Accurate assessment and prediction of upper limb function play a crucial role in the rehabilitation process of stroke patients. This assessment and prediction can be used to formulate stroke treatment plans or to measure the effectiveness of rehabilitation programs.

[0005] Meanwhile, traditional clinical evaluation methods for upper limb function have the disadvantages of being time-consuming, potentially involving subjective elements that require trained clinical experts, and making it difficult to produce continuous results. Additionally, three-dimensional motion analysis is a method that can measure upper limb function relatively objectively. However, it requires expensive testing equipment and a large testing space, as well as complex post-processing steps. Accordingly, various technological approaches are being attempted recently to overcome the limitations of traditional clinical evaluation methods, and automated evaluation methods utilizing deep learning technology are receiving particular attention. Prior art literature

[0006] Registered Patent Publication No. 10-2620129 (2024.01.03.) The problem to be solved

[0007] One objective of the present disclosure is to provide an electronic device that accurately and objectively evaluates the upper limb function of a stroke patient by applying a posture estimation model to a two-dimensional image of a test movement to predict upper limb function, and a method for predicting upper limb function using the same.

[0008] Another objective of the present disclosure is to provide an electronic device that performs FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis on upper limb function of a stroke patient using a two-dimensional image of a test movement, and a method for predicting upper limb function using the same.

[0009] However, the problems that this disclosure aims to solve are not limited to those mentioned above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0010] An electronic device for predicting upper limb function of a stroke patient according to the present disclosure for achieving the technical problem described above may include a memory for storing at least one instruction and at least one processor for executing said at least one instruction. The at least one processor may collect a two-dimensional image of the stroke patient performing a test operation using an image capturing unit, generate 2D keypoint data from the two-dimensional image using a pose estimation model, and perform at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that learns the correlation between the 2D keypoint data and upper limb function.

[0011] In one embodiment, the test action may include a first task of extending the arm to grasp a cup, a second task of bending the arm to bring the cup toward the body, a third task of extending the arm again to place the cup on the table after finishing the drinking action, and a fourth task of returning the arm to its original position.

[0012] In one embodiment, the at least one processor can collect the two-dimensional image by using the image capturing unit to capture the stroke patient performing the test operation in a single sagittal view at a frame rate of 30 frames per second or more and at a resolution of 1920x1080 pixels or more.

[0013] In one embodiment, the at least one processor may use an RTMPose model based on a COCO-wholebody dataset. The RTMPose model may identify the 2D image based on an MMPose framework and extract 2D keypoint data from the 2D image.

[0014] In one embodiment, the 2D keypoint data may include 2D coordinate data of at least one of the head, body, shoulder, elbow, wrist, and finger of the stroke patient included in the 2D image.

[0015] In one embodiment, the deep learning model can label clinical measurements corresponding to the 2D keypoint data for the test action. The clinical measurements for the upper limb function may include at least one of an FMA-UE classification result based on clinical measurements, a BBT classification result based on clinical measurements, a spatiotemporal parameter regression value based on clinical measurements, and an APS regression value based on clinical measurements.

[0016] In one embodiment, the at least one processor inputs the 2D keypoint data into the deep learning model, generates a feature map based on the features of the 2D keypoint data extracted while passing through the convolution layer of the CNN, converts the feature map into a vector to generate feature values ​​of the 2D keypoint data, and performs at least one of the FMA-UE classification, the BBT classification, the spatiotemporal parameter regression analysis, and the APS regression analysis based on the feature values.

[0017] In one embodiment, when performing the FMA-UE classification, the at least one processor may calculate the upper limb function of the stroke patient as a score between 0 and 66 based on a predetermined motor evaluation, and based on the score, classify the upper limb stroke patient into at least one of severe, moderate, and mild.

[0018] In one embodiment, when performing the BBT classification, the at least one processor evaluates the upper limb function of the stroke patient based on the number of blocks that the stroke patient can move to another place with one hand within a predetermined time, and classifies the upper limb stroke patient into at least one of normal, mild, and severe.

[0019] In one embodiment, when performing the spatiotemporal parameter regression analysis, the at least one processor extracts at least one of MT (Movement Time), IoC (Index of Curvature), and NoMU (Number of Movement Units) from the 2D keypoint data, and can analyze the upper limb function of the stroke patient according to time and space based on at least one of the MT, the IoC, and the NoMU.

[0020] In one embodiment, when performing the APS regression analysis, the at least one processor can generate an APS (Arm Profile Score) based on the 2D keypoint data and analyze the upper limb function of the stroke patient based on the APS.

[0021] Additionally, the method for predicting upper limb function of a stroke patient according to the present disclosure may include the operation of collecting a two-dimensional image of the stroke patient performing a test movement using an image capturing unit, the operation of generating 2D keypoint data from the two-dimensional image using a pose estimation model, and the operation of performing at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that learns the correlation between the 2D keypoint data and upper limb function.

[0022] In one embodiment, the test action may include a first task of extending the arm to grasp a cup, a second task of bending the arm to bring the cup toward the body, a third task of extending the arm again to place the cup on the table after finishing the drinking action, and a fourth task of returning the arm to its original position.

[0023] In one embodiment, the operation of collecting a two-dimensional image of the stroke patient can be performed by using the image capturing unit to capture the stroke patient performing the test operation in a single sagittal view, at a frame rate of 30 frames per second or more, and at a resolution of 1920x1080 pixels or more, thereby collecting the two-dimensional image.

[0024] In one embodiment, the operation of generating 2D keypoint data from the 2D image may utilize an RTMPose model based on the COCO-wholebody dataset. The RTMPose model may identify the 2D image based on the MMPose framework and extract 2D keypoint data from the 2D image.

[0025] In one embodiment, the 2D keypoint data may include at least one 2D coordinate data of the head, body, shoulder, elbow, wrist, and fingers of the stroke patient included in the 2D image.

[0026] In one embodiment, the deep learning model can label clinical measurements corresponding to the 2D keypoint data for the test action. The clinical measurements for the upper limb function may include at least one of an FMA-UE classification result based on clinical measurements, a BBT classification result based on clinical measurements, a spatiotemporal parameter regression value based on clinical measurements, and an APS regression value based on clinical measurements.

[0027] In one embodiment, the operation of performing at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that has learned the correlation between the 2D keypoint data and upper limb functions may involve inputting the 2D keypoint data into the deep learning model, generating a feature map based on the features of the 2D keypoint data extracted while passing through a convolution layer of a CNN, converting the feature map into a vector to generate feature values ​​of the 2D keypoint data, and performing at least one of the FMA-UE classification, the BBT classification, the spatiotemporal parameter regression analysis, and the APS regression analysis based on the feature values.

[0028] In one embodiment, the operation of performing the FMA-UE classification using the deep learning model calculates the upper limb function of the stroke patient as a score between 0 and 66 based on a predetermined exercise evaluation, and based on the score, classifies the upper limb stroke patient into at least one of severe, moderate, and mild.

[0029] In one embodiment, the operation of performing the BBT classification using the deep learning model evaluates the upper limb function of the stroke patient based on the number of blocks that the stroke patient can move to another place with one hand within a predetermined time, and classifies the upper limb stroke patient into at least one of normal, mild, and severe.

[0030] In one embodiment, the operation of performing the spatiotemporal parameter regression analysis using the deep learning model extracts at least one of MT (Movement Time), IoC (Index of Curvature), and NoMU (Number of Movement Units) from the 2D keypoint data, and can analyze the upper limb function of the stroke patient according to time and space based on at least one of the MT, the IoC, and the NoMU.

[0031] In one embodiment, the operation of performing the APS regression analysis using the deep learning model can generate an APS (Arm Profile Score) based on the 2D keypoint data and analyze the upper limb function of the stroke patient based on the APS.

[0032] In addition to this, a computer program stored on a computer-readable recording medium for implementing the present disclosure may be further provided.

[0033] In addition to this, a computer-readable recording medium for recording a computer program for implementing the present disclosure may be further provided. Effects of the invention

[0034] According to the means for solving the problem described above in the present disclosure, the electronic device of the present disclosure and the method for predicting upper limb function using the same can accurately and objectively evaluate the upper limb function of a stroke patient by applying a posture estimation model to a two-dimensional image of a test movement to predict upper limb function.

[0035] In addition, the electronic device of the present disclosure and the upper limb function prediction method using it can perform FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis on the upper limb function of a stroke patient using a two-dimensional image of a test movement.

[0036] Accordingly, the electronic device of the present disclosure and the method for predicting upper limb function using the same can predict and evaluate the upper limb function of a stroke patient based on a deep learning model, thereby analyzing the upper limb function of a stroke patient at a low cost, overcoming spatiotemporal constraints, at a fast speed, and with high accuracy, and providing a customized treatment strategy for the stroke patient.

[0037] The effects of the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below. Brief explanation of the drawing

[0038] FIG. 1 is a drawing showing the block configuration of an electronic device of the present disclosure. FIG. 2 is a conceptual diagram illustrating the operation of an electronic device of the present disclosure. FIG. 3 is a flowchart illustrating the operation of the electronic device of the present disclosure. FIG. 4 is a diagram illustrating the process of upper limb function prediction using a two-dimensional image of the present disclosure. Figure 5 is a diagram showing the collection of two-dimensional images through the shooting of a test operation. FIG. 6 is a drawing showing a test operation according to one embodiment of the present disclosure. Figure 7 is a diagram showing a pose estimation model that generates 2D keypoint data from a 2D image. Figure 8 is a diagram showing 2D keypoint data generated from a 2D image through a pose estimation model. Figure 9 is a diagram showing a CNN model that performs FMA-UE classification and BBT classification. Figure 10 is a diagram showing a CNN model that performs spatiotemporal parameter regression analysis and APS regression analysis. FIG. 11 is a diagram showing the evaluation results for the FMA-UE classification of the electronic device of the present disclosure. FIG. 12 is a diagram showing the evaluation results for the BBT classification of the electronic device of the present disclosure. FIG. 13 is a diagram showing the evaluation results of the spatiotemporal parameter regression analysis of the electronic device of the present disclosure. FIG. 14 is a diagram showing the evaluation results of the APS regression analysis of the electronic device of the present disclosure. Specific details for implementing the invention

[0039] Throughout this disclosure, the same reference numerals denote the same components. This disclosure does not describe all elements of the embodiments, and general content in the art to which this disclosure pertains or content that overlaps between embodiments is omitted. The terms 'part, module, component, block' as used in the specification may be implemented in software or hardware, and depending on the embodiments, a plurality of 'parts, modules, components, blocks' may be implemented as a single component, or a single 'part, module, component, block' may include a plurality of components.

[0040] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are directly connected but also cases where they are indirectly connected, and indirect connections include connections made via a wireless communication network.

[0041] Furthermore, when it is stated that a part "includes" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0042] Throughout the specification, when it is stated that a component is located "on" another component, this includes not only cases where a component is in contact with another component, but also cases where another component exists between the two components.

[0043] The terms first, second, etc. are used to distinguish one component from another, and the components are not limited by the aforementioned terms.

[0044] Singular expressions include plural expressions unless there is an obvious exception in the context.

[0045] In each step, identification codes are used for convenience of explanation and do not describe the order of the steps; the steps may be performed differently from the specified order unless a specific order is clearly indicated in the context.

[0046] The operating principles and embodiments of the present disclosure will be described below with reference to the attached drawings.

[0047] In this specification, the term "device according to the present disclosure" includes all various devices capable of performing computational processing and providing results to a user. For example, the device according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may be in the form of any one of these.

[0048] Here, the computer may include, for example, a notebook, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

[0049] The above server device is a server that processes information by communicating with an external device, and may include an application server, a computing server, a database server, a file server, a game server, a mail server, a proxy server, and a web server.

[0050] The above portable terminal may include, for example, all types of handheld-based wireless communication devices such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminals, smartphones, etc., as well as wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted devices (HMDs).

[0051] Functions related to artificial intelligence according to the present disclosure are operated through a processor and memory. The processor may be composed of one or more processors. In this case, the one or more processors may be general-purpose processors such as CPUs, APs, and DSPs (Digital Signal Processors), graphics-dedicated processors such as GPUs and VPUs (Vision Processing Units), or artificial intelligence-dedicated processors such as NPUs. The one or more processors control the processing of input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the one or more processors are artificial intelligence-dedicated processors, the artificial intelligence-dedicated processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model.

[0052] The predefined rules of operation or artificial intelligence models are characterized by being created through learning. Here, being created through learning means that a basic artificial intelligence model is trained using a number of training data by a learning algorithm, thereby creating predefined rules of operation or artificial intelligence models configured to perform desired characteristics (or objectives). Such learning may be performed on the device itself where the artificial intelligence according to the present disclosure is executed, or it may be performed through a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but are not limited to the examples described above.

[0053] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations through operations between the results of previous layers and the multiple weights. The multiple weights possessed by the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Artificial neural networks may include deep neural networks (DNNs), such as Convolutional Neural Networks (CNNs), Deep Neural Networks (DNNs), RecuREnt Neural Networks (RNNs), Restricted Boltzmann Machines (RBMs), Deep Belief Networks (DBNs), Bidirectional RecuREnt Deep Neural Networks (BRDNNs), or Deep Q-Networks, but are not limited to the examples mentioned above.

[0054] According to an exemplary embodiment of the present disclosure, a processor can implement artificial intelligence. Artificial intelligence refers to a machine learning method based on an artificial neural network that enables a machine to learn by mimicking human biological neurons. Methodologies of artificial intelligence can be classified according to the learning method into supervised learning, where input and output data are provided together as training data and the solution (output data) to the problem (input data) is predetermined; unsupervised learning, where only input data is provided without output data and the solution (output data) to the problem (input data) is not predetermined; and reinforcement learning, where a reward is given from an external environment whenever an action is taken from the current state, and learning proceeds in a direction that maximizes such reward. In addition, artificial intelligence methodologies can be classified according to the architecture, which is the structure of the learning model. The architectures of widely used deep learning technologies can be classified into Convolutional Neural Networks (CNN), RecuREnt Neural Networks (RNN), Transformers, and Generative Adversarial Networks (GAN).

[0055] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0056] FIG. 1 is a drawing showing the block configuration of the electronic device (100) of the present disclosure, and FIG. 2 is a conceptual diagram showing the operation of the electronic device (100) of the present disclosure.

[0057] Referring to FIG. 1, an electronic device (100) according to one embodiment may include a processor (120), a memory (110), and a communication unit (not shown). The configuration shown in FIG. 1 illustrates a configuration for describing an embodiment according to the present disclosure, and other configurations (e.g., a communication unit) for performing the functions of the electronic device may be added in addition to the illustrated configuration, and the illustrated configurations may be omitted.

[0058] A memory (110) according to one embodiment is a storage medium used by an electronic device (100) and can store data such as at least one instruction or setting information corresponding to at least one program. The program may include an operating system (OS) program and various application programs.

[0059] In one embodiment, the memory (110) may include at least one type of storage medium among a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), RAM (random access memory, RAM), SRAM (static random access memory), ROM (read only memory, ROM), EEPROM (electrically erasable programmable ROM), PROM (programmable ROM), magnetic memory, a magnetic disk, and an optical disk.

[0060] In one embodiment, the communication unit may include a wireless communication unit (e.g., a cellular communication module, a short-range wireless communication module, or a GNSS (global navigation satellite system) communication module) or a wired communication unit. The communication unit may communicate with an external electronic device through a network (e.g., a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a wide-area communication network such as a computer network (e.g., a LAN or WAN).

[0061] In one embodiment, the communication unit may support 5G networks and next-generation communication technologies, for example, new radio access technology. NR access technology may support high-speed transmission of high-capacity data (enhanced mobile broadband (eMBB)), minimization of terminal power and connection of multiple terminals (massive machine type communications (mMTC)), or high reliability and low-latency (ultra-reliable and low-latency communications (URLLC)). The communication unit may support a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate, for example. The communication unit may support various technologies for securing performance in the high-frequency band, for example, beamforming, massive MIMO (multiple-input and multiple-output), full-dimensional MIMO (FD-MIMO), array antenna, analog beamforming, or large-scale antenna.

[0062] A processor (120) according to one embodiment can, for example, execute software (e.g., a program) to control at least one other component (e.g., a hardware or software component) of an electronic device (100) connected to the processor (120) and can perform various data processing or operations.

[0063] In one embodiment, the processor (120) may store commands or data received from another component (e.g., a communication unit) in volatile memory as at least part of data processing or computation, process the commands or data stored in volatile memory, and store the result data in non-volatile memory.

[0064] In one embodiment, the processor (120) may include a main processor (e.g., a central processing unit or an application processor) or an auxiliary processor that can operate independently or together with it (e.g., a graphics processing unit, a neural processing unit (NPU), or a communication processor). For example, the processor (120) may include at least one of a CPU (Central Processing Unit), an AP (Application Processor), or a microprocessor.

[0065] A processor (120) according to one embodiment can perform the operation of an electronic device (100) described below through the drawings. For example, as shown in FIG. 2, the processor (120) can collect a two-dimensional image of the stroke patient performing a test operation using an image capturing unit. The processor (120) can generate 2D keypoint data from the two-dimensional image using a pose estimation model. The processor (120) can output a predicted value for the upper limb function of the stroke patient using a deep learning model that has learned the correlation between the 2D keypoint data and the upper limb function. For example, the processor (120) can perform at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis.

[0066] In this way, the processor (120) can accurately and objectively evaluate the upper limb function of a stroke patient by applying a posture estimation model to a two-dimensional image of a test movement to predict upper limb function.

[0067] Additionally, the processor (120) can perform FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis on upper limb function of a stroke patient using a two-dimensional image of the test movement.

[0068] Accordingly, the processor (120) can predict and evaluate the upper limb function of a stroke patient based on a deep learning model, thereby analyzing the upper limb function of a stroke patient at a low cost, high speed, and high accuracy, and providing a customized treatment strategy to the stroke patient.

[0069] FIG. 3 is a flowchart showing the operation of an electronic device (100) of the present disclosure, FIG. 4 is a diagram showing the process of upper limb function prediction using 2D keypoint data of the present disclosure, FIG. 5 is a diagram showing the collection of a 2D image through the shooting of a test operation, FIG. 6 is a diagram showing a test operation according to one embodiment of the present disclosure, FIG. 7 is a diagram showing a pose estimation model that generates 2D keypoint data from a 2D image, and FIG. 8 is a diagram showing 2D keypoint data generated from a 2D image through a pose estimation model.

[0070] Referring to FIGS. 3 and 4, the electronic device (100) can accurately and objectively evaluate the upper limb function of a stroke patient by applying a posture estimation model to a two-dimensional image of a test movement to predict the upper limb function.

[0071] Specifically, the electronic device (100) can collect a two-dimensional image of the stroke patient performing a test operation using an image capturing unit (operation 310), generate 2D keypoint data from the two-dimensional image using a posture estimation model (operation 320), and perform at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that learns the correlation between the 2D keypoint data and upper limb function (operation 330).

[0072] According to one example, in operation 310, the electronic device (100) can collect a two-dimensional image of the stroke patient performing the test operation using an image capturing unit.

[0073] The video capturing unit may include various camera systems. The video capturing unit may capture two-dimensional images of a test operation. For example, the video capturing unit may include at least one of a fixed camera system, a multi-camera system, a wearable camera, and a smartphone.

[0074] For example, a fixed camera system can install a camera at a fixed position within the test space and capture the test movements of a stroke patient. For example, a multi-camera system can use multiple cameras to capture the test movements of a stroke patient from various angles. For example, a wearable camera can capture the stroke patient's test movements from a subjective perspective by being worn on the body by a stroke patient or a clinical professional. For example, a smartphone can easily capture the stroke patient's test movements using its built-in camera.

[0075] The test motion may include a series of movements to efficiently analyze the upper limb function of a stroke patient. For example, the test motion may be the stroke patient drinking water from a cup on a table.

[0076] As shown in FIG. 5, the electronic device can use an image capturing unit to capture the stroke patient performing the test operation in a single sagittal view. For example, the electronic device can use an image capturing unit to capture the stroke patient performing the test operation at a frame rate of 30 frames per second or more. The two-dimensional image can be collected by capturing at a resolution of 1920x1080 pixels or higher.

[0077] A two-dimensional image can display the visual content of the image (e.g., test movements) by configuring it into two dimensions, the x-axis and the y-axis. For example, the two-dimensional image can be collected through the image capturing unit. For example, the two-dimensional image may include visual information necessary for evaluating upper limb function in a stroke patient. For example, the two-dimensional image may include test movements of a stroke patient.

[0078] In one embodiment, as shown in FIG. 6, the test motion may include a first task of extending the arm to grasp the cup, a second task of bending the arm to bring the cup toward the body, a third task of extending the arm again to place the cup on the table after finishing the drinking motion, and a fourth task of returning the arm to its original position.

[0079] For example, the first task could be the action of extending an arm to grab a cup in order to lift it (e.g., Extend Arm, Grasp object).

[0080] For example, the second task could be the action of bending the arm to bring the cup to the mouth to drink water from the cup (e.g., Flex Arm, Target self).

[0081] For example, the third task may be the action of extending the arm back to the table to place the cup held in the hand back on the table after drinking water (e.g., Extend Arm, Target table).

[0082] For example, the fourth task could be the action of returning the arm to its original position (e.g., Retract Arm) to complete the action of drinking water after placing the cup on the table.

[0083] That is, the electronic device (100) can record sequential first to fourth tasks of a stroke patient using 3D motion capture and collect 2D keypoint data on the stroke patient's test movements based on 3D motion capture.

[0084] According to one example, in operation 320, the electronic device (100) can generate 2D keypoint data from the 2D image using a pose estimation model.

[0085] A posture estimation model may be an algorithm or software that recognizes and analyzes the posture of a stroke patient from a two-dimensional image. For example, a posture estimation model may receive two-dimensional image data as input and identify and extract key points (e.g., major joint locations) corresponding to the body structure and movement of a stroke patient.

[0086] The electronic device (100) may use an RTMPose model based on the COCO-wholebody dataset as a pose estimation model. The COCO-wholebody dataset can improve the accuracy of pose estimation by including various keypoint information that comprehensively represents the entire body of a person.

[0087] As shown in Fig. 7, the RTMPose model can identify the 2D image based on the MMPose framework and extract 2D keypoint data from the 2D image. The RTMPose model can use the COCO-wholebody dataset as training data to accurately estimate the posture of a stroke patient in an actual 2D image and extract the location of each keypoint as 2D coordinate data.

[0088] 2D keypoint data can represent changes in the body position of a stroke patient in a two-dimensional coordinate system based on the x-axis and y-axis. For example, 2D keypoint data can record information on the body position that changes over time as time-series data.

[0089] 2D keypoint data may be a set of 2D coordinate data representing the location of a person's body parts included in a 2D image. 2D keypoint data can accurately represent the location of major body parts of a stroke patient, such as the head, body, shoulders, elbows, wrists, and fingers.

[0090] For example, as shown in FIG. 8, the location of each keypoint in the 2D image of the test motion can be expressed as x-axis and y-axis coordinates in a 2D coordinate system. For example, the 2D keypoint data may include at least one 2D coordinate data of the head, body, shoulder, elbow, wrist, and fingers of the stroke patient included in the 2D image.

[0091] According to one example, in operation 330, the electronic device (100) can perform at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that has learned the correlation between 2D keypoint data and upper limb function.

[0092] The deep learning model can learn the correlation between the 2D keypoint data for the test action and the clinical measurements for the upper limb function. Here, the deep learning model may be a single artificial intelligence model or may be implemented as multiple artificial intelligence models. The artificial intelligence model may be composed of a neural network (or artificial neural network) and may include statistical learning algorithms in machine learning and cognitive science that mimic biological neurons. A neural network may refer to a model that possesses problem-solving capabilities by having artificial neurons (nodes) that form a network through synaptic connections and change the strength of the synaptic connections through learning. The neurons of a neural network may include combinations of weights or biases. A neural network may include one or more layers composed of one or more neurons or nodes. For example, the device may include an input layer, a hidden layer, and an output layer. The neural network constituting the device can infer a result (output) to be predicted from an arbitrary input by changing the weights of the neurons through learning.

[0093] At least one processor included in the electronic device (100) can generate a neural network, train or learn a neural network, perform operations based on received input data, generate an information signal based on the results of the operation, or retrain the neural network. The neural network models may include, but are not limited to, various types of models such as Convolutional Neural Networks (CNN), Region with Convolutional Neural Networks (R-CNN), Region Proposal Networks (RPN), RecuRent Neural Networks (RNN), Stacking-based Deep Neural Networks (S-DNN), State-Space Dynamic Neural Networks (S-SDNN), Deconvolution Networks, Deep Belief Networks (DBN), Restructured Boltzmann Machines (RBM), Fully Convolutional Networks, Long Short-Term Memory Networks (LSTM), and Classification Networks, such as GoogleNet, AlexNet, and VGG Network. The processor may include one or more processors for performing operations according to the neural network models. There is. For example, neural networks can include deep neural networks.

[0094] Neural networks include CNN (Convolutional Neural Network), RNN (RecuREnt Neural Network), perceptron, multilayer perceptron, FF (Feed Forward), RBF (Radial Basis Network), DFF (Deep Feed Forward), LSTM (Long Short Term Memory), GRU (Gated RecuREnt Unit), AE (Auto Encoder), VAE (Variational Auto Encoder), DAE (Denoising Auto Encoder), SAE (Sparse Auto Encoder), MC (Markov Chain), HN (Hopfield Network), BM (Boltzmann Machine), RBM (Restricted Boltzmann Machine), DBN (Depp Belief Network), DCN (Deep Convolutional Network), DN (Deconvolutional Network), DCIGN (Deep Convolutional Inverse Graphics) Network), GAN (Generative Adversarial Network), LSM (Liquid State Machine), ELM (Extreme Learning Machine), A person skilled in the art will understand that any neural network may be included, but is not limited to, ESN (Echo State Network), DRN (Deep Residual Network), DNC (Differentiable Neural Computer), NTM (Neural Turning Machine), CN (Capsule Network), KN (Kohonen Network), and AN (Attention Network).

[0095] According to an exemplary embodiment of the present disclosure, at least one processor included in an electronic device (100) is a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), RecuREnt Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA, Text Analysis, Dialog System, GPT-3, GPT-4 for natural language processing, Visual Analytics, Visual Understanding, Video Synthesis for vision processing, Anomaly Detection for ResNet data intelligence, Various artificial intelligence structures and algorithms, such as prediction, time-series forecasting, optimization, recommendation, and data creation, may be used, but are not limited thereto.

[0096] In one embodiment, the deep learning model can label the clinical measurements corresponding to the 2D keypoint data. For example, the deep learning model can label the clinical measurements that match the collected 2D keypoint data.

[0097] Here, clinical measurements of upper limb function may include at least one of the FMA-UE classification result based on clinical measurement, the BBT classification result based on clinical measurement, the spatiotemporal parameter regression analysis value based on clinical measurement, and the APS regression analysis value based on clinical measurement.

[0098] For example, the FMA-UE classification result can be provided as a score in which a clinical expert has precisely evaluated various upper extremity motor functions of a stroke patient. The FMA-UE classification result measures the overall motor function of the upper extremities, including the motor abilities of the patient's arms, wrists, hands, and fingers, and can be calculated as a score between 0 (no function) and 66 (full function).

[0099] For example, BBT classification results can be provided as a numerical evaluation of upper limb functional ability by counting the number of blocks a stroke patient can move with one hand within a specific time. BBT classification results measure hand and arm dexterity, coordination, and speed, and can reflect the ability of stroke patients to perform basic movements required in daily life.

[0100] For example, spatiotemporal parameter regression values ​​may include at least one of MT (Movement Time), IoC (Index of Curvature), and NoMU (Number of Movement Units). MT may refer to the total time taken for a stroke patient to initiate and complete a specific movement. MT can be used to evaluate the speed and efficiency of the stroke patient's movement. IoC is an indicator representing the curvature of the stroke patient's movement path and can assess the accuracy and smoothness of each movement. For example, a high IoC value may indicate that more coordination is required or that there are many unnecessary movements. NoMU represents the number of accelerations and decelerations that occur during the performance of a specific movement and can evaluate the continuity and coordination ability of the stroke patient's movement. For example, a low NoMU value may indicate efficient and coordinated movement.

[0101] For example, APS regression values ​​can be a quantitative assessment of the overall profile of arm movement in stroke patients. APS regression values ​​can comprehensively reflect the range of motion, speed, and coordination of the arm. For instance, APS regression values ​​can be used as an indicator to track the degree of improvement in upper limb function during the rehabilitation process of stroke patients.

[0102] Figure 9 is a diagram showing a CNN model that performs FMA-UE classification and BBT classification.

[0103] Referring to FIG. 9, the electronic device (100) can perform at least one of FMA-UE classification and BBT classification using a deep learning model that learns the correlation between 2D keypoint data and upper limb functions.

[0104] Specifically, the deep learning model can perform the FMA-UE classification and the BBT classification through inference using a convolutional neural network (CNN). For example, the deep learning model can generate feature values ​​of 2D keypoint data using a convolutional layer, an activation function, a pooling layer, and a fully connected layer.

[0105] For example, the electronic device (100) can input the 2D keypoint data into the deep learning model. The electronic device (100) can generate a feature map based on the features of the 2D keypoint data extracted while passing through the convolution layer of the CNN.

[0106] The electronic device (100) can convert the feature map into a vector to generate feature values ​​of the 2D keypoint data. The electronic device (100) can perform the FMA-UE classification and the BBT classification based on the feature values.

[0107] In one embodiment, when performing FMA-UE classification, the electronic device (100) can calculate the upper limb function of the stroke patient as a score between 0 and 66 based on a predetermined exercise evaluation.

[0108] For example, the electronic device (100) can generate feature values ​​from 2D keypoint data for a test movement and quantify the upper limb function of the stroke patient based on the feature values. Based on the score, the electronic device (100) can classify the upper limb stroke patient into at least one of severe, moderate, and mild.

[0109] In one embodiment, when performing BBT classification, the electronic device (100) can evaluate the upper limb function of the stroke patient based on the number of blocks that the stroke patient can move to another place with one hand within a predetermined time.

[0110] For example, the electronic device (100) can generate feature values ​​from 2D keypoint data for a test operation and predict the number of blocks that a stroke patient can move to another place with one hand within a predetermined time based on the feature values. Based on the predicted values, the electronic device (100) can classify the upper limb stroke patient into at least one of normal, mild, and severe.

[0111] Figure 10 is a diagram showing a CNN model that performs spatiotemporal parameter regression analysis and APS regression analysis.

[0112] Referring to FIG. 10, the electronic device (100) can perform at least one of spatiotemporal parameter regression analysis and APS regression analysis using a deep learning model that has learned the correlation between 2D keypoint data and upper limb function.

[0113] Specifically, the deep learning model can perform the spatiotemporal parameter regression analysis and the APS regression analysis through inference using a convolutional neural network (CNN). For example, the deep learning model can generate feature values ​​of 2D keypoint data using a convolution layer, an activation function, a pooling layer, and a fully connected layer.

[0114] For example, the electronic device (100) can input the 2D keypoint data into the deep learning model. The electronic device (100) can generate a feature map based on the features of the 2D keypoint data extracted while passing through the convolution layer of the CNN.

[0115] The electronic device (100) can convert the feature map into a vector to generate feature values ​​of the 2D keypoint data. The electronic device (100) can perform the spatiotemporal parameter regression analysis and the APS regression analysis based on the feature values.

[0116] In one embodiment, when performing the spatiotemporal parameter regression analysis, the electronic device (100) may extract at least one of MT (Movement Time), IoC (Index of Curvature), and NoMU (Number of Movement Units) from the 2D keypoint data. MT may represent the total time taken for a stroke patient to start and complete a specific movement. IoC is an indicator representing the curvature of the stroke patient's movement path, which can evaluate the accuracy and smoothness of each movement. NoMU represents the number of accelerations and decelerations that occur while performing a specific movement, which can evaluate the continuity and coordination ability of the stroke patient's movement. The electronic device (100) may analyze the upper limb function of the stroke patient over time and space based on at least one of the MT, IoC, and NoMU.

[0117] In one embodiment, when performing the APS regression analysis, the electronic device (100) can generate an Arm Profile Score (APS) based on the 2D keypoint data. The APS regression analysis value may be a numerical value that quantitatively evaluates the overall profile of the stroke patient's arm movement. The APS regression analysis value may comprehensively reflect the range of motion, speed, and coordination ability of the arm. The electronic device (100) can analyze the upper limb function of the stroke patient based on the APS.

[0118] FIG. 11 is a drawing showing the evaluation results for the FMA-UE classification of the electronic device (100) of the present disclosure.

[0119] Referring to FIG. 11, the electronic device (100) can perform FMA-UE classification using a deep learning model that learns the correlation between 2D keypoint data and upper limb functions.

[0120] As illustrated in FIG. 11, the electronic device (100) performed FMA-UE classification using 330 two-dimensional image datasets (e.g., 265 training datasets and 65 test datasets), and the accuracy, precision, recall, and F1-score of the FMA-UE classification results were evaluated, respectively.

[0121] As shown in the diagram of FIG. 11, for FMA-UE classification, the accuracy was evaluated as 89.23%, the precision as 88.39%, the recall as 85.97%, and the F1-score as 87.16%. That is, it can be confirmed that the electronic device (100) of the present disclosure can perform high-accuracy FMA-UE classification using a deep learning model.

[0122] FIG. 12 is a drawing showing the evaluation results for BBT classification of the electronic device (100) of the present disclosure.

[0123] Referring to FIG. 12, the electronic device (100) can perform BBT classification using a deep learning model that learns the correlation between 2D keypoint data and upper limb functions.

[0124] As illustrated in FIG. 12, the electronic device (100) performed BBT classification using 330 two-dimensional image datasets (e.g., 265 training datasets and 65 test datasets), and the accuracy, precision, recall, and F1-score of the BBT classification results were evaluated, respectively.

[0125] As shown in the figure of FIG. 12, for BBT classification, the accuracy was evaluated as 76.92%, the precision as 73.79%, the recall as 75.51%, and the F1-score as 74.64%. That is, it can be confirmed that the electronic device (100) of the present disclosure can perform BBT classification with reliable accuracy using a deep learning model.

[0126] FIG. 13 is a diagram showing the evaluation results of the spatiotemporal parameter regression analysis of the electronic device (100) of the present disclosure.

[0127] Referring to FIG. 13, the electronic device (100) can perform spatiotemporal parameter regression analysis using a deep learning model that learns the correlation between 2D keypoint data and upper limb functions.

[0128] As illustrated in FIG. 13, the electronic device (100) performed spatiotemporal parameter regression analysis using 196 two-dimensional image datasets (e.g., 117 training datasets and 40 test datasets), and the correlation coefficient for the results of the spatiotemporal parameter regression analysis was evaluated.

[0129] As shown in the graph of FIG. 13, the correlation coefficient of MT (Movement Time) was 0.528, indicating appropriate correlation, the correlation coefficient of IoC (Index of Curvature) was 0.703, indicating strong correlation, and the correlation coefficient of NoMU (Number of Movement Units) was 0.625, indicating appropriate correlation. That is, it can be confirmed that the electronic device (100) of the present disclosure can perform spatiotemporal parameter regression analysis with reliable accuracy using a deep learning model.

[0130] FIG. 14 is a diagram showing the evaluation results of the APS regression analysis of the electronic device (100) of the present disclosure.

[0131] Referring to FIG. 14, the electronic device (100) can perform APS regression analysis using a deep learning model that learns the correlation between 2D keypoint data and upper limb function.

[0132] As illustrated in FIG. 14, the electronic device (100) performed APS regression analysis using 196 two-dimensional image datasets (e.g., 117 training datasets and 40 test datasets), and the correlation coefficient for the results of the APS regression analysis was evaluated.

[0133] As shown in the graph of FIG. 14, the correlation coefficient for the APS regression analysis was 0.569, indicating appropriate correlation. That is, it can be confirmed that the electronic device (100) of the present disclosure can perform APS regression analysis with reliable accuracy using a deep learning model.

[0134] In this way, the electronic device of the present disclosure and the upper limb function prediction method using it can accurately and objectively evaluate the upper limb function of a stroke patient by predicting upper limb function through a two-dimensional image of a test movement.

[0135] In addition, the electronic device of the present disclosure and the upper limb function prediction method using it can perform FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis on the upper limb function of a stroke patient using a two-dimensional image of a test movement.

[0136] Accordingly, the electronic device of the present disclosure and the method for predicting upper limb function using the same can predict and evaluate the upper limb function of a stroke patient based on a deep learning model, thereby analyzing the upper limb function of a stroke patient at low cost, high speed, and high accuracy, and providing a customized treatment strategy for the stroke patient.

[0137] However, as this has been explained above, a redundant explanation thereof will be omitted.

[0138] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0139] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0140] Although the embodiments have been described above with reference to the limited drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results may be achieved even if the described techniques are performed in a different order than described, and / or if the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents. Therefore, other implementations, other embodiments, and equivalents to the claims below are also within the scope of the claims. Explanation of the symbols

[0141] 100: Electronic device 110: Memory 120: Processor

Claims

Claim 1 An electronic device for predicting upper limb function of a stroke patient, comprising: a memory for storing at least one instruction; and at least one processor for executing said at least one instruction, wherein the at least one processor collects a two-dimensional image of the stroke patient performing a test operation using an image capturing unit, generates 2D keypoint data from the two-dimensional image using a posture estimation model, and performs FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that learns the correlation between the 2D keypoint data and upper limb function, wherein the 2D keypoint data records position information of the stroke patient's body changing over time as time-series data in a two-dimensional coordinate system based on the x-axis and y-axis, and the deep learning model comprises: a CNN model that extracts feature values ​​from the 2D keypoint data and performs the FMA-UE classification and the BBT classification based on the extracted feature values; and a CNN model that extracts feature values ​​from the 2D keypoint data and performs the spatiotemporal parameter regression analysis and the APS regression analysis based on the extracted feature values. Claim 2 An electronic device according to claim 1, wherein the test operation comprises a first task of extending an arm to grasp a cup, a second task of bending an arm to bring the cup toward the body, a third task of extending the arm again to place the cup on a table after completing the drinking operation, and a fourth task of returning the arm to its original position. Claim 3 An electronic device according to claim 1, wherein at least one processor collects the two-dimensional image by using the image capturing unit to capture the stroke patient performing the test operation in a single sagittal view at a frame rate of 30 frames per second or more and a resolution of 1920x1080 pixels or more. Claim 4 An electronic device according to claim 1, wherein at least one processor uses an RTMPose model based on a COCO-wholebody dataset, and the RTMPose model identifies the two-dimensional image based on an MMPose framework and extracts the 2D keypoint data from the two-dimensional image. Claim 5 An electronic device according to claim 4, wherein the 2D keypoint data comprises 2D coordinate data of at least one of the head, body, shoulder, elbow, wrist, and fingers of the stroke patient included in the 2D image. Claim 6 An electronic device according to claim 1, wherein the deep learning model labels clinical measurements corresponding to the 2D keypoint data for the test operation, and the clinical measurements for the upper limb function include at least one of an FMA-UE classification result based on clinical measurements, a BBT classification result based on clinical measurements, a spatiotemporal parameter regression analysis value based on clinical measurements, and an APS regression analysis value based on clinical measurements. Claim 7 An electronic device according to claim 1, wherein the at least one processor inputs the 2D keypoint data into the deep learning model, generates a feature map based on features of the 2D keypoint data extracted while passing through a convolution layer of the CNN, converts the feature map into a vector to generate feature values ​​of the 2D keypoint data, and performs at least one of the FMA-UE classification, the BBT classification, the spatiotemporal parameter regression analysis, and the APS regression analysis based on the feature values. Claim 8 An electronic device according to claim 1, wherein when performing the FMA-UE classification, the at least one processor calculates the upper limb function of the stroke patient as a score between 0 and 66 based on a predetermined motor evaluation, and classifies the upper limb stroke patient into at least one of severe, moderate, and mild based on the score. Claim 9 An electronic device according to claim 1, wherein when performing the BBT classification, the at least one processor evaluates the upper limb function of the stroke patient based on the number of blocks the stroke patient can move to another place with one hand within a predetermined time, and classifies the upper limb stroke patient into at least one of normal, mild, and severe. Claim 10 An electronic device according to claim 1, wherein when performing the spatiotemporal parameter regression analysis, the at least one processor extracts at least one of MT (Movement Time), IoC (Index of Curvature), and NoMU (Number of Movement Units) from the 2D keypoint data, and analyzes the upper limb function of the stroke patient according to time and space based on at least one of the MT, the IoC, and the NoMU. Claim 11 An electronic device according to claim 1, wherein when performing the APS regression analysis, the at least one processor generates an APS (Arm Profile Score) based on the 2D keypoint data and analyzes the upper limb function of the stroke patient based on the APS. Claim 12 A method for predicting upper limb function in a stroke patient, comprising: collecting a two-dimensional image of the stroke patient performing a test movement using an image capturing unit; generating 2D keypoint data from the two-dimensional image using a posture estimation model; and performing FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that has learned the correlation between the 2D keypoint data and upper limb function; wherein the 2D keypoint data is recorded as time-series data of the stroke patient's body position information changing over time as a two-dimensional coordinate system based on the x-axis and y-axis, and the deep learning model comprises: a CNN model that extracts feature values ​​from the 2D keypoint data and performs the FMA-UE classification and the BBT classification based on the extracted feature values; and a CNN model that extracts feature values ​​from the 2D keypoint data and performs the spatiotemporal parameter regression analysis and the APS regression analysis based on the extracted feature values. Claim 13 A method for predicting upper limb function according to claim 12, wherein the test motion comprises a first task of extending the arm to grasp a cup, a second task of bending the arm to bring the cup toward the body, a third task of extending the arm again to place the cup on a table after completing the drinking motion, and a fourth task of returning the arm to its original position. Claim 14 In claim 12, the operation of collecting a two-dimensional image of the stroke patient is to collect the two-dimensional image by using the image capturing unit to capture the stroke patient performing the test operation in a single sagittal view at a frame rate of 30 frames per second or more and at a resolution of 1920x1080 pixels or more. Claim 15 In claim 12, the operation of generating 2D keypoint data from the 2D image utilizes an RTMPose model based on the COCO-wholebody dataset, and the RTMPose model identifies the 2D image based on the MMPose framework and extracts the 2D keypoint data from the 2D image, a method for predicting upper limb function. Claim 16 A method for predicting upper limb function according to claim 15, wherein the 2D keypoint data comprises 2D coordinate data of at least one of the head, body, shoulder, elbow, wrist, and fingers of the stroke patient included in the 2D image. Claim 17 A method for predicting upper limb function according to claim 12, wherein the deep learning model labels clinical measurements corresponding to the 2D keypoint data for the test action, and the clinical measurements for the upper limb function include at least one of an FMA-UE classification result based on clinical measurements, a BBT classification result based on clinical measurements, a spatiotemporal parameter regression analysis value based on clinical measurements, and an APS regression analysis value based on clinical measurements. Claim 18 In claim 12, the operation of performing at least one of FMA-UE classification, BBT classification, spatiotemporal parameter regression analysis, and APS regression analysis using a deep learning model that has learned the correlation between the 2D keypoint data and the upper limb function comprises inputting the 2D keypoint data into the deep learning model, generating a feature map based on features of the 2D keypoint data extracted while passing through a convolution layer of a CNN, converting the feature map into a vector to generate feature values ​​of the 2D keypoint data, and performing at least one of the FMA-UE classification, the BBT classification, the spatiotemporal parameter regression analysis, and the APS regression analysis based on the feature values. Claim 19 In claim 12, the operation of performing the FMA-UE classification using the deep learning model calculates the upper limb function of the stroke patient as a score between 0 and 66 based on a predetermined motor evaluation, and classifies the upper limb stroke patient into at least one of severe, moderate, and mild based on the score, a method for predicting upper limb function. Claim 20 In claim 12, the operation of performing the BBT classification using the deep learning model evaluates the upper limb function of the stroke patient based on the number of blocks the stroke patient can move to another place with one hand within a predetermined time, and classifies the upper limb stroke patient into at least one of normal, mild, and severe, a method for predicting upper limb function. Claim 21 In claim 12, the operation of performing the spatiotemporal parameter regression analysis using the deep learning model comprises extracting at least one of MT (Movement Time), IoC (Index of Curvature), and NoMU (Number of Movement Units) from the 2D keypoint data, and analyzing the upper limb function of the stroke patient according to time and space based on at least one of the MT, the IoC, and the NoMU, a method for predicting upper limb function. Claim 22 In claim 12, the operation of performing the APS regression analysis using the deep learning model comprises generating an APS (Arm Profile Score) based on the 2D keypoint data and analyzing the upper limb function of the stroke patient based on the APS, a method for predicting upper limb function.

Images