Artificial intelligence-based robot-assisted orthopedic exercise system and method

Abstract

Disclosed is an exercise system including: a walking exercise device including an effector having at least one link that drives a pedal on which a patient stands; and a system operation device for controlling a main effector according to a training plan for each individual to conduct walking training for the patient, wherein the system operation device includes: an index prediction unit having an index prediction model for predicting a walk evaluation index of the patient on the basis of basic information including personal information and initial walking state information of the patient; a training planning unit for establishing an individual training plan for each patient on the basis of the predicted waling evaluation index; a training evaluation unit for evaluating a training result on the basis of training result data according to a result of rehabilitation training of the patient performed by the walking exercise unit; and a reporting unit for performing reporting on the training result on the basis of the evaluation of the training result.

Description

AI-based robot-assisted orthopedic exercise system and method

[0001] The present disclosure relates to a robot-assisted orthopedic exercise system, and more particularly, to a patient-friendly AI-based assistive orthopedic exercise system capable of automatically establishing and executing a personalized orthopedic exercise plan based on artificial intelligence (AI) and automatically modifying the exercise plan according to the rehabilitation effect of the exercise, and a control method thereof.

[0002] The Robot-Assisted Orthopedic Exercise System was developed as a device to aid in gait rehabilitation for patients with walking difficulties. This system improves walking function and helps restore lower body muscle strength in patients with walking difficulties. It is an end-effector-type robotic device that supports and assists the patient's body movements, enabling them to perform gait rehabilitation more safely and effectively.

[0003] These systems can be adjusted according to the patient's physical ability and condition, and provide gait training by simulating or assisting walking movements, and also detect and compensate for the patient's movements to help improve walking ability more safely and efficiently and restore independence in daily life.

[0004] In these existing systems, a therapist is present to guide and assist the rehabilitation exercise process. However, in some cases, patients are unable to adapt to the operation of the pedals and are unable to respond or conform to the movement of the pedals. In addition, there are cases where effective rehabilitation is difficult even after a long period of walking training.

[0005] Therefore, the development of a rehabilitation training system that enables safer and more effective rehabilitation training, thereby helping patients quickly rehabilitate and return to their daily lives, is very beneficial.

[0006] The present disclosure presents an artificial intelligence-based robot-assisted orthopedic exercise system and method capable of establishing a patient-friendly gait training plan.

[0007] The present disclosure presents an artificial intelligence-based robot-assisted orthopedic exercise system and method thereof, which can reduce the intervention of a therapist by prescription of a medical professional.

[0008] The present disclosure presents a patient-personalized and friendly artificial intelligence-based robot-assisted orthopedic exercise system and method thereof.

[0009] An artificial intelligence-based robot-assisted orthopedic exercise system according to the present disclosure:

[0010] An artificial intelligence-based robot-assisted orthopedic exercise system comprising a walking exercise device having a pedal on which a patient stands, an actuator having one or more links for driving the pedal, and a system operating device that controls the main actuator according to a training plan established for each patient to perform walking training for a patient standing on the pedal.

[0011] The above system operating device;

[0012] An indicator prediction unit having an indicator prediction model that predicts a gait evaluation indicator of the patient based on the patient's initial gait status information or gait training result information after initial gait training;

[0013] A training planning unit that establishes a training plan for the patient performed by the gait exercise unit based on the predicted gait evaluation index;

[0014] A training evaluation unit that evaluates the training results based on training result data according to the results of the patient's rehabilitation training performed by the above walking exercise unit; and

[0015] A reporting unit is provided to perform reporting on training results based on evaluation of the above training results.

[0016] According to one or more embodiments, the indicator prediction unit may have, as the indicator prediction model, at least one of a Recurrent Neural Network (RNN) model and a Long Short-Term Memory Model (LSTM).

[0017] According to one or more embodiments, the training planning unit may have a multi-input, multi or single output model as the training planning model.

[0018] According to one or more embodiments, the device may further include a detection unit that detects at least one of Ground Reaction Force (GRF), Body Weight Support (BWS), Upper body inclination or tilting, Joint angle, Cadence, Stride length, and Ankle angle of a patient undergoing gait training as training information.

[0019] According to one or more embodiments, the device may further include a monitoring unit that monitors the status of rehabilitation training performed by the gait exercise unit according to the training plan, obtains information during the training, and generates training status information related to the patient's gait training therefrom.

[0020] According to one or more embodiments, the monitoring unit may provide the patient with training guidelines or recommended guidelines or current training status for improving gait related to the gait training based on the information during the training.

[0021] According to one or more embodiments, the monitoring unit may further include an HMD (Head-Mounted Display) that presents necessary information related to the patient's walking motion, training status information, or recommended guidelines related to training to the patient.

[0022] According to one or more embodiments, the device may further include a head-mounted display (HMD) that presents at least one of necessary information related to the patient's walking motion, training status information, or recommended guidelines related to training to the patient.

[0023] According to one or more embodiments, the training evaluation unit may include an anomaly detection model based on time series information that analyzes the current training result as a training evaluation model, or analyzes the trend of the result according to repeated training by analyzing the previous training result and the current training result, and performs a future training prediction.

[0024] According to one or more embodiments, the reporting unit, as a reporting model, can analyze previous training results and current training results to generate reporting content in the form of text, images, or videos.

[0025] Method for providing artificial intelligence-based robot-assisted orthopedic exercise according to the present disclosure:

[0026] A step of predicting a gait evaluation index of a patient based on information on the patient's initial gait status or information on the results of gait training after initial gait training, by an indicator prediction unit;

[0027] A step in which the training planning department establishes a training plan for each patient based on the above gait evaluation index;

[0028] A step of performing rehabilitation training of a patient according to the training plan, wherein a walking exercise device having a pedal on which a patient stands, for example, an end-effector type, and an actuator having one or more links for driving the pedal;

[0029] A step in which the training evaluation department evaluates the training results based on the training result data according to the training results; and

[0030] The reporting unit includes a step of performing a report on the training results based on the evaluation of the training results.

[0031] According to one or more embodiments of the above method, the indicator prediction unit can predict the patient's gait evaluation indicator using at least one of a Recurrent Neural Network (RNN) model or a Long Short-Term Memory model as the indicator prediction model.

[0032] According to one or more embodiments of the above method, the training planning unit can establish the training plan using a multi-input, multi or single output model.

[0033] According to one or more embodiments of the above method, the detection unit can detect at least one of GRF (Ground Reaction Force), BWS (Body Weight Support), upper body inclination or tilting, joint angle, cadence, stride length, and ankle angle of a patient undergoing gait training as training information.

[0034] According to one or more embodiments of the above method, the monitoring unit can monitor the status of rehabilitation training performed by the gait exercise unit according to the training plan, obtain information during the training, and generate training status information related to the patient's gait training therefrom.

[0035] According to one or more embodiments of the above method, the monitoring unit may provide the patient with training guidelines or recommended guidelines or current training status for improving gait related to the gait training based on the information during the training.

[0036] According to one or more embodiments of the above method, a head-mounted display (HMD) can present necessary information related to the patient's gait movement, training status information, or recommended guidelines related to training to the patient.

[0037] According to one or more embodiments of the above method, the time series information-based anomaly detection model as the training evaluation model can perform future training prediction by analyzing the current training result or analyzing the trend of the result according to repeated training by analyzing the previous training result and the current training result.

[0038] According to one or more embodiments of the above method, the reporting unit can analyze previous training results and current training results to generate reporting content in the form of text, images, or videos.

[0039] Figure 1 is a block diagram of an artificial intelligence-based robot-assisted orthopedic exercise system according to the present disclosure.

[0040] FIG. 2 is a schematic perspective view of an artificial intelligence-based robot-assisted orthopedic exercise system according to one embodiment of the present disclosure;

[0041] FIG. 3 is a side view of an artificial intelligence-based robot-assisted orthopedic exercise system according to one embodiment of the present disclosure;

[0042] FIG. 4 is a front view of an artificial intelligence-based robot-assisted orthopedic exercise system according to one embodiment of the present disclosure;

[0043] Figure 5 illustrates a scene in which a patient is training while wearing an HMD (Head-mounted Display) in a gait training system according to the present disclosure.

[0044] FIG. 6 is a schematic perspective view of a walking motion unit of an artificial intelligence-based robot-assisted walking training system according to the present disclosure.

[0045] Figure 7 illustrates a normal gait cycle, showing the posture of the feet, knees, and thighs during normal walking.

[0046] Figure 8 illustrates the regulations at each position in the walking pattern.

[0047] FIG. 9 illustrates functional parts based on the software and hardware structure of the system control device according to the present disclosure.

[0048] FIG. 10 illustrates the configuration of a data set applied to a gait evaluation index prediction model applied to an artificial intelligence-based robot-assisted orthopedic exercise method and system according to one embodiment of the present disclosure.

[0049] FIG. 11 is a flowchart of a training process for a new patient in an artificial intelligence-based robot-assisted orthopedic exercise presentation method according to one embodiment of the present disclosure.

[0050] FIG. 12 is an overall flowchart of an artificial intelligence-based robot-assisted orthopedic exercise presentation method according to one embodiment of the present disclosure.

[0051] FIG. 13 illustrates a code snippet for a class of LSTM (Long Short-Term Memory) models in an artificial intelligence-based robot-assisted orthopedic exercise method and system according to one embodiment of the present disclosure.

[0052] FIG. 14 illustrates a code snippet defining a forward propagation process of a model in an artificial intelligence-based robot-assisted orthopedic movement method and system according to one embodiment of the present disclosure.

[0053] FIG. 15 illustrates a code snippet for the definition of a Python class that implements early stopping in an artificial intelligence-based robot-assisted orthopedic exercise method and system according to one embodiment of the present disclosure.

[0054] FIG. 16 illustrates a code snippet for the definition of train_model, a function for model learning, in an artificial intelligence-based robot-assisted orthopedic movement method and system according to one embodiment of the present disclosure.

[0055] FIG. 17 illustrates a code snippet for a data evaluation function that evaluates new data from a gait training patient and predicts the gait evaluation index (FAC, Functional Ambulation Categories) of the patient in an artificial intelligence-based robot-assisted orthopedic exercise method and system according to one embodiment of the present disclosure.

[0056] FIG. 18 illustrates a code snippet for the execution of an LSTM model in an artificial intelligence-based robot-assisted orthopedic movement method and system according to one embodiment of the present disclosure.

[0057] FIG. 19 is a code snippet for inputting data for training a model according to one embodiment of the present disclosure and generating a dataframe accumulating the data.

[0058] FIG. 20 is a code snippet for declaration of a transformer model, learning settings, model generation, etc. according to one embodiment of the present disclosure.

[0059] FIG. 21 is a code snippet for training and test data prediction of a transformer model according to one embodiment of the present disclosure;

[0060] FIG. 22 illustrates a graphical interface showing various parameters of gait training according to one embodiment of the present disclosure, and

[0061] Figure 23 is a butterfly diagram showing the average plantar pressure distribution of the patient during training as a result of the patient's gait training.

[0062] FIG. 24 is a code snippet of the declaration part of the dataset class for training model 3 according to one embodiment of the present disclosure.

[0063] Figure 25 is a code snippet defining Model 3 according to one embodiment of the present disclosure;

[0064] FIG. 26 is a code snippet of a feedback generation function that outputs feedback provided to a patient according to one embodiment of the present disclosure;

[0065] FIG. 27 is a code snippet of a real-time data processing function according to one embodiment of the present disclosure;

[0066] Figure 28 is a code snippet of a function for learning Model 3 according to one embodiment of the present disclosure.

[0067] FIG. 29 is a code snippet of a function for evaluating a trained model 3 according to one embodiment of the present disclosure, and

[0068] FIG. 30 is a snippet of execution code for generating model 3 trained according to one embodiment of the present disclosure.

[0069] Figure 31 is a code snippet of a data preprocessing and learning pipeline utilizing an LSTM model for anomaly detection and trend prediction according to the present disclosure.

[0070] Figure 32 is a code snippet illustrating the declaration of an LSTM-based Autoencoder class and a Seq2Seq class for anomaly detection and trend prediction according to the present disclosure, and the initialization process of two models utilizing these classes.

[0071] Figure 33 is a function for training the above-mentioned anomaly detection model (M4a), which is one element of the M4 model according to the present disclosure.<train_anomaly_model> Illustrate the definition of a class.

[0072] Figure 34 illustrates the definition of a function for training a trend prediction model (M4b) in the M4 model according to the present disclosure.

[0073] Figure 35 illustrates the definition of a function evaluate_anomaly_model for evaluating an anomaly detection model (M4a) in the M4 model according to the present disclosure.

[0074] FIG. 36 is a pseudo code snippet of a function that visualizes and analyzes test results and suggests improvement suggestions in the M4 model according to the present disclosure.

[0075] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, the embodiments of the present invention may be modified in various different forms, and the scope of the present invention should not be construed as being limited by the embodiments described below. It is preferable to interpret that the embodiments of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art. Like reference numerals denote like elements throughout. Furthermore, various elements and areas in the drawings are schematically drawn. Therefore, the present invention is not limited by the relative sizes or intervals drawn in the accompanying drawings.

[0076] While terms such as "first" and "second" may be used to describe various components, these components are not limited by these terms. These terms are used solely to distinguish one component from another. For example, a first component could be referred to as a second component, and vice versa, without departing from the scope of the present invention.

[0077] The terminology used in this application is only used to describe specific embodiments and is not intended to limit the inventive concept. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the expressions "comprises" or "has" indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but do not preclude the possibility of the presence or addition of one or more other features, numbers, operations, components, parts, or combinations thereof.

[0078] Unless otherwise defined, all terms used herein, including technical and scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which the present invention pertains. Furthermore, it is to be understood that commonly used terms, such as those defined in dictionaries, should be interpreted to have a meaning consistent with their meaning within the relevant technical context, and should not be interpreted in an overly formal sense unless explicitly defined herein.

[0079] In some embodiments, where implementations are otherwise feasible, specific process sequences may be performed in a different order than described. For example, two processes described in succession may be performed substantially simultaneously, or in a reverse order from the described order.

[0080] FIG. 1 is a block diagram of an artificial intelligence-based robot-assisted orthopedic movement system according to the present disclosure, FIG. 2 illustrates an embodiment of an artificial intelligence-based robot-assisted orthopedic movement system according to the present disclosure, FIG. 3 is a side view thereof, and FIG. 4 is a front view thereof.

[0081] First, referring to FIG. 1, the exercise system (100) includes a system control unit (111) with a composite configuration of hardware and software, a gait training apparatus (105) controlled by the system control unit, a main or control monitor (110) that controls the system control unit (111), a status display monitor (113) that displays the patient's exercise status, etc., a composite input unit (114) that inputs various electrical and optical data including a keyboard (112), and an HMD (Head-Mounted Display) 115 that the patient wears as an additional option. The system control unit (111) according to the present disclosure has a structure of a computer system based on a virtual agent that includes one or more models trained in machine learning or deep learning to perform data processing and prediction.

[0082] The above-mentioned walking exercise unit (105) is equipped with two pedals on the left and right sides for the patient to stand on and a driving device for operating the two pedals, and is equipped with a training-state detection unit (105a) equipped with various sensors for detecting the patient's status information during walking training, either on the walking exercise unit (105) itself or around it.

[0083] The above training-state detection unit (105a) may include a signal processor that generates a signal to be sent to the system control device (111) from a signal from the sensor as well as a sensor that detects a signal. The HMD (115) is part of the monitor device in the exercise system (100) of the present disclosure, and provides the patient with information necessary during training.

[0084] Additionally, the system control device (100) may further be provided with an optical detection device for detecting the patient's movement status, for example, a video camera such as a webcam, which is an optical input device.

[0085] Referring to FIGS. 2 to 4, the exercise system (100) is a type of robot that provides walking training to a patient and provides two left and right walking movement units (105) controlled by the system control device (111). The two walking movement units (105) are arranged between the left and right main body bases (101) and the right main body bases (101) that are arranged side by side at a predetermined interval. The two pedals (105a) of the two left and right walking movement units (105) can move in harmony with each other in response to a three-degree of freedom movement trajectory, such as rotation of the left and right ankles and up and down and forward and backward movements of the left and right feet, when walking, and in some cases, they can also move in an intentionally disharmonious manner.

[0086] The pedals are moved by a control signal from the system control device (111) to enable the patient to perform a walking exercise suitable for the patient. The patient stands on them and performs a walking exercise in accordance with a training walking trajectory determined by the movement of the pedals. An ankle band that can be installed here secures the patient's foot to the pedal in an appropriately loose or appropriately tight manner to allow the patient's foot to move within a certain range.

[0087] The movement of these two pedals (105a) is controlled by the system control device to accommodate the abnormal gait pattern while training the gait pattern to be closer to a normal gait pattern for the purpose of gait training of the patient.

[0088] For this gait training, the left walking movement unit (105) has a pedal (105a) and an operating link (105b) to which it is dynamically connected, and the operating link (105b) is coupled to the left body base (101) via a first driving device (105c) provided on the left body base (101) so as to enable linear reciprocating motion and rotational motion, and the right walking movement unit (105) is also coupled to the right body base (101) via a first driving device (105c) of the right body base (101) so as to enable reciprocating motion and rotational motion. The pedal (105a) is connected to the operating link (105b) via a driving device, for example, a driving device capable of controlling a rotation angle, and the pedal (105a) is forced to move relative to the operating link (105b) within a predetermined angular range by the driving device.

[0089] The linear reciprocating motion of each of the walking motion parts (105) by the first driving device is performed independently, but moves away from each other or closer to each other and then intersects each other in response to the movement of the left and right feet during walking. The first driving device (105c) causes the linear reciprocating motion and rotational motion of the operating link (105b), and may include a plurality of driving devices. For example, the first driving device (105c) may have a linear motion driving device for controlling the linear reciprocating motion of the operating link or the entire walking motion part (105) equipped therewith, and a rotational or rotary motion driving device for controlling the rotational motion of the operating link (105b).

[0090] In front of the above walking exercise unit (105), an assembly is provided in which a saddle (104) and a safety bar (106) thereon or a fence part (107) including the same for supporting the chest, etc. are combined into one, and this assembly is installed with respect to a lift device (102) via an up-down frame (103). The lift device (102) is an assembly raising / lowering device that adjusts the height of the assembly to match the patient's physical condition. A system status display part (109) facing the patient is provided at the top of this lift device (102).

[0091] At the front of the system (100) viewed by the patient, a lifting or fixed support column (113a) is installed, and a status display monitor (113) that displays the operating status of the entire system is connected thereto.

[0092] FIG. 5 illustrates a scene in which a patient (1) is training while wearing an HMD (115) described below in a gait training system (100) according to the present disclosure. As illustrated in FIG. 5 , for example, in the case of a severely ill patient, the patient (1) can perform gait training while holding a safety bar (106) without being on a saddle (104). Depending on the condition of the patient (1), for example, in the case of a mildly ill patient, the saddle (104) can be folded, and thus, the patient (1) can perform gait training without relying on the saddle (104).

[0093] The pedal (105a) is operated by the system control device to force the patient to walk regardless of the patient's will. At this time, pressure sensors, etc. are installed on the saddle (104) and the pedal (105a) to detect the load applied to the saddle (104) and the load applied to the pedal during walking training. In particular, a plurality of pressure sensors are provided before and after the pedal to detect the local pressure applied to the pedal, thereby detecting the degree of pressure applied to the sole of the foot or the presence of contact with the pedal.

[0094] Figure 6 is a schematic perspective view of the walking motion unit (105) dynamically coupled to the main body base (101).

[0095] Referring to FIG. 6, the operating link (105b) is coupled to the main body base (101) via a first driving device (105c) to enable linear reciprocating motion (LM) and rotational motion (RM1). According to one embodiment, the first driving device (105c) is a driving motor, and a separate reciprocating motion driving device may be connected to one side or the lower side thereof. Meanwhile, the pedal (105a) is coupled to the operating link (105b) via a second driving device (105d) to enable rotational motion (RM2) of a predetermined angle with respect to the operating link (105b), and a band capable of binding a patient's foot or ankle may be installed thereon.

[0096] Figure 7 illustrates a normal gait cycle, showing the posture of the feet, knees, and thighs during normal gait, and Figure 8 illustrates the regulations for each position in the gait pattern.

[0097] Referring to Figure 7, in one gait cycle, the stance phase is the section where the foot touches the ground, and the swing phase is the section where the foot is lifted off the ground.

[0098] Within each gait cycle, there are three tasks:

[0099] 1. Weight Acceptance

[0100] This period has two parts: initial contact (when the foot first touches the ground) and loading response (when the sole of the foot touches the ground).

[0101] 2. Single Limb Support

[0102] This is the mid-stance period, where the sole of the foot touches the ground, the foot of the opposite leg leaves the ground, and the heel is lifted and the opposite leg swings.

[0103] 3. Limb Advancement

[0104] This period is the period when the other foot is planted, and includes the free swing, where the toes of the front foot lift off the ground as the back foot leaves the ground; the mid swing, where the foot that was on the ground earlier begins to lift off while the feet are together; and the terminal swing, where the heel of the front foot begins to touch the ground as the back foot pushes off the ground.

[0105] The above gait pattern is a normal gait pattern, and patients are trained to acquire this normal gait pattern.

[0106] However, for patients who have difficulty walking normally, their feet may become misaligned during gait training. For example, during the mid-stance cycle, the heel of the rear foot may not lift off the ground during the heel lift phase, but remain on the ground. This is because the patient's body does not follow the normal gait pattern. The robot of the present disclosure can forcibly bend the knee or ankle joint during this phase to match the patient's gait pattern, thereby forcing the heel of the rear foot to lift off the ground. All of these forced joints can be performed within the cycle, allowing the patient to achieve a similarly normal gait pattern despite their physical limitations during gait training.

[0107] Below, a system control device that effectively, accurately, and efficiently performs gait training for rehabilitation training of the above-mentioned patient is described.

[0108] The system control device in the present disclosure controls the walking movement part largely through software and partially through hardware.

[0109] These system control devices operate based on hardware, namely, a computer system for executing software. Therefore, the system control device according to the present disclosure may include all or part of a general computer system. The software utilizes multiple artificial intelligence models to predict a patient's gait evaluation index from initial gait status information or gait training result information after gait training, establish an individualized training plan for each patient based on the gait evaluation index, evaluate the training results based on training result data according to the training results, and report the training results based on the evaluation of the training results.

[0110] According to the present disclosure, training of a general deep learning-based model is performed with each model for each function, and training using the trained models and applying the same are performed to establish, execute, evaluate, and report a patient's training plan, and in this process, update of the training plan for long-term training is performed.

[0111] FIG. 9 illustrates functional parts based on the software and hardware structure of the system control device (111) according to the present disclosure.

[0112] As illustrated in FIG. 9, the system control device (111) may include an input unit (111a), an indicator prediction unit (111b), a training planning unit (111c), a training progress unit (111d), a training evaluation unit (111e), and a reporting unit (111f).

[0113] The above input unit (111a) may include at least a keyboard, or an input unit (111a) including a keyboard, a webcam, and a communication unit, and an index prediction model (M1) that predicts a patient's gait evaluation index required for establishing a patient's training plan.

[0114] The above training planning unit (111b) may include a training planning model (M2) that plans training for individual patients according to their symptoms by referring to the indicators predicted by the indicator prediction unit (111b).

[0115] The above training progress unit (111d) may include a training progress model (M3) that monitors the training status according to the training plan and generates feedback on the training status to guide the patient and medical staff.

[0116] The above training evaluation unit (111e) may include an evaluation model (M4) that evaluates the results of training performed by the training progress unit (111d) and predicts future trends.

[0117] The above reporting unit (111f) may include a reporting model (M5) that reports the results of training obtained from the training evaluation unit (111e) to patients and training personnel in various forms.

[0118] In addition to the elements listed above, the system control device (111) may further include other additional software and / or hardware depending on the design of the system control device (111).

[0119] FIG. 10 is a schematic flowchart of a training process by the system control device (111) in a method for providing artificial intelligence-based gait training according to the present disclosure, which is performed by an exercise system (100) having the aforementioned system control device (111).

[0120] Referring to Fig. 10, in the training process, for a new patient, the process proceeds to step S13 after going through the data preparation stage by entering data (S11) and generating an initial gait evaluation index (S12). In the case of an already registered patient, the process proceeds to step S13 after verifying personal information and then calling up the existing gait evaluation index stored in the system.

[0121] In step S13, a training plan is generated using initial or existing gait assessment metrics.

[0122] In step S14, the patient's gait training is performed according to the above training plan.

[0123] At step S15, an intermediate gait evaluation index is generated based on the current gait training results.

[0124] In step S16, the training results are analyzed by comparing the current gait evaluation index based on the current training results with the existing gait evaluation index.

[0125] Step S17 is where various forms of reports are generated and presented based on the analysis of training results. After completing this step, a decision is made as to whether to continue the training, and the training either continues or ends. If continuation is selected, the training proceeds to step S13 or S14, depending on whether the training plan has been updated.

[0126] Figure 11 is a flow chart of a more specific training process for new training patients.

[0127] Each step schematically illustrated in Figure 11 can be used to perform gait training on a patient one or more times, for example, using five different models (M1, M2, M3, M4, M5). In other embodiments, some of the models may be excluded.

[0128] When training begins, new patients begin with step S11 below, while previously registered patients, after verifying and selecting their information, continue with the training plan creation and update step S13 below. That is, as previously mentioned in the description of Fig. 7, steps S11 and S12 are performed once initially, and thereafter, when training begins, patients whose personal information has been specified begin the training process from step S13.

[0129] Step S11: An initial test data set for gait training for one patient is input.

[0130] S12: Input the input initial test data set into the gait evaluation index prediction model (M1) to generate an initial gait evaluation index.

[0131] S13: The above gait evaluation indicators are input into the training plan generation model (M2) to generate a training plan. If this process is not the initial training plan generation process but rather a generation process during an iterative training process, the training plan from the previous process is updated.

[0132] S14: Once a training plan is created or established, the patient's gait training is performed using it. This training is conducted by a walking exercise unit that the patient is riding. During the training process, the training status is monitored, and information is provided to the patient according to the training status, along with encouraging audio and / or text information. During the monitoring process, a webcam or Kinect sensor, which are components of the training status detection unit, are used to acquire real-time gait images of the patient. From this, a machine learning-based model is used to extract key points and other major joint locations of the patient, thereby displaying the skeleton, extracting joint angle information, and extracting gait trajectories.

[0133] This information can be presented through a status display monitor (113, Fig. 4), or alternatively, can be presented to the patient through an HMD (115, Fig. 4) worn by the patient. The model M3 applied here can be an interactive model, such as a chatbot, for example.

[0134] S15: In this step, an intermediate gait assessment index is generated based on the current gait training results. The gait assessment index prediction model (M1), previously used to predict the initial gait assessment index, is applied in this step.

[0135] S16: In this step, the training results are analyzed by comparing the current gait evaluation index based on the current training results with the existing gait evaluation index. The trained training analysis model (M4) is used to analyze the training results.

[0136] S17: This step involves creating and presenting various types of reports based on the analysis of training results. A model trained to generate reports from training results, such as a multimodal model (M5), can be applied at this stage.

[0137] After this step, it is determined whether to continue training, and if continuing training is selected, it moves to step S13 or S14. If the current training results require an update to the training plan, it moves to S13. Otherwise, if training is to be performed again with the current training plan, S14 moves directly to the gait training process.

[0138] Below, the aforementioned artificial intelligence models M1, M2, M3, M4, M5, etc., which are included in each part of the system control device, are described.

[0139]

[0140] I. Model 1 (M1)

[0141] Model 1 (M1) is a model that predicts the patient's gait evaluation index and can be either a Recurrent Neural Network (RNN) model or a Long Short-Term Memory Model (LSTM).

[0142] One embodiment of the present disclosure applies LSTM. LSTM is a type of recurrent neural network (RNN), a model designed to address long-term dependency problems, and has a structure capable of effectively learning sequence information in data.

[0143] During training of these LSTMs, dropout and early-stopping techniques are applied. Dropout is a regularization technique used to prevent overfitting of neural networks. It randomly disables (drops) some neurons during the training process to prevent the model from overfitting to specific patterns. This technique can be applied to the embodiments of the present disclosure. Early-stopping is a technique used to prevent overfitting during model training. It terminates training early if the performance of the validation data does not improve.

[0144] Figure 12 illustrates the composition of a data set from a patient for generating an initial gait evaluation index by the gait evaluation index prediction model (M1).

[0145] Basic input data is basic information entered into the exercise device (100) in the medical hospital, and this must be managed in compliance with personal information protection procedures.

[0146] This dataset includes patient information data, including patient personal information, patient symptom information, gait evaluation indices (FAC, Functional Ambulation Categories) determined by medical staff, and Berg Balance Scale (BBS).

[0147] The above patient personal information includes age, sex, height, weight, leg length, foot size, and symptoms, and the symptoms information includes patient information such as symptoms and references for each symptom, date of symptom onset, and lower extremity symptoms (both feet, left foot, right foot).

[0148] In addition, it includes clinical evaluation indices using clinical evaluation tools such as FAC based on medical staff's diagnosis results, and pre-gait training data.

[0149] Meanwhile, as gait information data, a data set measuring the results of gait training while walking on a flat surface before boarding a gait rehabilitation training robot is included, including information on upper body inclination and lower extremity joint angles during walking. Furthermore, a data set measuring the results of gait training after boarding a gait rehabilitation training robot is included, including a training result data set while boarding the exercise system of the present disclosure. This includes information on average GRF (Ground Reaction Force), COP (Center of Pressure), BWS (Body Weight Support), lower extremity joint angles, and left-right balance for each gait cycle (0-99%) during walking.

[0150] Meanwhile, as optional input data, information provided by the hospital or patient includes doctor comments, gait images, gait training videos (MOVs), and training measurement information, such as GRF information measured during walking on a GRF measurement pad, COP-based average (Butterfly Diagram) image information during walking, and upper body tilt information.

[0151] Therefore, Model 1 (M1) is trained to input the above data and output a gait evaluation index.

[0152] FAC, one of the input data, has six indicators and can be organized as follows.

[0153] FAC 0: Non-ambulatory - Inability to walk.

[0154] FAC 1: The ability to walk is present, but walking ability is very limited and requires walking assistance.

[0155] FAC 2: The ability to walk is weak and walking assistance is required.

[0156] FAC 3: Some walking ability has been restored, but some walking assistance is required.

[0157] FAC 4: Walking ability is almost restored, and independent walking is possible without assistance.

[0158] FAC 5: Functional walking ability is fully restored and the patient can walk independently without assistance.

[0159] Meanwhile, the BBS consists of several items designed to assess the ability to maintain balance while performing various activities. These items include the ability to stand, transition from sitting to standing, maintain a sitting-standing posture, and turn the head while standing. Each item is rated on a scale of 0 to 4, resulting in a total score of 56, with higher scores indicating better balance.

[0160] Model 1 predicts the gait evaluation index of patients undergoing gait rehabilitation training by applying the grade reflecting the gait evaluation index standards applied in the medical field (FAC, BBS, etc.), and this can be a Recurrent Neural Network (RNN) model or a Long Short-Term Memory model (LSTM).

[0161] As a model training method to increase the judgment resolution in preparation for FAC (applied at 6 levels), the time-series gait training result data (3 minutes each of flat ground and stair walking training) obtained from a group of non-disabled people (multiple people) using this disclosed exercise system is set as the supervised learning model group (correct answer: 100%), and the time-series gait training result data of a person with walking disabilities is digitized from 0 to 100% using an anomaly detection model based on time-series information to determine the gait grade.

[0162] The model's output data can vary depending on the design. However, when the FAC rating is applied, the measured values ​​(%) reflecting the time-series information-based anomaly detection model (Anomaly Detection Model) can be compared with the measured values ​​(100%) of the non-disabled person and the measured values ​​(%) of the disabled person, as shown below.

[0163] yes)

[0164] 0 ~ 20% = FAC 0

[0165] 21 ~ 40% = FAC 1

[0166] 41 ~ 60% = FAC 2

[0167] 61 ~ 70% = FAC 3

[0168] 71 ~ 80% = FAC 4

[0169] 81 ~ 100% = FAC 5

[0170]

[0171] In cases where the FAC grade is not applied, the walking grade can be determined more precisely by directly applying the measurement value (%) that reflects the time series information-based anomaly detection model (Anomaly Detection model) with the disabled measurement value corresponding to the non-disabled measurement value (100%).

[0172]

[0173] An example of a model performed as above is described below.

[0174] Figure 13 illustrates a code snippet for initializing the ImprovedLSTMModelWithDropout class, which improves the Long Short-Term Memory (LSTM) model by adding a dropout layer. This code was written using PyTorch.

[0175] The ImprovedLSTMModelWithDropout class defines an LSTM model by inheriting nn.Module.

[0176] This class has an initialization method "__init__" and a "forward" method, and the parameters of the initialization method are as follows:

[0177] - input_size: Number of features in input data

[0178] - hidden_size: Size of the LSTM hidden state

[0179] - num_layers: number of LSTM layers

[0180] - output_size: Number of features in the output data

[0181] - dropout: dropout rate

[0182] And, his instance variables are as follows:

[0183] - self.lstm: defines the LSTM layer

[0184] - self.fc1: Definition of the first fully connected layer

[0185] - self.dropout: define the dropout layer, and

[0186] - self.fc2: defines the second fully connected layer

[0187]

[0188] Figure 14 illustrates a code snippet defining the forward propagation process of the model.

[0189] It involves a series of steps that process the input data to generate final predictions, with each step transforming the input data to incrementally generate the model's predictions.

[0190] Meanwhile, the main processing steps of the forward method are as follows:

[0191] (1) Forward Step 1:

[0192] h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)

[0193] c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)

[0194] - In this step, the initial hidden state (h0) and cell state (c0) are defined. The objects h0 and c0 are tensors representing the initial hidden state and cell state of the LSTM network, respectively. These are state values ​​that are set before the network processes the first sequence, providing the initial state for each layer and each element of the LSTM. In addition, the torch.zeros function is used to create tensors with all elements initialized to 0. These objects, or tensors, have a specific batch size (batch size, x.size(0)) and the size of the hidden state (self.hidden_size), and their dimensions are set according to the number of network layers (self.num_layers). Furthermore, the to(x.device) method is used to move the created tensor to the same device as the input data x (e.g., CPU or GPU).

[0195] (2) Forward Step 2:

[0196] out = self.lstm(x, (h0, c0))

[0197] In this step, the input data x and the initial state (h0, c0) are passed to the LSTM layer for processing. Here, x is the input data tensor (batch size, sequence length, number of input features), out: the output of the LSTM layer. It is a tensor in the form of (batch size, sequence length, size of hidden state).

[0198] (3) Forward Step 3:

[0199] out = self.fc1(out[:, -1, :])

[0200] out[:, -1, :]: Use only the output of the last timestep of the LSTM. This output passes through the first fully connected layer (self.fc1).

[0201] (4) Forward Step 4:

[0202] out = torch.relu(out)

[0203] The ReLU activation function is applied to the output of the first fully connected layer. This adds nonlinearity and enhances the expressive power of the model.

[0204] (5) Forward Step 5:

[0205] self.dropout(out)

[0206] Dropout is applied to randomly disable some neurons.

[0207] (6) Forward Step 6:

[0208] self.fc2(out)

[0209] The output of the dropout layer is passed to the second fully connected layer (self.fc2) to generate the final prediction.

[0210] (7) Forward Step 7:

[0211] return out

[0212] At this stage, the final predicted value of the model is returned.

[0213]

[0214] In this way, the forward method defines the process by which input data is transformed into the final output value through LSTM, a fully connected layer, an activation function, and dropout, with each step gradually transforming the data to generate the final prediction.

[0215] Figure 15 illustrates a code snippet for the definition of an Early Stopping class.

[0216] This class is used to prevent overfitting and support efficient learning during the training process of machine learning models. Its main components and functions are as follows.

[0217] (1) Initialization method (__init__)

[0218] def __init__(self, patience=15, delta=0.0005):

[0219] - patience (int): An instance variable that sets how many epochs to wait for when performance does not improve. The default value is 15, which is passed as a parameter.

[0220] - delta (float): An instance variable that stores the minimum change to be recognized as a performance improvement. This value defines how much the validation loss must improve compared to the previous minimum loss. The default value is 0.0005, which is also passed as a parameter.

[0221] - best_score (float): An instance variable that stores the best observed performance score (negative value of validation loss).

[0222] - early_stop (bool): An instance variable that stores a flag indicating whether to terminate early.

[0223] - val_loss_min (float): An instance variable that stores the minimum observed validation loss.

[0224] - counter (int): An instance variable that stores the number of epochs at which performance does not improve.

[0225] (2) Call method (__call__)

[0226] def __call__(self, val_loss, model):

[0227] - val_loss (float): Validation loss at the current epoch.

[0228] - model (torch.nn.Module): The PyTorch model object being evaluated.

[0229] This method evaluates the model's validation loss after each epoch to check for early stopping conditions. If performance does not improve by more than delta compared to the previous best, a counter is incremented. If this counter reaches patience, the early_stop flag is set to True to stop training. However, if performance improves, the counter is reset and the model state is saved as a checkpoint.

[0230]

[0231] (3) Checkpoint save method (save_checkpoint)

[0232] def save_checkpoint(self, val_loss, model):

[0233] This method saves the model's state to a file (checkpoint.pt). This is used when the model needs to be reloaded later or when you want to maintain a state with better performance. It also updates val_loss_min with the latest validation loss value.

[0234] The primary purpose of this early stopping class is to facilitate efficient learning, prevent overfitting, reduce unnecessary training time, and maintain the model's generalization ability. Early stopping is a crucial technique in many machine learning scenarios, and is particularly useful in deep learning models that require high computational costs.

[0235] Figure 16 illustrates a code snippet for the definition of train_model, a function for model training.

[0236] def train_model(model, X, y, num_epochs=100, learning_rate=0.0001):

[0237] - The function train_model is used to train a model, and receives model (PyTorch model object to train), X (input data tensor), y (output data tensor), num_epochs (number of epochs to train, default 100), and learning_rate (learning rate, default 0.0001) as parameters.

[0238] (1) Loss function: Mean squared error (MSE)

[0239] criterion = nn.MSELoss()

[0240] - A loss function is defined. Here, the mean squared error (MSE) loss function is used. MSE is the average of the squared differences between the predicted and actual values, and is primarily used in regression problems.

[0241] (2) Create an optimizer class

[0242] optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

[0243] - Create an Adam optimizer object to update the model's weights using the torch.optim.Adam class. The optimizer uses the learning rate learning_rate to adjust the model's weights along the gradient.

[0244] (3) Creation of an early termination class instance

[0245] early_stopping = EarlyStopping(patience=15, delta=0.0005)

[0246] - Create an instance of the EarlyStopping class for early termination. patience is the number of epochs to wait for even if the loss does not improve, and delta is the minimum amount of change to be considered an improvement.

[0247] (4) Learning loop

[0248] for epoch in tqdm(range(num_epochs), desc="Training Progress"):

[0249] This is a loop in which learning progresses. It repeats a total of num_epochs and uses the tqdm library to visually display the learning progress.

[0250] (4.1) Set the model to learning mode

[0251] model.train()

[0252] - Switch the model to training mode. This is necessary to make layers like dropout and batch normalization behave differently in training mode.

[0253] (4.2) Generating predicted values

[0254] outputs = model(X)

[0255] - Generate predicted values ​​by passing input data X to the model. Outputs are the predicted results of the model.

[0256] (4.3) Initializing the gradient of the optimizer

[0257] optimizer.zero_grad()

[0258] - Initializes the gradient buffer of the optimizer, which is needed to initialize the gradient before computing it via backpropagation.

[0259] (4.4) Loss calculation

[0260] loss = criterion(outputs, y)

[0261] - The loss between the predicted values ​​(outputs) and the actual values ​​(y) is calculated. Here, the MSE loss function is used to calculate the loss value.

[0262] (4.5) Calculating the loss slope

[0263] loss.backward()

[0264] - The gradient of the loss is calculated through backpropagation, and the gradient of each parameter is calculated at each step.

[0265] (4.6) Weight update

[0266] optimizer.step()

[0267] - Update the model weights using the optimizer based on the calculated gradient.

[0268] (4.7) Save current loss value

[0269] val_loss = loss.item()

[0270] - Convert the current loss value to a scalar value and store it. This will later be used as a criterion for early termination.

[0271] (4.8) Check early termination conditions

[0272] early_stopping(val_loss, model)

[0273] - Determines the early stopping condition using the loss value of the current epoch. The early_stopping object tracks the number of epochs without improvement and stops model training when the condition is met.

[0274] (4.9) Output early termination message

[0275] if early_stopping.early_stop: print(" / nEarly stopping")

[0276] - At this stage, if early_stopping.early_stop is True, an early termination message is output and the loop is exited, stopping learning.

[0277] (5) Output learning completion message

[0278] print('Model training complete')

[0279] - Finally, a message is printed indicating that learning is complete.

[0280]

[0281] The code snippet described step-by-step above implements a basic framework for training a PyTorch model using early stopping. The MSE loss function and the Adam optimizer are commonly used in regression tasks, and the early stopping mechanism helps prevent overfitting by stopping training when the validation loss no longer improves significantly.

[0282]

[0283] Figure 17 illustrates a code snippet for a data evaluation function that evaluates new data from a gait training patient and predicts the FAC of that patient.

[0284] This example function uses a PyTorch model to generate predictions for new data and calculates the similarity by comparing it with the actual values.

[0285] (1) Data loading:

[0286] def evaluate_new_data(file_path, model, data_mean, data_std, device):

[0287] The function evaluate_new_data reads a CSV file from file_path and converts it into a DataFrame by selecting only the necessary columns. The necessary columns are, for example, left Current Gait Cycle Rate (p_leftCurrentGaitCycleRate), left Current Ground Reaction Force (p_leftCurrentGRF), right Current Gait Cycle Rate (p_rightCurrentGaitCycleRate), and right Current GRF (p_rightCurrentGRF).

[0288] # Data preprocessing:

[0289] new_data['p_leftCurrentGaitCycleRate'] =

[0290] new_data['p_leftCurrentGaitCycleRate'].astype(int)

[0291] new_data['p_rightCurrentGaitCycleRate'] =

[0292] new_data['p_rightCurrentGaitCycleRate'].astype(int)

[0293] Convert p_leftCurrentGaitCycleRate and p_rightCurrentGaitCycleRate to integers and then perform normalization.

[0294] (2) Prepare model input data:

[0295] Separate the input data (X_new) and output data (y_new) from the normalized data. The input data includes p_leftCurrentGaitCycleRate and p_rightCurrentGaitCycleRate, and the output data includes p_leftCurrentGRF and p_rightCurrentGRF.

[0296] (2) Model prediction:

[0297] The input data is converted to a PyTorch tensor and passed to the model to generate predictions. The model is set to eval mode, and gradient computation is disabled using the torch.no_grad() method.

[0298] (3) Calculating the results:

[0299] The difference between the predicted and actual values ​​is calculated in percentage terms to obtain the average error rate. Similarity is then calculated based on this difference.

[0300] (4) FAC rating determination:

[0301] The patient's FAC grade is determined based on the similarity. For example, if the similarity is 0-40%, it is assigned FAC 1, etc.

[0302]

[0303] Figure 18 illustrates a code snippet for executing the above LSTM model.

[0304] (1) Specify required columns and set file path

[0305] In this process, the required data columns (required_columns) and file path (file_path) are first specified.

[0306] - file_paths: Paths to multiple CSV files to be used for model training.

[0307] - new_data_path and eval_data_path: Paths for new data and evaluation data

[0308] - model_save_path: Specifies the checkpoint file path for the trained model.

[0309] (2) Data loading and preprocessing

[0310] combined_data = load_and_combine_csv(file_paths, required_columns)

[0311] X, y, data_mean, data_std = preprocess_data(combined_data)

[0312] - load_and_combine_csv: A function that reads and combines data from multiple CSV files.

[0313] - preprocess_data: Preprocesses the combined data and returns the input data X, output data y, data mean data_mean, and data standard deviation data_std.

[0314] (3) Processor (GPU or CPU) usage settings

[0315] device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

[0316] - In this process, GPU memory is released, - If GPU is available, CUDA is set as the device, otherwise CPU is set as the device, and data is transferred to the set device.

[0317] (4) Model initialization and training

[0318] An improved LSTM model with dropout is created using the ImprovedLSTMModelWithDropout class, and the model is trained using training data using the train_model function. In this example, training is performed for 100 epochs, and the learning rate is set to 0.0001.

[0319] (5) Data evaluation

[0320] At this stage, a data evaluation function (e.g., the evaluate_new_data function described above) evaluates the data, and the results are stored in the similarity_percentage and FAC grade (fac_grade) variables.

[0321] In this step, data evaluation is performed by a data evaluation function, for example, the function evaluate_new_data described above, and the results are stored in the similarity_percentage and fac_grade variables.

[0322] This code snippet loads and preprocesses data from a CSV file containing information used to assess FAC for a given patient's gait training. It then trains an LSTM model, evaluates new data using the trained model, and calculates the similarity between the predicted and actual results to determine an FAC rating. This can be used as initial data for gait training, such as gait analysis, or to assess advanced FAC for patients undergoing gait training.

[0323]

[0324] II. Model 2 (M2)

[0325] Model 2 is an AI agent that automatically generates a gait training plan for each patient symptom for a gait rehabilitation training robot in an artificial intelligence-based robot-assisted orthopedic exercise system according to the present disclosure. This model may be, for example, a Transformer model, and according to another embodiment, LSTM or GRU (Gated Recurrent Unit) may also be applied.

[0326] Input data for this model 2 include 1) patient information, 2) prior gait training outcome data, and 3) gait training standard protocol and training plan definition data.

[0327] The above patient information includes the gender, weight, foot size, year of birth, and FAC (Foot Assessment Criteria) grade assigned to the patient when registering the patient in the training system. The above pre-training result data includes a dataset including training protocols such as stair climbing / descending and slope climbing / descending, training time, and training parameters such as stride length, stride height, initial contact angle, toe-off angle, ground reaction force, and gait cycle. In addition, the above gait training standard protocol and training plan definition data are data determined by medical staff.

[0328] Model 2 receives data such as the FAC of the patient as above, patient information, and current training result information, and derives (outputs) a training plan, such as a training protocol for walking on flat ground, climbing / descending stairs, climbing / descending slopes, training time, training parameters, etc., and this can be a multi-input, single, or multi-output model.

[0329] In another embodiment, the inputs may be current gait evaluation indices and patient information, and the output may be a standard training model group (Classification), and in another embodiment, the inputs may be current gait evaluation indices, patient information, and training information, and the output may be patient-tailored training plan information.

[0330] When applying an AI autonomous judgment training plan, the output data of Model 2 is a Training Plan Definition Dataset that reflects the results of previously performed training for each patient. Real-time training plan modifications, which reflect training status information during each patient's training, include settings for training time, training parameters, training protocol changes, and robot stop / start.

[0331] The M2 model, designed for AI autonomous judgment training plan generation, is an AI agent that uses patient gait training data to predict training parameters. Like the aforementioned M1 model, it can be created and trained using Python and its underlying framework, PyTorch. For this purpose, several libraries, including PyTorch, can be imported.

[0332] - import os: module for working with files and directories

[0333] - import torch: a library for basic tensor operations and neural network construction.

[0334] - import torch.nn as nn: neural network module (e.g., defining layers)

[0335] - import torch.optim as optim: Optimization algorithm for model training (e.g., Adam, SGD, etc.)

[0336] - from torch.utils.data import DataLoader, TensorDataset: Modules for data processing (batch generation, etc.)

[0337] - import pandas as pd: A library for creating and manipulating data frames.

[0338] The Python code that imports the above library sequentially includes the following steps for declaring and training the M2 model.

[0339] (1) Data input and data frame creation

[0340] Figure 19 is a code snippet from the entire Python code for creating the M2 model, including data input for training the model and creation of a dataframe that accumulates the data.

[0341] The above M2 model is an instance of the Transformer model, utilizing PyTorch's neural network module, nn, to predict training parameters using a patient's gait training data. As described sequentially below, the input data is provided in the form of multiple CSV files, which are preprocessed and combined into a two-dimensional dataframe before being used as training data.

[0342] (1.1) Setting the directory path containing CSV files

[0343] csv_directory = 'train_data_csv'

[0344] (1.2) List initialization

[0345] all_dataframes = []

[0346] (1.3) Read all CSV files in the directory and combine them into one data frame.

[0347] for filename in os.listdir(csv_directory):

[0348] if filename.endswith('.csv'):

[0349] file_path = os.path.join(csv_directory, filename)

[0350] df = pd.read_csv(file_path)

[0351] all_dataframes.append(df)

[0352] (1.4) Combine all data into one data frame

[0353] combined_dataframe = pd.concat(all_dataframes, ignore_index=True)

[0354]

[0355] (2) data preprocessing and tensor conversion

[0356] (2.1) The definition of a word

[0357] relevant_columns = [

[0358] 'p_patientGender', 'p_weight', 'p_footSize', 'p_patientBirthYear', 'p_fac',

[0359] 'p_trainingCurrentCadance', 'p_strideLength', 'p_strideHeight',

[0360] 'p_initialContactAngle', 'p_toeOffAngle',

[0361] 'p_leftCurrentGRF', 'p_rightCurrentGRF',

[0362] 'p_leftCurrentGaitCycleRate', 'p_rightCurrentGaitCycleRate'

[0363] ]

[0364] The input data includes patient information such as gender 'p_patientGender', weight 'p_weight', foot size 'p_footSize', birth year 'p_patientBirthYear', and FAC 'p_fac', as defined in the array definition statement above, and pre-training training result data such as steps per minute 'p_trainingCurrentCadance', stride length 'p_strideLength', stride height 'p_strideHeight', initial contact angle 'p_initialContactAngle', toe-off angle 'p_toeOffAngle', current left foot GRF 'p_leftCurrentGRF', right foot GRF 'p_rightCurrentGRF', left foot gait cycle rate 'p_leftCurrentGaitCycleRate', and left foot gait cycle rate 'p_rightCurrentGaitCycleRate'.

[0365] (2.2) Copy data frame with selected columns

[0366] combined_dataframe_relevant = combined_dataframe[relevant_columns].copy()

[0367] (2.3) Data preprocessing

[0368] combined_dataframe_relevant['p_patientGender'] = combined_dataframe_relevant['p_patientGender'].map({'Male': 0, 'Female': 1})

[0369] In the preprocessing process, male "male" is converted to "0", female "female" is converted to "1", and missing values ​​(NaN, Not a Number) are replaced with the mean value of each column by the method "fillna".

[0370] combined_dataframe_relevant.fillna(combined_dataframe_relevant.mean(), inplace=True)

[0371] (2.4) Convert to PyTorch tensor “X” and create tensor “Y”

[0372] mean_target = combined_dataframe_relevant[['p_trainingCurrentCadance', 'p_stepLength', 'p_stepHeight', 'p_initialContactAngle', 'p_toeOffAngle']].mean().values

[0373] The preprocessed data is converted to a PyTorch tensor "X", then the mean value of the target parameters to be predicted, mean_target, is calculated, and a tensor "Y" with the same target value for all samples is created.

[0374] Y = torch.tensor([mean_target] * len(X), dtype=torch.float32)

[0375]

[0376] (3) Definition of Transformer-based model class

[0377] Figure 20 is a code snippet from the entire Python code, including declaration of the transformer model, learning settings, and model creation.

[0378] class TransformerModel(nn.Module):

[0379] The TransformerModel class is defined, which inherits PyTorch's nn.Module.

[0380] The initialization method __init__ of TransformerModel is the constructor method of the class, which defines the structure of the model and each layer, and performs the initialization function of the base class by calling the initialization method super(TransformerModel, self).__init__ of the parent class, and its parameters are as follows.

[0381] - input_dim: Specifies the dimension (number of features) of the input data.

[0382] -output_dim: Specifies the output dimension of the model (the number of parameters to be predicted).

[0383] - num_heads: Specifies the number of heads in multi-head attention, and the default value is 1.

[0384] - num_layers: Specifies the number of Transformer encoder layers, the default value is 2.

[0385] - hidden_dim: Sets the dimension of the forward propagation network, the default value is 128.

[0386]

[0387] The class instance nn.TransformerEncoderLayer defines self.encoder_layer , a Transformer encoder layer provided by PyTorch, as follows:

[0388] self.encoder_layer = nn.TransformerEncoderLayer(d_model=input_dim, nhead=num_heads, dim_feedforward=hidden_dim)

[0389] The parameters of this instance are as follows:

[0390] - d_model=input_dim: Input dimension.

[0391] -head=num_heads: Number of heads in multi-head attention.

[0392] - dim_feedforward=hidden_dim: The dimension of the forward propagation network.

[0393] A fully connected layer (fc) is defined by the method nn.Linear , where the parameters input_dim is the input dimension, output_dim is the final output dimension, and is the number of training plan parameters to predict.

[0394] Meanwhile, the forward propagation method of the above TransformerMode preprocesses the input data and returns it. To do this, first, a dimension is added to the input data x, and then the input x with the added sequence dimension is passed to the Transformer_encoder to be converted into an encoded output. Then, after the sequence dimension is removed from the encoded output x by the squeeze method, the transformed data x is returned as the output through the fully connected layer.

[0395] (4) Training Settings

[0396] As shown in Figure 20, the hyperparameters in the learning settings are set as follows.

[0397] - input_dim = X.shape[1] # Dimension of input data (number of features)

[0398] - output_dim = 5 # Dimension of output data (number of parameters to predict)

[0399] - batch_size = 64 # Batch size (number of data samples to learn at once)

[0400] - learning_rate = 0.001 # learning rate (size of change when weights are updated)

[0401] - epochs = 100 # Number of learning iterations

[0402]

[0403] After this process, the dataset and data loader are created as shown below.

[0404] - dataset = TensorDataset(X, Y) # Define dataset

[0405] - train_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True) # Create a data loader

[0406] - model = trasnformerMode(input_dim, output_dim): Create an instance of the M2 model

[0407]

[0408] In the following process, the model to be learned is initialized and the loss function and optimizer are set. In this embodiment, the M2 model is initialized as a transformer model, and then "MSE" is set as the loss function and "Adam" is set as the optimizer, and then model learning is performed.

[0409] (5) Model Training

[0410] Figure 21 is a code snippet for training and testing data prediction of the model.

[0411] As illustrated in FIG. 21, according to one embodiment of the present disclosure, the learning of the model is repeatedly performed for 100 epochs preset in the above step.

[0412] -for epoch in range(epochs): Starts a loop to repeat the entire learning process. Epochs defines the number of times to repeat the learning process.

[0413] - model.train() : Sets the model to training mode. This mode allows certain layers, such as dropout and batch normalization, to operate during training.

[0414] -running_loss = 0.0: Initializes a variable to store the total loss of the current epoch, and the average loss is calculated using this variable at the end of each epoch.

[0415] - for inputs, targets in train_loader: Loads input data and target data in batches in train_loader. The data loader is responsible for repeatedly loading the dataset in batches.

[0416] - optimizer.zero_grad(): Initializes the gradient of the optimizer. Since PyTorch accumulates previous gradients by default, the gradient must be initialized for each batch.

[0417] - outputs = model(inputs): The input data is passed through the model to calculate the predicted values ​​(outputs), and the model's forward method is called at this stage.

[0418] -loss = criterion(outputs, targets): Calculates the loss between the predicted values ​​and the actual target values. Here, criterion defines the loss function (e.g., MSELoss).

[0419] - loss.backward(): Performs backpropagation based on the loss to calculate the gradient for each parameter of the model.

[0420] -optimizer.step(): Updates the model's parameters based on the gradient calculated using the optimizer.

[0421] -running_loss += loss.item(): Accumulates the loss value of the current batch and adds it to running_loss, which allows you to calculate the average loss at the end of an epoch. And,

[0422] -print(): Print the average loss for each epoch, dividing running_loss by the length of train_loader to calculate the average, format it, and print it. For example, if the real-time loss is 250.123 and the length of train_loader is 50, the output for the 6th epoch out of 100 epochs will be "Epoch [6 / 100], Training Loss: 5.0025".

[0423] The above model learning process is performed repeatedly, calculating and outputting the loss at each epoch, where the optimizer and loss function are used to update the model's parameters, and the progress of learning can be monitored through the loss value.

[0424] (6) Predicting Test Data

[0425] As shown in Figure 21, the prediction of test data using the M2 model obtained through the above process is also performed on the input data of the CSV file.

[0426] - test_data = pd.read_csv('test.csv'): Read test data file

[0427]

[0428] - test_data_relevant = test_data[relevant_columns].copy(): Select the necessary columns from the data in test data preprocessing.

[0429] test_data_relevant['p_patientGender'] = test_data_relevant['p_patientGender'].map({'Male': 0, 'Female': 1}): Convert gender data to numbers

[0430] test_data_relevant.fillna(test_data_relevant.mean(), inplace=True): Processing missing values

[0431]

[0432] X_test = torch.tensor(test_data_relevant.values, dtype=torch.float32): Convert test data to PyTorch tensor

[0433] (7) Model prediction

[0434] model.eval() : Set the model to evaluation mode (disable dropout, etc.)

[0435] with torch.no_grad(): disable gradient calculation

[0436] prediction = model(X_test).mean(dim=0).numpy(): Calculate the predicted value and then calculate the mean.

[0437] (8) Output prediction results

[0438] predicted_df = pd.DataFrame([prediction], columns=['p_trainingCurrentCadance', 'p_stepLength', 'p_stepHeight', 'p_initialContactAngle', 'p_toeOffAngle'])

[0439] print(predicted_df) : Convert the prediction results to a data frame and output them.

[0440]

[0441] The model trained as above is used as a model to establish a patient-tailored training plan.

[0442] In actual usage, in order to establish a training plan by reflecting the information on the results of previously performed training, the most recent data is extracted from the existing training data, prediction is performed using the M2 model, and the result values ​​obtained after the prediction are input into new training to proceed with training.

[0443] When developing a training plan using the M2 model, if patient training information is available, this information is input into the M2 model to extract an autonomous training plan. If necessary, this information is input into the Expert System to further enhance reliability.

[0444] The Expert System includes a reference model trained according to theoretical standards. This reference model has the same structure as the M2 model and outputs a reference training plan based on a dataset defining a standard gait training protocol and training plan determined by a medical professional. This output is matched with training plan establishment information corresponding to patient information and the assessed gait grade evaluation indicators. This reference model is trained using patient information (gender, weight, foot size, year of birth, FAC grade, etc.), training protocol, training time, and training parameters set by the medical staff.

[0445] This baseline model can be applied to the pre-built AI autonomous judgment training plan establishment model and to assess the reliability of the aforementioned M2 model. To this end, the patient's previous training information, previously input into the pre-built AI autonomous judgment training plan establishment model and the aforementioned M2 model, is input into the baseline model to extract a baseline training plan.

[0446] This baseline training plan is compared with the autonomous training plan output from the M2 model, allowing its reliability to be assessed and the autonomous training plan to be appropriately modified based on the level of reliability. In actual training, either the baseline training plan or the autonomous training plan can be selected. The baseline model can be applied as an optional element to more precisely compensate or correct the autonomous training plan.

[0447] In another embodiment, the autonomous training plan can be adjusted based on the reference training plan. This adjustment can be performed by a final training plan model that outputs an optimal training plan from the outputs of the M2 model and the reference model. This final training plan model can be trained using data obtained by comparing the aforementioned M2 model and the reference model.

[0448]

[0449] III. Model 3 (M3)

[0450] Model M3 is a training progress model that monitors and guides training progress during gait rehabilitation training. Model M3 could be, for example, a chatbot-type model. Input data for this model includes patient information, robot status information, and real-time input / output information during gait. Model M3 can be implemented with an AI agent based on a deep learning model selected from among the Multi-Layer Perceptron (MLP) and Open Vision-Language Model (VLM).

[0451] Figure 22 illustrates a graphical interface showing various parameters of gait training.

[0452] Real-time input / output information during gait training includes training protocols, training parameters, and training data at regular step intervals detected in real time.

[0453] Figure 23 is a butterfly diagram showing the average plantar pressure distribution of a patient during training as a result of the patient's gait training.

[0454] This training data may include real-time data acquired from a patient undergoing gait training. This data may include at least one characteristic information from among GRF (Ground Reaction Force, %), BWS (Body Weight Support, %), upper body inclination or tilting (degree), joint angle (degree), cadence, left gait cycle (0-99%), right gait cycle (0-99%), stride length, ankle angle, oxygen saturation data, and heart rate data.

[0455] Training parameters may include step length (cm), step height (cm), initial contact angle (degree), and toe off angle (degree).

[0456] As a separate option, when a video or voice recording device with a microphone, such as a webcam, is installed in the system, the patient's facial expression, voice and gesture information, and the therapist's voice and gesture information can be obtained and used as input data.

[0457] To apply these separate options, an image analysis unit may be provided that analyzes images from the camera to analyze the patient's facial expression or gesture, for example, an image analysis unit having an image analysis model trained to analyze the patient's facial expression or gesture from images.

[0458] The above model M3 can be trained to receive data from the image analysis unit and generate output according to the patient's training status together with the input data.

[0459] The output data of such model M3 may include state information during training, i.e., feedback on real-time training data and corrections to current training parameters.

[0460] Status feedback may include audio feedback such as an artificial intelligence voice or a beep sound, a user interface, and a status message output to a virtual reality HMD, for example, "Nice to meet you, Mr. Smith" when starting training after a patient boards, "Your training is going really well today" when training is in a similar pattern to the training data, "Press your left foot more" when the maximum GRF of the left foot is 10% or more lower than the maximum GRF of the right foot per 1 gait cycle, "Righten your upper body" when the upper body is gradually leaning forward or to the side per 10 gait cycles, "Cheer up" when the average GRF value before 10 gait cycles is greater than the average value per minute of the current gait cycle, "Your heart rate is rising" when the heart rate increases rapidly in 1 gait cycle, etc., and according to another embodiment, a request (request) for real-time changes in training parameters such as increasing or decreasing the speed or stopping the robot may be made through the user interface based on the results of monitoring the training status.

[0461] The training parameter modifications that may be included in the output data of the above model M3 may include the step length, step height, initial contact angle, and toe-off angle as recommended training parameters that can perform gait training close to the correct answer.

[0462] The above model M3 can perform the function of a training support agent through real-time interaction with patients / therapists, and can provide training status information by having an artificial intelligence agent compare and judge sequential training results and real-time training results during training.

[0463] The above model M3 can apply a multi-layer perceptron (MLP), which is a type of feedforward neural network, as described below.

[0464] Figure 24 is a code snippet of the declaration part of the dataset class for training model M3.

[0465] The program for training the above model M3 is written in Python, for example, and a number of libraries are imported as follows.

[0466] import os

[0467] import pandas as pd

[0468] import torch

[0469] import torch.nn as nn

[0470] import torch.optim as optim

[0471] from torch.utils.data import Dataset, DataLoader

[0472]

[0473] Referring to the above, "the os module for interacting with the operating system in Python is imported, and pandas, a Python library for data analysis and manipulation, is assigned with the alias 'pd'. In addition, torch, which constitutes a deep learning framework as a basic module of the PyTorch library and supports tensor operations, automatic differentiation, GPU acceleration, etc., is also called. torch.nn, a neural network module of PyTorch, is imported and assigned with the alias 'nn'.

[0474] The GaitCycleDataset class reads .csv files from a specified folder and consolidates the data into one. During this process, missing values ​​are filled with zeros, and the feature data (input) and target data (output) are normalized and stored separately. This class provides the __getitem__ method, which returns each data point as a tensor.

[0475]

[0476] The above data set has the following elements:

[0477] - Weight: p_weight

[0478] - Age: p_age

[0479] - Height: height'

[0480] - Left current GRF: p_leftCurrentGRF

[0481] - Right current GRF: p_rightCurrentGRF

[0482] - Current BWS: p_bwsCurrentSupport

[0483] - Body tilt: p_bodytilting

[0484] - Left ankle angle: p_kinectLeftAnkleAngle

[0485] - Right ankle angle: p_kinectRightAnkleAngle

[0486] - Heart rate: p_heartrate

[0487] - Oxygen saturation: p_spO2

[0488] - Left current gait cycle: p_leftCurrentGaitCycleRate

[0489] - Right current gait cycle: p_rightCurrentGaitCycleRate

[0490] - Current training steps per minute: p_trainingCurrentCadance

[0491]

[0492] The dataset outputs step length (p_stepLength), step height (p_stepHeight), initial contact angle (p_initialContactAngle), and toe-off angle (p_toeOffAngle) as targets.

[0493] Figure 25 is a code snippet defining GaitCycleModel, which is model M3.

[0494] Referring to Figure 25, the GaitCycleModel class has a multi-layer perceptron (MLP) structure. Model 3 of the MLP structure is a feedforward neural network consisting of an input layer, two hidden layers, and an output layer, with each hidden layer using the ReLU activation function.

[0495] Figure 26 is a code snippet of the feedback generation function, generate_feedback. The feedback generation function generates various feedbacks to be provided to pedestrians based on real-time input data and model output values.

[0496] For example, depending on the training status as described above, when training begins after boarding the patient, "Nice to meet you, Mr. Smith"; when training is in a similar pattern to the training data, "Today's training status is really good"; when the maximum GRF of the left foot is 10% or more lower than the maximum GRF of the right foot per 1 gait cycle, "Press down on your left foot more"; when the upper body is increasingly leaning forward or to the side per 10 gait cycles, "Rise your upper body"; when the average GRF value 10 gait cycles ago is greater than the average value per minute of the current gait cycle, "Cheer up"; when the heart rate increases rapidly in 1 gait cycle, "Your heart rate is increasing"; etc., feedback is provided to the patient.

[0497] Figure 27 is a code snippet for the real-time data processing function process_realtime_data.

[0498] The function process_realtime_data feeds data into the real-time model 3 and returns predictions and feedback.

[0499] The parameter row is input data in the form of a series, so data can be entered into the inputs variable as shown below, for example.

[0500] inputs = [70, 30, 170, 50, 55, 30, 8, 10, 12, 100, 98, 60, 62, 90]

[0501] The above inputs are converted into PyTorch tensors via torch.tensor, and these tensors are provided as inputs to Model 3, which then feeds back outputs corresponding to the inputs.

[0502] The outputs fed back by model M3 are input to the above-described generate_feedback function along with the above-described inputs, and this function returns feedback corresponding to the input inputs and outputs.

[0503] Finally, the function process_realtime_data returns the outputs and feedback, allowing the patient to be provided with the necessary feedback during real-time training.

[0504]

[0505] Figure 28 is a code snippet of the function train_model for training model M3.

[0506] The above function train_model is a function that implements a basic learning loop for training a model with given data. This function uses a PyTorch model with training data to repeatedly update weights and outputs the learning progress in epoch units.

[0507] Referring to Figure 28, the model learning function train_model trains the model using the given data, loss function, and optimization algorithm, and its parameters are as follows.

[0508] - model: PyTorch model to train.

[0509] - dataloader: PyTorch's DataLoader that provides the training dataset.

[0510] - criterion: loss function

[0511] - optimizer: optimization algorithm for weight updates

[0512] - epochs=10: Number of learning iterations (default is 10)

[0513] This function implements a model training loop, processing data from the dataset in batches and generating predictions through forward propagation through the model. A criterion is used as a loss function to calculate the error (loss) between the model's predictions and the actual values.

[0514] The loss value is converted into a gradient for the model's weights (parameters) through the backpropagation algorithm, and the weights are updated through an optimization algorithm. Here, the accumulated loss value over one epoch (one pass through the entire dataset) is calculated and output to monitor the learning status.

[0515]

[0516] Figure 29 is a code snippet of the evaluate_mode function that evaluates the trained model.

[0517] The evaluate_model function is a function for evaluating the performance of a learned model. It repeatedly processes data through a given data loader, and compares the model's predicted values ​​with actual values ​​or generates feedback.

[0518] This process switches the model to evaluation mode, disables gradient calculation, fetches inputs and actual values ​​from the data loader in batches to generate predictions, generates feedback based on the inputs and predicted values ​​of the first sample, and outputs the predicted values, actual values, and generated feedback.

[0519]

[0520] Figure 30 is a snippet of execution code for generating Model 3 according to the present disclosure.

[0521]

[0522] This executable code also loads the dataset through dataloader, and specifically includes the following processes:

[0523] - Load dataset and create data loader

[0524] - Model initialization and training

[0525] - Evaluation of the learned model

[0526] - Real-time data processing and feedback provision

[0527] - Saving the learned model M3

[0528] The above model M3 is initialized by the GaitCycleModel class and then trained by the train_model function. In this process, nn.MSELoss is used as a loss function to calculate the error between the predicted value and the actual value, and Adam is selected as the optimizer to update the weights.

[0529]

[0530] The overall flow of the above-described executable code is as follows.

[0531] The above execution code uses the GaitCycleDataset class to load data stored in the training data folder, initializes a model based on the data, and then performs the learning, evaluation, real-time data processing, and model storage processes.

[0532]

[0533] 1. Data preparation

[0534] Specify the folder path where the training data is stored in the folder_path variable, read the .csv files in that folder, and create a dataset using the GaitCycleDataset class. Then, use PyTorch's DataLoader class to divide this dataset into batches of size 16, shuffle it, and create a DataLoader object to be used for training.

[0535] 2. Model initialization

[0536] The input data size (input_size) is set to the number of feature data columns in the dataset, and the output data size (output_size) is set to the number of target data columns. Here, the hidden layer size (hidden_size) is set to 64, and based on this, the GaitCycleModel with a multi-layer perceptron structure is initialized.

[0537] 3. Setting the loss function and optimizer

[0538] The loss function uses nn.MSELoss to calculate the mean squared error between the predicted and actual values. Adam is selected as the optimizer, and the learning rate (lr) is set to 0.001.

[0539] 4. Model M3 training

[0540] After printing the message "Training Model...", call the train_model function to train the model for 30 epochs.

[0541] 4. Model M3 Evaluation

[0542] Print the message "Evaluating Model..." and call the evaluate_model function to evaluate the performance of the trained model.

[0543] 5. Real-time data processing:

[0544] After printing the message "Processing Real-time Data...", iterates through the feature data (dataset.features) of the dataset one row at a time and calls the process_realtime_data function.

[0545] The function generates feedback based on the input data and the model's output values, and outputs the generated feedback and predicted values.

[0546] 6. Save the model:

[0547] Save the trained model weights to the "gait_cycle_model.pth" file using torch.save. Then, output the message "Model Saved!" and terminate the execution.

[0548]

[0549] IV. Model 4 (M4)

[0550] This model 4 is an AI agent that evaluates training results and predicts future trends. The input data can be past training result data and current (real-time) training result data, and in this case, the output data is trend analysis and future prediction of the past training results and current training result data.

[0551] This training data may include real-time data acquired from a patient undergoing gait training. This data may include at least one of the following: Ground Reaction Force (GRF, %), Body Weight Support (BWS, %), Upper body inclination or tilting (degree), Joint angle (degree), Cadence, Left gait cycle (0-99%), Right gait cycle (0-99%), Stride length, Ankle angle, Oxygen saturation data, and Heart rate data.

[0552] Training parameters may include step length (cm), step height (cm), initial contact angle (degree), and toe off angle (degree).

[0553] This model 4 is an anomaly detection model based on time series information, and can include algorithms such as LSTM (Long Short-Term Memory), which is an extension of RNN (Recurrent Neural Network) and can learn long sequence data and handle long-term dependencies, or VAE (Variational Autoencoder) that is extended to properly process time series data.

[0554] This model, Model 4, is designed to analyze current training results or compare them with previous training results to analyze trends in results following repeated training, and predict future training outcomes based on these results. This model can output graphs analyzing the previous training average trend line, the current training average trend line, and the predicted future training trends. To achieve this, it compares and analyzes trends between previous and current training results, and predicts future training outcomes based on trends in previous training outcomes.

[0555] The above model 4 can be configured to include an anomaly detection model (M4a) and a trend prediction model (M4b).

[0556] The anomaly detection model (M4a) utilizes algorithms such as the LSTM Autoencoder or Variational Autoencoder (VAE) to detect abnormal patterns in real-time training data. This model detects anomalies by setting dynamic thresholds based on the Reconstruction Error (RE), effectively analyzing the patient's current training status.

[0557] The trend prediction model (M4b) leverages a Seq2Seq-based LSTM / GRU or Transformer-based model to predict future training performance based on past and current data. This model uses a sliding window technique to sample time-series data and analyzes multiple input features to learn correlations between data. Furthermore, the two models operate in conjunction with each other, incorporating the results of the anomaly detection model (M4a) into the weights of the trend prediction model (M4b), thereby improving prediction accuracy.

[0558] M4's output data may include visualization data in graph form, results analysis data, comprehensive evaluation indicators, and improvement suggestion data.

[0559] - Visualization data in graph form: Includes the training average trend line, the current training average trend line, and the future training performance prediction trend line, allowing users to intuitively understand changes in training performance over time.

[0560] - Results Analysis Data: Analyzes the differences between previous and current training to provide the rate of improvement or decrease in key indicators. Additionally, by highlighting anomalies detected in specific sections, problems encountered during training can be clearly identified.

[0561] - Comprehensive evaluation index and improvement suggestion data: The comprehensive evaluation index evaluates the patient's performance on a scale of 0 to 100 and suggests specific adjustment recommendations for training parameters that require improvement (e.g., Step Length, Step Height, etc.).

[0562] Figure 31 shows a code snippet of a data preprocessing and learning pipeline utilizing LSTM model M4 for anomaly detection and trend prediction according to the present disclosure.

[0563] The program for training the above model M4 is written in Python, for example, and multiple libraries can be imported as follows.

[0564] The OS module for interacting with the operating system, pandas as pd for data analysis and processing, torch, the core module of the PyTorch framework, torch.nn, which provides neural network components, the torch.optim module, which provides optimization algorithms for training deep learning models, and torch.utils.data, which provides tools for efficiently managing data and processing it in batches, can be imported.

[0565]

[0566] Hyperparameters can be set, for example, as follows:

[0567] Number of features in input data: input_size = 15:

[0568] LSTM hidden layer size: hidden_size = 64

[0569] Number of layers in LSTM: num_layers = 2

[0570] Sequence length: seq_length = 20

[0571] Batch size: batch_size = 32

[0572] Learning rate: learning_rate = 0.001

[0573] Number of epochs: epochs = 20

[0574]

[0575] A data processing pipeline may include the following steps:

[0576] 1. Data loading and preprocessing

[0577] In this process, all CSV files stored in a specified folder, for example, the "training_dataset" folder, are read and merged into a single Pandas DataFrame, including the steps of filtering .csv files in the training_dataset folder, reading and collecting data files, and forming a dataframe in which the data in the data files are integrated (merged) into one and used for analysis and processing.

[0578] - Forming a dictionary (columns_to_use_updated) of a data structure

[0579] The above dictionary stores data as a pair of input keys and output values, and the name to be used for actual output can be obtained as the output value through the input key.

[0580]

[0581] columns_to_use_updated = {

[0582] 'p_trainingCompletionNumber': 'Gait Cycle Identifier',

[0583] 'p_leftCurrentGRF': 'Left GRF',

[0584] 'p_rightCurrentGRF': 'Right GRF',

[0585] 'p_bwsCurrentSupport': 'BWS',

[0586] 'p_trainingCurrentCadance': 'Cadence',

[0587] 'p_leftCurrentGaitCycleRate': 'Left Gait Cycle',

[0588] 'p_rightCurrentGaitCycleRate': 'Right Gait Cycle',

[0589] 'p_stepLength': 'Step Length',

[0590] 'p_stepHeight': 'Step Height',

[0591] 'p_initialContactAngle': 'Initial Contact Angle',

[0592] 'p_toeOffAngle': 'Toe Off Angle',

[0593] 'p_footSize': 'Foot Size',

[0594] 'p_weight': 'Weight',

[0595] 'p_age': 'Age',

[0596] 'p_height': 'Height',

[0597] 'p_fac': 'FAC Grade'

[0598] }

[0599]

[0600] 2. Data separation:

[0601] In this step, some of the selected data (filtered_data) from the entire data is divided into training data (train_data) and test data (test_data), and in the example code below, it is divided in a ratio of 7:3.

[0602]

[0603] train_data, test_data = train_test_split(filtered_data, test_size=0.3, random_state=42)

[0604]

[0605] 3. Grouping:

[0606] Group data based on the "Gait Cycle Identifier", which is the output value of the dictionary.

[0607]

[0608] train_grouped = train_data.groupby("Gait Cycle Identifier")

[0609] test_grouped = test_data.groupby("Gait Cycle Identifier")

[0610]

[0611] Figure 32 is a code snippet illustrating the declaration of an LSTM-based Autoencoder class and a Seq2Seq class for anomaly detection and trend prediction according to the present disclosure, and the initialization process of two models utilizing these classes.

[0612] The initialized anomaly detection model (M4a) is anomaly_model , and the trend prediction model (M4b) is prediction_model . The loss function for each model is the Mean Squared Error (MSE), and the Adam optimization algorithm is applied to update the learning parameters of each model.

[0613]

[0614] The code snippet below may contain a series of command statements that train and evaluate the anomaly detection model (M4a) and trend prediction model (M4b) of the M4 model, and visualize the results.

[0615]

[0616] train_anomaly_model(train_grouped, epochs)

[0617] train_prediction_model(train_grouped, epochs)

[0618] evaluate_anomaly_model(test_grouped)

[0619] visualize_analyze_and_suggest_results(test_grouped)

[0620]

[0621] Above, the functions train_anomaly_model and train_prediction_model train the anomaly detection model and trend prediction model, respectively, the function evaluate_anomaly_model evaluates the performance of the anomaly detection model, and the function visualize_analyze_and_suggest_results visualizes and analyzes the test results and presents improvement suggestions.

[0622]

[0623] Figure 33 is a function for training the above-mentioned anomaly detection model (M4a), which is one element of the M4 model according to the present disclosure.<train_anomaly_model> Illustrate the definition of a class.

[0624] Referring to Figure 33, model M4a learns the pattern of normal data and then detects abnormal behavior by exhibiting high loss for abnormal data, and is trained through the following process.

[0625] - Data preprocessing: Separate data into groups and convert to tensor format.

[0626] - Model training: Input data into the model to generate predictions, calculate the loss between the predictions and the actual values, and update the model weights through backpropagation.

[0627] - Progress output: Monitor the learning process by outputting the loss value every 5 epochs.

[0628]

[0629] Figure 34 illustrates the definition of a function for training a trend prediction model (M4b) in the M4 model according to the present disclosure.

[0630] Referring to Figure 34, the learning of model M4a includes the following processes.

[0631] - Learning by dividing data into grouped units

[0632] - Convert each group data into a tensor and pass it to the model to generate predicted values.

[0633] - Calculate the loss between the predicted value and the actual value

[0634] - Calculate the gradient according to the loss through backpropagation and update the model weights based on this.

[0635] - Monitor learning progress by outputting loss values ​​every 5 epochs.

[0636]

[0637] Figure 35 illustrates the definition of a function evaluate_anomaly_model for evaluating an anomaly detection model (M4a) in the M4 model according to the present disclosure.

[0638] This function includes the following evaluation process of M4a:

[0639] - Data preparation: Process data in groups and move to evaluation device.

[0640] - Perform model reconstruction: The model reconstructs the input data to generate output values.

[0641] - Calculate reconstruction loss: Calculate and store the loss between input data and output values.

[0642] - Average loss output: Calculate and output the average reconstruction loss for all group data.

[0643] Figure 36 is a pseudocode snippet of the visualize_analyze_and_suggest_results function, which visualizes and analyzes test results and suggests improvement suggestions in the M4 model according to the present disclosure. Some of the pseudocode in Figure 36 can be concretized as follows.

[0644]

[0645] - INITIALIZE analysis_results = [], overall_score = 0, num_cycles = len(test_grouped):

[0646]

[0647] analysis_results = []

[0648] overall_score = 0

[0649] num_cycles = len(test_grouped)

[0650]

[0651] The above code initializes the list to store the analysis results (analysis_results), the total score (overall_score), and the number of groups of test data (num_cycles).

[0652]

[0653] - EXTRACT relevant features FROM group INTO group_tensor:

[0654]

[0655] group = group[['Left GRF', 'Right GRF', 'BWS', ...]].values ​​group_tensor = torch.tensor(group, dtype=torch.float32).unsqueeze(0).to(device)

[0656]

[0657] The above code filters the group data to extract only the necessary features (group) and converts it to a PyTorch tensor (group_tensor).

[0658]

[0659] - PREDICT predictions USING prediction_model(group_tensor):

[0660]

[0661] with torch.no_grad():

[0662] predictions = prediction_model(group_tensor).squeeze(0).cpu().numpy()

[0663]

[0664] The above code uses a trend prediction model (prediction_model) to predict future values ​​for the current group.

[0665]

[0666] - COMPUTE mean_diff, std_diff, mse, mae FROM differences:

[0667]

[0668] mean_diff = np.mean(differences, axis=0)

[0669] std_diff = np.std(differences, axis=0)

[0670] mse = np.mean(np.square(differences), axis=0)

[0671] mae = np.mean(np.abs(differences), axis=0)

[0672]

[0673] The above code calculates the mean difference (mean_diff), standard deviation (std_diff), mean squared error (mse), and mean absolute error (mae) based on the differences.

[0674]

[0675] - COMPUTE cycle_score BASED ON mse AND ADD TO overall_score:

[0676]

[0677] cycle_score = max(0, 100 - (np.mean(mse) * 10))

[0678] overall_score += cycle_score

[0679]

[0680] The above code calculates the score of the current group (cycle_score) and adds it to the overall score (overall_score).

[0681]

[0682] - APPEND metrics TO analysis_results:

[0683]

[0684] analysis_results.append({ "Gait Cycle": key, "Mean Difference": mean_diff.tolist(), ...})

[0685]

[0686] The above code saves the calculated metrics (mean difference, standard deviation, loss value, etc.) to the analysis results list (analysis_results).

[0687]

[0688] - PLOT “Actual vs Predicted” graph FOR current group:

[0689]

[0690] plt.figure(figsize=(10, 6))

[0691] for i, feature in enumerate(['Left GRF', ...]):

[0692] plt.plot(group[:, i], label=f"Actual {feature}")

[0693] plt.plot(predictions[:, i], linestyle='--', label=f"Predicted {feature}")

[0694] plt.show()

[0695]

[0696] The above code visualizes a graph comparing actual and predicted values ​​for the current group.

[0697]

[0698] - INITIALIZE suggestions = [] AND CHECK deviation FOR monitored_features:

[0699]

[0700] suggestions = []

[0701] for i, feature in enumerate(['Step Length', ...]):

[0702] if np.abs(mean_diff[i]) > 0.1 * np.mean(group[:, i]):

[0703] suggestions.append(f"Consider adjusting {feature}. Current deviation: {mean_diff[i]:.2f}")

[0704]

[0705] The above code generates suggestions when the deviation of a specific feature (Step Length, Step Height, etc.) exceeds a threshold.

[0706]

[0707] - PRINT overall_score AND analysis_results:

[0708]

[0709] print(f"Overall Performance Score (0-100): {overall_score:.2f}")

[0710] print(" / nDetailed Analysis Results:")

[0711]

[0712] The above code prints the overall performance score and detailed analysis results.

[0713]

[0714] As described above, in the model 4 (M4) according to the present disclosure, the anomaly detection model (M4a) evaluates the extent to which the model has learned normal data patterns through the reconstruction loss (Rconstruction Loss). A low reconstruction loss indicates that the anomaly detection model has learned the normal data patterns well, and a decrease in loss per epoch indicates that the anomaly detection model is gradually demonstrating better performance. The trend prediction model (M4b) indicates how accurately the future is predicted based on past data through the prediction loss. A decrease in loss per epoch indicates that the trend prediction model has learned the correlation between data well.

[0715] The evaluation of these models (M4a, M4b) uses average reconstruction loss, cycle score, and graphical visual comparison.

[0716] The average reconstruction loss is calculated on the test data, and a lower value indicates a better performance of the model in reconstructing normal data.

[0717] The cycle score is an indicator of gait cycle performance, measured on a scale of 0 to 100. A low cycle score indicates poor predictive performance (indicating how accurately the gait cycle can be predicted) or that the data contains outliers. The cycle score can be used to identify problematic sections of a specific cycle.

[0718] Visual comparisons can be performed by overlaying actual and predicted values ​​for each gait cycle on a single display, displaying them as distinct linear lines (e.g., dotted and solid lines). The closer the actual and predicted lines overlap, the better the model's performance. Significant deviations in a specific characteristic may indicate the need for additional model adjustments for that characteristic.

[0719] The results analysis comprehensively evaluates the model's performance, including the mean difference between the actual and predicted values ​​of each feature (the lower the value, the better the prediction accuracy), the MSE and MAE that evaluate the prediction error (the lower the value, the better the performance), and the standard deviation that indicates the variability of the prediction error (the lower the value, the more stable the performance).

[0720]

[0721] V. Model 5 (M5)

[0722] Model 5 is an artificial intelligence model that supports smart training result reporting. The input data is pre- and post-training result data and report generation type, and the output is images such as text, images, and videos.

[0723] The input report generation type is a simple or multimodal Generative Pre-trained Transformer (GPT) model-based prompt input. The outputs include text in PDF format, images of patient-specific training results and status (various backgrounds and progress states), or videos of patient-specific gait trajectories. Another output is video-based results based on patient gait training analysis results, such as virtual avatar movement, self-figure movement, and video output in the form of a butterfly diagram.

[0724] In gait rehabilitation training systems, a "butterfly diagram" is a visual tool primarily used for gait analysis and assessment. This diagram visualizes the foot's contact pattern and pressure on the ground, helping to analyze the gait cycle, including footprint analysis, gait cycle assessment, gait performance measurement, and left-right asymmetry analysis. These diagrams are often called "butterfly diagrams" because their footprint shape and pressure distribution resemble the shape of a butterfly's wings. An example of a butterfly diagram is shown in Figure 23.

[0725] In the case of multimodal support GPT, it will be possible to induce prompt-based report writing (text, image video) for the above work.

[0726] In addition to the outputs mentioned above, tasks such as training and report output via voice output can be enabled. This will enable smart reporting of patient status and training results after training, enabling more intuitive and effective delivery of training results to patients and increasing training achievement and satisfaction.

[0727]

[0728] Below, an exemplary configuration of Model 5 according to the present disclosure will be described.

[0729] Input data for AI agents applying Model 5 may include pre- and post-training result data, report generation type, and prompt input.

[0730] Pre- and post-training results data, including past and present training results, are integrated with training parameters and real-time data. Key data items may include, as previously mentioned, key data items such as step length, step height, gravimetric response (GRF), gait cycle (BWS), ankle angle, and gait cycle.

[0731] The output of the above Model 5, i.e. the report generation type, can include a basic report that provides a simple text or image-based summary of results, and a multimodal report containing text, images, and videos generated from GPT-based prompt input, and can diversify the data analysis and visualization method depending on the user-selected type.

[0732] Prompts for generating GPT-based reports using prompt input for multimodal reporting. For example, "Please visualize the patient's gait trajectory and output it as a video."

[0733] The above model 5 (M5) may include a report automation model, a video-based analysis model, and a gait evaluation index prediction model.

[0734] - Report Automation Model:

[0735] Leveraging a multimodal GPT-based model, it analyzes input data and automatically generates training results in text, image, and video formats. This model understands training data and can generate reports in appropriate formats based on given prompts.

[0736] - Video-based analysis model

[0737] This model analyzes and visualizes the patient's gait trajectory based on time series data, enabling, for example, the simulation of the movement of a virtual avatar, the actual movement of a patient's figure, or the visualization of gait patterns such as "butterfly diagrams."

[0738] - Walking evaluation index prediction model

[0739] This model applies a deep learning model based on LSTM and Transformer to predict gait evaluation indices, automatically establishes a training plan based on the prediction results, and derives improvement points.

[0740] The output of the above Model 5 (M5) may include training results in report form; real-time voice output, video-based visualization of training results, and smart reporting.

[0741] The training results in the form of reports may include text reports in PDF format containing training performance summaries, key indicator analysis results, image reports containing visual data reflecting the patient's various backgrounds and progress, video reports presenting gait trajectory simulations, for example, self-portrait or virtual avatar gait movements, butterfly diagrams, etc.

[0742] As a real-time voice output, the training progress and results can be provided as real-time voice guidance, for example, current training status information such as "The current gait cycle average speed is 90% of the target."

[0743] Video-based visualized training results may include video-enhanced results using a virtual avatar or self-image, and visual presentation of gait cycle asymmetry and rehabilitation effects using “butterfly diagrams.”

[0744] Smart Reporting can improve patient training performance and satisfaction by providing personalized training performance summaries and improvement suggestions based on patient status and training outcomes, and by providing intuitive and easy-to-understand reporting data.

[0745]

[0746] According to the gait training system and method of the present disclosure, which apply the aforementioned multi-stage AI agents, i.e., Models 1 to 5 (M1, M2, M3, M4, M5), it is possible to establish a patient-friendly gait training plan, and to efficiently conduct personalized and patient-friendly rehabilitation training for each patient. The rehabilitation training according to the system and method utilizes an artificial intelligence model for each stage to accurately predict gait evaluation indices, automatically establish a training plan based on the predicted values, and more accurately evaluate the training results.

[0747] Rehabilitation training using these systems and methods can accurately predict gait evaluation indices using artificial intelligence models at each stage, automatically establish training plans based on these, and more accurately evaluate training results, thereby reducing the intervention of therapists according to medical staff's prescriptions compared to existing methods and systems.

[0748] While exemplary embodiments of the present invention have been described in detail above, those skilled in the art will appreciate that various modifications and variations may be made to the invention without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, modifications to future embodiments of the present invention will not depart from the scope of the invention.

Claims

1. An artificial intelligence-based robot-assisted orthopedic exercise system comprising a walking exercise device having a pedal on which a patient stands, an actuator having one or more links that drive the pedal, and a system operating device that controls the main actuator according to a training plan established for each patient to perform walking training for a patient standing on the pedal. The above system operating device; An indicator prediction unit having an indicator prediction model that predicts a gait evaluation indicator of the patient based on the patient's initial gait status information or gait training result information after initial gait training; A training planning unit that establishes a training plan for the patient performed by the gait exercise unit based on the predicted gait evaluation index; A training evaluation unit that evaluates the training results based on training result data according to the results of the patient's rehabilitation training performed by the above walking exercise unit; and An artificial intelligence-based robot-assisted orthopedic exercise system comprising a reporting unit that performs reporting on training results based on evaluation of the above training results.

2. In paragraph 1, An artificial intelligence-based robot-assisted orthopedic movement system, wherein the above-mentioned indicator prediction unit has one of a RNN (Recurrent Neural Network) model or an LSTM (Long Short-Term Memory model) as the above-mentioned indicator prediction model.

3. In paragraph 1, The above training planning unit is an artificial intelligence-based robot-assisted orthopedic exercise system having a multi-input, multi or single output model as a training planning model.

4. In paragraph 1, An artificial intelligence-based robot-assisted orthopedic exercise system further comprising a detection unit that detects at least one piece of information from among GRF (Ground Reaction Force), BWS (Body Weight Support), upper body inclination or tilting, joint angle, cadence, stride length, and ankle angle of a patient undergoing gait training as training information.

5. In paragraph 4, An artificial intelligence-based robot-assisted orthopedic exercise system further comprising a monitoring unit that monitors the status of rehabilitation training performed by the gait exercise unit according to the training plan, obtains information during the training, and generates training status information related to the patient's gait training from the information.

6. In paragraph 5, The monitoring unit is an artificial intelligence-based robot-assisted orthopedic exercise system that provides training guidelines or recommended guidelines or current training status to the patient to improve gait related to the gait training based on information during the training.

7. In paragraph 5 or 6, An artificial intelligence-based robot-assisted orthopedic exercise system, wherein the monitoring unit further comprises an HMD (Head-Mounted Display) that provides necessary information related to the patient's walking movement, training status information, or recommended guidelines related to training to the patient.

8. In paragraph 1, An artificial intelligence-based robot-assisted orthopedic exercise system further comprising an HMD (Head-Mounted Display) that presents at least one of necessary information related to the patient's walking exercise, training status information, or recommended guidelines related to training to the patient.

9. In paragraph 1, The above training evaluation unit is an artificial intelligence-based robot-assisted orthopedic movement system, which includes an anomaly detection model based on time series information that analyzes the current training result as a training evaluation model, or analyzes the trend of the result according to repeated training by analyzing the previous training result and the current training result, and performs a future training prediction.

10. In the first paragraph, the above-mentioned reporting unit is a robot-assisted orthopedic exercise system based on intelligent robots that analyzes previous training results and current training results to generate report contents in the form of text, images, and videos.

11. A step of predicting a walking evaluation index of a patient based on the patient's initial walking status information or walking training result information after initial walking training, by the index prediction unit; A step in which the training planning department establishes a training plan for the patient based on the above gait evaluation index; A walking exercise device having an end-effector type pedal and an actuator having one or more links driving the pedal, the device performing rehabilitation training of the patient according to the training plan; A step in which the training evaluation department evaluates the training results based on the training result data according to the training results; and A method for providing artificial intelligence-based robot-assisted orthopedic exercise, comprising: a step of a reporting unit performing a report on training results based on an evaluation of the training results; 12. In paragraph 11, The above-mentioned index prediction unit is an artificial intelligence-based robot-assisted orthopedic exercise provision method, wherein either an RNN (Recurrent Neural Network) model or an LSTM (Long Short-Term Memory model) is used as an index prediction model to predict the patient's gait evaluation index.

13. In paragraph 11, The above training planning unit is an artificial intelligence-based robot-assisted orthopedic exercise providing method, which establishes the training plan using a multi-input, multi or single output model.

14. In paragraph 11, A method for providing robot-assisted orthopedic exercise based on artificial intelligence, wherein a detection unit detects at least one piece of information from among GRF (Ground Reaction Force), BWS (Body Weight Support), upper body inclination or tilting, joint angle, cadence, stride length, and ankle angle of a patient undergoing gait training as training information.

15. In paragraph 14, A method for providing artificial intelligence-based robot-assisted orthopedic exercise, wherein the monitoring unit monitors the status of rehabilitation training performed by the gait exercise unit according to the training plan, obtains information during the training, and generates training status information related to the patient's gait training therefrom.

16. In paragraph 14, A method for providing artificial intelligence-based robot-assisted orthopedic exercise, wherein the monitoring unit provides the patient with training guidelines or recommended guidelines or the current training status for improving gait related to the gait training based on the information during the training.

17. In paragraph 15 or 16, A method for providing artificial intelligence-based robot-assisted orthopedic exercise, wherein an HMD (Head-Mounted Display) provides necessary information related to the patient's walking exercise, training status information, or recommended guidelines related to training to the patient.

18. In paragraph 11, A method for providing an artificial intelligence-based robot-assisted orthopedic movement, wherein the time series information-based anomaly detection model as the above training evaluation model analyzes the current training result or analyzes the trend of the result according to repeated training by analyzing the previous training result and the current training result, thereby performing a future training prediction.

19. In paragraph 11, A method for providing artificial intelligence-based robot-assisted orthopedic exercise, wherein the reporting unit analyzes previous training results and current training results to generate reporting content in the form of text, images, and videos.

Images