Robot-assisted shaping motion system and method based on artificial intelligence

Abstract

The disclosed exercise system includes: a walking exercise device having a pedal treaded by a patient and one or more linked pedal driving portions driving the pedal; the system control device is used for carrying out walking training on the patient according to the training plan, and the system control device comprises an index prediction part which is configured to determine a walking evaluation index of the patient based on initial walking state information of the patient or a walking training result after the walking training; a training planning section configured to establish a patient individual-targeted training plan performed by the walking movement section based on a walking training result; establishing a patient individual targeted training plan performed by the walking motion section based on the walking training result; a training performing part configured to perform training on a patient according to a training plan, monitor training data about a real-time training state from the patient in real time, and generate feedback about the training data; and a training evaluation unit that evaluates a training result on the basis of a training result in accordance with a rehabilitation training result of the patient performed by the walking exercise unit.

Description

Technical Field

[0001] This disclosure relates to a robot-assisted orthopedic movement system, specifically a patient-friendly, AI-based assisted orthopedic movement system and its control method, which can automatically establish and implement a personalized orthopedic movement plan based on artificial intelligence (AI) and automatically modify the movement plan according to the rehabilitation effects caused by the movement. Background Technology

[0002] A robotic-assisted orthopedic kinetic system has been developed to assist patients with mobility impairments in their rehabilitation. This system helps improve mobility and restore lower body muscle strength in patients with mobility impairments. As a so-called end-effector-type robotic device, it supports and assists the patient's body movements, enabling them to undergo gait rehabilitation more safely and effectively.

[0003] This system helps improve the mobility of patients with walking difficulties and restore lower body muscle strength. As a so-called end effector-type robotic device, it supports and assists patients in their body movements, enabling them to undergo walking rehabilitation more safely and effectively.

[0004] In existing systems, therapists typically stand by and guide and assist the rehabilitation exercise process. In some cases, some patients are unable to adapt to the pedal movement and cannot conform to or comply with it. Furthermore, even with prolonged walking training, effective rehabilitation may not be achieved.

[0005] Therefore, the development of a rehabilitation training system that can perform safer and more effective rehabilitation training and thereby help patients recover quickly and return to their daily lives is of great benefit. Summary of the Invention

[0006] Technical issues

[0007] This disclosure discloses an AI-based robot-assisted orthopedic movement system and method that enables the establishment of patient-friendly walking training programs.

[0008] This disclosure discloses an artificial intelligence-based robot-assisted orthopedic motion system and method, wherein medical personnel intervention can be reduced through medical personnel prescriptions.

[0009] This disclosure discloses a patient-personalized and friendly AI-based robot-assisted orthopedic motion system and method thereof.

[0010] Technical solution

[0011] According to the AI-based robot-assisted orthopedic motion system disclosed herein,

[0012] This refers to an AI-based robot-assisted orthopedic movement system, comprising a gait training apparatus and a system control unit. The gait training apparatus has a pedal that the patient steps on and a pedal driver with one or more links configured to drive the pedal. The system control unit is configured to control the pedal driver to perform gait training for the patient stepping on the pedal according to a training plan established for the individual.

[0013] The system control device has

[0014] The indicator prediction unit is configured to determine the patient's walking assessment indicators based on the patient's initial walking status information or walking training results information after initial walking training.

[0015] The training planning department is configured to establish a patient-specific training plan to be executed by the gait movement department based on the results of the gait training.

[0016] The training execution unit is configured to perform training on the patient according to the training plan, monitor training data from the patient regarding their real-time training status in real time, and generate feedback on the training data; and

[0017] The training assessment department evaluates the training results based on training result data from the rehabilitation training performed on the patient by the gait department.

[0018] According to one or more embodiments,

[0019] The training data may include any one of the following: Ground Reaction Force (GRF, %), Body Weight Support (BWS, %), Upper Body Inclination or Tilting (°, degree), Joint Angle (°, degree), Steps Per Minute (Cadence), Left Gait Cycle (0-99%), Right Gait Cycle (0-99%), Stride Length, Ankle Angle, Oxygen Saturation, and Heart Rate.

[0020] According to one or more embodiments,

[0021] The training model can be configured to receive at least one training parameter among step length (cm), step height (cm), initial contact angle (° (degree)), and toe off-angle (° (degree)) and output a correction value for the at least one training parameter based on the training data.

[0022] According to one or more embodiments,

[0023] The training mode can be any deep learning model, such as Multi-Layer Perceptron (MLP) or Open Vision-Language Model (Open VLM).

[0024] According to one or more embodiments,

[0025] Video or audio information of patients or therapists assisting in training can be obtained and used as input data for the training model, which can then generate output data corresponding to the input data.

[0026] According to one or more embodiments,

[0027] The training plan may further include a reference model, which uses patient information, training procedures, training time, and training parameters set by medical personnel for training.

[0028] The training planning unit can output a final training plan based on the output correction or compensation of the autonomous training plan model and the benchmark training plan model.

[0029] According to one or more embodiments,

[0030] The training planning unit may further include a final training plan model, the output of which is a final training plan based on the outputs of the autonomous training plan model and the benchmark training plan model.

[0031] According to one or more embodiments,

[0032] It may further include a monitoring unit that monitors the status of rehabilitation training performed by the gait unit according to the training plan, while obtaining information during the training and generating training status information about the patient's gait training based on this information.

[0033] According to one or more embodiments,

[0034] The monitoring department can provide the patient with training guidelines or recommendations related to improving walking or the current training status based on the information obtained during the training.

[0035] According to one or more embodiments,

[0036] The training evaluation unit may include an anomaly detection model based on time-series information, which analyzes the current training results or analyzes the evolution of the results of repeated training based on the previous and current training results to make predictions for future training.

[0037] The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to this disclosure includes:

[0038] The step of predicting the patient's walking assessment indicators by the indicator prediction unit based on the patient's initial walking status information or walking training results information after initial walking training;

[0039] The steps for the training planning department to establish a training plan for each patient based on the walking assessment indicators using a self-discipline training plan model;

[0040] During the rehabilitation training of the patient performed by the walking motion unit, which has an end effector-type pedal and one or more pedal drive units configured to drive the pedal, according to the training plan, the training execution unit monitors training data about the real-time training status from the patient in real time, and the training execution model set in the training execution unit generates feedback about the training data; and

[0041] The step of evaluating the training results by the Training Evaluation Department based on the training result data.

[0042] In one or more embodiments of an AI-based method for providing robot-assisted orthopedic movement, the training data may include at least one of GRF, BWS, upper body tilt, joint angle, steps per minute, left foot walking cycle, right foot walking cycle, stride length, ankle angle, oxygen saturation data, and heart rate data.

[0043] In a method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to one or more embodiments, the training model may receive at least one training parameter among step length (cm), step height (cm), initial contact angle (degree), and toe off angle (degree) and output a correction value for the at least one training parameter based on training data.

[0044] In the method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to one or more embodiments, the training model can be any deep learning model among Multi-Layer Perceptron (MLP) and Open Vision-Language Model (Open VLM).

[0045] In the AI-based robot-assisted orthopedic motion provision method according to one or more embodiments,

[0046] Video or audio information about the patient during training or the therapist assisting the patient during training can be obtained through video or audio recording devices.

[0047] The video or audio information is analyzed by the analysis device, and...

[0048] The training model can use the analysis as input data to generate output data corresponding to the input data.

[0049] In the method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to one or more embodiments,

[0050] The training plan further includes: a self-disciplined training plan model, which establishes a personalized training plan for trained patients based on initial walking status information or on walking training results; and

[0051] A reference training model is a training program based on patient information, training procedures, training time, training parameters, etc., set by medical personnel.

[0052] The training planning unit can output a final training plan based on the output correction or compensation of the autonomous training plan model and the benchmark training plan model.

[0053] In the method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to one or more embodiments,

[0054] The testing department can detect at least one of the following information during gait training: GRF, BWS, upper body tilt, joint angle, steps per minute, left foot gait cycle, right foot gait cycle, stride length, ankle angle, oxygen saturation data, and heart rate data.

[0055] In the AI-based robot-assisted orthopedic motion provision method according to one or more embodiments,

[0056] The supervisory department can monitor the status of rehabilitation training performed by the gait department according to the training plan, obtain information during the training, and thereby generate training status information about the patient's gait training.

[0057] The reporting department can analyze previous and current training results to generate reports in text, image, and video formats.

[0058] In the method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to one or more embodiments,

[0059] The HMD can provide the patient with necessary information related to their walking movement, training status information, or training-related recommendations.

[0060] Beneficial effects

[0061] According to the present invention, a patient-friendly gait training program can be established, which can reduce the workload of therapists compared with existing technologies, and can effectively conduct personalized and patient-friendly rehabilitation training. In rehabilitation training based on this system and method, artificial intelligence models for each step can be used to predict the correct gait assessment indicators, automatically establish training plans based on these indicators, and more accurately evaluate training results. Attached Figure Description

[0062] Figure 1 The present disclosure provides a block diagram of an AI-based robot-assisted orthopedic motion system.

[0063] Figure 2 This is a schematic perspective view of an artificial intelligence-based robot-assisted orthopedic motion system according to an embodiment of the present disclosure.

[0064] Figure 3 This is a side view of an artificial intelligence-based robot-assisted orthopedic motion system according to an embodiment of the present disclosure.

[0065] Figure 4 This is a front view of an artificial intelligence-based robot-assisted orthopedic motion system according to an embodiment of the present disclosure.

[0066] Figure 5An example is shown depicting a patient undergoing training while wearing a head-mounted display (HMD) in a walking training system according to this disclosure.

[0067] Figure 6 A schematic perspective view of the walking motion unit in the AI-based robot-assisted walking training system according to this disclosure.

[0068] Figure 7 This illustrates an example of a normal walking cycle, showing the posture of the feet, knees, and thighs during normal walking.

[0069] Figure 8 This shows the rules for each position in the walking mode.

[0070] Figure 9 Examples of functional components based on software and hardware architecture in a system control apparatus according to the present disclosure are shown.

[0071] Figure 10 The diagram illustrates the composition of a dataset applied to a gait assessment index prediction model, which is used in an AI-based robot-assisted orthopedic motion method and system according to an embodiment of this disclosure.

[0072] Figure 11 This is a flowchart illustrating the training process for a new patient in the motion cues method for robot-assisted orthopedic surgery based on thermal intelligence, according to this disclosure.

[0073] Figure 12 This is an overall flowchart of a motion cues method for robot-assisted plastic surgery based on artificial intelligence, according to an embodiment of the present disclosure.

[0074] Figure 13 The illustration shows an example of a code segment related to a Long Short-Term Memory (LSTM) model in a motion method and system for robot-assisted orthopedic surgery based on an embodiment of the present disclosure.

[0075] Figure 14 An example of a code segment illustrating the forward propagation process of a model defined in a system based on an artificial intelligence-based robot-assisted orthopedic procedure, according to an embodiment of the present disclosure, is shown.

[0076] Figure 15 This illustration shows a code segment illustrating a motion method and system based on artificial intelligence for robot-assisted orthopedic surgery according to an embodiment of the present disclosure, defining a Python class that implements early stopping.

[0077] Figure 16The illustration shows an example of a code segment defining a train_model in a motion method and system for robot-assisted plastic surgery based on an embodiment of the present disclosure, wherein the train_model is a function used for model training.

[0078] Figure 17 The illustration shows an example of a code segment in an AI-based robot-assisted orthopedic motion method and system according to an embodiment of the present disclosure, which evaluates new data from patients undergoing gait training to predict corresponding patients' gait assessment categories (FAC).

[0079] Figure 18 The illustration shows an example of a motion method and system for robot-assisted plastic surgery based on artificial intelligence, according to an embodiment of the present disclosure, and an example of code segment related to the execution of an LSTM model.

[0080] Figure 19 The code snippet illustrates a method for generating a data frame from data inputs for model training, according to an embodiment of the present disclosure.

[0081] Figure 20 This is a code segment relating to a converter model declaration, learning settings, model generation, etc., according to an embodiment of this disclosure.

[0082] Figure 21 This is a code segment regarding training and testing data prediction of a converter model according to an embodiment of this disclosure.

[0083] Figure 22 A graphical interface illustrating various parameters for walking training according to an embodiment of the present disclosure is shown, and,

[0084] Figure 23 The results of the patient's walking training are shown, with a basalt-shaped network illustrating the distribution of the patient's average plantar pressure during training.

[0085] Figure 24 This is a code segment of the declaration section for a dataset class used to train model 3 according to an embodiment of the present disclosure.

[0086] Figure 25 To define a code segment of Model 3 according to an embodiment of this disclosure,

[0087] Figure 26 This is a code segment that outputs feedback generation function for providing feedback to the patient according to an embodiment of the present disclosure.

[0088] Figure 27 This is a code segment of a real-time array processing function according to an embodiment of the present disclosure.

[0089] Figure 28 This is a code segment used to train a model 3 according to an embodiment of the present disclosure.

[0090] Figure 29 This is a code segment for a function that evaluates a trained model 3 according to an embodiment of this disclosure, and...

[0091] Figure 30 This is a code segment used to generate execution code for a model 3 trained according to an embodiment of this disclosure.

[0092] Figure 31 The code snippet provided is for a data preprocessing and learning pipeline using an LSTM model for anomaly detection and evolution prediction, in accordance with this disclosure.

[0093] Figure 32 The following is a declaration of the LSTM-based Autoencoder class and Seq2Seq class for anomaly detection and evolution prediction according to this disclosure, along with an example code segment illustrating the two model initialization processes using these classes.

[0094] Figure 33 The function shown is for training the anomaly detection model M4a, which is an element of the M4 model according to this disclosure.<train_anomaly_model> Class definition.

[0095] Figure 34 An example of the definition of a function for training an evolutionary prediction model M4b in the M4 model according to this disclosure is shown.

[0096] Figure 35 An example is shown of the definition of the function evaluate_anomaly_model for evaluating the anomaly detection model M4a according to the M4 model of this disclosure.

[0097] Figure 36 This section shows a code segment of pseudocode for a function in the M4 model of this disclosure that visualizes, analyzes, and suggests improvement solutions for test results. Detailed Implementation

[0098] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. However, embodiments of the present invention can be modified in various different forms, and the scope of the present invention should not be construed as limited to the embodiments described in detail below. The embodiments of the present invention are preferably provided to provide a more complete description of the present invention to those skilled in the art. The same reference numerals always refer to the same elements. Furthermore, the various elements and regions in the drawings are shown schematically. Therefore, the present invention is not limited to the relative dimensions or spacing shown in the drawings.

[0099] The terms "first," "second," etc., can be used to describe various constituent elements, but the constituent elements are not limited to these terms. These terms are used only to distinguish one constituent element from others. For example, without departing from the scope of the invention, a first constituent element can be named a second constituent element, and vice versa.

[0100] The terminology used in this application is for describing specific embodiments only and is not intended to limit the concept of the invention. Unless the context clearly indicates otherwise, singular expressions also include plural expressions. In this application, expressions such as "comprising" or "having" should be understood as specifying the presence of a particular, number, step, operation, constituent element, component, or combination thereof described in the specification, rather than precluding the presence or additional possibilities of one or more other features or numbers, operations, constituent elements, components, or combinations thereof.

[0101] Unless otherwise defined, all terms used herein, including technical and scientific terms, shall have the same meaning as commonly understood by one of ordinary skill in the art to which the concepts of this invention are described. Furthermore, it should be understood that commonly used terms, such as those defined in dictionaries, shall be interpreted as having a meaning consistent with their meaning in the relevant technical context, and shall not be interpreted as having an overly formal meaning unless expressly defined herein.

[0102] In different implementations of certain embodiments, a particular process sequence may be performed differently than the described sequence. For example, two processes described consecutively may be performed substantially simultaneously, or they may be performed in the reverse order of the described sequence.

[0103] Figure 1 The present disclosure provides a block diagram of an AI-based robot-assisted orthopedic motion system. Figure 2 This illustration shows an embodiment of an AI-based robot-assisted orthopedic motion system according to the present disclosure. Figure 3 Here is its side view. Figure 4 This is a front view.

[0104] First, refer to Figure 1The motion system 100 includes a system control unit 111 comprised of hardware and software, a gait training apparatus 105 controlled therefrom, a main monitor or control monitor 110 configured to control the system control unit 111, a status display monitor 113 configured to indicate the patient's motion status, etc., a composite input unit 114 configured to input various data such as electrical and optical data, including a keyboard 112, and, as an additional option, a head-mounted display (HMD) 115 worn by the patient. The system control unit 111 according to this disclosure has a computer system architecture based on a virtual agent, which includes one or more models trained by machine learning or deep learning to perform data processing and prediction.

[0105] The walking unit 105 consists of two pedals stepped on by the patient and a driving device configured to operate the two pedals. The walking unit 105 itself or its surroundings has a training-state detection unit 105a', which has various sensors configured to detect patient state information during walking training.

[0106] In addition to the sensors configured to detect signals, the training state detection unit 105a' may include a signal processor configured to generate a signal to be transmitted to the system control device 111 based on the signal from the sensors. The HMD 115 is configured to prompt the patient with information needed during training and is part of the monitoring device in the walking training system 100 of this disclosure.

[0107] In addition, the walking training system 100 may be further equipped with an optical detection device configured to detect the patient's movement status, such as a webcam or video camera that serves as an optical input device.

[0108] Reference Figures 2 to 4 The walking training system 100 is a robot that provides walking training for patients, and provides two walking motion units 105, one for left and one for right, controlled by the system control device 111. The two walking motion units 105 are arranged side-by-side between a left body base 101 and a right body base 101, spaced apart by a predetermined interval. The two pedals 105a of the two walking motion units 105 can move harmoniously with each other during walking, corresponding to the rotation of the left and right ankles and the three-degree-of-freedom movement of the left and right feet (up, down, forward, and backward), but in some cases, they can also move uncoordinatedly.

[0109] The pedal moves in response to a control signal from the system control device 111 to enable walking movements suitable for the patient, who can step on it and follow a training walking trajectory determined based on the pedal movement. An ankle strap, which may be provided here, secures the foot to the pedal with appropriate looseness or tightness, allowing the patient's foot to move within a certain range.

[0110] The movement of these two pedals 105a is used for the patient's walking training. Through the system control device, it is controlled to accommodate abnormal walking patterns while simultaneously bringing the walking pattern closer to a normal walking pattern.

[0111] For this walking training, the left-side walking motion unit 105 has a pedal 105a and a dynamically connected operating link 105b. The operating link 105b is connected to the left-side body base 101 via a first drive device 105c disposed on the left-side body base 101 to enable linear reciprocating motion and rotational motion. Similarly, the right-side walking motion unit 105 is also connected to the right-side body base 101 via a first drive device 105 to enable reciprocating motion and rotational motion. The pedal 105a is connected to the operating link 105b via a drive device, such as a drive device that can control the rotation angle. Within a certain angle range, the pedal 105a is forced to move relative to the operating link 105b by the drive device.

[0112] The linear reciprocating motions of the walking motion units 105, executed by the first drive device, are performed independently; however, they intersect each other during walking, corresponding to the movements of the left and right feet, either spaced apart or approaching each other. The first drive device 105c, which causes the linear reciprocating and rotational motions of the operating link 105b, may include a plurality of driving devices. For example, the first drive device 105c may have a combination of a linear motion drive device and a rotational or turning motion drive device, wherein the linear motion drive device controls the linear reciprocating motion of the operating link or the walking motion unit 105 as a whole, and the rotational or turning motion drive device controls the rotational operation of the operating link 105b.

[0113] A component is provided in front of the walking movement unit 105, wherein the saddle 104 and the safety bar 106 thereon or the guardrail 107 including thereon and supporting the chest, etc., are integrated into one unit. This component is mounted on the lifting device 102 via a lifting frame 103. The lifting device 10 is a component lifting device that adjusts the height of the component to fit the patient's body condition. A system status display 109 is provided at the upper end of this lifting device 102, facing the patient.

[0114] The patient's gaze is directed at the foremost part of the walking training system 100, which is equipped with a lifting or fixed support column 113a, to which a status display monitor 113 indicating the overall system operation status is attached.

[0115] Figure 5 An exemplary scene is shown of a patient 1 wearing the HMD115 (described later) during training in the walking training system 100 according to this disclosure. Figure 5 As shown, for example, for a critically ill patient, patient 1 can perform gait training while seated on saddle 104 and holding onto safety bar 106. Depending on the patient 1's condition, for example, for a mildly ill patient, the saddle 104 can be folded so that patient 1 can perform gait training without relying on the saddle 104.

[0116] The pedal 105 is operated by the system control device to force the patient to walk independently of the patient. At this time, pressure sensors are provided on the saddle 104 and the pedal 105a to detect the force applied to the saddle 104 and the force applied to the pedal during the walking training state. In particular, a plurality of pressure sensors are provided in front of and behind the pedal to detect the local pressure applied to the pedal to detect the degree of pressure applied by the sole of the foot to the pedal or whether there is contact.

[0117] Figure 6 A schematic perspective view of the walking motion unit 105 dynamically attached to the body base 101.

[0118] Reference Figure 6 The operating link 105b is connected to the body base 101 via a first drive device 105c to perform linear reciprocating motion (LM) and rotational motion (RM1). According to one embodiment, the first drive device 105c can be a drive motor, and a separate reciprocating motion drive device can be connected to one side or the lower part of it. On the other hand, the pedal 105a is connected to the operating link 105b via a second drive device 105d to perform rotational motion (RM2) relative to the operating link 105b at a certain angle; a strap for securing the patient's foot or ankle may be provided here.

[0119] Figure 7 This example illustrates a normal walking cycle, showing the posture of the feet, knees, and thighs during normal walking. Figure 8 This shows the rules for each position in the walking mode.

[0120] Reference Figure 7In a single gait cycle, the stance phase is the period when the feet are in contact with the ground, while the swing phase is the period when the feet are separated from the ground.

[0121] Each walking cycle involves the following three tasks.

[0122] 1. Weight Acceptance Period

[0123] This period includes an initial contact (the initial contact of the foot with the ground) interval and a loading response (the period when the sole of the foot touches the ground) interval.

[0124] 2. Single Limb Support Period

[0125] This period is the midstance of standing, which is the instant when one foot touches the ground and the other foot is separated from the ground, and is the interval when the heel is raised and the other foot swings.

[0126] 3. Limb Advancement

[0127] This period is the period of stepping with the other foot, including the pre-swing, in which the rear foot separates from the ground while the toes of the front foot that were in contact with the ground leave the ground, the mid-swing, in which the foot that was in contact with the ground begins to leave the ground while the two feet are together, and the terminal swing, in which the heel of the front foot begins to touch the ground while the rear foot pushes against the ground.

[0128] The walking pattern described above is the normal walking pattern, and patients are trained to acquire this normal walking pattern.

[0129] However, for patients who have difficulty walking normally, their feet may become incoherent and inconsistent with each cycle during gait training. For example, during the mid-stance phase, when the heel of the rear foot should be off the ground, the heel may remain on the ground. This is because the patient's body does not follow a normal gait pattern. The robot of this disclosure can forcibly flex the knee or ankle joint during this phase, causing the heel of the rear foot to conform to the patient's gait pattern and leave the ground. This forced joint movement can be performed in each cycle, thus ensuring that even if the patient has physical limitations during gait training, they still exhibit a gait pattern at least similar to a normal one.

[0130] The following describes a system control device for effectively and correctly performing gait training for the patient's rehabilitation training.

[0131] The system control device in this disclosure controls the walking motion unit in a substantial part via software and in a partial part via hardware.

[0132] This system control device operates based on hardware, which is a computer system for executing software. Therefore, the system control device according to this disclosure can have a whole or part of a conventional computer system. The software uses multiple artificial intelligence models to predict the patient's walking assessment indicators from initial walking status information or walking training results information after walking training, establishes a training plan for the individual patient based on the walking assessment indicators, evaluates the training results based on training result data, and reports the training results based on the evaluation of the training results.

[0133] According to this disclosure, the training of conventional deep learning-based models is performed by each functional model, using the trained model for training, and applying it to establish, implement, evaluate and report on patient training plans, and in the process, training plan updates are also performed for long-term training.

[0134] Figure 9 An example is shown of the functional parts of the system control device 111 according to the present disclosure, which are based on software and hardware architecture.

[0135] like Figure 9 As shown, the system control device 111 may include an input unit 111a, an indicator prediction unit 111b, a training planning unit 111c, a training execution unit 111d, a training evaluation unit 111e, and a reporting unit 111f, etc.

[0136] The input unit 111a may include an input unit 111a and an indicator prediction model M1. The input unit includes at least a keyboard or keyboard, a webcam, and a communication unit. The indicator model M1 predicts the patient walking assessment indicators required to establish a patient training plan.

[0137] The training planning unit 111b may include a training planning model M2, which is configured to plan individual patient training based on patient symptoms with reference to indicators predicted by the indicator prediction unit 111b.

[0138] The training execution unit 111d may include a training execution model M3, which is configured to detect the training status according to the training plan, generate feedback on the training status, and prompt the patient and medical personnel.

[0139] The training evaluation unit 111e may include an evaluation model M4, which evaluates the results of the training performed by the training unit 111d and estimates future evolution.

[0140] The reporting unit 111f may include a reporting model M5, which reports the training results obtained from the training evaluation unit 111e to patients and training-related personnel in various forms.

[0141] In addition to the elements listed above, the system control device 111 may further include other additional software and / or hardware depending on the design of the system control device 111.

[0142] Figure 11 This is a schematic flowchart of the training process performed by the system control device in the AI-based walking training method according to the present disclosure, which is executed by the motion system 100 having the aforementioned system control device 111.

[0143] Reference Figure 11 In the training process, for new patients, after the data preparation step through the data input (S11) and initial walking assessment index generation (S12) process, they proceed to step S13. For registered patients, after confirming their personal information, they recall the existing walking assessment indexes stored in the system and proceed to step S13.

[0144] In step S13, a training plan is generated that uses initial or existing walking assessment metrics.

[0145] In step S14, patient walking training is performed according to the training plan.

[0146] In step S15, an intermediate walking evaluation metric is generated based on the current walking training results.

[0147] In step S16, the training results are analyzed by comparing the current walking evaluation index based on the current training results with the existing walking evaluation index.

[0148] Step S17 involves generating and presenting various forms of reports based on the training results analysis. After this step, it is determined whether to continue training, whether to continue training, or whether to terminate the training. If training continues, the process moves to step S13 or S14 depending on whether the training plan needs to be updated.

[0149] Figure 11 A flowchart for a more specific training process for newly trained patients.

[0150] In reference Figure 11 In the steps described in the outline, for example, five models M1, M2, M3, M4, and M5 can be used sequentially to perform one or more walking training sessions on the patient. According to other embodiments, some of the models can be excluded.

[0151] At the start of training, new patients proceed through process S11 below, while registered patients, after confirming and selecting their personal information, proceed through the training plan generation and update phase in step S13 below. That is, as already stated in the relevant... Figure 11 As mentioned in the description, S11 and S12 are the initial processes performed once. Then, at the start of training, the individual patient has been assigned to perform the training process starting from step S13.

[0152] Step S11: Input the initial test dataset used to train a patient to walk.

[0153] S12: Input the initial test dataset into the walking assessment index prediction model M1 to generate the initial walking assessment index.

[0154] S13: Input the walking evaluation index into the training plan generation model M2 to generate a training plan. At this time, if this process is not the initial training plan generation process but the generation process during repeated training, the training plan in the earlier process is updated.

[0155] S14: A training plan is generated or established, and then patient walking training using it is executed. This training is performed by the walking training unit the patient is riding in. During the training process, the training status is monitored, and information is provided to the patient based on the training status, along with motivating audio and / or text prompts. During monitoring, real-time video of the patient's walking can be acquired using a webcam or Kinect sensor included in the components of the training status detection unit, and thereby a machine learning-based model is used to extract key points and other major joint positions of the patient to represent the skeleton and extract joint angle information and walking trajectory.

[0156] This information can be displayed on monitor 113 (status representation monitor 113) Figure 4 This suggests that, on the other hand, the HMD115 worn by the patient can be used as a reference. Figure 4 This prompts the patient. The model M3 used here can be a conversational model, such as a chatbot.

[0157] S15: In this step, intermediate walking evaluation indices are generated based on the current walking training results. The walking evaluation index prediction model M1, previously used in the initial walking evaluation index prediction, is applied in this step.

[0158] S16: In this step, the training results are analyzed by comparing the current walking evaluation metric based on the current training results with existing walking evaluation metrics. The training analysis model M4, trained specifically for analyzing the training results, is used at this stage.

[0159] S17: This step involves generating and prompting various forms of reports based on the analysis of the training results. In this step, a model trained to generate reports based on the training results, such as the multimodal model M5, can be applied.

[0160] After this step, determine whether to continue training. If you choose to continue training, move to step S13 or S14. If the training plan needs to be updated based on the current results, move to S13. Otherwise, if you want to retrain according to the current training plan, immediately move to the walking training process in S14.

[0161] The artificial intelligence models mentioned above, including M1, M2, M3, M4, and M5, which are the various parts of the system control device, will be described below.

[0162] I. First Model M1

[0163] The first model, M1, is a model configured to predict patients' walking assessment indicators. It can be either a Recurrent Neural Network (RNN) model or a Long Short-Term Memory (LSTM) model.

[0164] According to an embodiment of this disclosure, LSTM is applied. LSTM is a type of recurrent neural network (RNN), a model designed to address long-term dependency problems, and has a structure capable of effectively learning the sequential information of data.

[0165] When learning this LSTM, dropout and early stopping methods are applied. Dropout is a normalization method to prevent overfitting of neural networks. During the learning process, some neurons are arbitrarily deactivated (dropped) to prevent the model from overfitting to a specific pattern; this method can be applied to embodiments of this disclosure. Early stopping is a method to prevent overfitting during the model learning process, wherein learning is terminated early if the performance on the test data does not improve.

[0166] Figure 12 An exemplary diagram illustrates the composition of a patient dataset used to generate initial walking assessment indicators by the walking assessment indicator prediction model M1.

[0167] The basic input data consists of basic information about the exercise device 100 entered by the treating hospital, which should be managed in accordance with personal information protection procedures.

[0168] This dataset includes patient personal information, patient symptom information, functional ambulation categories (FAC) determined by medical personnel, and the Berg Balance Scale (BBS) as patient information data.

[0169] The patient's personal information includes age, gender, height, weight, leg length, shoe size, and symptoms. Symptom information includes the symptoms and reference details for each symptom, the date of onset of the symptoms, and the location of the symptoms in the lower limbs (both feet, left foot, right foot), etc.

[0170] Furthermore, it also includes the diagnostic results of medical personnel, such as forestry assessment indicators based on clinical assessment tools like FAC and pre-walking training data, etc.

[0171] On the other hand, the data includes walking training results measured on flat ground before using the walking rehabilitation training robot, which includes information on upper body tilt and lower limb joint angles during walking. Furthermore, the data includes walking training results measured after using the walking rehabilitation training robot, which includes the motor system training results dataset disclosed herein. This includes information on average ground reaction force (GRF), center of pressure (COP), body weight support (BWS), lower limb joint angles, and left-right balance during each walking cycle (Gait Cycle: 0 to 99%).

[0172] On the other hand, as optional input data, information provided by hospitals or patients includes doctor comments, walking images, walking training videos (MOV), and training measurement information, such as GRF information measured during walking on a GRF measurement mat, butterfly diagram image information based on COP during walking, and upper body tilt information.

[0173] Therefore, the first model M1 is trained to receive the data and output a walking evaluation metric.

[0174] The FAC, which is one type of input data, has six metrics, which can be summarized as follows.

[0175] FAC0: Unable to walk - A state where one is unable to walk.

[0176] FAC1: Although there is a walking function, the walking ability is very limited and walking assistance is required.

[0177] FAC2: A state where walking ability is weak and walking assistance is required.

[0178] FAC3: Walking ability has recovered somewhat, but some walking assistance is still needed.

[0179] FAC4: A state in which walking ability has almost recovered and one can walk independently without walking aids.

[0180] FAC5: The patient has fully recovered functional walking ability and is able to walk independently without the need for walking assistance.

[0181] On the other hand, the BBS consists of various items used to assess the ability to maintain balance while performing various activities. These items include standing posture, the ability to transition to a squatting standing posture, the ability to maintain a squatting standing posture, and the ability to turn one's head while standing. Each item is rated on a scale of 0 to 4, with a total score of 56 points. Higher scores represent better balance ability.

[0182] The first model applies and utilizes the benchmark response (FAC, BBS, etc.) levels of walking assessment indicators used in the medical field to predict the walking assessment indicators of patients undergoing walking rehabilitation training. It can be a recurrent neural network (RNN) model or a long short-term memory network (LSTM) model.

[0183] As a model training scheme to improve the resolution of judgment in the FAC (Application Level 6), the temporal walking training data (3 minutes each of flat ground and stair walking training) obtained from non-disabled groups (multiple people) by the exercise system disclosed herein can be used as a supervised learning model group (correct answer: 100%), and an anomaly detection model based on temporal information can be used to quantify the temporal walking training results data of walking disabled people from 0 to 100% to determine the walking level.

[0184] The model output data can be changed into various forms according to the design. However, if the FAC level is applied, the disability measurement value corresponding to the non-disabled measurement value (100%) and the measurement value (%) reflected by the time-series information-based anomaly detection model are compared and judged as follows.

[0185] example)

[0186] 0 ~ 20% = FAC 0

[0187] 21 ~ 40% = FAC 1

[0188] 41 ~ 60% = FAC 2

[0189] 61-70% = FAC 3

[0190] 71 ~ 80% = FAC 4

[0191] 81 ~ 100% = FAC 5

[0192] If the FAC level is not applied, the disability level can be more accurately determined by directly applying an anomaly detection model based on time-series information that corresponds to the measurement value (%) of non-disabled persons (100%).

[0193] The model executed as described above is described below as an example.

[0194] Example 13 shows a code snippet for initializing the `ImprovedLSTMModelWithDropout` class, which improves a Long Short-Term Memory (LSTM) network model and adds a random dropout layer. This code was written using PyTorch.

[0195] The ImprovedLSTMModelWithDropout class inherits from nn.Module to define LSTM models.

[0196] This class has an initialization method "__init__" and a "forward" method. The parameters of the initialization method are as follows.

[0197] -input_size: Number of input data features

[0198] -hidden_size: Size of the LSTM hidden state

[0199] -num_layers: The number of LSTM layers

[0200] -output_size: Number of output data characteristics

[0201] -dropout: Random dropout rate

[0202] Furthermore, its instance variables are as follows.

[0203] -self.lstm: Defines the LSTM hierarchy

[0204] -self.fcl: Defines the first fully connected layer.

[0205] -self.dropout: Defines the random dropout level, and

[0206] -self.fc2: Defines the second fully connected layer.

[0207] Figure 14 An example code snippet is shown that defines the forward propagation process of the model.

[0208] This involves a series of steps to process the input data to generate the final predicted value, transforming the input data and gradually generating the model's predicted value at each step.

[0209] On the other hand, the main processing steps of the forward method are as follows.

[0210] (1) First step of feedforward:

[0211] h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)

[0212] c0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(x.device)

[0213] - In this step, the initial hidden state (h0) and cell state (c0) are defined. Objects h0 and c0 are tensors representing the initial hidden state and cell state of the LSTM network, respectively, providing initial state values ​​for each layer and element of the LSTM network before the network processes the initial sequence. Tensors are generated using the torch.zeros function, where all elements are initialized to 0. These objects, i.e., tensors, have a specific batch size (x.size(0)) and the size of the hidden state (self.hidden_size), and their dimensions are set according to the number of network layers (self-num_layers). Furthermore, the tensors generated using the to(x.device) method are moved to the same device as the input data x (e.g., CPU or GPU).

[0214] (2) Feedforward second step:

[0215] out = self.lstm(x, (h0, c0))

[0216] In this step, the input data x and the initial state (h0, c0) are passed to the LSTM layer for processing. Here, x: the input data tensor (batch size, sequence length, number of input features), and out: the output of the LSTM layer, a tensor in the form of (batch size, sequence length, size of hidden states).

[0217] (3) Feedforward third step

[0218] out = self.fc1(out[:, -1, :])

[0219] out[:,-1,:]: Only uses the output of the final time step of the LSTM. This output passes through the first fully connected layer (self.fcl).

[0220] (4) Fourth step of feedforward:

[0221] out = torch.relu(out)

[0222] The ReLU activation function is applied to the output of the first fully connected layer. This adds non-linearity to improve the model's expressiveness.

[0223] (5) Fifth step of feedforward:

[0224] self.dropout(out)

[0225] Random inactivation is used to arbitrarily inactivate some neurons.

[0226] (6) Feedforward sixth step:

[0227] self.fc2(out)

[0228] The output of the randomly deactivated layer is passed to the second fully connected layer (self.fc2) to generate the final prediction.

[0229] (7) Seventh step of feedforward:

[0230] return out

[0231] Only in this step is the model's final prediction returned.

[0232] Thus, the forward method defines the process by which input data is transformed into the final output value through LSTM and fully connected layers, activation functions, and random deactivation. Each step is used to gradually transform the data to generate the final predicted value.

[0233] Figure 15 An example code snippet is shown regarding the definition of the Early Stopping class.

[0234] This class is used to prevent overfitting of machine learning models during training and to support effective learning. Its main components and functions are as follows.

[0235] (1) Initialization method (__init__)

[0236] def __init__(self, patience=15, delta=0.0005):

[0237] -patient(int): An instance variable used to set how many times to wait if there is no performance improvement. The basic setting value is 15, which is received as a parameter.

[0238] -delta(float): An instance variable used to store the minimum change that is considered a performance improvement. This value defines how much the test loss should improve compared to the previous minimum loss. The default value is 0.0005, which is also received as a parameter.

[0239] -best_score(float): An instance variable used to store the highest observed performance score (the negative value of the validation loss).

[0240] -early_stop(bool): An instance variable used to store a flag representing whether early stopping is enabled.

[0241] -val_loss_min(float): An instance variable used to store the minimum observed test loss.

[0242] -counter(int): An instance variable used to store the number of rounds that do not improve performance.

[0243] (2) Calling the method (__call__)

[0244] def __call__(self, val_loss, model):

[0245] -val_loss(float): The test loss in the current round.

[0246] -model(torch.nn.Module): The PyTorch model object being evaluated.

[0247] This method evaluates the model's validation loss after each round to check for early stopping conditions. If performance does not improve by delta or more from the previous highest level, the counter is incremented; if the counter reaches patience, the early_stop flag is set to True, causing learning to stop. However, if performance improves, the counter is reset and the model state is stored as a checkpoint.

[0248] (3) Checkpoint storage method (save_checkpoint)

[0249] def save_checkpoint(self, val_loss, model):

[0250] In this method, the model state is stored as a document (checkpoint.pt). This is used later when the model is reloaded or when a better performance state is desired, to update val_loss_min with the latest test loss value.

[0251] The main purpose of early stopping is to aid effective learning and prevent overfitting, reduce unnecessary learning time, and help the model maintain its generalization ability. Early stopping is a key technique in many machine learning scripts, especially for deep learning models that require high computational resources.

[0252] Figure 16 An example code snippet is shown regarding the definition of the function train_model used for model training.

[0253] def train_model(model, X, y, num_epochs=100, learning_rate=0.0001):

[0254] The function `train_model` is used to train a model. It takes `model` (the PyTorch model object to be trained), `X` (the input data tensor), `y` (the output data tensor), `num_epochs` (the number of training epochs, default value 100), and `learning_rate` (the learning rate, default value 0.0001) as parameters.

[0255] (1) Loss function: mean-square error (MSE)

[0256] criterion = nn.MSELoss()

[0257] - Define the loss function. Here, we use the mean squared error (MSE) loss function. MSE is the average of the squared differences between the predicted and actual values, and is primarily used for regression problems.

[0258] (2) Generate optimizer class

[0259] optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

[0260] - The `torch.optim.Adam` class is used to generate Adam optimizer objects for updating the model formaldehyde. The optimizer adjusts the model formaldehyde along the gradient using the learning rate `learning_rate`.

[0261] (3) Generate an instance of the early stopping class.

[0262] early_stopping = EarlyStopping(patience=15, delta=0.0005)

[0263] - Generate an instance of the EarlyStopping class for early stopping. patience is the number of rounds to wait even if the loss cannot be improved, and delta is the minimum change that will be considered an improvement.

[0264] (4) Learning Cycle

[0265] for epoch in tqdm(range(num_epochs), desc="Training Progress"):

[0266] The code above is a loop statement for learning. It repeats a total of num_epochs and uses the tqdm library to visually represent the learning progress.

[0267] (4.1) Set the model to learning mode

[0268] model.train()

[0269] The model is converted to a learning mode. This is to allow layers such as Dropout or batch normalization to operate differently in the learning mode.

[0270] (4.2) Generate predicted values

[0271] outputs = model(X)

[0272] The input data X is passed to the model to generate predicted values. The outputs are the model's prediction results.

[0273] (4.3) Initialize the optimizer gradient

[0274] optimizer.zero_grad()

[0275] - Initialize the optimizer's skew buffer. This is necessary to initialize the skew before calculating the skew via backpropagation.

[0276] (4.4) Calculate the loss

[0277] loss = criterion(outputs, y)

[0278] - Calculate the loss between the predicted values ​​(outputs) and the actual values ​​(y). Here, the MSE loss function is used to calculate the loss value.

[0279] (4.5) Calculate the loss slope

[0280] loss.backward()

[0281] - The slope of the loss is calculated through backpropagation, and the slope of each parameter is calculated in the step.

[0282] (4.6) Update weighted

[0283] optimizer.step()

[0284] - Update the model's weights using the optimizer based on the calculated tilt.

[0285] (4.7) Store the current loss value

[0286] val_loss = loss.item()

[0287] - Convert the current loss value into a scalar value for storage. This will later be used as a standard for early stopping.

[0288] (4.8) Confirm the conditions for early termination

[0289] early_stopping(val_loss, model)

[0290] - Use the loss value of the current round to confirm the early stopping condition. Track the number of rounds in which the early_stopping object does not improve; if the condition is met, terminate model learning.

[0291] (4.9) Output early stop message

[0292] if early_stopping.early_stop: print("\nEarly stopping")

[0293] - In this step, if early_stopping.early_stop is True, an early stop message is output and the learning process is terminated by breaking free from the loop statement.

[0294] (5) Output the learning end message

[0295] print('Model learning has ended')

[0296] - The final output is a message indicating the end of the learning process.

[0297] The code snippets described step-by-step above implement the basic structure for training a PyTorch model using early stopping. The MSE loss function and Adam optimizer are commonly used in regression work, while the early stopping mechanism helps prevent overfitting by stopping learning when the test loss no longer shows significant improvement.

[0298] Figure 17 An exemplary code snippet is shown regarding a data evaluation function for predicting a patient's FAC by assessing new data from a patient undergoing gait training.

[0299] The function shown uses a PyTorch model to generate predicted values ​​for new data and calculates similarity by comparing them with the actual values.

[0300] (1) Loading data

[0301] def evaluate_new_data(file_path, model, data_mean, data_std, device):

[0302] The function `evaluate_new_data` reads a CSV document from `file_path`, selects only the necessary columns, and transforms it into a DataFrame. The necessary columns might be, for example, the left current Gait Cycle Rate (p_leftCurrentGaitCycleRate), the left current Ground Reaction Force (p_leftCurrentGRF), the right current Gait Cycle Rate (p_rightCurrentGaitCycleRate), and the right current Ground Reaction Force (p_rightCurrentGRF).

[0303] #Data Preprocessing:

[0304] new_data['p_leftCurrentGaitCycleRate'] =

[0305] new_data['p_leftCurrentGaitCycleRate'].astype(int)

[0306] new_data['p_rightCurrentGaitCycleRate'] =

[0307] new_data['p_rightCurrentGaitCycleRate'].astype(int)

[0308] Normalization is performed after converting p_leftCurrentGaitCycleRate and p_rightCurrentGaitCycleRate to integer types.

[0309] (2) Prepare model input data:

[0310] The input data X_new and output data y_new are separated from the normalized data. The input data includes p_leftCurrentGaitCycleRate and p_rightCurrentGaitCycleRate, and the output data includes p_leftCurrentGRF and p_rightCurrentGRF.

[0311] (2) Prediction model:

[0312] The input data is converted into a PyTorch tensor and passed to the model to generate predictions. The model is set to eval mode, and the gradient computation is deactivated using the torch.no_grad() method.

[0313] (3) Calculation results

[0314] The average error rate is calculated by determining the difference between the predicted and actual values ​​using percentage units. Similarity is then calculated based on this.

[0315] (4) Determine the FAC level:

[0316] The patient's FAC grade is determined based on similarity. For example, when the similarity is between 0 and 40%, the rules for FAC1, etc., are followed.

[0317] Figure 18 An example code snippet is shown regarding the execution of the LSTM model.

[0318] (1) Specify the required columns and set the file path

[0319] In this process, the required data columns (required_columns) and file paths (file_path) are executed first.

[0320] -file_paths: Paths to the various CSV files used for model learning.

[0321] -new_data_path and eval_data_path: Paths for new data and evaluation data.

[0322] -model_save_path: Specifies the path to the checkpoint file of the trained model.

[0323] (2) Load and preprocess data

[0324] combined_data = load_and_combine_csv(file_paths, required_columns)

[0325] X, y, data_mean, data_std = preprocess_data(combined_data)

[0326] -load_and_combine_csv: A function that reads and combines data from multiple CSV files.

[0327] -preprocess_data: Preprocesses the combined data to return the input data X, output data y, data mean data_mean, and data standard deviation data_std.

[0328] (3) Configure the processor (GPU or CPU) to use

[0329] device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

[0330] - During this process, the GPU memory is released. - If the GPU is available, CUDA is set as the device; otherwise, the CPU is set as the device, and data is transferred to the set device.

[0331] (4) Model initialization and learning

[0332] An improved LSTM model, including early stopping, is generated using the `ImprovedSLTMModelWithDropout` class, and model training using the training data is performed using the `train_model` function. In this embodiment, the learning rate is set to 0.0001 during the 100 training epochs.

[0333] (5) Data evaluation

[0334] In this step, the data is evaluated by a data evaluation function (e.g., the evaluate_new_data function described above), and the results are stored in the similarity (similarity_percentage) and FAC grade (fac_grade) parameters.

[0335] In this step, data evaluation is performed by a data evaluation function, such as the function evaluate_new_data described above, and the results are stored as similarity_percentage and fac_grade parameters.

[0336] This code snippet loads and preprocesses data from a CSV file containing information for assessing the FAC (Features, Acute, and Cognitive Capabilities) during gait training for a given patient. It then trains an LSTM model, evaluates new data using the trained model, and calculates the similarity between predicted and actual results to determine the FAC grade. This can be used for initial data on patient gait training, such as gait analysis, or for later assessment of a patient's FAC development during gait training.

[0337] II. Second Model M2

[0338] The second model is an AI agent that automatically generates a walking training plan based on the patient's symptoms in the AI-based robot-assisted orthopedic movement system according to the present disclosure. This model can be, for example, a Transformer model, and according to other embodiments, LSTM or Gated Recurrent Unit (GRU) can also be applied.

[0339] The input data for this second model includes 1) patient information, 2) pre-walking training results data, and 3) walking training standard procedures and training plan definition data.

[0340] The patient information includes the gender, weight, shoe size, year of birth, and FAC (Fast Walking Assessment) level assigned to the patient during registration with the training system. The preliminary training results data includes training procedures such as climbing / descending stairs and sloping terrain, training time, and datasets of training parameters such as stride length, stride height, initial contact angle, toe-off angle, ground reaction force, and gait cycle. Furthermore, the standard walking training procedures and training plan definition data are data determined by medical personnel.

[0341] The second model receives the aforementioned data, along with the currently determined patient FAC, patient information, and current training results, to derive (output) a training plan, such as training procedures, training time, and training parameters for walking on flat ground, climbing up / down stairs, and climbing up / down sloping ground. This can be a multi-input, single-output, or multi-output model.

[0342] According to another embodiment, the input is the current walking assessment indicators and patient information, and the output is the standard training model group (Classification). According to other embodiments, the input is the current walking assessment indicators, patient information, and training information, and the output can be patient-tailored training plan information.

[0343] When applying AI-driven autonomous judgment of the training plan, the output data of the second model is a training plan definition dataset reflecting the training results of each patient. Corrections to the real-time training plan, reflecting the training status of each patient, include changes to training time, training parameters, training procedures, and settings for robot stop / start.

[0344] Model M2, used to generate AI self-discipline judgment training plans, is an AI agent that uses patient walking training data to predict training parameters. It can be generated and trained using Python and PyTorch, just like model M1 described above. Various libraries, including PyTorch, can be imported for this purpose.

[0345] -import os: Module for working with files and directories.

[0346] -import torch: A library for basic tensor operations and building neural networks.

[0347] -import torch.nn as nn: Neural network module (e.g., hierarchy definition)

[0348] -import torch.optim as optim: Optimization algorithms used for model learning (e.g., Adam, SGD, etc.)

[0349] -from torch.utils.data import DataLoader, TensorDataset: Modules for data processing (generating batches, etc.)

[0350] -import pandas as pd: A library used to create and manipulate data frames.

[0351] The Python code that imports the libraries described above includes the following procedures for declaring and training the M2 model.

[0352] (1) Input data and generate a data frame

[0353] Figure 19 This shows a code snippet from the entire Python code used to form the M2 model, detailing the data inputs for model training and the generation of the data frame from these inputs.

[0354] The M2 model is an instance of a Transformer model using the PyTorch neural network model nn for predicting training parameters using patient walking training data. As described below, the input data is provided in the form of multiple CSV files, which are combined into a two-dimensional data frame through a preprocessing process and used as training data.

[0355] (1.1) Set the directory path where the CSV file is located.

[0356] csv_directory = 'train_data_csv'

[0357] (1.2) List initialization

[0358] all_dataframes = []

[0359] (1.3) Read all CSV files in the directory and combine them into a data frame.

[0360] for filename in os.listdir(csv_directory):

[0361] if filename.endswith('.csv'):

[0362] file_path = os.path.join(csv_directory, filename)

[0363] df = pd.read_csv(file_path)

[0364] all_dataframes.append(df)

[0365] (1.4) Combine all the data into a data frame.

[0366] combined_dataframe = pd.concat(all_dataframes, ignore_index=True)

[0367] (2) Preprocess the data and convert it into tensors

[0368] (2.1) Definition of a column

[0369] relevant_columns = [

[0370] 'p_patientGender', 'p_weight', 'p_footSize', 'p_patientBirthYear', 'p_fac',

[0371] 'p_trainingCurrentCadance', 'p_strideLength', 'p_strideHeight',

[0372] 'p_initialContactAngle', 'p_toeOffAngle',

[0373] 'p_leftCurrentGRF', 'p_rightCurrentGRF',

[0374] 'p_leftCurrentGaitCycleRate', 'p_rightCurrentGaitCycleRate' ]

[0376] As defined in the array definition statement above, the input data includes patient information such as gender 'p_patientGender', weight 'p_weight', shoe size 'p_footSize', birth year 'p_patientBirthYear', and FAC 'p_fac', and as preliminary training result data, steps per minute 'p_trainingCurrentCadance', stride length 'p_strideLength', stride height 'p_strideHeight', initial contact angle 'p_initialContactAngle', toe-off angle 'p_toeOffAngle', current left foot GRF 'p_leftCurrentGRF', right foot GRF 'p_rightCurrentGRF', left foot walking cycle count 'p_leftCurrentGaitCycleRate', and left foot walking cycle count 'p_rightCurrentGaitCycleRate'.

[0377] (2.2) Copy the data frame to the selected columns

[0378] combined_dataframe_relevant = combined_dataframe[relevant_columns].copy()

[0379] (2.3) Data Preprocessing

[0380] combined_dataframe_relevant['p_patientGender'] = combined_dataframe_relevant['p_patientGender'].map({'Male': 0, 'Female': 1})

[0381] During preprocessing, "male" and "female" are transformed into "0" and "1" respectively, and unknown values ​​(NaN, Not a Number) are replaced by the average value of each column using the "fillna" method.

[0382] combined_dataframe_relevant.fillna(combined_dataframe_relevant.mean(), inplace=True)

[0383] (2.4) Convert to PyTorch tensor "X" and generate tensor "Y".

[0384] mean_target = combined_dataframe_relevant[['p_trainingCurrentCadance', 'p_stepLength', 'p_stepHeight', 'p_initialContactAngle', 'p_toeOffAngle']].mean().values

[0385] The preprocessed data is converted into a PyTorch tensor "X". Then, the mean_target of the target parameter to be predicted is calculated to generate a tensor "Y", which has the same target value for all samples.

[0386] Y = torch.tensor([mean_target] * len(X), dtype=torch.float32)

[0387] (3) Define a Transformer-based model class

[0388] Figure 20 This is a code segment concerning the declaration, learning settings, model generation, etc. of the Transformer model in the entire Python code.

[0389] class TransformerModel(nn.Module):

[0390] Define a TransformerModel class that inherits from PyTorch's nn.Module.

[0391] The `__init__` method of `TransformerModel` serves as the class generator method, defining the model structure and its hierarchical levels. It calls the parent level's initialization method `super(TransformerModel, self).__init__` to perform the initialization of the basic level, with the following parameters.

[0392] -input_dim: Specifies the dimension (number of features) of the input data.

[0393] -output_dim: Specifies the output dimension of the model (the number of parameters to be predicted).

[0394] -num_heads: Specifies the number of heads in the multi-head attention mechanism. The default value is 1.

[0395] -num_layers: Specifies the number of layers in the Transformer encoder. The default value is 2.

[0396] -hidden_dim: Sets the dimension of the forward propagation network, with a default value of 128.

[0397] The class instance nn.TransformerEncoderLayer is defined as follows: self.encoder_layer is the Transformer encoder layer provided by PyTorch.

[0398] self.encoder_layer = nn.TransformerEncoderLayer(d_model=input_dim,nhead=num_heads, dim_feedforward=hidden_dim)

[0399] The parameters for this example are as follows.

[0400] -d_model=input_dim: Input dimension.

[0401] -head=num_heads: The number of heads for the multi-head concentration mechanism.

[0402] -dim_feedforward=hidden_dim: The dimension of the forward feedforward network.

[0403] The fully connected layer fc is defined by the method nn.Linear, where the parameter input_dim is the input dimension, output_dim is the final output dimension, and the number of training plan parameters to be predicted is 0.

[0404] On the other hand, the forward propagation method of the TransformerMode preprocesses and returns the input data. To do this, a dimension is first added to the input data x, and the input x with the added temporal dimension is passed to the Transformer_encoder to be transformed into an encoded output. Then, the temporal dimension is removed from the encoded output x using the squeeze method, and the transformed data x is returned as output through a fully connected layer.

[0405] (4) Training Settings

[0406] like Figure 20 As shown, the hyperparameters are set as follows in the settings used for training.

[0407] -input_dim=X.shape[1] # Dimension of the input data (number of features)

[0408] -output_dim=5 # Dimension of the output data (number of parameters to be predicted)

[0409] -batch_size=64 # Batch size (number of data samples in one training session)

[0410] -learning_rate: 0.001 #Learning rate (the magnitude of change during weighted updates)

[0411] -epochs=100 # Number of learning repetitions

[0412] Following this process, the dataset and data loader are generated as follows.

[0413] -dataset=TenserDataset(X,Y) # Define the dataset

[0414] - train_loader = DataLoader(dataset, batch_size=batch_size, shuffle=True) # Generate the data loader

[0415] - model = trasnformerMode(input_dim, output_dim): Generates an M2 model instance.

[0416] The following process sets the initialization of the model to be trained, the loss function, and the optimizer. In this embodiment, after initializing the M2 model as a transformer model, "MSE" is set as the loss function and "Adam" is set as the optimizer, and then model training is performed.

[0417] (5) Model Training

[0418] Figure 21 This shows a code snippet related to model training and test data prediction.

[0419] like Figure 21 As shown, according to one embodiment, model training is repeated during the preset 100 epochs in the above steps.

[0420] - for epoch in range(epochs): Starts the loop used to repeat the entire training process. The number of epochs defines the number of times the training is repeated.

[0421] - model.train(): Sets the model to training mode. In this mode, specific layers such as dropout and batch normalization are enabled during training.

[0422] - running_loss = 0.0: Initializes the variable used to store the total loss of the current round, and uses this variable to calculate the average loss at the end of each round.

[0423] - for inputs, targets in train_loader: Loads input data and target data in batches in train_loader. The data loader is used to split the dataset in batches and repeatedly retrieve the dataset.

[0424] - optimizer.zero_grad(): Initializes the optimizer's gradients. PyTorch essentially accumulates previous gradients, therefore, initialization is required for each batch.

[0425] - outputs = model(inputs): This step uses the model to calculate predicted values ​​(outputs) from the input data. The forward method of the model is called in this step.

[0426] - loss = criterion(outputs, targets): Calculates the loss between the predicted values ​​and the actual target values. Here, criterion defines the loss function (e.g., MSELoss).

[0427] - loss.backward(): Performs backpropagation based on the loss to calculate the gradients of the parameters in the model.

[0428] - optimizer.step(): Updates the model's parameters based on the gradients computed using the optimizer.

[0429] `- running_loss += loss.item()`: Adds the accumulated loss value of the current batch to `running_loss`, allowing the average loss to be calculated at the end of the round. Furthermore,

[0430] - print(): Outputs the average loss for each epoch. It divides the running_loss by the length of the train_loader, calculates the average, and formats and outputs it. For example, the output of the 6th epoch out of a total of 100 epochs has a real-time loss of 250.123 and a train_loader length of 50, and will output "Epoch[6 / 100],Training Loss:5.0025".

[0431] By repeatedly performing the above model training process and calculating and outputting the loss in each round, the model parameters can be updated using the optimizer and loss function, and the training process can be supervised by the loss value.

[0432] (6) Predicting Test Data

[0433] like Figure 21 As shown, the test data prediction of the M2 model obtained through the above process is also performed on the input data of the CSV file.

[0434] - test_data = pd.read_csv('test.csv'): Reads the test data file.

[0435] - `test_data_relevant = test_data[relevant_columns].copy()`: In the test data processing, select the desired columns from the data.

[0436] `test_data_relevant['p_patientGender'] = test_data_relevant['p_patientGender'].map({'Male': 0, 'Female': 1}):` converts gender data to numbers.

[0437] `test_data_relevant.fillna(test_data_relevant.mean(), inplace=True)`: Handles unknown values.

[0438] `X_test = torch.tensor(test_data_relevant.values, dtype=torch.float32):` converts the test data into a PyTorch tensor.

[0439] (7) Model prediction

[0440] model.eval(): Sets the model to evaluation mode (deactivates dropout and other similar methods).

[0441] with torch.no_grad(): Sets the gradient to not be calculated.

[0442] `prediction = model(X_test).mean(dim=0).numpy()`: Calculates the average value after calculating the predicted value.

[0443] (8) Output prediction results

[0444] predicted_df = pd.DataFrame([prediction], columns=['p_trainingCurrentCadance', 'p_stepLength', 'p_stepHeight', 'p_initialContactAngle', 'p_toeOffAngle'])

[0445] print(predicted_df): Converts the prediction results into a data frame and outputs them.

[0446] The model trained as described above is used as a model for establishing patient-specific training plans.

[0447] In practical applications, to establish a training plan that reflects the results of the training already performed, the most recent data is extracted from the existing training data to make predictions using the M2 model, and the predicted results are then input into the new training model for further training.

[0448] When using the M2 model to establish a training plan, if previous patient training information exists, this information is input into the M2 model to extract the autonomous training plan. Furthermore, if necessary, an expert system is used to further enhance reliability.

[0449] The expert system includes a reference model trained according to theoretical standards. This reference model has the same structure as the M2 model and outputs a dataset defining the standard walking training procedure and training plan, determined by medical personnel, as the standard training plan. This output is matched with the training plan information corresponding to patient information and the determined walking level assessment indicators. The reference model is trained using patient information (gender, weight, shoe size, year of birth, FAC level, etc.), training procedures, training time, and training parameters set by medical personnel.

[0450] This standard model can be used to establish a pre-built AI self-discipline judgment training plan and a model to judge the reliability of the aforementioned M2 model. To this end, previous training information from patients who had previously been input into the pre-built AI self-discipline judgment training plan establishment model and the aforementioned M2 model is input into the standard model to extract the standard training plan.

[0451] This standard training plan differs from the autonomous training plan output from the M2 model, thus allowing for assessment of reliability and appropriate modification of the autonomous training plan based on the level of reliability. In actual training, either the standard training plan or the autonomous training plan can be selected. The standard model is a selective element that can be used as a means to more accurately compensate for or correct the autonomous training plan.

[0452] According to another embodiment, the autonomous training plan can be corrected based on the standard training plan. This correction can be performed by a final training plan model that outputs the optimal training plan from the outputs of the M2 model and the standard model. This final training plan model can be trained using data obtained by comparing the M2 model and the standard model described above.

[0453] III. Third Model M3

[0454] Model M3 is the training model, meaning it's a model that monitors and prompts the training status during gait rehabilitation training. Model M3 can be, for example, a chatbot-type model. Input data for this model includes patient information, robot status information, and real-time input / output information during gait. An AI agent based on a deep learning model chosen from options such as Multi-Layer Perceptron (MLP) and Open Visual-Language Model (VLM) can be applied to Model M3.

[0455] Figure 22 An example graphical interface is shown, which displays various parameters for walking training.

[0456] Real-time input / output information in walking training includes training procedures, training parameters, and training data of a certain step interval detected in real time.

[0457] Figure 23 A basalt diagram is used to illustrate the distribution of average plantar pressure during training as a result of patients' walking training.

[0458] The training data can include real-time data obtained from patients during gait training. This data can include any one of the following characteristics: Ground Reaction Force (GRF, %), Body Weight Support (BWS, %), Upper Body Inclination or Tilting (°, degree), Joint Angle (°, degree), Steps per Minute (Cadence), Left Gait Cycle (0-99%), Right Gait Cycle (0-99%), Stride Length, Ankle Angle, Oxygen Saturation, and Heart Rate.

[0459] Training parameters may include stride length (cm), step height (cm), initial contact angle (° (degree)), and toe off angle (° (degree)).

[0460] When a network camera or other audio recording device with video or microphone is set up as a separate option in the system, patient facial expressions, voices, and postures, as well as the voices and postures of the therapists, can be obtained and used as input data.

[0461] To apply this separate option, a video analysis unit, such as a video analysis unit having a video analysis model trained to analyze patient expressions or postures from video, can be provided to analyze video from the camera.

[0462] The model M3 can be trained to receive data from the video analysis unit and generate an output based on the patient's training state together with the input data.

[0463] The output data of this model M3 can include training state information, namely feedback on real-time training data and correction values ​​for the current training parameters.

[0464] The status feedback can output audio feedback such as AI voice and beep sound, and status messages to the user interface such as virtual reality HMD. For example, it can play the following audio, video, or text message: "Welcome Mr. Smith" when the patient starts training after boarding; "Excellent training status today" when training is performed in a state similar to the training data; "Press the left foot a little more" when the maximum GRF value of the left foot is at least 10% lower than the maximum GRF value of the right foot in each walking cycle; "Stand up" if the upper body gradually leans forward or sideways every 10 walking cycles; "Keep going" if the average GRF value 10 walking cycles ago is greater than the average value per minute of the current walking cycle; "Heart rate is increasing" when the heart rate increases sharply in a walking cycle. According to other embodiments, the training parameters such as speed increase or decrease and robot straightening can be changed in real time according to the supervision results of the training status through user information requests (requirements).

[0465] The training parameter correction values ​​that can be included in the output data of the model M3 may include the stride length, stride height, initial contact angle, toe departure angle, etc., as recommended training parameters that can perform walking training close to the correct solution.

[0466] The model M3 can perform training support agent functions through real-time interaction with patients / therapists. During training, the AI ​​agent provides feedback on training status information by comparing and judging previous training results and real-time training results in sequence.

[0467] As described below, the model M3 can utilize a multi-layer perceptron (MLP), which is a type of feedforward neural network.

[0468] Figure 24 The code snippet shown is from the declaration section of the dataset class used to train model M3.

[0469] The program used to train the model M3 can be made into Python, for example, by importing multiple versions of the program as follows.

[0470] import os

[0471] import pandas as pd

[0472] import torch

[0473] import torch.nn as nn

[0474] import torch.optim as optim

[0475] from torch.utils.data import Dataset, DataLoader

[0476] Referring to the above, "The os module for operating the system and interaction is imported from Python, and the pandas Python library for analyzing and manipulating data is assigned the alias 'pd'. Furthermore, torch, as a basic module of the PyTorch library, forms the deep learning framework and supports Zhang Liang arithmetic, automatic differentiation, GPU acceleration, etc. torch.nn, as the neural network module of PyTorch, is imported and assigned the alias 'nn'."

[0477] The GaitCycleDataset class reads .csv files from a specified folder and merges the data into a single dataset. During this process, missing values ​​are filled with 0, and the feature data (input) and target data (output) are normalized and stored separately. This class provides the method __getitem__, which returns a tensor containing a number of data points.

[0478] The dataset has the following elements.

[0479] -weight: p_weight

[0480] - Age: p_age

[0481] - Height

[0482] - Current GRF on the left: p_leftCurrentGRF

[0483] - Current GRF on the right: p_rightCurrentGRF

[0484] -Currently BWS: p_bwsCurrentSupport

[0485] -Body tilt: p_bodytilting

[0486] - Left ankle angle: p_kinectLeftAnkleAngle

[0487] -Right ankle angle: p_kinectRightAnkleAngle

[0488] Heart rate: p_heartrate

[0489] -Oxygen saturation: p_sp02

[0490] - Current walking cycle on the left: p_leftCurrentGaitCycleRate

[0491] - Current walking cycle on the right: p_rightCurrentGaitCycleRate

[0492] - Current steps per minute: p_trainingCurrentCadance

[0493] As the target of the dataset, output step length (p_stepLength), step height (p_stepHeight), initial contact angle (p_initialContactAngle), and toe-off angle (p_toeOffAngle).

[0494] Figure 25 The code snippet shown is defined as GaitCycleModel as model M3.

[0495] Reference Figure 25 The GaitCycleModel class has a Multi-Layer Perceptron (MLP) structure. The third model with an MLP structure is a feedforward neural network consisting of an input layer, two hidden layers, and an output layer, using the ReLU activation function in each hidden layer.

[0496] Figure 26 The code snippet shown is for the feedback generation function `generate_feedback`. This function generates various feedbacks to be provided to the walker based on real-time input data and the model's output.

[0497] For example, as described above, based on the training status, the patient is given feedback as follows: "Welcome Mr. Smith" when the patient boards the train and training begins; "Excellent training status today" when training is conducted in a state similar to the training data; "Apply pressure to the left foot more frequently" when the maximum GRF value of the left foot is at least 10% lower than the maximum GRF value of the right foot in each gait cycle; "Upright posture" when the upper body gradually leans forward or sideways in every 10 gait cycles; "Keep going" when the average GRF value 10 gait cycles ago is greater than the average value per minute in the current gait cycle; and "Heart rate is increasing" when the heart rate increases sharply in one gait cycle.

[0498] Figure 27The code snippet shown is for the real-time data processing function process_realtime_data.

[0499] The function process_realtime_data inputs data into a third model in real time to return predicted values ​​and feedback.

[0500] The parameter 'row' is used as input data and is in series format. Therefore, the data can be input into the 'inputs' variable as follows.

[0501] inputs = [70, 30, 170, 50, 55, 30, 8, 10, 12, 100, 98, 60, 62, 90]

[0502] The inputs are converted into PyTorch tensors using torch.tensor, which are then provided as inputs to the third model. Consequently, the third model outputs corresponding to these inputs.

[0503] The outputs fed back by model M3 and the inputs are input into the generate_feedback function mentioned above. This function feeds back feedback corresponding to the inputs and outputs.

[0504] Finally, the function process_realtime_data returns the outputs and feedback to indicate the feedback needed by the patient during real-time training.

[0505] Figure 28 This is the code segment for the function train_model used to train model M3.

[0506] The function `train_model` is a basic learning loop that implements training a model using given data. This function repeatedly updates the weights of the PyTorch model with the training data and outputs the training progress status for each epoch.

[0507] Reference Figure 28 The model training function `train_model` trains the model using the given data, loss function, and optimization algorithm. Its parameters are as follows.

[0508] -model: The PyTorch model to be learned

[0509] -dataloader: Provides a PyTorch DataLoader for learning datasets.

[0510] -criterion: loss function

[0511] -optimizer: Used to update the weighted optimization algorithm.

[0512] -epochs=10: Number of learning repetitions (default value is 10)

[0513] This function implements the model training loop, processing data in batches from the dataset and generating predicted values ​​through forward propagation using the model. It uses the criterion as the loss function to calculate the error (loss) between the model's predicted values ​​and the actual values.

[0514] The loss value is converted into a gradient of the model's weighted values ​​(parameters) through backpropagation, and the weights are updated through optimization algorithms. Here, the accumulated loss value over one epoch (learning the entire dataset once) is calculated and output to supervise the learning state.

[0515] Figure 29 The code segment for the evaluate_mode function used to evaluate the trained model.

[0516] The `evaluate_model` function is used to evaluate the performance of a trained model. It performs a process of comparing the model's predictions with the actual values ​​or generating feedback while repeatedly processing the data using a given dataloader.

[0517] The process involves converting the model to evaluation mode to deactivate gradient calculation, obtaining inputs and actual values ​​in batches from the data loader to generate predicted values, generating feedback based on the input and predicted values ​​of the first sample, and outputting the predicted values, actual values, and generated feedback.

[0518] Figure 30 The code segment used to generate execution code according to this disclosure is shown.

[0519] The executable code also loads the dataset through the dataloader, specifically including the following process.

[0520] - Load the dataset and generate a data loader

[0521] - Initialize and train the model

[0522] - Evaluate the trained model

[0523] - Process data in real time and provide feedback

[0524] - Store the trained model M3

[0525] The model M3 is initialized using the GaitCycleModel class and trained by the train_model function. During this process, nn.MSELoss is used as the loss function to calculate the error between the predicted and actual values, and Adam is selected as the optimizer for finer weighting.

[0526] The entire process of the exported executable code is as follows.

[0527] The execution code uses the GaitCycleDataset class to call the data stored in the learning data folder, and then performs training, evaluation, real-time data processing and model storage processes based on this data after initializing the model.

[0528] 1. Prepare data

[0529] Specify the folder path for storing the learning data as the `folter_path` variable, and read the `.csv` file in that folder to generate the dataset using the `GaitCycleDataset` class. Then, use PyTorch's `DataLoader` class to split the dataset into batches of size 16 and reposition them to generate `DataLoader` objects for training.

[0530] 2. Model Initialization

[0531] The input data size (input_size) is set to the number of feature data columns in the dataset, and the output data size (output_size) is set to the number of target data columns. Here, the hidden layer size (hidden_size) is set to 64, and the GaitCyclemodel with multi-layer perceptron results is initialized based on this.

[0532] 3. Define the loss function and optimizer

[0533] We use nn.MSELoss as the loss function to calculate the mean squared error between the predicted and actual values. We choose Adam as the optimizer and set the learning rate (lr) to 0.001.

[0534] 4. Training model M3

[0535] After outputting the "Training Model..." message, the train_model function is called to train the model for 30 epochs.

[0536] 4. Evaluation Model M3

[0537] Output the "Evaluating Model..." message and call the evaluate_model function to evaluate the performance of the trained model.

[0538] 5. Real-time data processing:

[0539] After outputting the "Processing Real-time Data..." message, the feature data (dataset.features) of the dataset is repeated line by line and the process_realtime_data function is called.

[0540] This function generates feedback based on the input data and the model's output, and outputs the generated feedback and predicted values.

[0541] 6. Storage Model:

[0542] Use `torch.save` to save the weighted output of the trained model to the file "gait_cycle_model.pth". Then output the message "Model Saved!" and terminate execution.

[0543] IV. Fourth Model M4

[0544] The fourth model is an AI agent that evaluates training results and predicts future evolution. Existing (past) training result data and current (real-time) training result data can be used as input data. In this case, the output data is the evolution analysis of past and current training result data and the prediction of the future.

[0545] The training data can include real-time data obtained from patients during gait training. This data can include any one of the following: Ground Reaction Force (GRF, %), Body Weight Support (BWS, %), Upper Body Inclination or Tilting (°, degree), Joint Angle (°, degree), Steps per Minute (Cadence), Left Gait Cycle (0-99%), Right Gait Cycle (0-99%), Stride Length, Ankle Angle, Oxygen Saturation, and Heart Rate.

[0546] Training parameters may include stride length (cm), step height (cm), initial contact angle (° (degree)), and toe off angle (° (degree)).

[0547] This fourth model is an anomaly detection model based on time-series information. For example, it may include the following algorithms: Long Short-Term Memory (LSTM) network, which is an extension of Recurrent Neural Network (RNN) to learn long-series data and can handle long-term dependencies, or Variational Autoencoder (VAE).

[0548] The fourth model is designed to analyze the evolution of repeated training results by comparing current and previous training results, and to predict future training outcomes based on this analysis. This model can output graphs showing the average evolution of previous training results, the average current training result, and the predicted evolution for future training. To this end, it compares and analyzes the evolution of previous and current training results, and predicts future training outcomes based on previous results.

[0549] The fourth model can be configured to include an anomaly detection model M4a and an evolution prediction model M4b.

[0550] The anomaly detection model M4a uses algorithms such as LSTM Autoencoder or Variational Autoencoder (VAE) to detect anomalous patterns from real-time training data. This model uses a dynamic threshold based on reconstruction error (RE) to detect anomalous data, thereby effectively analyzing the patient's current training status.

[0551] The evolution prediction model M4b uses either a Seq2Seq-based LSTM / GRU or a Transformer-based model to predict future training results based on past and current data. This model employs a sliding window technique to sample time-series data and analyze multi-input characteristics to learn the correlations between data points. Furthermore, the two models operate in tandem, incorporating the results of the anomaly detection model M4a into the weighted calculation of the evolution prediction model M4b to improve prediction accuracy.

[0552] M4's output data can include visual data in the form of charts, results analysis data, comprehensive evaluation indicators, and improvement suggestion data.

[0553] - Visualized data in chart form: including the average evolution line of previous training, the average evolution line of current training, and the predicted evolution line of future training results, which allows users to intuitively grasp the changes in training results with events.

[0554] - Results Analysis Data: Analyzing the differences between previous and current training provides the improvement or reduction rate of key metrics. Furthermore, problems occurring during training can be explicitly identified by adding emphasis to outlier data detected within specific intervals.

[0555] - Comprehensive assessment indicators and improvement suggestions: The comprehensive assessment indicators evaluate patient outcomes on a scale of 0 to 100, and provide specific adjustment suggestions for training parameters that need improvement (e.g., Step Length, Step Height).

[0556] Figure 31 The code segment shown is for a data preprocessing and learning pipeline using the LSTM model M4 for anomaly detection and evolution prediction, according to this disclosure.

[0557] The program used to train the model M4, for example, can be made in Python and can import multiple libraries as follows.

[0558] You can import OS modules for interacting with the operating system, pandas aspd for data analysis and care, torch as the core module of the PyTorch framework, torch.nn which provides the building blocks of neural networks, torch.optim which provides optimization algorithms for training deep models, torch.utils.data which provides tools for effectively managing data and processing it in batches, etc.

[0559] Hyperparameters can be set as follows, for example.

[0560] Number of features in the input data: input_size = 15

[0561] LSTM hidden size: hidden size = 64

[0562] Number of layers in LSTM: num_layers = 2

[0563] Sequence length: seq_length = 20

[0564] Batch size: batch_size = 32

[0565] Learning rate: learning_rate = 0.001

[0566] Number of rounds: epochs = 20

[0567] A data processing pipeline can include the following processes.

[0568] 1. Data loading and preprocessing

[0569] The process includes steps such as reading all CSV files stored in a specified folder, such as the "training_dataset" folder, merging them into a single Pandas DataFrame, filtering the .csv files in the training_dataset folder, reading data files and collecting data, and forming a data frame by integrating (merging) the data from the data files into a single data frame for analysis and processing.

[0570] - A dictionary (columns_to_use_updated) that forms the data structure.

[0571] The dictionary stores data as pairs of input keys and output values, where the name used for the actual output can be obtained by inputting the key as the output value.

[0572] columns_to_use_updated = {

[0573] 'p_trainingCompletionNumber': 'Gait Cycle Identifier',

[0574] 'p_leftCurrentGRF': 'Left GRF',

[0575] 'p_rightCurrentGRF': 'Right GRF',

[0576] 'p_bwsCurrentSupport': 'BWS',

[0577] 'p_trainingCurrentCadance': 'Cadence',

[0578] 'p_leftCurrentGaitCycleRate': 'Left Gait Cycle',

[0579] 'p_rightCurrentGaitCycleRate': 'Right Gait Cycle',

[0580] 'p_stepLength': 'Step Length',

[0581] 'p_stepHeight': 'Step Height',

[0582] 'p_initialContactAngle': 'Initial Contact Angle',

[0583] 'p_toeOffAngle': 'Toe Off Angle',

[0584] 'p_footSize': 'Foot Size',

[0585] 'p_weight': 'Weight',

[0586] 'p_age': 'Age',

[0587] 'p_height': 'Height',

[0588] 'p_fac': 'FAC Grade'

[0589] }

[0590] 2. Data separation:

[0591] In this step, some selected data (filtered_data) in the entire dataset is split into training data (train_data) and test data (test_data), which is split into a 7:3 ratio in the example code below.

[0592] train_data, test_data = train_test_split(filtered_data, test_size=0.3, random_state=42)

[0593] 3. Grouping:

[0594] The data is grouped based on the “Gait Cycle Identifier” output as a dictionary value.

[0595] train_grouped = train_data.groupby("Gait Cycle Identifier")

[0596] test_grouped = test_data.groupby("Gait Cycle Identifier")

[0597] Figure 32 The following is a declaration of the LSTM-based Autoencoder class and Seq2Seq class for anomaly detection and evolution prediction according to this disclosure, along with an example code segment illustrating the two model initialization processes using these classes.

[0598] The initial anomaly detection model M4a is named `anomaly_model`, and the evolution prediction model M4b is named `prediction_model`. Mean Square Error (MSE) is used as the loss function for each model, and Adam is applied as the optimization algorithm to update the learning parameters of each model.

[0599] The following code snippet may include a series of command statements that perform training and evaluation of the anomaly detection model M4a and the evolution prediction model M4b in the M4 model and visualize the results.

[0600] train_anomaly_model(train_grouped, epochs)

[0601] train_prediction_model(train_grouped, epochs)

[0602] evaluate_anomaly_model(test_grouped)

[0603] visualize_analyze_and_suggest_results(test_grouped)

[0604] In the above text, the functions train_anomaly_model and train_prediction_model train the anomaly detection model and the evolution prediction model, respectively. The function evaluate_anomaly_model evaluates the performance of the anomaly detection model, and the function visualize_analyze_and_suggest_results visualizes and analyzes the test results and provides suggestions for improvement.

[0605] Figure 33The function shown is for training the anomaly detection model M4a, which is an element of the M4 model according to this disclosure.<train_anomaly_model> Class definition.

[0606] Reference Figure 33 After learning the patterns of normal data, the M4a model exhibits high loss on abnormal data to detect abnormal behavior, and is trained through the following process.

[0607] - Data preprocessing: Grouping the data and converting it into tensor form

[0608] -Model training: Input data into the model to generate predicted values, calculate the loss between the predicted and actual values, and update the model weights through backpropagation.

[0609] - Output status: Output the loss value every 5 rounds to supervise the learning process.

[0610] Figure 34 An example of the definition of a function for training an evolutionary prediction model M4b in the M4 model according to this disclosure is shown.

[0611] Reference Figure 34 The learning process of model M4a includes the following steps.

[0612] - Learn by splitting the data into groups.

[0613] - Transform each set of data into tensors and then pass them to the model to generate predicted values.

[0614] - Calculate the loss between the predicted value and the actual value.

[0615] - Calculate the gradient with respect to the loss through backpropagation and update the model weights based on this.

[0616] - Output the loss value every 5 rounds to supervise the learning process.

[0617] Figure 35 An example is shown of the definition of the function evaluate_anomaly_model for evaluating the anomaly detection model M4a according to the M4 model of this disclosure.

[0618] This parameter includes the following evaluation process for M4a.

[0619] - Data preparation: Grouping, processing, and moving the data to the evaluation device.

[0620] - Perform model reconstruction: Reconstruct the input data using the model to generate output values.

[0621] - Calculate reconstruction loss: Calculate and store the loss between the input data and the output value.

[0622] - Output average loss: Calculate and output the average reconstruction loss for all grouped data.

[0623] Figure 36 This diagram shows a pseudocode segment of the function `visualize_analyze_and_suggest_results`, which visualizes and analyzes test results and suggests improvement recommendations in the M4 model according to this disclosure. (Refer to such...) Figure 36 , one The pseudocode can be concretized as follows.

[0624] - INITIALIZE analysis_results = [], overall_score = 0, num_cycles =len(test_grouped):

[0625] analysis_results = []

[0626] overall_score = 0

[0627] num_cycles = len(test_grouped)

[0628] The code above initializes the list that will store the analysis results (analysis_results), the total score (overall_score), and the number of test data sets (num_cycles).

[0629] - EXTRACT relevant features FROM group INTO group_tensor:

[0630] group = group[['Left GRF', 'Right GRF', 'BWS', ...]].values ​​group_tensor = torch.tensor(group, dtype=torch.float32).unsqueeze(0).to(device)

[0631] The code above filters the grouped data and extracts only the features needed from the group, transforming them into a PyTorch tensor (group_tensor).

[0632] - PREDICT predictions USING prediction_model(group_tensor):

[0633] with torch.no_grad():

[0634] predictions = prediction_model(group_tensor).squeeze(0).cpu().numpy()

[0635] The code above uses an evolution prediction model to predict the future values ​​of the current group.

[0636] - COMPUTE mean_diff, std_diff, mse, mae FROM differences:

[0637] mean_diff = np.mean(differences, axis=0)

[0638] std_diff = np.std(differences, axis=0)

[0639] mse = np.mean(np.square(differences), axis=0)

[0640] mae = np.mean(np.abs(differences), axis=0)

[0641] The code above calculates the mean difference (mean_diff), standard deviation (std_diff), mean squared error (mse), and mean absolute error (mae) based on differences.

[0642] - COMPUTE cycle_score BASED ON mse AND ADD TO overall_score:

[0643] cycle_score = max(0, 100 - (np.mean(mse) * 10))

[0644] overall_score += cycle_score

[0645] The code above calculates the score of the current group (cycle_score) and appends it to the total score (overall_score).

[0646] - APPEND metrics TO analysis_results:

[0647] analysis_results.append({ "Gait Cycle": key, "Mean Difference": mean_diff.tolist(), ...})

[0648] The code above stores the calculated metrics (mean deviation, standard deviation, loss value, etc.) in the analysis results directory (analysis_results).

[0649] - PLOT "Actual vs Predicted" graph FOR current group:

[0650] plt.figure(figsize=(10, 6))

[0651] for i, feature in enumerate(['Left GRF', ...]):

[0652] plt.plot(group[:, i], label=f"Actual {feature}")

[0653] plt.plot(predictions[:, i], linestyle='--', label=f"Predicted{feature}")

[0654] plt.show()

[0655] The code above visualizes a chart obtained by comparing the actual and predicted values ​​in the current group.

[0656] - INITIALIZE suggestions = [] AND CHECK deviation FOR monitored_features:

[0657] suggestions = []

[0658] for i, feature in enumerate(['Step Length', ...]):

[0659] if np.abs(mean_diff[i]) > 0.1 * np.mean(group[:, i]):

[0660] suggestions.append(f"Consider adjusting {feature}. Currentdeviation: {mean_diff[i]:.2f}")

[0661] The code above generates suggestions when the deviation of characteristic features (stride length, stride height, etc.) exceeds a threshold.

[0662] - PRINT overall_score AND analysis_results:

[0663] print(f"Overall Performance Score (0-100): {overall_score:.2f}")

[0664] print("\nDetailed Analysis Results:")

[0665] The code above outputs the total functional score and detailed analysis results.

[0666] As described above, in the fourth model M4 according to this disclosure, the anomaly detection model M4a evaluates the degree to which the model learns normal data patterns through reconstruction loss (Rconstruction Loss). A low reconstruction loss indicates that the anomaly detection model stably learns normal data patterns, and the decrease in loss with each epoch indicates that the anomaly detection model gradually achieves better performance. The evolution prediction model M4b represents the accuracy of predicting the future based on past data through prediction loss. The decrease in loss at each epoch indicates that the evolution prediction model stably learns the correlations between data.

[0667] Average reconstruction loss, cycle score, and graphical visual comparisons were used to evaluate these models M4a and M4b.

[0668] The average reconstruction loss is calculated on the test data. The lower the average reconstruction loss, the higher the performance of the model in reconstructing normal data.

[0669] The cycle score is an indicator of the performance of the Gait Cycle on a scale of 0 to 100. A low cycle score may indicate poor predictive performance in terms of how accurately the Gait Cycle is predicted or that the data includes outliers. Problem intervals for specific cycles can be identified through the cycle score.

[0670] Visual comparison involves overlaying the actual and predicted values ​​for each walking cycle on a single display, represented using lines that can distinguish them from each other (e.g., dashed and solid lines). In this case, the better the overlap between the lines of actual and predicted values, the better the model performance. If a large deviation occurs in a particular characteristic, further model adjustments may be needed for that characteristic.

[0671] The results analysis comprehensively evaluates model performance, including the mean difference between the actual and predicted values ​​of each characteristic (the lower the value, the better the prediction accuracy), the MSE and MAE that evaluate prediction error (lower values ​​indicate better performance), and the standard deviation that represents the variability of prediction error (lower values ​​indicate stable performance).

[0672] V. Fifth Model M5

[0673] The fifth model is an AI model that can only train results reports. The input data are the training results before and after, the report generation type, and the output is text, images, videos, etc.

[0674] The input report generation type can be either simple or based on multimodal support for a generative pre-trained transformer (GPT) model with prompts. The output can be, for example, text in PDF format, images of each patient's training results and status (various backgrounds and progression states), or videos of each patient's walking trajectory. Another output is video-based results based on patient walking training analysis, such as virtualized limb movement, self-image movement, or output of a pheasant-shaped video.

[0675] The "flounder diagram" in a gait rehabilitation training system is a visual tool primarily used for gait analysis and assessment. This diagram visualizes the patterns and pressure of the foot contacting the ground to aid in the analysis of gait cycles, such as footprint analysis, gait cycle assessment, cost-effectiveness measurement, and left-right gait asymmetry. This type of diagram is often named "flounder diagram" because the footprint shape and pressure distribution are typically represented as resembling the shape of butterfly wings. Figure 23 An exemplary diagram of a flounder is shown.

[0676] For multi-mode support of GPT, it is possible to induce the creation of prompt-based reporting data for the workpiece.

[0677] Furthermore, in addition to the outputs mentioned above, training and reporting outputs via voice output are also permitted. This allows for intelligent reporting of patient status and training results after training, enabling a more intuitive and effective delivery of training outcomes to patients, and potentially increasing training achievement and satisfaction.

[0678] The exemplary configuration of the fifth model according to this disclosure is described below.

[0679] The input data for the AI ​​agent applying the fifth model can include pre- and post-training results, report generation type, and prompt words.

[0680] The training results data includes past and present training results, integrating training parameters and real-time data. Key data items, as mentioned above, may include Step Length, Step Height, GRF, BWS, ankle angle, and walking cycle.

[0681] The output of the fifth model, i.e. the report generation type, can include a basic report that provides a result summary based on Dancun text or images, a multi-modal report that includes text, images, and videos generated through GPT-based prompt word input, and can diversify data analysis and visualization analysis according to the user's selected type.

[0682] The input of prompt words for multimodal reporting refers to the prompt words used to generate GPT-based reports. For example, it could be provided as "Please visualize the patient's walking trajectory and output it as a video".

[0683] The fifth model M5 may include a report automation model, a video-based analytics model, and a walking assessment metric prediction model.

[0684] -Report automation model:

[0685] The model uses a multi-modal support GPT-based approach to analyze input data and automatically generates training results in text, image, and video formats. This model can understand the training data and create appropriate reports based on given prompts.

[0686] - Video-based analysis model

[0687] The model analyzes and produces videos of patients' walking trajectories based on time-series data. For example, it can visualize the movement of a virtual god, the actual shape of the patient's movement, or walking patterns such as "flounder diagrams".

[0688] - Walking assessment index prediction model

[0689] This model uses LSTM and Transformer-based deep learning models to predict walking evaluation metrics, and automatically establishes a training plan and derives improvement points based on the prediction results.

[0690] The output of the fifth model M5 may include training results in report form; real-time audio output; training results visualized based on video; and intelligent reports.

[0691] The training results in the report format may include a summary of training results, a text report in PDF format including the analysis results of key indicators, an image report including visual data reflecting various backgrounds and progression statuses of patients, a walking trajectory simulation, a video report such as walking movements that indicate the shape of the person or a virtual avatar, a flounder diagram, etc.

[0692] As a real-time audio output, it provides real-time audio guidance on the training status and results, such as prompting current training status information such as "The average speed of the current walking cycle is 90% of the target".

[0693] Training results based on video visualization can include results from videos created using virtual avatars or the user's own shape, and "flounder" visual cues that indicate the asymmetry of the gait cycle and the effectiveness of rehabilitation.

[0694] Intelligent reporting can suggest personalized training outcome summaries and improvement directions based on patient status and training results, and provide intuitive and mobile reporting data to improve patient training achievement and satisfaction.

[0695] The gait training system and method disclosed herein, which utilizes AI agents—models M1, M2, M3, M4, and M5—that employ the aforementioned steps, enables the establishment of patient-friendly gait training plans and facilitates effective, personalized, and patient-friendly rehabilitation training. In rehabilitation training based on this system and method, the artificial intelligence model for each step can predict accurate gait assessment indicators, automatically establish training plans accordingly, and more accurately evaluate training results.

[0696] By using this system and method for rehabilitation training, artificial intelligence models for each step can predict the correct walking assessment indicators, automatically establish training plans based on these indicators, and more accurately evaluate training results. Thus, compared with existing methods and systems, it is possible to reduce the intervention of therapists based on medical staff prescriptions.

[0697] As described in the preferred embodiments above, the preferred embodiments of the present invention have been described in detail. However, those skilled in the art can modify the present invention in various ways without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, changes to subsequent embodiments of the present invention will not depart from the technology of the present invention.

Claims

1. An artificial intelligence-based robot-assisted orthopedic movement system comprising a gait training apparatus and a system control unit, wherein the gait training apparatus has a pedal for a patient to step on and a pedal driver including one or more links configured to drive the pedal, and the system control unit is configured to control the pedal driver to perform gait training for the patient by stepping on the pedal according to a training plan established for the patient, wherein... The system control device has The indicator prediction unit is configured to determine the patient's walking assessment indicators based on the patient's initial walking status information or walking training results information after initial walking training. The training planning department is configured to establish a patient-specific training plan to be executed by the gait movement department based on the results of the gait training. The training unit is configured to perform training on the patient according to the training plan, monitor training data from the patient regarding the real-time training status in real time, and generate feedback on the training data. as well as The training assessment department evaluates the training results based on training result data from the rehabilitation training performed on the patient by the gait department.

2. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 1, wherein, The training data includes at least one of the following characteristics: ground reaction force (GRF), bodyweight support (BWS), upper body inclination or tilting, joint angle, steps per minute (Cadence), left gait cycle, right gait cycle, stride length, ankle angle, oxygen saturation, and heart rate.

3. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 1, wherein, The training model is configured to receive at least one training parameter among stride length (cm), step height (cm), initial contact angle (° (degree)), and toe off angle (° (degree)) and output correction values ​​for the at least one training parameter based on the training data.

4. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 1, wherein, The training model is any deep learning model among Multi-Layer Perceptron (MLP) and Open Vision-Language Model (VLM).

5. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 1, wherein, The video or audio information of the patient or the therapist assisting the patient in training is obtained and used as input data for the training model, and the training model is configured to generate output data corresponding to the input data.

6. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 1, wherein, The training planning department further includes a reference model, which is based on patient information, training procedures, training time, training parameters, etc., set by medical personnel. The training plan is configured to output a final training plan, which is based on the output correction or compensation of the autonomous training plan model and the benchmark training plan model.

7. The AI-based robot-assisted orthopedic motion system according to claim 5, wherein, The training planning unit further includes a final training plan model configured to output a final training plan based on the outputs of the autonomous training plan model and the baseline training plan model.

8. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 1, wherein, The monitoring unit is configured to monitor the status of rehabilitation training performed by the gait unit in accordance with the training plan, while obtaining information during the training, and thereby generating training status information about the patient's gait training.

9. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 8, wherein, The monitoring unit is configured to provide the patient with training guidelines, recommended guidelines, or current training status related to the walking training, based on information from the training.

10. The robot-assisted orthopedic motion system based on artificial intelligence according to claim 1, wherein, The training evaluation unit includes an anomaly detection model based on time-series information. This model analyzes the current training results or analyzes the previous and current training results to predict future training based on the evolution of the results of repeated training.

11. A method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence, comprising: The step of predicting the patient's walking assessment indicators by the indicator prediction unit based on the patient's initial walking status information or walking training results information after walking training; The steps for the training planning department to establish a training plan for the individual patient based on the walking assessment indicators using a self-discipline training plan model; During the rehabilitation training of the patient performed by the walking motion unit having an end effector type pedal and one or more pedal drive units configured to drive the pedal according to the training plan, the training execution unit monitors training data about the real-time training status from the patient in real time, and the training execution model set in the training execution unit generates feedback about the training data. The step of evaluating the training results by the training evaluation department based on the training result data according to the training results.

12. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 11, wherein, The training data includes at least one of the following: GRF, BWS, upper body tilt, joint angle, steps per minute, left foot walking cycle, right foot walking cycle, stride length, ankle angle, oxygen saturation data, and heart rate data.

13. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 11, wherein, The training model is configured to receive at least one training parameter among stride length (cm), step height (cm), initial contact angle (° (degree)), and toe off angle (° (degree)) and output correction values ​​for the at least one training parameter based on the training data.

14. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 11, wherein, The training model is any deep learning model among Multi-Layer Perceptron (MLP) and Open Vision-Language Model (VLM).

15. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 11, wherein, Video or audio information about a patient undergoing training or a therapist assisting in the training can be obtained via a video or audio recording device. The analysis device analyzes the video information or the audio information, and... The training model uses the analysis as input data to generate output data corresponding to the input data.

16. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 11, wherein, The training program further includes The self-discipline training plan model establishes a personalized training plan for patients based on initial walking status information or walking training results after walking training. as well as The reference model is trained based on patient information, training procedures, training time, and training parameters set by medical personnel. The training plan is configured to output a final training plan, which is based on the output correction or compensation of the autonomous training plan model and the benchmark training plan model.

17. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 12, wherein, The testing department will collect data on the patient's GRF, BWS, upper body tilt, joint angle, steps per minute, left foot walking cycle, right foot walking cycle, stride length, ankle angle, oxygen saturation, and heart rate during walking, and use at least one of these as training information.

18. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 11, wherein, While the supervisory department monitors the status of rehabilitation training performed by the gait department according to the training plan, it obtains information on the training process and thereby generates training status information about the patient's gait training.

19. The method for providing motion for robot-assisted orthopedic surgery based on artificial intelligence according to claim 11, wherein, The reporting department analyzes previous and current training results to generate reports in text, image, and video formats.

20. The method for providing motion for orthopedic surgery based on artificial intelligence robot assistance according to claim 11, wherein, The head-mounted display (HMD) provides the patient with information related to their walking movement, training status, or recommended guidelines for training.

Images