Digital twin-based educational mannequin simulation system and simulation method

Abstract

A digital twin-based educational mannequin simulation system according to an embodiment includes: an operator terminal that sets the degree of stiffness and rigidity for each joint of a patient; a simulation server that receives the degree of stiffness and rigidity set from the operator terminal and generates a digital twin-based educational simulation that matches the received degree of stiffness and rigidity; a mannequin simulator that provides tactile information implementing the degree of stiffness and rigidity of the patient according to the educational simulation to a trainee and collects hands-on data, which is joint-specific sensing information by the trainee; and an XR device that is worn by a trainee and projects a patient human body model according to the digital twin-based educational simulation onto the mannequin simulator to provide visual information regarding the degree of stiffness and rigidity of the patient to the trainee.

Description

Technology Field

[0001] The present disclosure relates to a digital twin-based educational mannequin simulation system and a simulation method, and specifically, to a digital twin-based educational simulation system and a simulation method comprising an XR (Extended Reality) device and an educational mannequin simulator.

[0002] This project (result) is the result of the Local Government-University Cooperation-based Regional Innovation Project, conducted in 2023 with funding from the Ministry of Education and support from the National Research Foundation of Korea. (Foundation Project Management Number: 2022RIS-006) Background Technology

[0004] Unless otherwise indicated in this specification, the contents described in this section are not prior art for the claims of this application, and are not to be recognized as prior art simply because they are included in this section.

[0005] XR (Extended Reality) is a term encompassing various real-world and digital technologies such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). It is a technology that combines the real and virtual worlds to provide a mixed reality experience. This XR technology primarily integrates diverse technologies, including computer graphics, sensors, displays, and artificial intelligence, to provide users with a sense of realism and interaction.

[0006] Virtual Reality (VR) is a technology that immerses users in a completely virtual environment. VR utilizes devices such as headsets and controllers to provide users with a fully virtual world, allowing them to interact within that world. Augmented Reality (AR) is a technology that enhances a user's reality by adding digital information or objects to the real world. Using smartphone cameras or AR headsets, virtual layers can be added to the surrounding environment or information can be displayed. Mixed Reality (MR) is a technology that combines the real world and the virtual world to provide a blended environment. Users can perceive and interact with both physical objects existing in the real world and virtual objects simultaneously. These XR technologies are being utilized in various fields, including entertainment, education, healthcare, industry, and telecommunications. For example, VR is used in gaming or simulation environments, while AR is utilized across diverse platforms ranging from smartphone applications to wearable devices. MR is used to manipulate virtual 3D objects within the real world or to integrate the real and virtual worlds. These technologies are creating new experiences and business models through innovative use cases.

[0007] Meanwhile, hands-on treatment by a therapist is essential during exercise, physical therapy, or rehabilitation. The hands-on method is a treatment process in which a therapist touches and corrects the patient's joints or muscles. However, in the case of novice therapists rather than experienced ones, the effectiveness of the treatment may be reduced due to touching other muscles or lacking proficiency in adjusting the intensity. While beginners or trainees improve their skills by gaining direct hands-on experience with patients, there is also a risk of patient injury when trainees or beginners perform hands-on treatment. Prior art literature

[0009] 1. Korean Patent Publication No. 10-2023-0101014 (July 6, 2023) 2. Korean Patent Registration No. 10-2011236 (August 8, 2019) The problem to be solved

[0010] The digital twin-based educational mannequin simulation system and simulation method according to the embodiment enable trainees to conduct hands-on training, such as manual diagnosis and treatment methods, under the observation of an educator using the digital twin-based educational mannequin simulator system.

[0011] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment project a patient body model similar to a real patient, implemented in a digital twin environment on an XR device worn by a trainee, onto a mannequin simulator. At the same time, the mannequin simulator provides the trainee with the patient's tremors, joint stiffness, or rigidity.

[0012] In addition, in the embodiment, the operator terminal adjusts the degree of shaking, rigidity, and stiffness to be similar to a real patient, and enables reproduction through a patient body model on a digital twin and a mannequin simulator in a real environment.

[0013] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment allow a trainee to receive visual information through an XR device regarding the degree of stiffness and rigidity of an actual patient joint implemented in a digital twin environment. Furthermore, the stiffness and rigidity of the patient joint can be received as tactile information through a mannequin simulator.

[0014] In addition, the digital twin-based educational mannequin simulation system and simulation method analyze the trainee's hands-on data after training to evaluate the trainee's current ability and training effectiveness, and provide the results to the trainee's terminal.

[0015] However, the problem to be solved according to one embodiment is not limited only to that mentioned above. means of solving the problem

[0017] A digital twin-based educational mannequin simulation system according to an embodiment includes: an operator terminal that sets the degree of stiffness and rigidity for each joint of a patient; a simulation server that receives the degree of stiffness and rigidity set from the operator terminal and generates a digital twin-based educational simulation that matches the received degree of stiffness and rigidity; a mannequin simulator that provides tactile information implementing the degree of stiffness and rigidity of the patient according to the educational simulation to a trainee and collects hands-on data, which is joint-specific sensing information by the trainee; and an XR device that is worn by a trainee and projects a patient human body model according to the digital twin-based educational simulation onto the mannequin simulator to provide visual information regarding the degree of stiffness and rigidity of the patient to the trainee. Effects of the invention

[0019] The digital twin-based educational mannequin simulation system and simulation method described above are designed to train rehabilitation medicine and physical therapists. They are utilized before performing treatment practice on actual patients with hemiplegia, and help prevent secondary accidents caused by inexperienced treatment.

[0020] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment can be used as educational aids for physical therapy and rehabilitation medicine students before practical training.

[0021] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment provide the same effect as performing hands-on training on actual patients under the observation of a skilled educator using an XR device.

[0022] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment enable visual and tactile training of the degree of stiffness or rigidity of a standardized joint, and the trainee can provide generalized treatment methods to patients.

[0023] In addition, the mannequin simulator provided in the embodiment can reproduce complex movements of stiffness or rigidity in the lower limbs as well as the upper limbs, allowing educators to experience various stiff or rigid joint characteristics of patients through the mannequin simulator.

[0024] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment enable trainees to effectively acquire the knowledge and skills necessary for the treatment and rehabilitation of patients.

[0025] The effects of the present invention are not limited to the effects described above, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description of the invention or the claims. Brief explanation of the drawing

[0027] FIG. 1 is a drawing showing a digital twin-based educational mannequin simulation system according to an embodiment. Figure 2 is a drawing showing a virtual environment output through an XR device. FIG. 3 is a drawing showing the passive joints and active joints of a mannequin simulator according to an embodiment. FIGS. 4 to 7 are drawings showing the passive and active joints of a mannequin simulator according to an embodiment in more detail. FIG. 8 is a diagram showing an algorithm for implementing complex stiffness and rigidity of an active joint by a mannequin simulator according to an embodiment. FIG. 9 is a block diagram of a simulation server according to an embodiment. FIG. 10 is a diagram illustrating a real-time communication algorithm that implements rigidity and stiffness including biological tremors by integrating real-time communication between a patient body model in a digital twin environment and a mannequin simulator installed in a real environment via an XR device in an embodiment. FIGS. 11 and 12 are drawings showing an output interface for an evaluation result according to an embodiment. FIG. 13 is a signal flow diagram of a digital twin-based educational mannequin simulation system according to an embodiment. Specific details for implementing the invention

[0028] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Identical or similar components regardless of drawing symbols are assigned the same reference number, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably solely for the ease of drafting the specification and do not inherently possess distinct meanings or roles. Furthermore, in describing embodiments disclosed in this specification, if it is determined that a detailed description of related prior art could obscure the essence of the embodiments disclosed in this specification, such detailed description will be omitted. Additionally, the attached drawings are intended only to facilitate understanding of the embodiments disclosed in this specification; the technical concept disclosed in this specification is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutions that fall within the spirit and technical scope of the present invention.

[0029] Terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. These terms are used solely for the purpose of distinguishing one component from another.

[0030] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0031] In this application, terms such as “comprising” or “having” are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0032] In this specification, the term "part" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware.

[0033] Some of the operations or functions described herein as being performed by a terminal, device, or device may instead be performed by a server connected to said terminal, device, or device. Likewise, some of the operations or functions described as being performed by a server may also be performed by a terminal, device, or device connected to said server.

[0034] Hereinafter, the present invention will be described in detail with reference to the attached drawings.

[0035] FIG. 1 is a diagram showing a digital twin-based educational mannequin simulation system according to an embodiment. Referring to FIG. 1, the digital twin-based educational mannequin simulation system according to an embodiment may be configured to include an operator terminal (200), a simulation server (100), a mannequin simulator (300), and an XR device (400).

[0036] The operator terminal (200) sets the degree of stiffness and rigidity for each joint of the patient. The simulation server (100) receives the degree of stiffness and rigidity set by the operator terminal (200) and generates a digital twin-based training simulation that matches the received degree of stiffness and rigidity. The mannequin simulator (300) provides tactile information implementing the degree of stiffness and rigidity of the patient according to the training simulation to the trainee and collects hands-on data, which is joint-specific sensing information by the trainee. The XR device (400) is worn by the trainee to project a patient body model according to the digital twin-based training simulation onto the mannequin simulator and provides visual information regarding the degree of stiffness and rigidity of the patient to the trainee.

[0037] In the embodiment, the operator terminal (200) can specify the degree of joint stiffness and rigidity of the patient by setting the MMT (Medical Research Council Muscle Strength Grading) grade and the MAS (Modified Ashworth Scale) grade through a user interface projected into a digital twin environment. In the embodiment, the MMT (Medical Research Council Muscle Strength Grading) grade and the MAS (Modified Ashworth Scale) grade, which indicate the degree of stiffness and rigidity by joint, are indicators representing muscle strength and muscle tone; MMT includes grades 1 to 5, and MAS includes grades 1 to 4. Additionally, in the embodiment, the patient to be simulated may be, for example, a hemiplegic patient.

[0038] In addition, in the embodiment, the operator terminal (200) can automatically set the degree of stiffness and rigidity of each joint of the patient according to a pre-prepared education program for training skilled rehabilitation medicine specialists and physical therapists. In addition, the operator terminal (200) can set the degree of stiffness and rigidity of each joint of the patient to be simulated by receiving input of the degree of stiffness and rigidity of each joint of the patient to be simulated through an interface from a pre-authenticated administrator or educator.

[0039] In addition, the operator terminal (200) can receive evaluation information from the simulation server (100) and monitor the training results.

[0040] In the embodiment, the simulation server (100) receives the degree of joint stiffness and rigidity set by the operator terminal (200) and controls the mannequin simulator (300) based on an educational simulation matched to the degree of stiffness and rigidity. In the embodiment, the simulation server (100) may store educational scenarios in advance and generate an educational simulation according to the educational scenario that matches the degree of stiffness and rigidity. In the embodiment, the educational simulation may include control signals for simulating the patient's physiological tremor through the mannequin simulator (300). For example, the educational simulation may include joint control signals of the mannequin simulator according to the degree of stiffness and rigidity. Additionally, the educational simulation may include control signals for indicating the biological tremor occurrence cycle and tremor intensity according to the patient's degree of stiffness and rigidity. Furthermore, in the embodiment, the simulation server (100) may generate an educational simulation according to the patient's degree of stiffness and rigidity and patient-specific disease information. In the embodiment, the educational simulation may be a digital twin-based patient body model and may include visual information regarding the patient's degree of stiffness and rigidity. In the embodiment, the simulation server (100) can receive hands-on data from the mannequin simulator (300) and generate real-time digital twin-based visual information. In the embodiment, the trainee's hands-on data is data including the trainee's hands-on position, pressure and rotation caused by the trainee's hands-on, etc., received from the mannequin simulator (300). In the embodiment, the simulation server (100) synthesizes the trainee's hands-on data collected after hands-on training to generate evaluation information, which is the trainee's training result.

[0041] Additionally, in the embodiment, the simulation server (100) can generate a medical simulation. In the embodiment, the medical simulation may include digital twin-based visual graphics and sound information to simulate an actual medical situation. In the embodiment, the medical simulation may be a digital twin-based simulation in which realistic graphics and sound are added to patient objects and hospital room objects according to a medical scenario. In the embodiment, the simulation server (100) can adjust the patient object so that the movement and shaking of the patient object are expressed differently according to the medical simulation. Through this, the interaction that may occur during the medical treatment of a trainee and a virtual patient can be provided realistically.

[0042] The mannequin simulator (300) provides tactile information to the trainee that implements the patient's biological tremors, joint stiffness, and degree of rigidity. In the embodiment, the mannequin simulator (300) may be configured to include an active joint that includes an actuator that generates movement by a control signal, and a passive joint that moves by an external force. In the embodiment, the passive joint may include a spring and a damping device to represent the patient's biological tremors. In addition, in the embodiment, the active joint may represent the patient's biological tremors, stiffness, and degree of rigidity.

[0043] The XR device (400) is worn by a trainee and projects a patient body model based on a digital twin-based training simulation onto a mannequin simulator (300) to provide the trainee with visual information regarding the degree of rigidity and stiffness of the patient.

[0044] Additionally, the XR device (400) projects a patient object included in the medical simulation onto a mannequin simulator (300), allowing the trainee to experience the characteristics of the patient implemented through the mannequin simulator (300) in virtual reality.

[0045] FIG. 2 is a diagram illustrating an example of a virtual environment output through an XR device. Referring to FIG. 2, the XR device (400) can project a patient body model onto a mannequin simulator (300). To this end, a simulation server (100) creates a patient body model based on previously collected patient images and patient disease information, and transmits the patient body model to the XR device (400). Additionally, the simulation server (100) can extract environmental objects such as a hospital room and a bed from actual patient medical images, reflect the extracted environmental objects in the educational situation to create a digital twin-based educational simulation that appears to be a virtual reality where actual medical treatment is being conducted, and transmit this to the XR device (400). The XR device (400) outputs the received educational simulation so that the trainee can experience the medical situation or hands-on practice more realistically during the training process. Furthermore, in the embodiment, as shown in FIG. 2, the XR device (400) can graphically output vital signals including the patient's body temperature, pulse, and electrocardiogram together in the virtual reality.

[0046] The digital twin-based educational mannequin simulation system and simulation method according to the embodiment enable a trainee to conduct hands-on training, such as manual diagnosis and treatment methods, under the observation of an operator using the digital twin-based educational mannequin simulator system.

[0047] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment project a patient body model similar to a real patient, implemented in a digital twin environment on an XR device worn by a trainee, onto a mannequin simulator. At the same time, the mannequin simulator provides the trainee with the patient's tremors, joint stiffness, or rigidity.

[0048] In addition, in the embodiment, the operator terminal adjusts the degree of shaking, rigidity, and stiffness to be similar to a real patient, and enables reproduction through a patient body model on a digital twin and a mannequin simulator in a real environment.

[0049] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment allow a trainee to receive visual information through an XR device regarding the degree of stiffness and rigidity of an actual patient joint implemented in a digital twin environment. Furthermore, the stiffness and rigidity of the patient joint can be received as tactile information through a mannequin simulator.

[0050] In addition, the digital twin-based educational mannequin simulation system and simulation method analyze the trainee's hands-on data after training to evaluate the trainee's current ability and training effectiveness, and provide the results to the trainee's terminal.

[0051] FIG. 3 is a diagram showing the passive joints and active joints of a mannequin simulator according to an embodiment.

[0052] Referring to FIG. 3, in the embodiment, the mannequin simulator may be configured to include at least one passive joint (310, 320), at least one active joint (330, 340), a sensor unit (not shown in the drawing), and a control unit (not shown in the drawing).

[0053] The manual joint (310, 320) is a joint composed of a driving device without including an actuator. In the embodiment, the driving device may include a spring and a damper device. In the embodiment, the manual joint (310, 320) is a joint intended to represent a general grade with no stiffness or rigidity.

[0054] Active joints (330, 340) are joints for implementing various grades of stiff and rigid joints.

[0055] The passive joints (310, 320) and active joints (330, 340) can reproduce the stiffness, rigidity, and biological tremors of the actual patient's joints using mechanical and control methods, respectively.

[0056] In the embodiment, the active joints (330, 340) include 12 degrees of freedom, and the passive joints (310, 320) include 12 degrees of freedom. Accordingly, the mannequin simulator can be implemented as educational mannequin simulator hardware having a total of 24 degrees of freedom.

[0057] FIGS. 4 to 7 are drawings showing the passive joints and active joints of a mannequin simulator according to an embodiment in more detail.

[0058] In the embodiment, the mannequin simulator is a total of 24 degrees of freedom system, consisting of 12 degrees of freedom for active joints and 12 degrees of freedom for passive joints. The active and passive joints provide 3 degrees of freedom for the shoulder, 1 degree of freedom for the elbow, 2 degrees of freedom for the wrist, 3 degrees of freedom for the hip joint, 1 degree of freedom for the knee joint, and 2 degrees of freedom for the ankle joint, respectively. In the embodiment, the mannequin simulator performs compound joint movements similar to those of a real patient, and the mannequin simulator implements various grades of stiffness and rigidity for each joint. In particular, it is possible to reproduce the degree of stiffness or rigidity of each joint differently.

[0059] Figures 4 and 5 are drawings showing the passive joint in more detail.

[0060] Referring to FIG. 4, the passive joint (310) of the shoulder and arm portion includes a string damper hybrid system (311). In an embodiment, the string damper hybrid system (311) may be implemented with 6 degrees of freedom, including a shoulder adduction and abduction implementation unit (1), a shoulder flexion and abduction implementation unit (2), an upper arm internal rotation and external rotation implementation unit (3), an elbow flexion and extension implementation unit (4), a wrist adduction and abduction implementation unit (5), and a wrist non-lateral flexion and plantar flexion implementation unit (6). The term “part” as used herein should be interpreted as including software, hardware, or a combination thereof, depending on the context in which the term is used.

[0061] Referring to FIG. 5, the passive joint (320) of the leg portion also includes a string damper hybrid system (321). In an embodiment, the string damper hybrid system (321) may be implemented with 6 degrees of freedom, including a hip joint flexion and abduction implementation unit (7), a hip joint internal rotation and external rotation implementation unit (8), a hip joint adduction and abduction implementation unit (9), a knee joint flexion and extension implementation unit (10), an ankle joint adduction and abduction implementation unit (11), and an ankle joint flexion and extension implementation unit (12).

[0062] FIGS. 6 and FIGS. 7 are drawings showing an active joint according to an embodiment.

[0063] Referring to FIG. 6, the active joint (330) of the shoulder and arm portion may be configured to include an actuator (331). In the embodiment, the actuator (331) may be implemented as, for example, a spring hybrid motor. In the embodiment, the actuator (331) of the shoulder and arm portion may be implemented with 6 degrees of freedom, including a motor (13) for flexion and extension of the shoulder, a motor (14) for adduction and abduction of the shoulder, a motor (15) for internal rotation and external rotation of the upper arm, a motor (16) for flexion and extension of the elbow, a motor (17) for adduction and abduction of the wrist, and a motor (18) for non-lateral flexion and lateral flexion of the wrist.

[0064] Referring to FIG. 7, the active joint (340) of the leg portion also includes an actuator (341) implemented with a spring hybrid motor. In the embodiment, the actuator (341) of the leg portion may be implemented with 6 degrees of freedom, including a motor (19) for hip joint flexion and extension, a motor (20) for hip joint internal and external rotation, a motor (21) for hip joint adduction and abduction, a motor (22) for knee flexion and extension, and a motor (23) for ankle joint adduction and abduction.

[0065] FIG. 8 is a diagram illustrating an algorithm for implementing complex stiffness and rigidity of an active joint using a mannequin simulator according to an embodiment. As shown in FIG. 8, in the embodiment, the patient's biological tremors can be implemented using a software method by utilizing the actuator of the active joint. In the embodiment, complex stiffness and rigidity of the active joint can be implemented by controlling the actuator according to an educational simulation.

[0066] FIG. 9 is a block diagram of a simulation server according to an embodiment.

[0067] In the embodiment, the simulation server (100) is a computing system that provides services to other computers or devices in a computer network or stores and manages data. The server (200) accepts requests from other computers or devices called clients and provides responses or data to those requests. The configuration of the simulation server (100) shown in FIG. 8 is merely a simplified example.

[0068] The communication unit (110) can be configured regardless of the mode of communication, such as wired or wireless, and can be configured with various communication networks, such as a Personal Area Network (PAN) or a Wide Area Network (WAN). Additionally, the communication unit (110) can operate based on the known World Wide Web (WWW) and may utilize wireless transmission technologies used for short-range communication, such as Infrared Data Association (IrDA) or Bluetooth. For example, the communication unit (110) may be responsible for transmitting and receiving data necessary to perform a technique according to one embodiment of the present disclosure.

[0069] In the embodiment, the communication unit (110) receives the set rigidity and rigidity degree from the operator terminal and receives hands-on data from the mannequin simulator.

[0070] Memory (120) may refer to any type of storage medium. For example, memory (120) may include at least one type of storage medium among flash memory type, hard disk type, multimedia card micro type, card type memory (e.g., SD or XD memory, etc.), RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk. Such memory (120) may also constitute the database shown in FIG. 1.

[0071] Memory (120) can store at least one instruction that can be executed by the processor (130). Additionally, memory (120) can store any form of information generated or determined by the processor (130) and any form of information received by the server (200). For example, memory (120) stores RM data and RM protocols according to the user, as will be described later. Additionally, memory (120) stores various types of modules, instruction sets, or models.

[0072] The processor (130) can perform technical features according to embodiments of the present disclosure to be described below by executing at least one instruction stored in memory (120). In one embodiment, the processor (130) may be composed of at least one core and may include a processor for data analysis and / or processing, such as a central processing unit (CPU) of a computer device, a general purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0073] The processor (130) controls the mannequin simulator by generating a control signal based on an educational simulation that matches the degree of stiffness and rigidity set at the operator terminal. For example, the processor (130) generates a digital twin-based educational simulation that matches the received degree of stiffness and rigidity based on an educational simulation according to the level of MMT and the level of MAS set at the operator terminal. Additionally, in the embodiment, the processor (130) can generate a mannequin simulator control signal according to the educational simulation matched to the degree of stiffness and rigidity and the body part where stiffness and rigidity are manifested. For example, the processor (130) can adjust the frequency, period, and intensity of stiffness and rigidity based on the educational simulation matched to the degree of stiffness and rigidity, and adjust the joint position controlled according to the body part.

[0074] In the embodiment, the processor (130) can generate a training simulation for controlling a mannequin simulator representing a patient after learning the training data. In the embodiment, the training data includes, but is not limited to, actual medical images of the patient and the degree of shaking, rigidity, and stiffness of the patient collected by a patient bio-information collection sensor.

[0075] For example, the processor (130) collects biological characteristics of the patient, such as tremors, stiffness, and degree of rigidity, through training data and selects a training simulation generation model. In the embodiment, the training simulation generation model may include a machine learning model. Subsequently, the model is trained using the collected training data and the patient's biological characteristics. In the embodiment, the model learns and understands characteristics such as the patient's tremors, stiffness, and degree of rigidity. Subsequently, the processor (130) uses the learned model to generate a training simulation that can be used to control a mannequin simulator representing the patient. The training simulation causes the mannequin simulator to mimic the patient's movements, postures, reactions, etc., based on the characteristics learned by the model.

[0076] Additionally, in the embodiment, the processor (130) can adjust the control signal controlling the mannequin simulator according to the patient's disease information, including the type and severity of the disease. The processor (130) can adjust the intensity and period of the control signal according to the patient's disease type and condition. For example, the processor (130) sets the intensity and period of the control signal through the intensity and period values ​​matched to the patient's disease type. Subsequently, the intensity and period values ​​matched to the patient's disease can be adjusted upward by a certain ratio according to the severity of the patient's disease. In the embodiment, if the severity of the disease falls within a first range, the processor (130) can increase the intensity and period matched to the patient's disease type by a first ratio (e.g., 20 percent), and if the severity of the disease falls within a second range, the intensity and period matched to the patient's disease type can increase by a second ratio (e.g., 10 percent).

[0077] Additionally, the processor (130) can communicate with a sensor attached to an actual patient to control the mannequin simulator to display in real time the tremor, rigidity, and stiffness information collected from the patient bio-information collection sensor. In an embodiment, when at least one of the tremor, rigidity, and stiffness information is collected from the patient bio-information collection sensor, the processor (130) calculates the intensity of the collected tremor, rigidity, and stiffness information and tracks the body location information where the tremor, rigidity, or stiffness occurred. Subsequently, based on the calculated intensity and the tracked body location, the signal intensity for tremor, rigidity, and stiffness control and the control position of the mannequin simulator are adjusted.

[0078] Additionally, the processor (130) adjusts the degree of rigidity and stiffness set by the operator terminal according to the shaking, rigidity, and stiffness information collected from the patient bio-information collection sensor. For example, the processor (130) calculates the intensity of shaking, rigidity, and stiffness according to the shaking, rigidity, and stiffness information collected from the patient bio-information collection sensor, and determines whether the calculated intensity falls within the intensity range of the MMT grade and MAS (Modified Ashworth Scale) grade set by the operator terminal. In the embodiment, if the calculated intensity falls outside the intensity range of the MMT grade and MAS (Modified Ashworth Scale) grade set by the operator terminal, the processor (130) may adjust the MMT grade or MAS grade to a grade that includes the calculated intensity.

[0079] Additionally, the processor (130) generates a patient object that reflects the actual appearance of each patient through the analysis of the patient's actual medical video and patient-specific image, and generates a patient-specific medical simulation by projecting the generated patient object onto a mannequin simulator. To this end, the processor (130) analyzes the actual medical video and patient image. In the embodiment, the patient's medical video and image are provided to a simulation server, and the processor (130) analyzes the provided video and image to identify the patient's actual appearance, posture, movements, etc. Subsequently, based on the results of the video and image analysis, the processor (130) generates a patient object for each patient. The patient object is a virtual representation capable of mimicking and representing the patient's actual characteristics. Subsequently, the processor (130) transmits the patient object to an EX terminal, and the EX terminal projects the received patient object onto a mannequin simulator to output a medical simulation.

[0080] FIG. 10 is a diagram illustrating a real-time communication algorithm that implements rigidity and stiffness including biological tremors by integrating a patient body model in a digital twin environment through an XR device and an educational mannequin simulator installed in a real environment through real-time communication in an embodiment. As shown in FIG. 10, in an embodiment, by projecting the patient body model onto the mannequin simulator and reflecting the virtual appearance of the patient onto the mannequin simulator, a simulation similar to a trainee interacting with a real patient can be output through the XR device.

[0081] Additionally, the processor (130) generates visual information to be displayed on the XR device by considering the hands-on data, and stores the trainee's hands-on data to use as the trainee's exercise data and as evaluation material for the trainee. To this end, the processor (130) collects the trainee's hands-on data from the mannequin simulator and evaluates the trainee's ability and training effectiveness based on the analysis results of the trainee's hands-on data. In the embodiment, the trainee's hands-on data includes the location of the muscle touched by the trainee, the pressing intensity, the direction of rotation, the rotation speed, etc., and can be collected from the sensor unit installed in the mannequin simulator. In the embodiment, the processor (130) compares the trainee's hands-on data with the reference hands-on data stored according to the degree of stiffness and rigidity set by the operator terminal and the type of disease of the patient, and evaluates the trainee's ability and training effectiveness based on the similarity between the two data.

[0082] FIGS. 11 and FIGS. 12 are drawings showing an output interface for evaluation results according to an embodiment.

[0083] Referring to FIG. 11, in the embodiment, the processor (130) generates evaluation results based on the sensing location of the trainee hands-on data and transmits the generated evaluation results to an operator terminal or a trainee terminal for output. In the embodiment, if the reference hands-on data and the trainee hands-on data are similar by a certain level or more, the processor (130) may output evaluation information (Good) indicating that the degree of stiffness has been alleviated. Additionally, as shown in FIG. 12, the evaluation level (A, B, C) of the trainee according to the range of similarity between the reference hands-on data and the trainee hands-on data may be output together. In the embodiment, if the similarity between the reference hands-on data and the trainee hands-on data exceeds a first threshold (e.g., 85 percent), the processor (130) evaluates the trainee as A, and if it exceeds a second threshold (75%), the trainee is evaluated as B. If the second threshold is not met, the trainee can be evaluated as C and this can be transmitted to the operator terminal or the trainee terminal.

[0084] Below, a digital twin-based educational mannequin simulation method will be described in turn. Since the operation (function) of the digital twin-based educational mannequin simulation method according to the embodiment is essentially the same as the function of the digital twin-based educational mannequin simulation system, descriptions that overlap with FIGS. 1 to 12 will be omitted.

[0085] FIG. 13 is a signal flow diagram of a digital twin-based educational mannequin simulation system according to an embodiment.

[0086] Referring to FIG. 13, in step S100, the degree of stiffness and rigidity for each joint of the patient is set at the operator terminal. In step S200, the simulation server receives the degree of stiffness and rigidity set from the operator terminal and generates a digital twin-based training simulation that matches the received degree of stiffness and rigidity. In step S300, tactile information implementing the degree of stiffness and rigidity of the patient according to the training simulation is provided to the trainee from the mannequin simulator, and in step S400, hands-on data, which is joint-specific sensing information by the trainee, is collected. In step S500, a patient body model according to the digital twin-based training simulation is projected onto the mannequin simulator by the XR device worn by the trainee, and visual information regarding the degree of stiffness and rigidity of the patient is provided to the trainee.

[0087] The digital twin-based educational mannequin simulation system and simulation method described above are designed to train rehabilitation medicine and physical therapists. They are utilized before performing treatment practice on actual patients with hemiplegia, and help prevent secondary accidents caused by inexperienced treatment.

[0088] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment can be used as educational aids for physical therapy and rehabilitation medicine students before practical training.

[0089] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment provide the same effect as performing hands-on training on actual patients under the observation of a skilled educator using an XR device.

[0090] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment enable visual and tactile training of the degree of stiffness or rigidity of a standardized joint, and the trainee can provide generalized treatment methods to patients.

[0091] In addition, the mannequin simulator provided in the embodiment can reproduce complex movements of stiffness or rigidity in the lower limbs as well as the upper limbs, allowing educators to experience various stiff or rigid joint characteristics of patients through the mannequin simulator.

[0092] In addition, the digital twin-based educational mannequin simulation system and simulation method according to the embodiment enable trainees to effectively acquire the knowledge and skills necessary for the treatment and rehabilitation of patients.

[0093] The disclosed content is merely illustrative and can be modified and implemented in various ways by a person skilled in the art without departing from the gist of the claim in the patent claims; therefore, the scope of protection of the disclosed content is not limited to the specific embodiments described above.

Claims

Claim 1 An operator terminal that specifies the degree of stiffness and rigidity for each joint of a patient by setting MMT (Medical Research Council Muscle Strength Grading) and MAS (Modified Ashworth Scale) grades through a user interface projected into a digital twin environment; a simulation server that receives the degree of stiffness and rigidity specified by the MMT and MAS grades set from the operator terminal and generates a digital twin-based educational simulation that matches the received degree of stiffness and rigidity; a mannequin simulator that provides tactile information implementing the degree of stiffness and rigidity of the patient according to the educational simulation to a trainee and collects hands-on data, which is joint-specific sensing information by the trainee; and an XR device worn by the trainee that projects a patient body model according to the digital twin-based educational simulation onto the mannequin simulator to provide visual information regarding the degree of stiffness and rigidity of the patient to the trainee. A digital twin-based educational mannequin simulation system, comprising, wherein the simulation server receives information on tremors, rigidity, and stiffness of the patient in real time from a patient bio-information collection sensor attached to the actual patient, calculates the intensity of the tremors, rigidity, and stiffness from the collected information, tracks the body position information of the actual patient where the information occurred in real time, and controls the control position and signal intensity of the mannequin simulator according to the calculated intensity and tracked body position to reproduce the actual patient's pathological condition in real time. Claim 2 A digital twin-based educational mannequin simulation system according to claim 1, wherein the mannequin simulator comprises: at least one passive joint including a spring and a damping device to implement the degree of biological tremors of the patient; at least one active joint including an actuator to implement the degree of biological tremors, rigidity, and stiffness of the patient; a sensor unit that senses the magnitude and direction of a force transmitted by a trainee for each joint to generate hands-on data; and a motion control unit that calculates the magnitude of a reaction force to be transmitted by the actuator of the active joint to the trainee based on the hands-on data and generates a control signal for a complex rigidity and stiffness movement to be implemented in the active joint accordingly. Claim 3 A digital twin-based educational mannequin simulation system according to claim 1, wherein the simulation server receives hands-on data from the mannequin simulator and generates visual information to be displayed on the XR device. Claim 4 delete Claim 5 In claim 1, the simulation server collects hands-on data from a trainee after hands-on training and generates evaluation information, which is the trainee's training result, a digital twin-based educational mannequin simulation system. Claim 6 (A) a step of specifying the degree of joint stiffness and rigidity of a patient by setting MMT (Medical Research Council Muscle Strength Grading) and MAS (Modified Ashworth Scale) grades through a user interface projected into a digital twin environment at an operator terminal; (B) a step of receiving the degree of stiffness and rigidity specified by the MMT and MAS grades set from the operator terminal at a simulation server, and generating a digital twin-based training simulation that matches the received degree of stiffness and rigidity; (C) a step of providing tactile information implementing the degree of stiffness and rigidity of the patient according to the training simulation at a mannequin simulator to a trainee, and collecting hands-on data, which is joint-specific sensing information by the trainee; and (D) a step of providing visual information regarding the degree of stiffness and rigidity of the patient to the trainee by projecting a patient body model according to the digital twin-based training simulation onto the mannequin simulator, which is worn by the trainee at an XR device; A digital twin-based educational mannequin simulation method, comprising: a simulation server receiving information on tremors, rigidity, and stiffness of a patient in real time from a patient bio-information collection sensor attached to a real patient; calculating the intensity of the tremors, rigidity, and stiffness from the collected information; tracking the body position information of the real patient where the information occurred in real time; and controlling the control position and signal intensity of the mannequin simulator to reproduce the pathological condition of the real patient in real time according to the calculated intensity and the tracked body position. Claim 7 In claim 6, the mannequin simulator comprises: at least one passive joint including a spring and a damping device to implement the degree of biological tremors of the patient; at least one active joint including an actuator to implement the degree of biological tremors, rigidity, and stiffness of the patient; a sensor unit that senses the magnitude and direction of a force transmitted by a trainee for each joint to generate hands-on data; and a motion control unit that calculates the magnitude of a reaction force to be transmitted by the actuator of the active joint to the trainee based on the hands-on data and generates a control signal for a complex rigidity and stiffness movement to be implemented in the active joint accordingly; a digital twin-based educational mannequin simulation method. Claim 8 A digital twin-based educational mannequin simulation method according to claim 6, further comprising, after step (C) above, generating visual information to be displayed on the XR device in the simulation server by considering the hands-on data. Claim 9 delete Claim 10 A digital twin-based educational mannequin simulation method according to claim 6, further comprising, after step (C) above, collecting hands-on data of the trainee after hands-on training on the simulation server and generating evaluation information which is the training result of the trainee.

Images