Simulation system, simulation method, and simulation program

Abstract

This system provides a simulation method that prevents the increasing discrepancy between the progress of the video and the actual operation in the surgical simulator, thereby maintaining the consistency of the simulation. [Solution] The system comprises a reference data acquisition unit 21a that acquires reference progress data indicating the standard progress of a procedure and operational load data related to the procedure; an analysis unit 21b that synchronizes the reference progress data and operational load data and extracts a load index; a control parameter calculation unit 21c that calculates control parameters that determine the response characteristics to the console based on the load index; and an output control unit 21d that controls the output to the console. The output control unit executes deceleration control to change the response characteristics when the load index is above a first threshold, and if the load index falls below a second threshold during the execution of deceleration control, it calculates the position deviation and transitions to synchronization recovery control to return the response characteristics, which have been modified to reduce the position deviation, back to the normal state.

Description

Technical Field

[0001] The present invention relates to a simulation system, a simulation method, and a simulation program.

Background Art

[0002] In recent years, in the medical field, surgical simulation systems using virtual reality (VR) technology and the like have been widely introduced for the purpose of improving the skills of physicians (trainees) and ensuring patient safety. In particular, in minimally invasive surgery using a surgical assistance robot system having a multi-joint arm, since the affected area cannot be directly touched, it is essential to acquire a precise operational sense via a console.

[0003] In order to safely and accurately perform procedures in actual surgery, it is strongly required to master not only visual information of the surgical field, but also the "sense of mechanical load (force sense)", such as the weight and hardness of tissue transmitted through a surgical tool, or the sense of resistance caused by interference with surrounding organs. Therefore, in conventional surgical simulators, force feedback presentation devices (haptic devices) in which a physical reaction force generation mechanism such as a motor is incorporated in a master controller or the like operated by a trainee have been generally used.

[0004] Furthermore, in order to learn the techniques of skilled surgeons, simulation methods have also been proposed that cooperate with data such as surgical videos of skilled surgeons to allow trainees to re-experience the procedures.

[0005] Furthermore, in education using simulators, techniques for quantitatively evaluating the operations of operating surgeons are also known. For example, Patent Document 1 discloses a method of storing data of simulated surgical tools and simulation models during surgical simulation, and quantitatively evaluating and scoring the operations of the operating surgeon based on these stored data and evaluation items after the simulation ends.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2020-071418 [Overview of the project] [Problems that the invention aims to solve]

[0007] In practical surgical simulators, including evaluation systems like the one described in Patent Document 1, providing not only visual information but also force feedback is crucial for improving surgical proficiency. However, configurations that use physical reaction force generation mechanisms such as motors and actuators to present mechanical loads such as resistance lead to larger and more complex console devices, resulting in increased manufacturing costs. Furthermore, there are technical constraints such as the burden of maintenance due to wear on movable parts and the mechanical delay (latency) during reaction force generation, which impairs the realism of the simulation.

[0008] To alleviate the burden on such physical mechanisms, one approach is to alter the response characteristics (such as response speed and delay) to operational inputs on a console or similar device, thereby allowing trainees to perceive a simulated mechanical load (such as weight or resistance).

[0009] However, in simulators that work in conjunction with reference data that progresses along a fixed time axis, such as surgical videos, simply slowing down or delaying the trainee's response using software control can lead to a physical time lag accumulating between the progression of the reference video (visual information) and the trainee's actual operations (somatosensory information). In particular, if such deceleration control is repeated in scenes with continuous loads such as tissue traction or suturing, the operations will lag significantly behind the progress of the video, ultimately resulting in a major failure in the simulation where "only the video progresses, and the trainee's operations cannot keep up at all." This remains an unresolved issue.

[0010] The present invention has been made in view of the above problems, and aims to provide a simulation system, simulation method, and simulation program that can simulate and effectively perceive a mechanical load (resistance) by software-modulating the response characteristics of the console without relying on a physical reaction force generation mechanism, while autonomously resolving the accumulation of discrepancies between the progression of the image and the operation caused by the modulation, and maintaining appropriate consistency even during long-term simulations. [Means for solving the problem]

[0011] According to the present invention, a simulation system having the following configuration is provided. [1] A simulation system comprising: a reference data acquisition unit that acquires reference progression data indicating the standard progression of a procedure and operational load data related to the procedure; an analysis unit that synchronizes the reference progression data and the operational load data and extracts a load index from the operational load data; a control parameter calculation unit that calculates control parameters that determine the response characteristics to a console for which an operational input is made, based on the load index; and an output control unit that controls the output to the console based on the control parameters, wherein the output control unit executes deceleration control to change the response characteristics to reduce the response speed to the operational input on the console or to impose a response delay when the load index is greater than or equal to a predetermined first threshold, and during the execution of the deceleration control, when the load index falls below a predetermined second threshold, the output control unit calculates the position deviation between the progression position of the reference progression data and the operational position on the console caused by the deceleration control, and transitions to synchronization recovery control to reduce the position deviation by temporarily increasing the reduced response speed above the normal state or reducing or eliminating the imposed response delay above the normal state. [2] A simulation system comprising: a reference data acquisition unit that acquires reference progression data indicating the standard progression of a procedure and operational load data related to the procedure; an analysis unit that synchronizes the reference progression data and the operational load data and extracts a load index from the operational load data; a control parameter calculation unit that calculates control parameters that determine the response characteristics to a console for which an operational input is made based on the load index; and an output control unit that controls the output to the console based on the control parameters, wherein the output control unit executes deceleration control that continuously reduces the response speed to the operational input on the console according to the magnitude of the load index, and sequentially calculates the position deviation between the progression position of the reference progression data and the operational position on the console caused by the execution of the deceleration control, and if the position deviation exceeds a predetermined tolerance range, executes synchronization recovery control that temporarily increases the reduced response speed or temporarily decreases the progression speed of the reference progression data in order to reduce the position deviation.

[0012] The following are examples of various embodiments of the present invention. The embodiments shown below can be combined with each other. [3] The simulation system according to [1] or [2], wherein the reference data acquisition unit further acquires a reference operation log of the procedure, and the analysis unit performs a synchronization process to synchronize the reference progress data, the operation load data, and the reference operation log, and performs a corrective behavior analysis based on the relationship between the operation load data and the reference operation log, and extracts the load index. [4] The simulation system according to any one of [1] to [3], wherein the control parameter calculation unit calculates a parameter to reduce the response speed based on the detection of an increase in traction force as the load index. [5] The simulation system according to any one of [1] to [4], wherein the control parameter calculation unit calculates a parameter to increase the operating resistance based on the detection of a fluctuation in the gripping force or the force acting in the release direction of the object being operated as the load index. [6] The simulation system according to any one of [1] to [5], wherein the control parameter calculation unit calculates a parameter to change the operating damping rate and slow down the response based on the detection of a sudden change in the direction of the operation. [7] The simulation system according to any one of [1] to [6], wherein the output control unit, while performing the deceleration control, delays the drawing of the object being operated on the console while maintaining the playback speed of the reference progress data, thereby intentionally creating an asynchronous state between the visual progress and the sense of operation. [8] The simulation system according to any one of [1] to [7], wherein the output control unit, while performing the deceleration control, reduces the response speed of the console operation while maintaining the playback speed of the reference progress data, thereby intentionally creating a desynchronized state between the visual progress and the operational sensation. [9] The simulation system according to [1] or any of [3] to [8] referencing [1], wherein the second threshold is set to a value lower than the first threshold, and the output control unit continues the deceleration control from the time the load index becomes equal to or greater than the first threshold and the deceleration control is started until the load index falls below the second threshold.

[10] The simulation system according to [3] or any of [4] to [9], which references [3], wherein the analysis unit includes a trained model that has been machine-trained using the reference progress data, the operation load data, and the reference operation log as training data, and uses the trained model to perform the corrected behavior analysis or the extraction of the load index.

[11] A simulation method performed by a computer, comprising the steps of: acquiring reference progression data indicating the standard progression of a procedure and operational load data related to the procedure; synchronizing the reference progression data and the operational load data and extracting a load index from the operational load data; calculating control parameters that determine the response characteristics to a console for operational input based on the load index; and controlling the output to the console based on the control parameters, wherein the step of controlling the output includes, if the load index is greater than or equal to a predetermined first threshold, executing deceleration control to change the response characteristics to reduce the response speed to operational input at the console or to impose a response delay; and during the execution of the deceleration control, if the load index falls below a predetermined second threshold, calculating the position deviation between the progression position of the reference progression data and the operational position at the console caused by the deceleration control, and transitioning to synchronization recovery control to reduce the position deviation by temporarily increasing the reduced response speed above the normal state or reducing or eliminating the imposed response delay above the normal state.

[12] A computer simulation method comprising: acquiring reference progression data indicating the standard progression of a procedure and operational load data related to the procedure; synchronizing the reference progression data and the operational load data and extracting a load index from the operational load data; calculating control parameters that determine the response characteristics to a console for operational input based on the load index; and controlling the output to the console based on the control parameters, wherein the step of controlling the output includes: executing deceleration control to continuously reduce the response speed to operational input at the console according to the magnitude of the load index; sequentially calculating the position deviation between the progression position of the reference progression data and the operational position at the console caused by the execution of the deceleration control; and executing synchronization recovery control to temporarily increase the reduced response speed or temporarily decrease the progression speed of the reference progression data in order to reduce the position deviation if the position deviation exceeds a predetermined tolerance range.

[13] A simulation program for causing a computer to perform the simulation method described in

[11] or

[12] . [Effects of the Invention]

[0013] According to the present invention, by changing the response characteristics of the console, it is possible to effectively perceive mechanical loads while preventing the widening of the discrepancy between the progress of images and other data and the operation caused by deceleration or delay, thereby maintaining the consistency of the simulation. Furthermore, since it does not rely on a physical reaction force generation mechanism, it is possible to achieve miniaturization of the device, reduction of manufacturing costs, and improvement of maintainability. [Brief explanation of the drawing]

[0014] [Figure 1] This block diagram shows an example of the overall hardware configuration of a surgical simulation system according to one embodiment of the present invention. [Figure 2] This is a block diagram showing an example of the hardware configuration of the surgical simulation device in this embodiment. [Figure 3] This is a block diagram illustrating an example of a functional configuration implemented by a control unit of the surgical simulation apparatus according to the present embodiment. [Figure 4] This is a time-series diagram illustrating the relationship between operation load data (such as gripping force or traction force) and modulation of control parameters (response speed) in the present embodiment. [Figure 5] This is a flowchart illustrating an example of the flow of processing executed by the surgical simulation apparatus according to the present embodiment.

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The surgical simulation system 1 according to the present embodiment is used in the medical field and the like, for learners such as doctors and medical students, or skilled operators themselves who confirm surgical techniques and equipment developers (hereinafter collectively referred to as "trainees" or "operators" in the present specification) to learn and train actual surgical techniques using virtual reality (VR) technology and the like. This system is not limited to a physical hardware configuration such as a controller of a specific manufacturer or a specific display device, and has a configuration that can be completed only by software-based control. That is, without relying on expensive, complex and large-scale hardware such as a physical reaction force generation mechanism, through "intentional and dynamic modulation of response characteristics" for a console that performs operation input, the present invention relates to a technology that makes it possible to artificially perceive the mechanical load transmitted from a surgical instrument in actual surgery (for example, the weight, hardness, elasticity of a tissue, or the sense of resistance when the tissue is pulled). Specifically, by utilizing the cross-modal effect generated by the intentional asynchrony of visual information and somatosensory information, highly realistic force feedback is realized only by software control. This hardware-independent characteristic makes it less susceptible to the impact of future device evolution and can support diverse business developments.

[0016] One of the problems with conventional simulators was the "accumulation of time lag between operations and videos etc." that occurs as a result of delayed operation responses. To solve this problem, the present embodiment employs a time management algorithm that executes deceleration control to present a pseudo haptic sensation when the load index (L) extracted from the operation load data is equal to or greater than a first threshold (T1), captures the timing at which the load index (L) drops below a second threshold (T2), that is, the moment the load is released, calculates the position deviation, and shifts to "synchronization recovery control" for resolving the position deviation so as to reduce it. This makes it possible to continuously maintain the consistency of the simulation without impairing the continuity of the procedure.

[0017] (Correspondence between the configuration of the claims and the configuration of the embodiment) Before describing the details below, the correspondence between each constituent element recited in the claims and the specific constituent elements in the present embodiment will be clarified in advance. The "simulation system" in the claims corresponds to the surgical simulation system 1 which is the entire system shown in Fig. 1 in the present embodiment. The "reference data acquisition unit", "analysis unit", "control parameter calculation unit", and "output control unit" in the claims respectively correspond to the reference data acquisition unit 21a, the analysis unit 21b, the control parameter calculation unit 21c, and the output control unit 21d, which are functional blocks implemented by software or hardware in the control unit 21 of the surgical simulation device 10 shown in Fig. 3 in the present embodiment. The "console" in the claims corresponds to the console device 30 shown in Fig. 1 in the present embodiment, on which a trainee actually performs operation inputs using their hands and feet.

[0018] (1. Overall configuration of surgical simulation system 1) Referring to Figure 1, the overall configuration of the surgical simulation system 1 according to this embodiment will be described. Figure 1 is a block diagram showing an example of the overall hardware configuration of the surgical simulation system 1 according to this embodiment. As shown in Figure 1, the surgical simulation system 1 consists of a surgical simulation device 10, which is the core information processing device of the entire system, a display device 20 connected to it via a communication line such as a network or dedicated cable for data communication, and a console device 30. For example, this system is suitably used to safely and realistically simulate minimally invasive surgery (such as endoscopic surgery) using a surgical support robot system with advanced multi-joint arms.

[0019] The surgical simulation device 10 accumulates and analyzes standard surgical data, including surgical record data (video data, robot arm dynamics data, operation log data, etc.) from actual surgeries performed in the past by skilled physicians (skilled surgeons), and controls the behavior of the console device 30 in real time based on the analysis results. The display device 20 is a monitor that presents trainees with images showing the progress of the procedure (camera images of the surgical field or images of a virtual space created with computer graphics) transmitted from the surgical simulation device 10. The console device 30 is modeled after, or identical to, the operating console of an actual surgical robot, and has master controllers (tweezers-type or joystick-type input devices) and foot pedals that are operated with the left and right hands. Trainees operate the console device 30 while viewing the images displayed on the display device 20, and relive the procedure.

[0020] (2. Hardware Configuration) Next, the physical hardware configuration of the surgical simulation device 10 will be described. Figure 2 is a block diagram showing an example of the hardware configuration of the surgical simulation device 10 in this embodiment. The surgical simulation device 10 is implemented using a general personal computer, workstation, or virtual server built on a cloud environment, and mainly comprises a control unit 11, a storage unit 12, a communication unit 13, and a system bus 14 for interconnecting these and transferring data.

[0021] The control unit 11 is a processing unit that comprehensively controls the operation of the entire device, and is composed of high-performance processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) capable of advanced parallel processing, and an NPU (Neural Processing Unit) specialized for AI (artificial intelligence) inference processing. The control unit 11 reads software such as simulation programs stored in the memory unit 12 into main memory such as RAM and executes it, thereby realizing various functions such as advanced synchronization processing and parameter calculation processing, which will be described later.

[0022] The memory unit 12 is a high-capacity, non-volatile storage device such as an HDD (Hard Disk Drive), SSD (Solid State Drive), or flash memory. In addition to the operating system (OS) and various application programs executed by the control unit 11, the memory unit 12 stores a large amount of "reference data" as a database. This data includes reference progression data that shows the standard progression of the procedure (high-resolution stereo surgical video data, CG trajectory data, etc.), operation load data related to the procedure (mechanical data such as gripping force, traction force, or force acting in the release direction of the forceps obtained from torque sensors, etc., installed on the arm joints of the surgical support robot), and reference operation logs (coordinate information in 3D space of the tip of the robot arm, posture information, operation acceleration, foot pedal depression amount, etc.). Furthermore, when data analysis using AI is performed in the modified example described later (modified example 22), the memory unit 12 also stores the weight parameters of the machine learning model (trained model).

[0023] The communication unit 13 is a communication module such as a network interface card (NIC) for high-speed and low-latency (e.g., within a few milliseconds) data transmission and reception between the display device 20 and the console device 30 via a wired LAN such as Ethernet® or optical fiber, or a wireless communication network such as Wi-Fi or 5G. To avoid compromising the realism of the simulation, the communication unit 13 is required to have high throughput and stability.

[0024] The system bus 14 is a data transmission path for exchanging data and control signals between various hardware components such as the control unit 11, storage unit 12, and communication unit 13.

[0025] (3. Detailed explanation of the functional configuration) Next, the functional configuration of the surgical simulation device 10 will be described in detail with reference to Figure 3. Figure 3 is a block diagram showing an example of the functional configuration realized by the control unit 21 of the surgical simulation device 10 in this embodiment. The control unit 21 referred to here is a general term for functional control blocks that are constructed in software by the control unit 11, which is hardware as shown in Figure 2, executing a program in the storage unit 12. The control unit 21 is broadly composed of four main functional units: a reference data acquisition unit 21a, an analysis unit 21b, a control parameter calculation unit 21c, and an output control unit 21d.

[0026] [Reference data acquisition unit 21a] The reference data acquisition unit 21a is responsible for acquiring data related to a specific procedure to be simulated from the storage unit 12 or an external server. Specifically, the reference data acquisition unit 21a acquires reference progression data that shows the standard progression of the procedure (for example, stereo surgical video data of the surgical field when a skilled surgeon actually performs surgery, or CG animation data based thereon, or a group of three-dimensional coordinate data showing the spatial trajectory of the procedure), as well as operational load data related to the procedure (for example, mechanical data including at least one of the gripping force, traction force, and force acting in the release direction recorded during the procedure). Furthermore, in this embodiment, the reference data acquisition unit 21a also simultaneously acquires a reference operation log of the procedure (time-series coordinate and acceleration data showing the minute movements of the robot arm and master controller). This data is important information that forms the basis for subsequent analysis processing.

[0027] [Analysis section 21b] The analysis unit 21b is responsible for integrating the diverse data acquired and quantifying the degree of "load" in the current simulation situation. First, the analysis unit 21b accurately synchronizes each frame of the reference progress data acquired by the reference data acquisition unit 21a with the operation load data and reference operation log recorded in milliseconds on the time axis using synchronization information such as timestamps added at the time of recording (synchronization process). At this time, in order to absorb the difference in sampling rates between, for example, a 60fps surgical video and operation load data sampled at 1000Hz, processing is performed to match the time resolution using linear interpolation or spline interpolation. As a result, an integrated dataset is constructed that shows "at this moment in the progress of the video, etc., this much force was applied, and the operation was moving in this way."

[0028] Next, the analysis unit 21b extracts a "load index" (represented as variable L in this specification) from the synchronized dataset, which serves as a control trigger. Here, the load index refers to a feature that quantitatively quantifies the physical resistance or weight that occurs during the procedure. In the basic configuration of this embodiment, the analysis unit 21b calculates the load index L from the operation load data and reference operation log using a pre-set calculation formula and judgment conditions (rule-based). For example, if the absolute value of the traction force exceeds a predetermined reference value and the operation acceleration decreases sharply (momentary stop of operation), the load index L is calculated to be high. This quantifies the load by combining the raw sensor values ​​and kinematic features. This makes it possible to capture corrective behaviors such as "fine-tuning of movement" and "adjustment of speed" that are unconsciously performed under high load using a statistical and mathematical approach, and to quantify them as actual resistance.

[0029] [Control parameter calculation unit 21c] The control parameter calculation unit 21c is responsible for calculating specific "control parameters" that determine the response characteristics to the console (console device 30) used for operation input, based on the load index L calculated moment by moment by the analysis unit 21b. The response characteristics here refer to how a virtual surgical instrument (forceps, etc.) reacts when a trainee moves the master controller, and specifically include response speed (represented as variable R in this specification), response delay (time lag), operating damping rate (damping coefficient), or sensitivity (gain).

[0030] The control parameter calculation unit 21c applies the following flexible and specific parameter calculation rules depending on the content and conditions of the load index L. (Rule A: Deceleration based on traction force) If the analysis unit 21b strongly detects an "increase in traction force" as the load index L, the control parameter calculation unit 21c calculates a parameter to either reduce the console's response speed R to the operation input (for example, reducing the gain from the usual 1.0 to 0.5) or to add a "response delay" that temporarily holds the input operation command in a ring buffer in memory and executes it after a certain period of time (for example, 50 milliseconds to 300 milliseconds) to represent the weight that the tissue is resisting the pull of. This expresses the inertia and viscosity when dragging heavy tissue. (Rule B: Resistance based on gripping force or force in the release direction) When the load index L detects "variation in gripping force (such as rebound when gripping hard tissue)" or "variation in force acting in the release direction (such as resistance when opening the forceps to separate tissue)," the control parameter calculation unit 21c calculates a parameter that increases the perceived resistance to the movement. Specifically, it reduces the gain of the closing angle (or opening angle) of the virtual forceps relative to the physical gripping amount (or opening amount) of the master controller to less than 1.0. This creates a visual-somatosensory gap where the trainee's fingers are deeply closed (or widely open) but the forceps on the screen do not follow, creating the illusion of resistance, such as "the tissue is hard and resisting" or "there is resistance to separating the tissue," even without a physical reaction force motor. (Rule C: Slowing down based on a change of direction) If the system detects a "sudden change in direction of operation" from the reference operation log and determines that a load is being applied at that point, the control parameter calculation unit 21c intentionally changes the motion damping rate (damping coefficient) significantly to represent inertial force and viscous resistance, calculating parameters that make the response heavier and slower. In other words, when the progress of the reference progress data in the simulation reaches a high-load situation where a sudden change in direction has occurred, the system temporarily slows down the console's response. This allows the trainee to virtually experience the "difficulty in moving (resistance) as if being hindered by surrounding tissues" that is unique to that situation. Thus, the control parameter calculation unit 21c calculates the optimal parameters not based on a single fixed value, but on dynamic setting rules corresponding to the specific actions of the procedure. Note that rules A to C are merely examples, and the combination of acquired indicators and calculated parameters can be changed as appropriate.

[0031] [Output control unit 21d] The output control unit 21d is the main component responsible for applying the control parameters calculated by the control parameter calculation unit 21c and actually controlling the output to the console device 30 (and display device 20). The output control unit 21d is in charge of the two important phases of the present invention: "deceleration control" and "synchronous recovery control".

[0032] First, the output control unit 21d determines whether the currently calculated load index L is equal to or greater than the "first threshold (referred to as variable T1 in this specification)" that is pre-set in the storage unit 12. This first threshold T1 functions for determining the transition to a high-load section and is determined by setting rules based on objective statistical data. In this specification, "pre-set" is a broad concept that includes not only cases where a fixed value is set before the start of the simulation, but also cases where the system dynamically calculates, determines, and updates the value according to the operator's attributes and usage conditions. If the load index L is equal to or greater than the first threshold T1, the output control unit 21d performs "deceleration control" to change the console's response characteristics to the operation input. Specifically, it reduces the response speed R from the normal state or adds a delay according to the calculated parameters.

[0033] Here, the output control unit 21d of this embodiment effectively utilizes a "cross-modal effect" that takes advantage of the illusions of the human brain. That is, during deceleration control, the output control unit 21d maintains the playback speed of the reference progress data (surgical video, etc.) output to the display device 20, while reducing only the drawing position of the object being operated on in the console device 30 (forceps, etc. in virtual space) and the response speed returned to the user's hand. This intentionally creates a desynchronization between the trainee's visual information (the video plays normally) and somatosensory information (the hand feels heavy and cannot be moved as intended). Because the human brain tends to prioritize visual information, when faced with this desynchronization, the user has the illusion that "my hand is being pulled by heavy tissue and cannot move." This is one of the features of this system, which presents force sensations software-wise without using motors or the like.

[0034] On the other hand, if this deceleration control is continued for a long time, a physical time lag will accumulate between the progress of the reference progress data, which is moving at a constant speed, and the operating position on the console, which is delayed due to the deceleration. Therefore, the output control unit 21d constantly monitors the load index L during the execution of deceleration control and determines whether the load index L has fallen below a predetermined "second threshold (represented as variable T2 in this specification)". An important setting rule here is that the second threshold T2 is set to a lower value than the first threshold T1 (T2 < T1). In other words, it is given a hysteresis (history phenomenon) characteristic. After the load index L exceeds T1 and deceleration control is started, the output control unit 21d does not return to the normal state even if L falls only slightly below T1, but continues deceleration control until it falls below T2, which can be judged as the load being released. This prevents the "chattering phenomenon," in which deceleration and normal response frequently switch and operation becomes jerky when the load fluctuates slightly near the threshold.

[0035] Then, when the load index L falls below the second threshold T2, the output control unit 21d transitions to "synchronous recovery control" to eliminate the accumulated deviation and return the modified response characteristics to the normal state. In this synchronous recovery control, the output control unit 21d does not simply return the response speed R to the normal state (baseline). If it simply returned it, the delay that had occurred up to that point would remain indefinitely. Therefore, the output control unit 21d calculates the positional deviation between the progress position of the reference progress data caused by the deceleration control (at least one of the progress degree on the time axis and the spatial coordinate position) and the operating position on the console, and performs a "fine adjustment" to temporarily increase the reduced response speed above the normal state in order to reduce the positional deviation. In other words, instead of abruptly overshooting the position, it performs a catch-up correction to recover the delay by temporarily overshooting only the speed gain. More specifically, the control parameter calculation unit 21c uses feedback control (such as PI control) to determine a gain increase amount (for example, R = 1.2 to 1.5, within a range that does not cause discomfort to the operator) and duration that are proportional to the amount of integral error, in order to converge the position deviation (error vector) to zero, thereby automatically and smoothly bringing the object being operated to the appropriate position (performing spatial alignment). Then, when the position deviation converges and resets to zero (in practice, within a predetermined small tolerance range), the gain is returned to the baseline of 1.0. This restores synchronization between the progress of the reference progress data and the progress of the operation, maintaining the consistency of the simulation without disrupting the flow of the procedure.

[0036] (4. Time-series data and control transitions) Next, the time management control performed by the output control unit 21d described above will be explained in detail on the time axis with reference to the graph in Figure 4. Figure 4 is a time series diagram showing the relationship between fluctuations in the operating load data (gripping force, traction force, or force acting in the release direction, etc.) and the modulation of the parameter (response speed) controlled accordingly in this embodiment.

[0037] The graph in Figure 4 has time (in milliseconds, for example) on the horizontal axis, and the thick solid line in the upper section shows the magnitude of the "operational load data (i.e., a value proportional to the load index L)" obtained by sensors and analysis. The thick dashed line in the lower section shows the magnitude of the "response speed R" commanded by the output control unit 21d to the console.

[0038] At the left edge of the graph (around time 0), the load is low and stable, so the response speed remains at a normal state (e.g., a horizontal baseline indicating a gain of 1.0). As time progresses, from time 20 to 40, the load increases due to the grasping and pulling of the virtual tissue. At this time, the internally calculated load index L increases.

[0039] The moment this load index L exceeds a predetermined first threshold T1 (for example, a line set at approximately 60% of the height of the upper graph), the output control unit 21d immediately starts the "deceleration phase (deceleration control)". In conjunction with this, looking at the lower graph, the dashed line indicating the response speed smoothly dips downward from the baseline, indicating that the "response speed is reduced". The maximum range of the reduced response speed and the added delay, i.e., the "upper modulation limit", is set within a range that satisfies "predetermined operability standards" that do not compromise safe operability. Specifically, limits are set such as a range where the amount of response speed reduction or sensitivity change does not fall below 50% of the normal amount, a range where the added delay does not exceed, for example, 50 milliseconds to 100 milliseconds, or a range of predetermined attenuation rates that are safe according to prior operator tests, so that the simulation does not break down. This prevents the device from becoming inoperable even under very high load conditions. In this section, the trainee feels a simulated force sensation of "heaviness in the hand" while maintaining safe operability.

[0040] Subsequently, just before time 40, the operational load on the upper graph drops sharply due to reasons such as the completion of the traction on the tissue and the release of tension. At this time, even if the load index L falls below the first threshold T1, the response speed does not recover immediately (to prevent chattering due to hysteresis). However, as it drops further and falls below the second threshold T2 (for example, the 20% line of the graph height), the output control unit 21d transitions to the "synchronous recovery phase (synchronous recovery control)".

[0041] Please pay attention to the section labeled "Synchronization restored after load reduction" in the lower graph. You can see that the response speed, which was at the bottom of the valley during the deceleration phase, rises, exceeding the baseline (normal state) and forming a temporary upward-bulging "peak." This is the aforementioned "fine-tuning that temporarily increases the reduced response speed above the normal state." In this section, the object being manipulated moves slightly more sensitively (faster) than usual, so the "delay in operation relative to progress data" accumulated during the deceleration phase is quickly resolved, and the discrepancy (position deviation) is reduced.

[0042] Subsequently, a small increase in load occurs again around time 60. Here again, when the load exceeds the first threshold T1, the system enters the "deceleration phase" and "decreases the response speed again." Then, when the load decreases, it enters the "synchronization recovery phase" again and "fine-tuning is performed to maintain synchronization." Thus, the graph in Figure 4 shows that the system continuously executes a set of time management actions in response to load fluctuations: "force feedback through deceleration" and "delay recovery through subsequent overshoot (synchronization recovery)."

[0043] (5. Simulation processing flow) Next, the specific processing steps performed by the control unit 21 of the surgical simulation device 10 will be described in detail with reference to the flowchart in Figure 5. Figure 5 is a flowchart showing an example of the processing flow performed by the surgical simulation device 10 according to this embodiment. This series of algorithms is executed repeatedly in real time as high-speed loop processing at predetermined minute time intervals (for example, predetermined control periods of 1 millisecond to 16 milliseconds) synchronized with the system clock during the execution of the simulation.

[0044] In step S101 (acquisition of reference data, etc.), the reference data acquisition unit 21a reads and acquires reference progress data corresponding to the current simulation playback time (timestamp t), operation load data recorded at the same time, and reference operation logs from the database of the storage unit 12.

[0045] In step S102 (extraction and calculation of load index), the analysis unit 21b takes the operation load data and reference operation log acquired in step S101 as input and executes processing using a predetermined calculation algorithm (calculation formula and threshold judgment rule) stored in the storage unit 12. This calculates and extracts a "load index L (for example, a continuous value from 0 to 100)" that takes into account corrective behavior (kinematic changes such as sudden speed reduction and fine trajectory correction).

[0046] In step S103 (determination of current control state), the control parameter calculation unit 21c and the output control unit 21d check a status flag indicating whether the system is currently "executing deceleration control" or "in a normal state (or synchronization recovery in progress)" and make a determination. If deceleration control is not being executed (the "NO" route in the diagram), the system is determined to be either not yet facing a load or has been released from a load, and the process proceeds to step S104. On the other hand, if deceleration control is already being executed (the "YES" route in the diagram), the system is determined to be currently experiencing a load, and the process proceeds to step S107.

[0047] In step S104 (high load determination by first threshold), i.e., when deceleration control is not in progress, the control parameter calculation unit 21c compares and determines whether the current load index L extracted in step S102 is greater than or equal to the "first threshold T1" which is the criterion for initiating deceleration (L ≥ T1). If the load index L is less than the first threshold T1 (the "NO" route in the diagram), there is no load, so the process proceeds to step S105 to maintain the normal state, calculates normal parameters with the response speed set to a reference value (e.g., 1.0), and then merges into step S109. On the other hand, if the load index L is greater than or equal to the first threshold T1 (the "YES" route in the diagram), it is determined that a high load has occurred, and the process proceeds to step S106 to start deceleration control.

[0048] In step S106 (calculation process for deceleration control parameters), that is, if the load newly exceeds T1 or if a high load continues, the control parameter calculation unit 21c calculates "control parameters for deceleration control" to lower the console's response speed R below a reference value (for example, set R=0.5) or to add an intentional delay time in order to create the illusion of weight or resistance. This calculation uses a function that continuously changes the deceleration rate according to the magnitude of the load index L. After the parameter calculation, the process proceeds to step S109.

[0049] In step S107 (load release determination by second threshold), i.e., if it was determined in step S103 that deceleration control was already in progress, the control parameter calculation unit 21c compares and determines whether the current load index L has fallen to below the "second threshold T2 (however T2 < T1)" which is the criterion for ending deceleration and moving to synchronization recovery (L < T2). If the load index L is still above the second threshold T2 (the "NO" route in the figure), it is determined that resistance is still continuing, and the unit returns to step S106 via the connecting mark (A) to continue calculating the parameters for deceleration control (maintaining the deceleration phase). On the other hand, if the load index L falls below the second threshold T2 (the "YES" route in the figure), it is determined that the load has been released, and the unit proceeds to step S108 to recover the misalignment.

[0050] In step S108 (calculation process for synchronous recovery control parameters), that is, when the load decreases to less than T2 and the deceleration phase ends, the control parameter calculation unit 21c calculates "control parameters for synchronous recovery control" to return the modified response characteristics to the normal state in order to eliminate the "delay in operation relative to progress data" accumulated by the deceleration control so far. Specifically, in this step, the positional deviation between the progress position of the reference progress data caused by the deceleration control and the operation position on the console is calculated, and a fine-tuning parameter is calculated to temporarily set the response speed R to a value higher than normal (for example, R=1.2) in order to reduce the positional deviation. This allows for proper alignment. After the parameter calculation, the process proceeds to step S109.

[0051] In step S109 (output control and end of one cycle), the output control unit 21d receives the latest control parameters (normal, deceleration, or synchronization recovery) calculated via one of the routes in steps S105, S106, or S108, and transmits them to the console device 30 via the communication unit 13 to execute and control the output. At the same time, it sends the next frame of video or the like to the display device 20. After that, it ends the processing of one cycle in that time step and starts processing again from step S101 at the next millisecond timing. Through this series of flows, the system achieves situation-appropriate time management. Note that the execution order of each step described in the flowchart of Figure 5 is merely an example, and it is possible to appropriately change the order of execution or execute multiple steps in parallel as long as no logical inconsistencies occur.

[0052] (6. Configuration and operation / effect of each claim corresponding to the embodiment) Based on the detailed description of the embodiments above, the specific configurations corresponding to each claim in the patent claims, and the unique functions and effects achieved therein, are summarized below.

[0053] [Configuration and effects corresponding to Claim 1 (Independent Claim: Threshold-type + Deviation-based Synchronization Recovery)] (Configuration): The simulation system according to this embodiment includes a reference data acquisition unit 21a that acquires reference progress data and operation load data, an analysis unit 21b that synchronizes these and extracts a load index L, a control parameter calculation unit 21c that determines response characteristics based on the load index L, and an output control unit 21d that controls the output to the console. The output control unit 21d is characterized by executing "deceleration control" when the load index L is equal to or greater than a first threshold, and if the load index L falls below a second threshold during execution, it calculates the position deviation and transitions to "synchronous recovery control" to reduce the position deviation. (Effect): With this configuration, the weight and resistance of tissues can be simulated and perceived simply by controlling the response speed and delay of the console, without using physical force feedback devices. Furthermore, instead of simply delaying, it transitions to "synchronization recovery control" that reduces the deviation after deceleration, thus preventing simulation failures caused by the accumulation of discrepancies between the video and the operation.

[0054] [Configuration and effects corresponding to Claim 2 (Independent Claim: Continuous Control Type + Deviation-Based Fail-Safe)] (Configuration): Instead of the state transition type control using the thresholds (T1, T2) described above (first embodiment), the output control unit 21d may perform deceleration control that reduces the response speed to the operation input on the console based on a continuous function according to the magnitude of the load index (second embodiment). In this case, if the position deviation exceeds a predetermined tolerance range, the output control unit 21d performs synchronization recovery control that temporarily increases the reduced response speed or, as a fail-safe, temporarily reduces the progress speed of the reference progress data in order to reduce the position deviation. (Effect): By covering both a continuous decrease in response speed without using a threshold and a fail-safe mechanism that corrects on the video side, etc., it is possible to provide a more stable system foundation against sudden operation delays while satisfying the requirement of uniformity.

[0055] [Configuration and effects corresponding to claim 3 (dependent claim: detailed configuration of the analysis unit)] (Configuration): The reference data acquisition unit 21a further acquires a reference operation log, and the analysis unit 21b performs a synchronization process to synchronize the reference progress data, operation load data, and reference operation log, and performs corrected behavior analysis and extraction of the load index L. (Effect): By analyzing not only mechanical data but also the subtle corrective movements of the hand in response to the load (operation logs) that are performed unconsciously, it becomes possible to identify the points where resistance is actually felt with greater accuracy, improving the realism of the simulation.

[0056] [Configuration and effects corresponding to claims 4-6 (dependent claims: parameter calculation)] (Configuration): The control parameter calculation unit 21c calculates parameters to reduce the response speed based on the detection of an increase in "traction force" as a load indicator, calculates parameters to increase the operating resistance based on the detection of fluctuations in "gripping force" or "force acting in the release direction of the operated object", or calculates parameters to slow down the response by changing the operating damping rate (damping coefficient) based on the detection of a "sudden change of direction" in the operation. (Effect): By assigning appropriate parameters according to the specific actions of the procedure, it is possible to reproduce the diverse mechanical nuances unique to the procedure in detail, rather than simply a monotonous delay.

[0057] [Configuration and effects corresponding to claims 7-8 (dependent claims: asynchronous vision)] (Configuration): During the execution of deceleration control, the output control unit 21d delays or reduces the drawing or response speed of the target being operated on the console while "maintaining the playback speed of the reference progress data," thereby intentionally creating a desynchronized state between the visual progress and the sense of operation. (Effect): By intentionally creating an asynchronous state where "video and other visuals remain at normal speed, but only the response of the user's own input is delayed," it is possible to effectively induce a "cross-modal effect" that utilizes the gap between visual information and somatosensory information.

[0058] [Configuration and effects corresponding to claim 9 (dependent claim: setting the threshold hysteresis)] (Configuration): The second threshold T2 is set to a lower value than the first threshold T1 (T2 < T1), and the output control unit 21d starts deceleration control when the load index L becomes T1 or higher, and continues the deceleration control until the load index L falls below T2. (Effect): A deliberate difference (hysteresis) is introduced between the first threshold at which deceleration begins and the second threshold at which the system returns to its normal state. This prevents the phenomenon of frequent control switching causing unnatural jerking (chattering) and provides a smooth operating experience.

[0059] [Configuration and effects corresponding to claim 10 (dependent claim: utilization of a trained model)] (Configuration): The analysis unit 21b includes a trained model (AI) that has been machine-trained using reference progress data, operation load data, and operation logs as training data, and uses this model to perform corrective behavior analysis or extraction of the load index L. This configuration is dependent only on claim 3 which includes the relevant configuration requirements. (Effect): By using AI (machine learning models), it becomes possible to accurately extract the movements of techniques and sensory elements that are difficult to judge with simple rule-based methods, and to build a personalized simulation environment.

[0060] (7. Variant) It should be noted that the present invention is not limited in any way to the specific configurations of the embodiments described above, and various modifications, applications, and combinations with other technologies are possible without departing from the technical idea and the scope of the claims, as exemplified below. The following lists the anticipated modifications, categorized as "hardware extension systems," "sensory feedback systems," "AI / evaluation systems," and "other applications."

[0061] [Variations of hardware expansion systems] (Variation 1) Hybrid configuration with physical devices: This system is not limited to completely eliminating the console's reaction force generation mechanism. For example, the master controller may be equipped with only a small eccentric motor or a vibrator such as a piezoelectric element that produces minute vibrations, and the output control unit 21d may transmit a high-frequency minute vibration signal in synchronization with the timing of the deceleration control. This allows for the combination of software-based cross-modal effects (illusion of weight due to response delay) and minor physical vibrations (friction when cutting bone or the click sensation of tissue popping), enabling a hybrid configuration that provides greater realism and immersion without using large motors.

[0062] (Modification 2) Reproduction of unique robotic kinematics: The memory unit 12 may be configured to pre-store kinematic models of unique arm joint singularities and unique gear backlash (play) that specific surgical assistance robots have as hardware. The analysis unit 21b predicts and analyzes these unique physical interferences from the operation trajectory and superimposes them on the deceleration control as minute noise in the control parameters or instantaneous maximum deceleration. This makes it possible to faithfully reproduce (emulate) the unique feel and quirks of specific, expensive hardware equipment in software, even while using a general-purpose console.

[0063] (Variation 3) Application to haptic gloves: Instead of a joystick-type master controller, the console device 30 may be configured to use a data glove (haptic glove) that controls the bending resistance of the fingertips with small wires or pneumatics as the input interface. In this case, the control parameter calculation unit 21c individually calculates the flexion depth of each finger, such as the thumb and index finger, and the degree of interference with the surgical instrument, and performs independent deceleration and resistance control at the finger joint level (for example, applying a strong delay only to the thumb side). This enables intuitive and precise force feedback in delicate procedures such as microscopic surgery and in VR environments that require natural hand movements.

[0064] (Modification 4) Utilization of HMDs: A VR / AR HMD may be used as the display device 20, and the trainee's head movements (head tracking data) obtained from a gyro sensor etc. built into the HMD may be linked with the parameter of the forceps response delay. For example, if it is detected that the trainee is turning their face away from the load generation point in the surgical field (not looking at it), the system will intentionally reduce the response speed of the forceps to an extreme degree. This physically suppresses dangerous blind operations when the eyes are taken off the surgical site, and ensures safe surgical education that maintains the correct posture and gaze at all times.

[0065] [Variations of sensory feedback systems] (Variation 5) Synchronization of auditory feedback: When deceleration control is performed (when the load increases), a configuration may be used in which an audio output module is used to amplify and modulate audio parameters such as "friction sounds and tension sounds (pitch and volume) as tissue is pulled" output from the display device 20 or headphones in proportion to the magnitude of the load index L. Because the human brain has a characteristic of strongly linking auditory information with physical tension, by generating a sophisticated cross-modal effect that synchronizes the three senses of vision (image), somatosensory (delay), and hearing, the illusion of "weight and stiffness" can be maximized even without physical reaction force.

[0066] (Experimental variation 6) Color and Texture Modulation of Surgical Field Images: During deceleration control, dynamic shader processing may be applied to the tissue texture drawn on the display device 20. For example, the redness of the RGB values ​​in areas under tension may be increased (to represent congestion or excessive tension), or highlights (glossiness or exudate) may be emphasized to add visual modulation. This allows for the intuitive communication of the load limit as visual warning information before actual physical tissue damage (rupture or tear) occurs, effectively guiding trainees to operate more cautiously.

[0067] (Example 7) Endoscope Camera Blur Effect: In high-load conditions where deceleration control is strongly active (such as excessive traction or dissection of indurated tissue), the output control unit 21d may be configured to intentionally apply camera shake or motion blur post-effects to the periphery of the image on the display device 20 in order to represent the physiological tremors (tremor) of the operating arm and the operator. This visual blur artificially amplifies the instability of the operation, thus having an educational effect by making trainees recognize that "any more forceful operation is dangerous" and encouraging them to voluntarily slow down the operation speed and stabilize.

[0068] (Variation 8) Linkage with virtual bleeding effect: When the load index L significantly exceeds the first threshold T1, reaching a dangerous area (e.g., the third threshold T3; T3 > T1), the system may determine that the limit of deceleration control has been exceeded and render a minute bleeding particle effect from the virtual tissue. The amount of bleeding is scaled according to the excess load. This allows the system to go beyond simply presenting resistance and provide direct and strong negative feedback (penalties) for inappropriate operations that exceed the allowable stress, strongly emphasizing gentle and careful techniques that are kind to the tissue.

[0069] [Variations of AI and evaluation systems] (Extreme variation 9) Extraction of load indicators using AI (trained model): In the above embodiment, the basic configuration was described in which the analysis unit 21b extracts the load indicator L using a pre-set calculation formula, etc. (rule-based). However, instead, a configuration may be used in which a trained model (AI) learned by deep learning, etc., is used as training data, based on the correlation between standard progress data (video), operation load data (sensor values), and operation logs (kinematics) from past skilled surgeons' techniques (corresponding to claim 10). This makes it possible to infer and extract with high accuracy the "tacit knowledge of load sensation unique to skilled surgeons (weight intuitively felt from the appearance of a specific tissue and arm angle)," which is difficult to quantify with simple mathematical formulas, from the video and the current operating posture.

[0070] (Variation 10) Predictive Recovery (RESCID): Instead of simply passively waiting for the load index L to fall below the second threshold T2, the system can be configured to use a time-series prediction AI model to constantly monitor the derivative values ​​of the operation log (decrease trend in acceleration and changes in trajectory) and predict a future state where the system will be completely free from load in a few milliseconds. Based on this prediction, RESCIDID can be smoothly initiated just before the actual load falls below the threshold (e.g., 50 milliseconds before), offsetting the unavoidable computational delays inherent in the system and network, and providing a natural and smooth operating feel that perfectly matches human predictions.

[0071] (Variation 11) Introduction of a tissue vulnerability index: The system may be configured to apply image recognition AI, such as semantic segmentation, to acquired surgical videos to identify the type of target tissue (fat, muscle, blood vessels, etc.) visible in the surgical field and calculate its "fragility (vulnerability index)" in real time. When manipulating an object determined to be highly vulnerable (fragile blood vessels, etc.), the control parameter calculation unit 21c non-linearly maximizes the degree of deceleration (damping coefficient) even for slight manipulations. This enables situation-adaptive simulations that are in line with the anatomical context, allowing for effective learning of appropriate handling techniques for each tissue.

[0072] (Example 12) An automated scoring system based on synchronization deviation may be configured to include an evaluation module that calculates the error amount by integrating the positional deviation between the "ideal recovery trajectory of an expert" and the "actual operation trajectory of a trainee" during the synchronization recovery phase over time. The smaller this error amount, the more it is judged that the trainee was able to "follow changes in load smoothly and safely," and the meticulousness of the technique and the risk of damage to tissue are quantitatively scored and displayed on a dashboard. This makes it possible to use this system not merely as an experiential tool, but as an objective and reproducible skill assessment tool.

[0073] (Example 13) Integration with eye tracking: The monitor section of the console device 30 may be equipped with an eye-tracking sensor using infrared light or the like to calculate the distance between the trainee's gaze coordinates and the current load generation point (e.g., the tip coordinates of a virtual surgical instrument). If a high load occurs while this distance exceeds a predetermined value (indicating incorrect gaze), the degree of deceleration control is increased even further than usual as a warning. This directly imprints the ironclad rule in surgical procedures, "Never take your eyes off the tip of a surgical instrument," into the trainee's bodily sensations in the form of physical difficulty in movement.

[0074] [Other applications] (Variation 14) Dynamic playback speed change of the progress data itself (fail-safe): In this embodiment, the playback speed of the reference progress data is maintained in order to produce a cross-modal effect. However, a fail-safe mechanism may be provided in the event that the trainee's operation is extremely delayed due to continuous high load and the position deviation completely exceeds a predetermined safety tolerance range. That is, the output control unit 21d intervenes and executes synchronization recovery control that temporarily reduces (slows down) the playback speed of the reference progress data (surgical video, etc.) itself, thereby reliably avoiding a fatal failure in which "only the video completely advances, making it impossible to proceed with the simulation." Note that the configuration of this modified example corresponds to a specific implementation of the synchronization recovery control that reduces the progress speed of the reference progress data, as described in the second embodiment (claim 2) above.

[0075] (Variation 15) Multi-user collaborative simulation: This configuration may involve not only a single surgeon, but also a multi-user environment where the surgeon and assistant log in simultaneously from separate consoles on the same network. In this environment, the mechanical data (load indicators) generated when the assistant pulls tissue in the virtual space is shared in real time with the surgeon's system and reflected in the response speed (deceleration control) of the surgeon's console. This allows users to experience the "sharing of tissue tension changes caused by the actions of others," which is essential in team-based medical care, and enables advanced collaborative action training.

[0076] (Variation 16) Dynamic threshold adjustment according to trainee proficiency: The trainee's account information may be configured to dynamically change the values ​​of the first threshold T1 and the second threshold T2 (scaffolding function) by managing "proficiency parameters" such as past simulation results and years of experience in the memory unit 12, and adjusting them according to the level. For beginners, the threshold can be set low to provide an early warning of deceleration (heaviness) and teach them to use a safety margin, while for experts, a high threshold close to the actual physical limit load can be set to provide an efficient educational program optimized for the learner's phase.

[0077] (Example 17) Introduction of fatigue simulation: In long-duration simulations, a timer unit may be included to measure the elapsed session time in order to realistically reproduce the physical and cognitive fatigue of the operator. After a certain period of time (e.g., 2 hours), parameter shift control is performed to gradually decrease the baseline response speed R (e.g., to 0.8 times) and gradually lower the first threshold T1. This allows the user to experience the difficulty of maintaining concentration and managing resources under harsh conditions where fatigue makes it difficult to move as intended.

[0078] (Example 18) Introduction of gamification elements: The system may be configured to internally award "safety points" based on the time and number of times the procedure segment is completed while maintaining a safe traction force (not exceeding T1 or reaching the T3 bleeding threshold). By implementing an achievement function that displays rankings and titles (badges) according to the points earned on the results screen after the simulation is completed, a mechanism can be created that enhances the trainee's motivation for continuous learning based on behavioral psychology and promotes spontaneous and repetitive training.

[0079] (Variation 19) Cloud-based parameter generation: The system architecture may also involve offloading the data synchronization and analysis processing (analysis unit 21b, control parameter calculation unit 21c) functions of the surgical simulation device 10 to a cloud server with high computing power using edge computing technology. The console device 30 transmits only lightweight operation logs and receives streaming control parameters from the cloud via a low-latency communication network such as 5G. This makes it possible to provide an advanced simulation environment with the latest AI models at all times while reducing the hardware costs of local terminals installed in each hospital and educational institution.

[0080] (Modification 20) Application to Network Latency Compensation in Remote Surgery: The time management algorithm of this system, consisting of "deceleration control" and "synchronous recovery control," is not limited to educational simulation applications. It can be directly applied as a safety control technology to conceal and compensate for communication delays (packet arrival fluctuations and latency) over WANs in actual robot-assisted remote surgery systems. By converting fluctuations in communication delay into a pseudo-"operational load (resistance)" in the console response and smoothly restoring synchronization during packet recovery, it provides a stable and intuitive operating environment without making the remote surgeon feel any unnatural jitter, resulting in beneficial ripple effects.

[0081] Although preferred embodiments and numerous modifications of the present invention have been described in detail above, the present invention is not limited thereto, and various other modifications and applications that can be easily inferred by those skilled in the art based on the claims are all included within the scope of the present invention. Furthermore, the simulation program according to the present invention may be provided by recording it on a non-transient computer-readable recording medium (e.g., optical disk, magnetic disk, flash memory, etc.), or it may be provided so that it can be downloaded from an external server via a communication network such as the Internet. [Explanation of Symbols]

[0082] 1: Surgical Simulation System 10: Surgical simulation device 11: Control Unit 12: Storage part 13: Communications Department 14: System bus 20: Display device (monitor) 21: Control Unit 21a: Reference data acquisition unit 21b: Analysis section 21c: Control parameter calculation unit 21d: Output control unit 30: Console device (console) T1: First threshold T2: Second threshold T3: Third threshold L: Load index R: Response speed

Claims

1. A reference data acquisition unit that acquires reference progression data indicating the standard progression of a procedure, and operational load data related to the procedure, An analysis unit that synchronizes the aforementioned reference progress data and the aforementioned operation load data, and extracts a load index from the operation load data, A control parameter calculation unit calculates control parameters that determine the response characteristics to the console used for operation input based on the load index, An output control unit that controls the output to the console based on the control parameters, A simulation system comprising, The output control unit, If the load index is greater than or equal to a predetermined first threshold, deceleration control is executed to reduce the response speed to the operation input on the console or to modify the response characteristics to impose a response delay. During the execution of the deceleration control, if the load index falls below a predetermined second threshold, the positional deviation between the progress position of the reference progress data caused by the deceleration control and the operating position on the console is calculated, and the system transitions to synchronization recovery control in which the reduced response speed is temporarily increased above the normal state, which is the reference response speed, or the applied response delay is reduced or eliminated above the normal state, which is the reference response delay amount, in order to reduce the positional deviation, and thereafter, based on the positional deviation converging within a predetermined tolerance range, the response speed or the response delay is returned to the normal state. Simulation system.

2. A reference data acquisition unit that acquires reference progression data indicating the standard progression of a procedure, and operational load data related to the procedure, An analysis unit that synchronizes the aforementioned reference progress data and the aforementioned operation load data, and extracts a load index from the operation load data, A control parameter calculation unit calculates control parameters that determine the response characteristics to the console used for operation input based on the load index, An output control unit that controls the output to the console based on the control parameters, A simulation system comprising, The output control unit, Depending on the magnitude of the load index, deceleration control is performed to continuously reduce the response speed to the operation input on the console. The system sequentially calculates the positional deviation between the position of the reference data and the operating position on the console, which is caused by the execution of the deceleration control. If the positional deviation exceeds a predetermined tolerance range, it performs synchronization recovery control to reduce the positional deviation by temporarily increasing the reduced response speed or temporarily decreasing the progress speed of the reference data. Subsequently, based on the positional deviation converging within a predetermined tolerance range, it restores the response speed or progress speed to the state before the execution of the synchronization recovery control. Simulation system.

3. The aforementioned reference data acquisition unit further acquires a reference operation log of the procedure, The analysis unit performs a synchronization process to synchronize the reference progress data, the operation load data, and the reference operation log, and performs a correction behavior analysis based on the relationship between the operation load data and the reference operation log, as well as extracting the load indicator. The simulation system according to claim 1 or 2.

4. The control parameter calculation unit calculates a parameter to reduce the response speed based on the detection of an increase in traction force as the load index. The simulation system according to claim 1 or 2.

5. The control parameter calculation unit calculates parameters to increase the operating resistance based on the detection of fluctuations in the gripping force or force acting in the release direction of the object being operated, as the load index. The simulation system according to claim 1 or 2.

6. The control parameter calculation unit calculates a parameter that changes the motion damping rate and slows down the response based on the detection of a sudden change in the direction of the operation. The simulation system according to claim 1 or 2.

7. The output control unit, while performing the deceleration control, delays the rendering of the target being operated on the console while maintaining the playback speed of the reference progress data, thereby intentionally creating a desynchronized state between the visual progress and the sense of operation. The simulation system according to claim 1 or 2.

8. The output control unit, while performing the deceleration control, maintains the playback speed of the reference progress data, and reduces the response speed of operations on the console, thereby intentionally creating a desynchronized state between the visual progress and the tactile sensation. The simulation system according to claim 1 or 2.

9. The second threshold is set to a value lower than the first threshold. The output control unit continues the deceleration control after the load index becomes equal to or greater than the first threshold and the deceleration control is initiated, until the load index falls below the second threshold. The simulation system according to claim 1.

10. The analysis unit includes a trained model that has been machine-trained using the reference progress data, the operation load data, and the reference operation log as training data, and performs the corrected behavior analysis and the extraction of the load index using the trained model. The simulation system according to claim 3.

11. A simulation method performed by a computer, A step of acquiring standard progression data that shows the standard progression of the procedure, and operational load data related to the procedure, The steps include synchronizing the aforementioned standard progress data and the aforementioned operation load data, and extracting a load index from the said operation load data, The steps include: calculating control parameters that determine the response characteristics to the console on which the operation input is performed, based on the load index; A step of controlling the output to the console based on the control parameters, Includes, The step of controlling the output is: If the load index is greater than or equal to a predetermined first threshold, deceleration control is executed to reduce the response speed to the operation input on the console or to modify the response characteristics to impose a response delay. During the execution of the deceleration control, if the load index falls below a predetermined second threshold, the positional deviation between the progress position of the reference progress data caused by the deceleration control and the operating position on the console is calculated, and the system transitions to synchronization recovery control in which the reduced response speed is temporarily increased above the normal state, which is the reference response speed, or the applied response delay is reduced or eliminated above the normal state, which is the reference response delay amount, in order to reduce the positional deviation, and thereafter, based on the positional deviation converging within a predetermined tolerance range, the response speed or the response delay is returned to the normal state. Simulation method.

12. A simulation method performed by a computer, A step of acquiring standard progression data that shows the standard progression of the procedure, and operational load data related to the procedure, The steps include synchronizing the aforementioned standard progress data and the aforementioned operation load data, and extracting a load index from the said operation load data, The steps include: calculating control parameters that determine the response characteristics to the console on which the operation input is performed, based on the load index; A step of controlling the output to the console based on the control parameters, Includes, The step of controlling the output is: Depending on the magnitude of the load index, deceleration control is performed to continuously reduce the response speed to the operation input on the console. The system sequentially calculates the positional deviation between the position of the reference data and the operating position on the console, which is caused by the execution of the deceleration control. If the positional deviation exceeds a predetermined tolerance range, it performs synchronization recovery control to reduce the positional deviation by temporarily increasing the reduced response speed or temporarily decreasing the progress speed of the reference data. Subsequently, based on the positional deviation converging within a predetermined tolerance range, it restores the response speed or progress speed to the state before the execution of the synchronization recovery control. Simulation method.

13. A simulation program for causing a computer to perform the simulation method described in claim 11 or 12.

Images