Surgical robot system for reproducing recorded path of medical instrument and control method thereof

Abstract

A surgical robot system according to an embodiment of the present invention comprises: a steerable medical instrument; one or more actuators for driving the medical instrument; a memory configured to store recording information of a path in which the medical instrument is steered in the body, wherein the recording information includes visual data of an anatomical structure steered by the medical instrument and control data of the medical instrument; and a control device which receives a path reproduction request command for reproducing the recorded path of the medical instrument, compares and analyzes the past record information stored in the memory with the current record information of the medical instrument to determine whether to approve the path reproduction request command, and then executes the path reproduction request command, wherein the control device includes a path reproduction determination module in which a decision-making algorithm having a condition enabling the path reproduction request command to be rejected is programmed.

Description

Surgical robot system and control method for reproducing recorded paths of medical devices

[0001] The present invention relates to a surgical robot system for remotely controlling a steerable medical instrument, and to a surgical robot system and control method capable of reproducing a recorded path of a medical instrument.

[0002]

[0003] Minimally invasive surgery is a surgical technique that involves making small incisions in the patient's body to insert surgical instruments and perform procedures. This surgical technique offers numerous advantages over open surgery, including faster patient recovery, reduced risk of infection, and improved aesthetics. With the continuous advancement of medical technology, minimally invasive surgery is evolving into robotic-assisted surgical systems capable of remote operations. For examples of robotic surgical systems for minimally invasive surgery, please refer to the applicant's previously filed application No. 10-2024-0170032.

[0004] Surgical robotic systems that remotely control medical instruments can be utilized not only for minimally invasive surgeries, but also for invasive procedures that penetrate the body's natural openings, steer to target tissue, and then perform the intended surgery. For examples of such surgical robotic systems, reference may be made to Korean Patent Publication No. 10-2024-0009905, a prior patent of the present applicant.

[0005] Medical personnel operating surgical robotic systems remotely control medical instruments through an operator console interface. During this process, the instruments are guided along complex anatomical paths and precisely inserted into the target tissue. The task of steering the instruments while avoiding collisions with tissues as they pass through complex anatomical structures places significant cognitive and mental stress on the medical personnel.

[0006] In some cases, surgical procedures may require re-insertion of medical instruments along previously traversed paths. Requiring medical personnel to manually control these repetitive paths each time can lead to significant fatigue and reduced surgical efficiency. If a surgical robotic system could automatically replicate existing paths and insert instruments in similar anatomical structures, surgical efficiency would be improved and surgical times would be reduced.

[0007] However, systematically implementing path replay poses significant technical challenges. Simply repeating recorded control commands, even for the same patient and the same internal path, can lead to unexpected collisions between medical devices and tissues. This is because the patient's internal environment is not a single rigid body but rather composed of flexible soft tissues. The soft tissue environment is not statically fixed, but constantly changes due to the patient's breathing, heartbeat, and other physiological factors.

[0008] External factors that alter the patient's internal environment include uncontrollable changes in the patient's breathing, heartbeat, and involuntary movements during the surgical procedure. These are referred to herein as "biologically induced changes." Even within the same anatomical structure, subtle changes in the path of the same anatomical structure due to biologically induced changes can lead to conflicts between medical devices and tissues when the same control commands are repeatedly played.

[0009] Therefore, there is a need for the development of an intelligent decision-making system that can safely reproduce recorded paths while minimizing the risk of collision by detecting and adapting to environmental changes of anatomical structures in real time.

[0010]

[0011] (Prior art literature)

[0012] (Patent Document)

[0013] Korean Patent No. 10-2196291

[0014] Korean Patent No. 10-2027422

[0015] Korean Patent Publication No. 10-2023-0113589

[0016]

[0017] The purpose of the present invention is to provide a control method and a surgical robot system capable of safely reproducing a recorded path of a medical device without risk of collision by detecting and adapting to environmental changes such as bio-induced deformation in real time.

[0018] A surgical robot system according to an embodiment of the present invention comprises: a steerable medical device; one or more actuators for driving the medical device; a memory, wherein the memory is configured to store recorded information of a path steered by the medical device within a body, the recorded information including visual data of an anatomical structure steered by the medical device and control data of the medical device; and a control device for receiving a path regeneration request command for regenerating a recorded path of the medical device, comparing and analyzing past recorded information stored in the memory with current recorded information of the medical device to determine whether to approve the path regeneration request command, and then executing the path regeneration request command; wherein the control device is characterized in that it includes a path regeneration judgment module programmed with a decision-making algorithm having a condition for rejecting the path regeneration request command.

[0019] In one embodiment, the control device may block input of a user's manual control command after approving the path regeneration request command and before the path regeneration request command is released, and may retrieve the control data stored in the memory and control the actuator to steer the medical device along the recorded path.

[0020] In one embodiment, the control device may analyze visual changes due to bio-induced deformation occurring in an anatomical structure to which the medical device is directed, and determine whether to approve the path regeneration request command based on the analysis results.

[0021] In one embodiment, the memory stores the record information in the form of time series data in which time indices are assigned to the visual data and the control data, respectively, and the stored record information may have a data structure configured such that the visual data and the control data having the same time indices are matched to each other.

[0022] In one embodiment, the surgical robot system further includes a record preprocessing module that preprocesses the record information stored in the memory, and the record preprocessing module can detect a pause section in which the medical device is not in motion in the temporal continuity of the visual data or the control data, and remove the visual data or control data of the detected pause section.

[0023] In one embodiment, the path regeneration determination module includes a first module programmed with a first decision-making algorithm that compares and analyzes past record information stored in the memory with current record information of the medical device in a first cycle; a second module programmed with a second decision-making algorithm that compares and analyzes past record information stored in the memory with current record information of the medical device in a second cycle; wherein the first decision-making algorithm and the second decision-making algorithm are based on different models and can independently determine whether to approve regeneration of the recorded path in different cycles.

[0024] In one embodiment, the path regeneration judgment module executes the first decision-making algorithm or the second decision-making algorithm when the path regeneration request command is received, and the first decision-making algorithm may be executed at a shorter cycle than the second decision-making algorithm and may be executed at a higher frequency per unit time.

[0025] In one embodiment, the path regeneration judgment module may block execution of the path regeneration request command if approval of the path regeneration request command is denied in either the first module or the second module.

[0026] In one embodiment, the first module may set a condition for rejecting the path regeneration request command based on a result determined in the first decision-making algorithm that the medical device is likely to collide with an anatomical structure when steered along the recorded path.

[0027] In one embodiment, the second module may analyze the visual similarity between the visual data of the current anatomical structure to which the medical device is directed and the recorded visual data in the second decision-making algorithm, and if the similarity is determined to be less than a reference value and thus dissimilar, a condition for rejecting the path reproduction request command may be set.

[0028] In one embodiment, the path reproduction judgment module is a short-term decision-making module that periodically determines whether to approve the path reproduction request command, and includes a first module programmed with a first decision-making algorithm, and the first decision-making algorithm has a neural network architecture including a Transformer model, and the Transformer model receives the visual data as a time-series sequence of images, receives record images, which are past recorded visual data, and replay images, which are current visual data, as input, and extracts a feature vector of an image in which an anatomical structure appears from each of the record images and the replay images, and evaluates a relationship between the feature vectors extracted from the past visual data and the current visual data to output correlation information.

[0029] In one embodiment, the first module may include a configuration for executing the first decision-making algorithm, a time series processing unit that generates a feature vector by passing each image from the record images, which are past recorded visual data, and the replay images, which are current visual data, through a convolutional layer and evaluates the relationship between the feature vectors to extract correlation information; and a decision-making unit that receives the correlation information and determines whether to approve the path playback request command based on a model that has been previously learned about whether a collision occurs between the medical device and the anatomical structure when a control command included in the control data is executed.

[0030] In one embodiment, the path reproduction judgment module is a long-term decision-making module that periodically determines whether to approve the path reproduction request command, and includes a second module programmed with a second decision-making algorithm; and the second decision-making algorithm includes a deep learning-based feature extraction model that extracts feature points considering the entire context of an image in which an anatomical structure appears from the visual data, and can determine similarity through matching between feature points extracted from the visual data recorded in the memory and the current visual data.

[0031] In one embodiment, the surgical robot system further includes an operating device that outputs the path regeneration request command to the control device, wherein the operating device includes an input unit that remains active in response to a continuous physical manipulation of a user; and an output unit configured to continuously output the path regeneration request command while the input unit is active, wherein the path regeneration request command can be stopped from being output as soon as the physical manipulation of the user is released.

[0032] In addition, the present invention is a processor encoded with a control command to operate one or more actuators to steer a medical device in a surgical robot system, the processor including a path regeneration request command for reproducing a recorded path of the medical device, a path regeneration judgment module that receives visual data of a path along which the medical device was steered in the past within the body and control data that drove the medical device from a memory, and analyzes the visual data of the current point in time of the medical device to determine whether to approve the path regeneration request command, wherein the path regeneration judgment module analyzes a visual change due to a bio-induced deformation occurring in an anatomical structure to which the medical device is steered, and rejects the path regeneration request command if there is a risk of the medical device colliding with the anatomical structure or if the current anatomical structure is dissimilar to the past anatomical structure by less than a predetermined similarity standard, thereby selectively reproducing a previously recorded path of the medical device in the body environment of the anatomical structure in which the bio-induced deformation occurred.

[0033] In addition, the present invention provides a control method performed by one or more processing devices configured in a surgical robot system, comprising: (a) a step of storing, in a memory, recorded information of a path along which a medical device is steered within a body, wherein the recorded information includes visual data of an anatomical structure steered by the medical device and control data of the medical device; (b) a step of receiving, in a control device, a path regeneration request command for regenerating the recorded path of the medical device; (c) a step of comparing and analyzing, in the control device, past recorded information stored in the memory with current recorded information of the medical device using a decision-making algorithm, and determining whether to approve the path regeneration request command based on a result of the comparative analysis; and (d) a step of controlling, in the control device, one or more actuators using the control data stored in the memory to steer the medical device along the recorded path; wherein step (c) is further characterized in that, when determining whether to approve the path regeneration request command, one or more conditions are set that can reject the path regeneration request command.

[0034]

[0035] The surgical robot system according to the present invention can automatically replay previously recorded paths of medical instruments. Rather than simply executing a user's path replay request command, the system analyzes environmental changes, such as patient-induced deformation, and selectively executes path replay only in situations where safety is ensured. Through this intelligent judgment function, the present invention can ensure the safe replay of recorded paths. Furthermore, by providing an automatic replay function for verified paths, the present invention significantly reduces the burden of repetitive manual manipulation on medical personnel while simultaneously enhancing surgical safety. This ultimately contributes to shortened surgical times, enhanced medical efficiency, and greater consistency in patient treatment outcomes.

[0036]

[0037] Figure 1 is an embodiment of a surgical robot system.

[0038] Figure 2 is a schematic diagram of a surgical robot system according to an embodiment of the present invention.

[0039] Figure 3 is a conceptual diagram illustrating problems that may arise due to bio-induced deformation during path regeneration.

[0040] Figure 4 is a block diagram of a surgical robot system according to one embodiment of the present invention.

[0041] Figure 5 is a flowchart of a path regeneration request command processing of a control device according to one embodiment of the present invention.

[0042] Figure 6 is a block diagram of a first module according to one embodiment of the present invention.

[0043] Figure 7 is a block diagram of a time series processing unit and a decision-making unit according to one embodiment of the present invention.

[0044] Figure 8 is a block diagram of a second module according to one embodiment of the present invention.

[0045] Figure 9 is an example of visual data for which feature point extraction and matching were performed in the second module of Figure 8.

[0046] Figure 10 is a conceptual diagram of reinforcement learning according to one embodiment.

[0047] Figure 11 is a block diagram of reinforcement learning applied to a first module according to one embodiment of the present invention.

[0048]

[0049] The various embodiments described in this document are exemplified for the purpose of clearly explaining the technical concepts of the present invention and disclosure, and are not intended to limit them to specific embodiments. The technical concepts of the present invention and disclosure include various modifications, equivalents, alternatives, and embodiments selectively combined from all or part of the embodiments described in this document. Furthermore, the scope of the technical concepts of the present invention and disclosure is not limited to the embodiments presented below or the specific descriptions thereof.

[0050] Terms used in this document, including technical or scientific terms, unless otherwise defined, may have the meaning commonly understood by one of ordinary skill in the art to which the present invention and disclosure pertain.

[0051] The expressions "includes," "may include," "comprises," "may have," "have," and "may have" used in this document imply the presence of a function, operation, or component as the target feature, and do not exclude the presence of other additional features. In other words, such expressions should be understood as open-ended terms that imply the possibility of including other embodiments.

[0052] The singular forms used in this document may include the plural form unless the context clearly indicates otherwise, and this also applies to the singular forms set forth in the claims.

[0053] As used herein, the expressions "A, B, and C," "A, B, or C," "A, B, and / or C," or "at least one of A, B, and C," "at least one of A, B, or C," "at least one of A, B, and / or C," "at least one selected from A, B, and C," "at least one selected from A, B, or C," "at least one selected from A, B, and / or C," and the like can mean each of the listed items or all possible combinations of the listed items. For example, "at least one selected from A and B" can refer to (1) A, (2) at least one of A, (3) B, (4) at least one of B, (5) at least one of A and at least one of B, (6) at least one of A and B, (7) at least one of B and A, and (8) both A and B.

[0054] The expression "based on" as used in this document is used to describe one or more factors that influence the decision, act of judgment, or action described in the phrase or sentence containing the expression, and this expression does not exclude additional factors that influence the decision, act of judgment, or action.

[0055] As used herein, the expression that a component (e.g., a first component) is “connected” or “connected” to another component (e.g., a second component) may mean that the component is directly connected or connected to the other component, as well as connected or connected via a new other component (e.g., a third component).

[0056] The expression "configured to" used in this document can have the meanings of "set to do", "having the ability to do", "changed to do", "made to do", and "capable of doing" depending on the context, and is distinct from the meaning of "consist".

[0057] Hereinafter, various embodiments of the present disclosure will be described with reference to the attached drawings. In the attached drawings and the description of the drawings, identical or substantially equivalent components may be assigned the same reference numerals. Furthermore, in the description of various embodiments below, duplicate descriptions of identical or corresponding components may be omitted, but this does not mean that the corresponding components are not included in the embodiments.

[0058] Surgical systems, such as those used in minimally invasive medical procedures, may include large and complex equipment to precisely control and operate relatively small tools or instruments.

[0059] A remotely operated surgical robot system with a single entry port can be used for various surgical procedures, allowing the use of various surgical instruments in a single system. Depending on the embodiment, the surgical robot system may be described as a remote surgical system capable of operating various medical instruments through a single entry port. The surgical robot system may also be described as a surgical robot system that handles the operation of a single medical instrument.

[0060] Depending on the characteristics of the shaft, the types of medical devices being operated can include overtubes, general surgical instruments, advanced surgical instruments, and camera instruments such as endoscopes. Medical devices can be controlled manually, computer-assisted, or remotely via an operator console.

[0061] A surgical robot system according to an embodiment of the present invention can perform surgery using a single entry port in various parts of a patient's body. Specifically, a medical instrument can be inserted through a patient's oral cavity, intercostal space, femoral region, or other natural openings or incisions in the body to perform the surgery. An overtube, which is an embodiment of the medical instrument, has multiple lumens formed therein, and an endoscope or a single-stage surgical instrument can be inserted through the lumens and then introduced into the body along with the overtube. Control from the operator console can be performed independently and remotely not only for the overtube but also for the surgical instrument inserted into the overtube.

[0062] The medical device referred to in this specification may collectively refer to an overtube and other surgical devices inserted into the overtube, and with respect to steering and control of the overtube, reference may be made to Korean prior registration patents Nos. 10-2740479 and 10-2684546 of the same applicant.

[0063] FIG. 1 illustrates an embodiment of a surgical robot system (1). Referring to FIG. 1, the surgical robot system (1) may include a positioning cart (3), a surgical instrument device (10), and an operator console (5). In an embodiment, the positioning cart (3) may be a mobile platform on which the surgical instrument device (10) is mounted. The positioning cart (3) may perform a function of precisely positioning the surgical instrument device (10) to a surgical site of a patient. The positioning cart (3) may be provided with a movable wheel at the bottom for access to an operating table. The positioning cart (3) may include a frame composed of a plurality of arms and links. The frame may be configured to enable the surgical instrument device (10) to move up and down, move toward and away from a patient. The frame of the positioning cart (3) may be configured in various forms in consideration of a kinematic structure that allows access to the operating table and approach the surgical instrument device (10) to the surgical site of a patient. A medical instrument may be mounted on the surgical instrument device (10). The surgical instrument device (10) may be provided in a form in which it is mounted on a frame with controlled degrees of freedom, as in the embodiment of FIG. 1, or may be provided in a form in which it is mounted on a fixed frame, as in the embodiment of FIG. 2. The medical instrument mounted on the surgical instrument device (10) may include an overtube and a surgical instrument.

[0064] The operator console (5) may refer to medical control equipment capable of confirming surgical-related images and controlling the positioning cart (3) and the surgical instrument device (10). The operator console (5) is a remote control device that controls the entire system and can be installed inside the surgical space or in a separate space. The operator console (5) may include one or more manipulators. One embodiment of the manipulator may be an input device such as a handgrip, a joystick, a trackball, a data glove, a trigger gun, or a manual controller. Another embodiment of the manipulator may be a voice recognition device or a touchscreen. Another embodiment of the manipulator may be a manipulator such as a clutch or a foot motion controller.

[0065] Although not shown in the drawings, depending on the embodiment, the surgical robot system (1) may further include a vision cart, which is an auxiliary imaging device. The vision cart can provide visual prompts and step-by-step surgical guidance through a touchpad or touchscreen monitor. In addition, medical imaging devices for surgical assistance, such as CT or X-ray, may be linked to the system, if necessary.

[0066] In an embodiment, the system configuration equipment including the positioning cart (3) and the operator console (5) may be equipped with one or more processors. The processor may process information input from various sensors. For example, the sensor information may include the state of the drape, the state of the positioning cart (3), the state of the surgical instrument device (10), and the state of the medical instrument.

[0067] In an embodiment, the processor may be mounted on a positioning cart (3), an operator console (5), a surgical instrument device (10), or may be mounted on each piece of equipment. The processor may be provided as a separate device, and in this case, data may be processed through wired or wireless communication with each piece of equipment constituting the surgical robot system (1).

[0068] Figure 2 is a schematic diagram of a surgical robot system (1) according to an embodiment of the present invention.

[0069] Referring to FIG. 2, the surgical instrument device (10) may include a medical instrument (100) and an actuator (200). The operator console (5) may include a display device (50), an input device (51), an operating device (52), a memory (400), and a control device (500).

[0070] The medical device (100) can be implemented as an elongated shaft shape having a steerable joint structure, a flexible structure, or a bendable structure. The medical device (100) can be controlled in three or more degrees of freedom, including pitch, yaw, and rolling, through a wire-based drive mechanism. As a specific embodiment, the medical device (100) can be implemented in a three-degree-of-freedom or four-degree-of-freedom configuration. In a three-degree-of-freedom configuration, one-bending (pitch, up / down direction), rolling, and forward / backward motions are implemented. In a four-degree-of-freedom configuration, two-bending (pitch, up / down direction), yaw (left / right direction), rolling, and forward / backward motions are implemented. Among the above degrees of freedom, the bending motion can be controlled through a wire mechanism.

[0071] In one embodiment, the medical device (100) is implemented in the form of an overtube, capable of guiding the surgical instrument into the body. An endoscopic probe can be inserted into the overtube via a separate shaft, or an endoscopic camera can be installed within the overtube itself to capture real-time images of the steered path. In this case, the inserted surgical instrument can be controlled independently of the overtube.

[0072] In another embodiment, the medical device (100) may be implemented as a single shaft that integrates an endoscope and a surgical instrument. For example, an integrated medical device comprising a shaft that outputs a basket or laser and a camera at its end may be utilized as a surgical device for removing kidney stones. Fig. 2 illustrates one such embodiment, a surgical device for removing kidney stones, as an example of the medical device (100).

[0073] As another embodiment, the medical device (100) may be implemented in a form that combines tissue collection for biopsy procedures with image acquisition functions. In this case, the medical device (100) may include a basket capable of collecting microscopic tissue samples, a mechanical mechanism, and a miniature camera that provides real-time images of surrounding tissue.

[0074] The image acquired through the camera provided in the medical device (100) can be configured as visual data in the form of an image or recorded image and can be transmitted to and stored in the memory (400).

[0075] The actuator (200) is a core component for driving the medical device (100), and may be equipped with one or more motors and encoders. The motors mounted on the actuator (200) can provide precise driving force for steering the medical device (100) in various directions. Individual actuators (200) can be designed to implement a single movement along a specific axis, and multiple actuators (200) can be organically combined to form an integrated motor mechanism. Through the cooperative operation of these motor mechanisms, the medical device (100) can be precisely steered along a complex anatomical path. The actuator (200) can be provided on the surgical device device (10).

[0076] In one embodiment, the actuator (200) may include a high-precision servo motor and an optical or magnetic encoder to provide real-time feedback on the position and orientation of the medical device (100). Such a feedback system may be essential for controlling the movement of the medical device (100) with micrometer-level accuracy. In another embodiment, the actuator (200) may be configured based on a stepper motor, enabling fine steering of the medical device (100) through precise, step-by-step movements. This configuration may be particularly suitable for manipulation in narrow cavities or around sensitive tissues.

[0077] The operator console (5) may include a display device (50), an input device (51), an operating device (52), a control device (500), and a memory (400). The devices exemplified in the operator console (5) may refer to the structures of the devices presented in the aforementioned Korean Patent Publication No. 10-2024-0009905 or Application No. 10-2024-0170032.

[0078] The display device (50) can display real-time visual data captured by an endoscopic camera mounted on a medical device (100) in high resolution. In addition to the visual data, the display device (50) can also display a user's operating environment and various control interfaces in an integrated manner. This display device (50) can support medical staff to intuitively monitor and control the surgical process. In one embodiment, the display device (50) is implemented as a medical display supporting 4K or 8K resolution, so that even the minute features of anatomical structures can be clearly expressed. In another embodiment, the display device (50) can be configured to include a touchscreen function so that medical staff can control the system by directly manipulating the screen. In addition, the display device (50) can be configured as a single display or multiple monitors to simultaneously display various information such as biosignals, patient information, and system status along with the endoscopic image.

[0079] The input device (51) can be defined as a precision manipulation tool operated by the user, a medical professional, by hand. The input device (51) can be provided in an ergonomic form, such as a trackball, a multi-axis joystick, or a precision gimbal, through which the medical professional can intuitively and accurately control the medical device (100). In one embodiment, the input device (51) is configured with multiple manipulation devices, one for the left hand and one for the right hand, to enable precise manipulation using both hands.

[0080] In another embodiment, the input device (51) may be implemented to include a haptic feedback function so that the user can feel the resistance of the tissue that the medical device (100) contacts with his or her hand. This haptic feedback function can help determine the physical characteristics of the tissue that are difficult to determine with visual information alone. The input device (51) can precisely detect the user's manual control commands, convert them into digital signals, and transmit them to the control device (500), thereby performing a key interface role that precisely controls the movement of the medical device (100).

[0081] The operating device (52) can function as a specialized interface device that outputs a path regeneration request command to the control device (500). The operating device (52) can be provided in the form of a foot pedal, clutch, or other means that a user can control with his or her feet. The operating device (52) can be configured separately from the input device (51) to provide specialized operating functions.

[0082] The operating device (52) may include an input unit that remains active in response to a user's continuous physical manipulation, and an output unit configured to continuously output a path regeneration request command while the input unit is active. The configuration of the operating device (52) may cause the output of the path regeneration request command to stop as soon as the user's physical manipulation is released.

[0083] In the embodiment of Fig. 2, the operating device (52) is implemented in the form of a foot pedal. Although not explicitly depicted in the drawing, the input unit may be realized as an internal configuration of the foot pedal, including a sensor and a signal processing circuit that detects pressure applied to the foot pedal and converts it into an electrical signal. In other words, the input unit may be implemented as a signal processing circuit of the foot pedal. When the user presses the foot pedal with a pressure exceeding a certain level, the input unit may generate an activation signal.

[0084] In this embodiment, a configuration is basically described in which an active signal is generated when the user steps on the foot pedal, and a stop signal (or cessation of the active signal) is generated when the user takes his / her foot off the foot pedal. However, as another embodiment of the present invention, the operating device (52) may be implemented in an alternative manner, that is, a configuration in which a stop signal is generated when the user steps on the foot pedal, and an active signal is maintained when the user takes his / her foot off the foot pedal, and such a modified embodiment may also be included in the technical scope of the present invention.

[0085] Likewise, an output unit not specified in the drawing may include a communication interface circuit that converts an activation signal generated from the input unit into a standardized command signal that can be recognized by the control device (500) and transmits it. The output unit, like the input unit, may be realized by the internal configuration of the foot pedal. That is, the output unit may be implemented by a signal processing circuit of the foot pedal. In the present embodiment, a signal activated through the foot pedal may be interpreted and processed by the control device (500) as a path regeneration request command.

[0086] As an example, the operating device (52) may include a multi-stage pressure detection function, allowing the speed and intensity of path regeneration to be adjusted based on the amount of pressure applied to the foot pedal. This variable control function allows the medical professional to finely adjust the speed of path regeneration depending on the situation, thereby enhancing the accuracy and safety of the procedure.

[0087] In one embodiment, the path regeneration request command may be activated only while the user continues to press the foot pedal, and may function as a dead-man switch mechanism, where the command output ceases as soon as the user lifts their foot from the foot pedal. This safety mechanism can ensure that a medical professional can immediately stop the automatic path regeneration and revert to manual control in the event of an unexpected situation.

[0088] The memory (400) is configured to store record information of a path steered by the medical device (100) within the body, and the record information may include visual data of an anatomical structure (20) through which the medical device (100) is steered and control data of the medical device (100).

[0089] The memory (400) stores record information in the form of time-series data, in which time indices are assigned to each visual data and control data, and the stored record information may have a data structure in which visual data and control data having the same time indices are matched to each other. This time-synchronized data structure can ensure an accurate correspondence between an anatomical image at a specific point in time and the control command at that moment.

[0090] In this embodiment, the recorded information may collectively refer to all types of data acquired by the medical device (100) while traveling along the internal path (21) of the anatomical structure (20). The recorded information may be broadly divided into two main categories: visual data and control data.

[0091] As an example of visual data, the image data may include high-resolution image data of an anatomical structure (20) and an internal passage (21) photographed by an endoscopic camera mounted on a medical device (100). As another example, the image data may include continuous recording image data of an anatomical structure (20) and an internal passage (21) photographed by an endoscopic camera, which may be captured at a frame rate of 30 fps or higher to enable smooth visual reproduction. As yet another example, the image data may include additional image data of an anatomical structure (20) acquired from external medical imaging equipment such as ultrasound, X-ray, CT, MRI, etc.

[0092] As an example of the control data, precise control signal information of each motor driving the actuator (200) may be included. This may include parameters such as the rotation speed, direction, and acceleration of the motor. In another example, it may be detailed information such as the actual position, rotation angle, and torque of the motor recorded in the encoder of the motor. In addition, the control data may include low-level command data such as current values, voltage values, and PWM (Pulse Width Modulation) signals that drive the motor, which may be essential elements for accurately reproducing the movement of the medical device (100).

[0093] Fig. 4 is a block diagram of a surgical robot system (1) according to one embodiment of the present invention. Fig. 4 illustrates the relationship between one or more processing devices required to perform a path regeneration request command.

[0094] The record preprocessing module (300) can preprocess the record information stored in the memory (400). This module is a specialized component responsible for preprocessing the collected record information and can function in two main operating modes.

[0095] In the first operating mode, the record information is first stored in the memory (400), and then the record preprocessing module (300) can perform a preprocessing operation on the stored data in a postprocessing manner. In the second operating mode, before the data is stored in the memory (400), the record preprocessing module (300) receives visual data from the endoscope camera and control data of the actuator (200) in real time, performs preprocessing immediately, and then transmits the already preprocessed record information to the memory (400) for storage.

[0096] The record preprocessing module (300) may be implemented as a processor or a dedicated signal processing chip (DSP), and depending on the system configuration, may be integrated into the operator console (5) or directly mounted on the surgical instrument device (10). In the latter case, the preprocessed data may be transmitted to the memory (400) via a high-speed communication system.

[0097] As an example, the record preprocessing module (300) may apply image processing algorithms such as noise removal, image stabilization, and resolution optimization to visual data. In addition, signal filtering, outlier removal, and data compression may be performed on control data.

[0098] The recording preprocessing module (300) can precisely detect idle sections, which are sections unrelated to the path operation of the medical device (100), in the temporal continuity of visual data or control data, and perform the function of removing the visual data and control data of the detected idle sections. This idle section detection and removal process can be understood as a unique preprocessing process of the recording preprocessing module (300) specifically required for efficient playback of the recorded path.

[0099] A path that a medical device (100) has been steered through may have pauses for various reasons. For example, during a surgical procedure, a medical professional may temporarily stop the path to explore the path, or may stop the medical device (100) at a specific location to closely observe an anatomical structure (20). From a control data perspective, these pauses may appear as periods in which no actuator control commands are input. Meanwhile, from a visual data perspective, pauses may appear in two ways. First, when the medical device (100) is completely stopped, it may appear as a static period in which the visual information hardly changes. Second, when a medical professional rotates the medical device (100) in place or makes slight movements to explore the structure, it may appear as a period in which the visual information changes but the medical device (100) does not actually advance along the path.

[0100] Therefore, the idle section detection logic of the record preprocessing module (300) can accurately detect a state in which the medical device (100) is not actually driving along the path by analyzing changes in visual data and changes in control data by connecting them with logical conditions of OR or AND. Specifically, in the case of visual data, a case in which the difference value between consecutive frames is below a set threshold value, and at the same time, in the case of control data, a case in which the amount of change in the actuator driving command is below a specific threshold value can be determined as an idle section.

[0101] As an example, the recording preprocessing module (300) can identify sections of the video where no significant movement occurs through motion vector analysis, and can determine idle sections by cross-checking these sections with the stagnant state of the control data. For sections detected as idle sections, a noise removal process can be performed to delete all matched visual data and control data by referencing the time index of the corresponding section. This idle section removal process can improve the efficiency and continuity of path playback by eliminating unnecessary downtime during playback of the recorded path. It can also contribute to optimizing data storage space.

[0102] The control device (500) receives a path regeneration request command for regenerating a recorded path of the medical device (100), compares and analyzes past record information stored in the memory (400) with current record information of the medical device (100), determines whether to approve the path regeneration request command, and then executes the path regeneration request command. The control device (500) may be implemented in the form of physical hardware such as a controller, a processor, a microprocessor, a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit). The control device (500) may include a memory, which is a non-transitory computer-readable storage medium for storing logic circuits and software instructions, and these instructions are configured to perform specific functions of the control device (500) described herein when executed by the processor.

[0103] In an implementation form, the memory and processor may be implemented as physically separate semiconductor circuits. In another implementation form, the memory and processor may be implemented together in a single integrated semiconductor circuit (SoC: System on Chip). The processor may be implemented as a single processor having a single-core or multi-core structure, or as a distributed processing architecture including multiple individual processors.

[0104] In particular, the components referred to as “control device” and “module” in the present invention are implemented as devices having a specific physical hardware structure as described above, and may mean specific circuits or programmed electronic devices having an actual physical configuration rather than simple functional blocks. For example, the “path regeneration judgment module (501)” to be described later may be implemented as a specialized semiconductor circuit implementing a neural network architecture, an FPGA (Field Programmable Gate Array), or a combination of a processor and memory programmed to execute a specific algorithm.

[0105] The control device (500) may include a high-performance processor encoded with control commands for operating the actuator (200). The control device (500) may transmit the generated control commands to the actuator (200) to precisely drive each motor. As a key feature of the control device (500), unlike general control commands, an exceptional processing process may be applied to a path regeneration request command. That is, a control command as a response to a path regeneration request command undergoes a special judgment process to evaluate the safety and suitability of the command before outputting it to the actuator (200).

[0106] The control device (500) can precisely analyze visual changes caused by bio-induced deformation occurring in an anatomical structure (20) to which a medical device (100) is directed, and determine whether to approve a path regeneration request command based on the analysis results. Here, bio-induced deformation may refer to changes in the position, shape, or characteristics of an anatomical structure caused by breathing, heartbeat, elasticity changes in tissues, flow of body fluids, etc.

[0107] Figure 3 is a conceptual diagram clearly visualizing problems that may arise due to bio-induced deformation during path regeneration. Figure 3 (a) conceptually represents a previously recorded path (P1) that successfully reached the destination point (T) along the internal path (21) of an anatomical structure (20).

[0108] In contrast, Fig. 3 (b) illustrates a potentially dangerous situation that can arise when attempting to reproduce the same previously recorded path (P1). It indicates that, as bio-induced deformations occur in the anatomical structure (20) due to respiration, heartbeat, and tissue elasticity changes, collisions with the structure may occur at multiple points (A1, A2, A3) when moving along the same path as in the past.

[0109] These bio-induced deformations, even at a minimal level, can alter the geometric characteristics of the internal path (21), which can lead to cascading problems when simply reproducing the recorded path. As shown in Fig. 3(b), if a collision occurs at point A1 due to the narrowed path caused by bio-deformation, this can sequentially lead to additional collisions at points A2 and A3. This cascading collision can seriously threaten patient safety and damage medical devices.

[0110] Accordingly, the present invention provides a sophisticated decision-making algorithm that detects and evaluates bio-induced deformations of anatomical structures in real time, enabling the system to continuously determine safety on its own even when a user requests path replay.

[0111] Therefore, a key criterion in the decision-making process of the control device (500) is whether the bio-induced deformation of the anatomical structure may have a negative impact on the safety or effectiveness of the medical device (100) when reproducing the recorded path. To assess this impact, the control device (500) can quantitatively analyze the difference between the current anatomical situation and the previously recorded situation through an algorithm. In one embodiment, the control device (500) can measure the change in position, size, surface characteristics, etc. of the anatomical structure between the past recording time and the current time, and automatically reject the path reproducing request command if such change exceeds a set safety threshold.

[0112] The control device (500) can completely block the user's manual control command input during the period from the time the path regeneration request command is approved until the command is released. During this period, the control device (500) sequentially retrieves control data stored in the memory (400) and precisely drives the actuator (200), thereby accurately steering the medical device (100) along the recorded path.

[0113] This control priority switching mechanism can serve as an important safety feature to ensure the stability and accuracy of the path regeneration mode. Specifically, the control device (500) can prevent unintended steering of the medical device (100) during the process of regenerating the recorded path by ignoring or blocking all active control commands from the user via the input device (51) while the path regeneration request command is approved and being executed.

[0114] In one embodiment, the control device (500) may be configured to temporarily buffer or ignore all signals received from the input device (51) while the path regeneration mode is activated. In another embodiment, the input device (51) itself may be physically or logically disabled to fundamentally prevent a user from unintentionally inputting control signals during path regeneration.

[0115] This manual control blocking feature can enhance the safety of the procedure by increasing the consistency and predictability of the path regeneration process, while eliminating potential risks caused by inadvertent user manipulation. Furthermore, this feature can free the medical professional from operating the medical device (100) during path regeneration, allowing them to focus on other important aspects of the procedure.

[0116] Importantly, this manual control blocking state automatically ends as soon as the path regeneration request command is released, allowing medical personnel to quickly transition to manual control mode whenever necessary. This flexible control mode transition can serve as a crucial safety feature in responding to emergencies or unexpected anatomical changes.

[0117] The control device (500) may include a path regeneration judgment module (501). A decision-making algorithm having conditions for rejecting a path regeneration request command may be programmed into the path regeneration judgment module (501). To effectively implement the function of judging a path regeneration request command, the path regeneration judgment module (501) may be composed of a first module (510) and a second module (520) having different characteristics and purposes.

[0118] The first module (510) may be programmed to execute a first decision-making algorithm that compares and analyzes past record information stored in the memory (400) with the current record information of the medical device (100) in a first cycle. The first module (510) may function as a short-term decision-making module that performs decision-making in a rapid cycle. In the present embodiment, the first cycle may be set to a frame rate of 10 Hz to 50 Hz, which means that decisions are made 10 to 50 times per second.

[0119] This frame rate is set based on the time axis of the entire recorded path, and the first decision algorithm can be automatically executed at every set first cycle of the entire path. For example, if the first cycle is set to 10 Hz, the first decision algorithm can execute the decision once every 0.1 seconds. This enables near-real-time safety assessment.

[0120] The core decision-making purpose of the first module (510) is to detect and evaluate in advance the possibility of collision with an anatomical structure (20) when the medical device (100) travels along the recorded path (21) of the internal body.

[0121] A second decision-making algorithm that compares and analyzes past record information stored in the memory (400) with the current record information of the medical device (100) in a second cycle may be programmed into the second module (520). The second module (520) may function as a long-term decision-making module that periodically determines whether to approve a path regeneration request command.

[0122] In this embodiment, the second cycle may refer to waypoint nodes strategically sampled from the entire recorded route. These waypoints may be set in units longer than the first cycle units of the recorded route and may serve to indicate important points on the route. It is noteworthy that the second cycle in this embodiment does not necessarily mean repetitive points at equal intervals. Unlike the first cycle, the second cycle is a concept that indicates that decision-making is performed at waypoints organically set according to the characteristics of the route, and may mean a specific point on the recorded route rather than a unit of time or frequency (Hz). However, the second cycle may be expressed in comparison to the first cycle, which means repetitive points at equal intervals, by forming the time from a specific point on the route to the next specific point longer than the first cycle.

[0123] For example, waypoints may be set at singular points such as a portion of a recorded path where a change in path occurs, a portion where the moving speed of a medical device (100) changes rapidly, a portion entering a complex anatomical structure such as a bronchial bifurcation or a blood vessel intersection, a boundary point where tissue properties change, or near a target point requiring surgical intervention.

[0124] These waypoints can be directly designated by the medical professional during the surgical planning phase, but the system can also automatically detect and set important points as waypoints by analyzing geometric changes in the path, changes in speed, and anatomical landmarks.

[0125] Due to the nature of waypoints, the path segment between consecutive waypoint nodes is formed to be larger than the unit in which the first cycle is executed. Therefore, the second decision-making algorithm is executed at a fundamentally different cycle from the first decision-making algorithm, and is executed only when the medical device (100) reaches the set waypoint node, so it can be executed significantly less frequently than the first decision-making algorithm.

[0126] As an example, the second module (520) can assess the overall context and structural similarity between the current anatomical environment and the environment at the time of past recording at each waypoint, thereby providing a more comprehensive assessment of the sustainability of the path replay. This can serve as an important safety mechanism complementing the high-frequency collision detection of the first module (510).

[0127] In summary, the first decision-making algorithm and the second decision-making algorithm set in the path determination module (501) according to the present embodiment are based on different models, and can independently determine whether to approve the reproduction of a path recorded at different cycles.

[0128] In one embodiment, when a route regeneration request command is received, the path regeneration judgment module (501) executes the first decision-making algorithm with priority over the second decision-making algorithm, and the first decision-making algorithm is executed at a shorter cycle than the second decision-making algorithm, so that it can be executed at a higher frequency per unit time.

[0129] Figure 5 is a flowchart of a route regeneration request command processing of a control device (500) according to one embodiment of the present invention. This can also be understood as a procedural flow of a control method for route regeneration performed by a processor of the control device (500).

[0130] First, a step (S10) of storing the recorded information of the path that drove the medical device (100) for path regeneration may be performed. In this step, the recorded information is stored in the memory (400), and a preprocessing process such as removing idle sections, as in the aforementioned embodiment, may be applied.

[0131] Thereafter, when the user physically operates the operating device (52), a path regeneration request command can be received by the control device (500). Upon receiving the command, the control device (500) performs a step of initializing the visual data image index of the recorded information, which can be viewed as an initialization process of internal parameters for decision-making. For example, a variable (nStep) indicating a path step can be initialized to 0. After initialization, the control device (500) can check whether the operating device (52) is continuously operated and the activation state of the path regeneration request command is maintained.

[0132] If the route regeneration request command is activated, a step (S30) may be executed in which the route regeneration judgment module (501) determines whether to approve the route regeneration command. During this process, the first decision-making algorithm of the first module (510) may be executed preferentially. The first decision-making algorithm is a short-term collision inference model that is executed at high frequency to assess immediate collision risk.

[0133] If the first decision-making algorithm determines that the path replay request command has been approved, the next step is to evaluate whether the medical device (100) has reached the waypoint node. If the waypoint has been reached, the second decision-making algorithm of the second module (520) may be executed. The second decision-making algorithm, a long-term inference model, can evaluate the overall similarity between the current anatomical environment and the environment recorded in the past.

[0134] If the route playback request command is also approved by the second decision-making algorithm, the process can proceed to step S40, where the route playback is executed. While the route playback is being executed, the variable (nStep) parameter is sequentially increased, and at each step, the variable (nStep) is evaluated to determine whether it has reached the final index of the recorded route. This is the end index review step, which determines whether the route playback is complete.

[0135] If the end index is reached, the path playback is completed and the process is terminated. If it is not reached, the activation status of the path playback request command is checked again and the approval decision step (S30) can be repeated.

[0136] The example of Fig. 5 specifically illustrates a control method when the medical device (100) reaches a waypoint, which is the second cycle, and both the first and second decision-making algorithms are executed. Whether the medical device (100) has reached the waypoint can be reviewed during the execution of the long-term inference model, and in general sections where the waypoint has not been reached, only the short-term collision inference model can be executed to make an approval decision.

[0137] In this embodiment, a decision-making algorithm is executed at a specific, predefined cycle. After the decision-making algorithm is executed, the recorded route can be continuously played until the next cycle is reached, provided that the route playback request command is activated. When a waypoint is reached, both the first decision-making algorithm and the second decision-making algorithm can be executed, and if any of the decision-making algorithms determines that the route playback request command cannot be approved, the route playback request command is rejected. The first and second decision-making algorithms are generally configured so that the first decision-making algorithm is executed first, but as another embodiment of the present invention, a configuration in which the second decision-making algorithm is executed first is also included within the technical scope of the present invention. Each of the above algorithm modules can be executed in parallel within a single processor or by utilizing multiple processor cores, and the judgment result of each module can be utilized to determine whether to grant final approval based on a logical AND or OR condition.

[0138] Below, the decision-making algorithm programmed in the path regeneration judgment module (501) is described in detail. The path regeneration judgment module (501) can block the execution of the path regeneration request command if approval of the path regeneration request command is rejected in either the first module (510) or the second module (520).

[0139] The first module (510) may set a condition for rejecting a path regeneration request command based on a result determined in the first decision-making algorithm that the medical device (100) is likely to collide with an anatomical structure when steered along the recorded path.

[0140] The first decision-making algorithm is composed of an advanced neural network architecture including a Transformer model, and the Transformer model can input two types of visual data (record images and replay images) having time-series characteristics. The first decision-making algorithm can analyze and output visual correlation information between these two types of visual data. As an embodiment of the present invention, the Vision Transformer model among the Transformer series can be specifically applied to the first decision-making algorithm. However, the technical idea of ​​the present invention is not limited thereto, and various image processing models based on neural network architectures that can effectively extract feature point information from an image through a convolution layer or an attention mechanism can be applied instead.

[0141] Figure 6 is a block diagram of a first module (510) according to one embodiment of the present invention. Referring to Figure 6, the first module (510) is configured to execute a first decision-making algorithm and may include a time series processing unit (511) and a decision-making unit (512).

[0142] Record images may refer to past visual data stored in memory (400), and replay images may refer to visual data acquired by a medical device (100) at the current location. The time series processing unit (511) may receive and process such past visual data (record images) and current visual data (replay images).

[0143] The time series processing unit (511) may be architecturally composed of a transformer encoder (5110) and a transformer decoder (5111). The transformer encoder (5110) is a model that deeply analyzes past visual data (record images), and the transformer decoder (5111) can be viewed as a model that analyzes current visual data (replay images) and evaluates the relationship with past data.

[0144] The processing process of the time series processing unit (511) is configured as follows. First, when visual data (record images), which are past record information, are input, the time series processing unit (511) performs an embedding process that generates tokens from an image sequence. Here, a token refers to an E-dimensional feature vector generated by passing each image in the image sequence through a convolutional layer. If there are S frames in the sequence, S corresponding tokens are generated. The embedding process can also be performed in parallel in the same manner for current visual data (replay images).

[0145] Once the embedding process is complete, the Transformer encoder (5110) of the time series processing unit (511) can execute a self-attention mechanism. The self-attention mechanism is an attention extraction algorithm that evaluates the relative importance of specific tokens within an image sequence, and can perform the function of selecting tokens with high informational value or semantic significance. The feature vector processed by the Transformer encoder (5110) can be passed to the Transformer decoder (5111) for subsequent processing.

[0146] The transformer decoder (5111) can first perform self-attention on the current visual data (replay images) to identify tokens with high semantic significance within the image sequence. Thereafter, a cross-attention mechanism can be executed to evaluate the relationship between the feature vectors of past visual data received from the transformer encoder (5110) and the feature vectors extracted from the current visual data. The cross-attention mechanism is a model that evaluates the relationship between the feature vectors extracted from two image sequences, through which visual similarity and correlation information between the two sequences can be extracted. The correlation information can be a three-dimensional tensor (BxSxE) composed of S E-dimensional vectors for each batch. The correlation information extracted in this way is transmitted to the decision-making unit (512) for final decision-making.

[0147] The decision-making unit (512) can determine whether to approve a path regeneration request command based on a model that has learned in advance whether a collision occurs between a medical device (100) and an anatomical structure when executing a control command included in the control data.

[0148] Fig. 7 is a block diagram of a time series processing unit (511) and a decision-making unit (512) according to another embodiment. Fig. 7 illustrates a self-attention mechanism to which a Swin transformer model is applied. Fig. 7 (a) is a conceptual diagram visually expressing the concept of self-attention executed in the transformer module of the time series processing unit (511), and Fig. 7 (b) illustrates a flowchart of the learning process and actual judgment and output processing performed in the decision-making unit (512).

[0149] Referring to Fig. 7 (a), an example in which an image of visual data is evenly divided into nine patches during the embedding process can be confirmed. Afterwards, when the self-attention mechanism is applied, the relationship between each patch is deeply analyzed, and through this, only important patches (indicated in white in the drawing) with high information value are selectively extracted. In one embodiment, the transformer module that performs self-attention in the time series processing unit (511) can be implemented in an encoder-decoder manner based on the Generative Pre-trained Transformer (GPT) architecture. In this structure, the encoder is responsible for processing the historical data of the past path, and the decoder can be responsible for interpreting the information for path reproduction in the current situation. The output of the transformer module can have a multidimensional tensor structure in the form of BxSxE, where B represents the batch size and represents the number of data samples processed simultaneously. E is the embedding dimension, which represents the dimension of the vector representing each patch, and S represents the number of images or tokens or patches in the sequence. Therefore, the output can be represented as a 3D tensor composed of S vectors with dimension E for each batch. Meanwhile, the input format of the transformer module can be configured in a multidimensional format of BxSxCxHxW. The input can receive S consecutive 1-channel (1ch) grayscale images of size 224x224. In a simplified embodiment that considers more computational efficiency, the output of the transformer module can also be represented as a 2D matrix in the form of ExS. In this case, S can be the number of images or tokens or patches in the sequence, and each patch or token can be represented as a vector of dimension E. This 2D representation method can provide sufficient information representation power while reducing computational complexity in a single batch processing scenario.Such tensor or matrix-type outputs can be utilized as core data for analyzing the characteristics of learning patches and determining the likelihood of collision in the subsequent decision-making unit (512). The critical patches selected in this manner are defined as relevance information and transmitted to the decision-making unit (512), which can then use a binary classifier to ultimately evaluate the situation as either collision (0) or safety (1).

[0150] Figure 7 (b) describes in more detail the internal structure and operating principles of the decision-making unit (512). The decision-making unit (512) operates based on previously accumulated learning data, and the processing flow of the learning stage (train) is illustrated on the left side of Figure 7 (b).

[0151] In the learning phase, correlation information extracted from record images and replay images is input to a decision maker, and learning can proceed to determine whether to execute a control command based on this. The output of the transformer module is correlation information including features for determining the possibility of collision according to a specific action, and can be input in the form of a feature point matrix such as a 3D tensor or a learning patch such as the present embodiment. The decision maker is trained by judging the appropriateness of executing the control command based on this. Through the learning process, the decision maker can acquire the ability to make safe control decisions in similar visual situations. This learning process is performed based on a sequence of images according to a time series, and reinforcement learning techniques can be applied.

[0152] During the learning process, if the path is safely continued based on the attributes of the relevant association information, the control command is learned to be followed. Conversely, if a collision occurs, the control command is not learned to be not executed. This learning can be performed by constructing a virtual environment (env) based on the presence or absence of a collision. The classifier has pre-learned information that determines the presence or absence of a collision. This pre-learned classifier is used to train the decision maker. As the learning process is repeated, the decision maker systematically learns the characteristic attributes of the association information and the situations in which collisions occur. As a result of this secondary learning, information can be generated regarding the differences in the association information of the replay image and the attributes of the association information of the record image. When determining whether to approve an actual path playback command, the deploy process, illustrated on the right side of Figure 7 (b), is executed. The decision maker compares the input association information and infers whether the differences in the attributes match the learned results related to the occurrence of a collision. Based on this, the decision maker can make a final judgment as to whether the current situation is a collision (0) or a safety (1).

[0153] Figure 8 is a block diagram of a second module (520) according to one embodiment of the present invention.

[0154] The second module (520) may be configured to analyze the visual similarity between the current anatomical structure to which the medical device (100) is directed and the recorded visual data in the second decision-making algorithm, and if the similarity is determined to be dissimilar and less than a reference value, a condition for rejecting the path reproduction request command may be set. The term "visual similarity" as used herein may be defined as including not only comparative analysis of visual images of anatomical structures, but also measurement of structural similarity between spatial mapping paths generated by processing visual data through SLAM (Simultaneous Localization and Mapping) or a similar algorithm. This is a concept that goes beyond simple image pixel-based comparison and encompasses topological and geometric similarity analysis of paths based on feature points extracted from visual data and their spatial arrangement.

[0155] The second decision-making algorithm includes a deep learning-based feature extraction model that extracts feature points considering the entire context of an image in which an anatomical structure appears from visual data, and can determine similarity through matching between feature points extracted from the visual data recorded in the memory (400) and the current visual data.

[0156] Figure 8 is a block diagram of a second module (520) according to one embodiment of the present invention. The second module (520) is configured to execute a second decision-making algorithm and may include a feature point extraction module (521) and a feature point matching module (522).

[0157] As described above, the second module (520) executes a process when reaching a node of the second cycle, the waypoint. The second module (520) receives and processes past recorded visual data (record images) and current visual data (replay images) stored in the memory (400).

[0158] In one embodiment, unlike the first decision-making algorithm, the second decision-making algorithm can perform analysis on a single image rather than a sequence-based one. Accordingly, one visual data (record image, replay image) from among past recorded visual data (record images) and current visual data (replay images) can be input to the feature extraction module (521).

[0159] This differs significantly from the approach of module 1 (510). Module 1 (510) analyzed the recorded image sequence from the past to the present, n steps from the time corresponding to the current control command. In contrast, module 2 (520) analyzes only a single image whose time exactly corresponds to the current control command.

[0160] This single image-based approach focuses on comprehensively understanding the status and characteristics of the overall anatomical structure, which can serve to complement the local collision detection capabilities of the first module (510).

[0161] The feature extraction module (521) can perform an advanced deep learning-based feature extraction function. In one embodiment, the feature extraction module (521) may use a pretrained SuperPoint model. In principle, the algorithm model configured in the feature extraction module (521) can be a network capable of extracting accurate feature points from an image.

[0162] However, while classical feature extraction algorithms (e.g., SIFT, ORB, FAST) primarily extract features based on local data changes (e.g., brightness changes, corners, etc.), the feature extraction network used in this embodiment should preferably be one that comprehensively reflects the structural features of the entire image to extract feature points. This approach can consider the overall context of anatomical structures, making it more suitable for medical image analysis. Specifically, the network can extract and provide the locations of feature points and feature information at those locations in vector form.

[0163] In one embodiment, the input data of the feature point extraction module (521) may have a tensor structure in the form of Bx1xHxW, which may be composed of one 224×224 grayscale resolution image. Here, B represents the batch size, and H×W represents the resolution of the image, which is set to 224×224 in this example.

[0164] The output of the feature point extraction module (521) may be composed of keypoints and descriptors. The keypoint output data may have a tensor structure in the form of B×N×2, where N is the number of extracted feature points, and each feature point is expressed as a two-dimensional coordinate (row, column). These coordinates indicate the point where the feature point is located in the image.

[0165] The descriptor output data can have a tensor structure of the form B×N×D, which means a D-dimensional feature vector corresponding to each of the N keypoints. By default, D is set to 256 dimensions, which means that the unique characteristics of each feature point are expressed with 256 numbers. The value of N can vary depending on the complexity of the image, but the important thing is that the number of keypoints and descriptors is always set to be the same.

[0166] The pretrained SuperPoint model forms a 256-dimensional feature vector space during the pre-training process and acquires the ability to extract vector representations that can effectively express various image features. These high-dimensional vector representations can play a key role in precisely assessing the similarity between past and current images in the subsequent step, the feature point matching module (522). It should be noted that the second module (520) extracts feature points from images of visual data, while the first module (510) analyzes patches that segment the visual data region.

[0167] Referring again to FIG. 8, the img (A, B) keypoint and img (A, B) descriptor, which are data output from the feature point extraction module (521) according to the above-described embodiment, are transmitted to the feature point matching module (522). Here, A represents feature point information extracted from past recorded visual data (Record Image), and B represents feature point information extracted from current visual data (Replay Image).

[0168] The feature matching module (522) can perform feature matching between two images using classical machine learning techniques. Specifically, this module can perform a mapping task to match the keypoints extracted from the two images. This matching process can be efficiently performed using the FLANN (Fast Library for Approximate Nearest Neighbors) matcher.

[0169] Duplicate feature point pairs that may occur during the matching process can be removed to increase data reliability. This process can be implemented using the "Drop Duplicates" function, which ensures a one-to-one correspondence between feature points. Once this preprocessing process is complete, the feature point matching module (522) extracts a homography, which represents the geometric transformation relationship between the two images.

[0170] In one embodiment, the feature point matching module (522) can perform homography matching to accurately recognize similar parts of the same anatomical structure even when viewed from different viewpoints or angles. This homography matching can be performed using the outlier-resistant Random Sample Consensus (RANSAC) algorithm.

[0171] The matching process can be performed based on N key points extracted from the feature point extraction module (521) and an N×D dimensional descriptor containing the characteristics of each key point. To increase matching accuracy, only matching pairs within the upper k% (k is a natural number) of the distance between matched point pairs can be selectively used. At this time, the k value can be appropriately set according to the system requirements and image characteristics.

[0172] After homography matching, the overall similarity between the two images can be calculated based on the number of keypoint features exceeding a set threshold. The threshold can also be appropriately set based on system requirements and image characteristics. This process can be implemented using the calculate similarity function, and the calculated similarity score can be used as a key indicator by the second decision-making algorithm to determine whether to approve a path replay request.

[0173] This feature point matching-based similarity evaluation can play an important role in complementing the local collision detection function of the first module (510) by evaluating the similarity of the overall structure and shape of anatomical structures.

[0174] The similarity score can be calculated by applying the following relationship.

[0175] [Relationship 1]

[0176]

[0177]

[0178] Figure 9 is an exemplary diagram showing the visual results of the feature extraction and matching process performed in the second module (520). In this diagram, feature points are clearly indicated by green dots, and feature points matched between the two images are expressed by connecting them with a solid line crossing the images.

[0179] FIG. 10 is a conceptual diagram of reinforcement learning according to one embodiment of the present invention, visually representing a reinforcement learning mechanism that can be applied when performing learning in the decision-making unit (512) of the first module (510).

[0180] In this embodiment, time-series-based episodes can be systematically structured to behave similarly to the actual environment (env) of an anatomical structure. Each episode is structured time-series based on consecutive time indices, such as t, t+1, and t+2, allowing for modeling the continuous movement of a medical device and the resulting changes in the anatomical environment.

[0181] During the learning process, a record image can only advance to the next image if the decision value is 1 (following the command). This implements a mechanism that trains the medical device to advance only when it can safely follow the path. Conversely, if the decision value is 0 (ignoring the command), the device is guided to maintain the current state or explore an alternative path.

[0182] To ensure diversity and richness of training data, replay images can be images at steps separated by a sine function pattern from the record image. In this case, the replay images used may not be actual captured images, but rather virtual visual data generated by algorithmically distorting the original image. This approach can enhance the ability to respond to various anatomical deformations that may occur in real clinical settings by simulating various deformations and distortions. Furthermore, because this method utilizes data augmentation techniques and is a reinforcement learning method, it can be particularly useful in the medical field, where actual clinical data is limited. It also enables the application of effective decision-making algorithms even in environments where securing sufficient real-world training data is difficult.

[0183] Figure 11 is a block diagram of reinforcement learning applied to a first module (510) according to one embodiment of the present invention. The reinforcement learning mechanism conceptually described in Figure 10 can be implemented as a dedicated processor module called a reinforcement learning unit (513) in actual implementation.

[0184] The reinforcement learning unit (513) may be configured based on an Actor-Critic architecture. In an embodiment, only the Actor component is activated in the stage where the reinforcement learning unit (513) is applied to a patient in an actual clinical environment, and the Critic component, which is a pre-learning unit (5130), may be used only in the learning stage and not in the actual patient application stage.

[0185] The reinforcement learning unit (513) illustrated in Fig. 11 receives as input a learning patch or feature point matrix, which is the output of a transformer model, or correlation information for determining the possibility of collision according to a specific action. Based on this input, the unit is composed of an actor that determines which action (path following / path stopping) the medical device (100) will take, and a critic that evaluates the value of a specific action in the current input state.

[0186] Looking at the specific processing of the actor network, a flattening operation is first performed on the input feature point matrix. This process converts the multidimensional tensor-type feature information into a one-dimensional vector. The flattened vector is then input into a multilayer perceptron (MLP) for in-depth feature extraction and analysis. The output of the MLP passes through an action sampler, which determines the final action (whether to follow a path).

[0187] The critic network receives both the currently input feature information and the actor's actions, quantitatively assessing the value of the state-action combination. Throughout the learning process, the actor network is continuously optimized based on this value assessment, and learning proceeds toward maximizing value.

[0188] Finally, after sufficient learning has been completed, the actor network is activated in a real clinical environment to determine whether a path replay request command can be safely executed.

[0189] A control method performed by one or more processing devices configured in a surgical robot system (1) may include step (S10) of storing recorded information of a path steered by a medical device in a memory, step (b) of receiving a path reproduction request command, step (S20) of determining whether the path reproduction request command is approved, and step (d) of steering the medical device along the recorded path. Step (a) (S10) is a step of storing recorded information of a path steered by a medical device (100) in a memory (400), wherein the recorded information may include visual data of an anatomical structure steered by the medical device (100) and control data of the medical device (100). Step (b) (S20) is a step of receiving a path reproduction request command for reproducing the recorded path of the medical device (100) from a control device (500). (c) Step (S30) is a step in which the control device (500) uses a decision-making algorithm to compare and analyze past record information stored in the memory (400) with the current record information of the medical device (100), and determines whether or not to approve the path regeneration request command based on the result of the comparative analysis. (d) Step (S40) is a step in which the control device (500) uses the control data stored in the memory (400) to drive one or more actuators (200) to steer the medical device (100) along the recorded path, if the path regeneration request command is approved. The control method performed by one or more processing devices configured in the surgical robot system (1) can be performed through the processor of the control device (500) described above.

[0190] While the technical concept of the present invention and disclosure has been illustrated by the embodiments described above, the technical concept of the present invention encompasses various substitutions, modifications, and variations that can be made within the scope understandable to those of ordinary skill in the art. Furthermore, it should be understood that such substitutions, modifications, and variations are encompassed within the scope of the appended claims.

[0191]

[0192] (Explanation of symbols)

[0193] 1: Surgical Robot System

[0194] 3: Positioning Cart

[0195] 5: Operator Console

[0196] 10: Surgical instrument device

[0197] 20: Anatomical structures

[0198] 21: Intravenous route

[0199] 50: Display device

[0200] 51: Input device

[0201] 52: Control device

[0202] 100: Medical devices

[0203] 200: Actuator

[0204] 300: Record Preprocessing Module

[0205] 400: Memory

[0206] 500: Control unit

[0207] 501: Path Regeneration Decision Module

[0208] 510: Module 1

[0209] 511: Time series processing unit

[0210] 512: Decision-making department

[0211] 520: Module 2

[0212] 521: Feature extraction module

[0213] 522: Feature Matching Module

[0214] 513: Reinforcement Learning Department

[0215] 5110: Transformer Encoder

[0216] 5111: Transformer Decoder

[0217] 5130: Pre-learning Department

[0218]

[0219] The present invention relates to a surgical robot system that remotely controls a steerable medical device, and has industrial applicability.

Claims

1. Steerable medical devices; One or more actuators for driving the medical device; As a memory, the memory is configured to store record information of a path steered by the medical device within the body, the record information including visual data of an anatomical structure steered by the medical device and control data of the medical device; and A control device that receives a path regeneration request command for regenerating a recorded path of the medical device, compares and analyzes past record information stored in the memory with current record information of the medical device, determines whether to approve the path regeneration request command, and then executes the path regeneration request command; The above control device, A surgical robot system characterized by including a path regeneration judgment module programmed with a decision-making algorithm having a condition for rejecting the above path regeneration request command.

2. In paragraph 1, The above control device, After approving the above path regeneration request command, input of the user's manual control command is blocked until the above path regeneration request command is released. A surgical robot system characterized in that it retrieves the control data stored in the memory and controls the actuator to steer the medical device along the recorded path.

3. In paragraph 1, The above control device, A surgical robot system characterized in that the medical device analyzes visual changes caused by bio-induced deformation occurring in an anatomical structure being controlled, and determines whether to approve the path regeneration request command based on the analysis results.

4. In paragraph 1, The above memory is, Store the record information in the form of time series data in which a time index is assigned to each of the visual data and the control data, A surgical robot system characterized in that the stored record information has a data structure in which the visual data and the control data having the same time index are matched with each other.

5. In paragraph 4, Further comprising a record preprocessing module that preprocesses the record information stored in the memory, The above record preprocessing module is, A surgical robot system characterized in that it detects a pause section in which the medical device is not in motion in the temporal continuity of the visual data or the control data, and removes the visual data or control data of the detected pause section.

6. In paragraph 1, The above path regeneration judgment module is, A first module programmed with a first decision-making algorithm that compares and analyzes past record information stored in the above memory and current record information of the medical device in a first cycle; A second module is programmed with a second decision-making algorithm that compares and analyzes past record information stored in the above memory and current record information of the medical device in a second cycle; A surgical robot system characterized in that the first decision-making algorithm and the second decision-making algorithm are based on different models and independently determine whether to approve reproduction of the recorded path at different cycles.

7. In paragraph 6, The above path regeneration judgment module is, When the above path regeneration request command is received, the first decision algorithm or the second decision algorithm is executed, A surgical robot system characterized in that the first decision-making algorithm is executed in a shorter cycle than the second decision-making algorithm and is executed at a higher frequency per unit time.

8. In paragraph 6, The above path regeneration judgment module is, If the route regeneration request command is rejected in either the first module or the second module, A surgical robot system characterized by blocking execution of the above path regeneration request command.

9. In paragraph 6, The above first module, A surgical robot system characterized in that, in the first decision-making algorithm, a condition is set for rejecting the path regeneration request command based on a result determined to be highly likely to collide with an anatomical structure when the medical device is steered along the recorded path.

10. In paragraph 6, The above second module, A surgical robot system characterized in that, in the second decision-making algorithm, the visual similarity between the visual data of the current anatomical structure to which the medical device is directed and the recorded visual data is analyzed, and if the similarity is determined to be less than a reference value and thus dissimilar, the condition for rejecting the path regeneration request command is set.

11. In paragraph 1, The above path regeneration judgment module is, A short-term decision-making module that periodically determines whether to approve the above-mentioned path regeneration request command, comprising a first module in which a first decision-making algorithm is programmed, The above first decision-making algorithm is, It has a neural network architecture that includes a Transformer model, The above transformer model receives the visual data as a sequence of time series images, and receives record images, which are recorded visual data from the past, and replay images, which are current visual data, as input. A surgical robot system characterized in that it extracts a feature vector of an image in which an anatomical structure appears from each of the above-described record images and the above-described replay images, evaluates the relationship between the feature vectors extracted from past visual data and current visual data, and outputs correlation information.

12. In paragraph 11, The above first module, A configuration for executing the above first decision-making algorithm, A time series processing unit that generates a feature vector by passing each image from the record images, which are recorded visual data of the past, and the replay images, which are current visual data, through a convolutional layer and extracts correlation information by evaluating the relationship between the feature vectors; and A surgical robot system characterized by comprising a decision-making unit that receives the above correlation information and determines whether to approve the path regeneration request command based on a model that has previously learned whether a collision occurs between the medical device and the anatomical structure when executing a control command included in the control data.

13. In paragraph 1, The above path regeneration judgment module is, A long-term decision-making module that periodically determines whether to approve the above-mentioned path regeneration request command, comprising a second module programmed with a second decision-making algorithm; The above second decision-making algorithm is, Includes a deep learning-based feature extraction model that extracts feature points considering the entire context of an image in which an anatomical structure appears from the above visual data, A surgical robot system characterized in that the similarity is determined through matching between the visual data recorded in the above memory and the feature points extracted from the current visual data.

14. In paragraph 1, Further comprising an operating device that outputs the path regeneration request command to the above control device; The above operating device, An input unit that remains active in response to continuous physical manipulation by the user; and An output unit configured to continuously output the path regeneration request command while the input unit is active; A surgical robot system characterized in that the above path regeneration request command stops outputting as soon as the user's physical manipulation is released.

15. In a processor encoded with control commands to operate one or more actuators to steer a medical device in a surgical robotic system, A path regeneration request command for regenerating a recorded path of the medical device is received, and a path regeneration judgment module is included that calls up visual data of a path that the medical device was steered in the body in the past and control data that drove the medical device from a memory, and analyzes visual data of the current point in time of the medical device to determine whether the path regeneration request command is approved. The above path regeneration judgment module is, By analyzing the visual changes caused by bio-induced deformation of the anatomical structure to which the medical device is directed, if there is a risk of collision between the medical device and the anatomical structure or if the current anatomical structure is less than a predetermined similarity standard with the past anatomical structure, the path regeneration request command is rejected. A processor characterized in that it selectively reproduces the previously recorded path of the medical device in the body environment of an anatomical structure in which a bio-induced deformation has occurred.

16. In a control method performed by one or more processing devices configured in a surgical robot system, A step of storing in memory the recorded information of the path steered by the medical device within the body, wherein the recorded information includes visual data of the anatomical structure steered by the medical device and control data of the medical device; (a) step; (b) step of receiving a path regeneration request command for regenerating the recorded path of the medical device from the control device; (c) step of comparing and analyzing past record information stored in the memory and current record information of the medical device using a decision-making algorithm in the control device, and determining whether to approve the path regeneration request command based on the result of the comparative analysis; and (d) step of steering the medical device along the recorded path by controlling one or more actuators using the control data stored in the memory in the control device when the above path regeneration request command is approved; A control method for a surgical robot system, characterized in that the step (c) above sets at least one condition that can reject the path regeneration request command when determining whether to approve the path regeneration request command.

Images