Spinal surgery guidance method based on augmented reality and artificial intelligence, and system thereof

Abstract

Provided are a spinal surgery guidance method based on augmented reality and artificial intelligence, and a system thereof. A spinal surgery guidance method according to some embodiments may comprise the steps of: acquiring an actual image of spinal surgery of a patient; recognizing, in the actual image, a position and a direction where a screw is to be inserted into a spinal region of the patient through an oriented bounding box-based object detection model; and providing guide information about the spinal surgery to medical staff performing the spinal surgery on the basis of the recognized position and direction. According to this method, safety and accuracy of spinal surgery can be greatly improved.

Description

Augmented reality and artificial intelligence-based spinal surgery guidance method and system thereof

[0001] The disclosure relates to a method and system for guiding spinal surgery using augmented reality and artificial intelligence technologies.

[0002] Spinal fusion is a highly difficult surgical procedure that stabilizes (fixes) the patient's spine by inserting screws and cages.

[0003] During spinal fusion surgery, medical staff typically use an X-ray machine in the operating room to confirm the position of the screws and cages.

[0004] However, this method has the problem of exposing patients to excessive radiation.

[0005] To address radiation exposure issues, free hand techniques can be used.

[0006] The freehand technique is a technique in which the medical staff inserts screws and cages while directly observing the anatomical structure of the spine.

[0007] However, these techniques require extensive experience in spinal fusion surgery, and even with skilled medical staff, the surgical results can vary significantly depending on the patient's condition (e.g., spinal structure, etc.) and the medical staff's condition.

[0008] The technical problem to be solved through several embodiments of the present disclosure is to provide a guide method and system that can improve the safety and accuracy of spinal surgery for various patients.

[0009] The technical problems of the present disclosure are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art of the present disclosure from the description below.

[0010] A spinal surgery guide method according to some embodiments of the present disclosure for solving the above-described technical problem may include a step of acquiring an actual image of a patient's spinal surgery, a step of recognizing a position and direction of a screw inserted into a spinal region of the patient in the actual image through an object detection model based on an oriented bounding box, and a step of providing guide information for the spinal surgery to a medical staff performing the spinal surgery based on the recognized position and direction, in a method performed by at least one processor.

[0011] In some embodiments, the step of recognizing the position and direction of the screw may include the step of predicting a directional bounding box for the screw through the object detection model, and the step of recognizing the direction of the screw based on an angular difference between the predicted directional bounding box and a horizontal bounding box for the screw.

[0012] In some embodiments, the step of recognizing the position and direction of the screw may include the step of predicting a directional bounding box for a screw driver used for inserting the screw through the object detection model, and the step of recognizing the direction of the screw based on an angular difference between the predicted directional bounding box and a horizontal bounding box for the screw driver.

[0013] In some embodiments, the step of recognizing the position and direction of the screw may include the step of predicting a directional bounding box for a hand portion of the medical professional holding the screwdriver through the object detection model, wherein the screwdriver is used for inserting the screw, and the step of recognizing the direction of the screw based on an angular difference between the predicted directional bounding box and a horizontal bounding box for the hand portion.

[0014] In some embodiments, the step of providing guide information for the spinal surgery may include the steps of obtaining a virtual spinal structure image of the patient, generating an augmented reality image by aligning the virtual spinal structure image with the actual image, and providing the guide information to the medical staff based on the augmented reality image.

[0015] In some embodiments, the augmented reality image may be provided to the medical staff via a head mounted display (HMD).

[0016] In some embodiments, the step of providing guide information for the spinal surgery may include the step of providing an alarm to the medical staff when the difference between the recognized direction and the preset recommended direction is greater than a reference value.

[0017] In some embodiments, the step of providing guide information for the spinal surgery may include the step of generating an augmented reality image by adding a first indicator indicating the recognized direction and a second indicator indicating a preset recommended direction to the actual image, and the step of providing the augmented reality image to the medical staff.

[0018] In some embodiments, the first indicator or the second indicator may be added to the actual image when the difference between the recognized direction and the recommended direction is greater than a threshold value.

[0019] In some embodiments, the spinal surgery guide method may further include a step of recognizing a posture of a hand of the medical staff holding a screwdriver through a deep learning model, wherein the screwdriver is used to insert the screw, a step of adding information about the recommended posture to the actual image to generate an augmented reality image when a difference between the recognized posture and a predefined recommended posture is greater than or equal to a reference value, and a step of providing the augmented reality image to the medical staff.

[0020] In some embodiments, the spinal surgery guide method may further include a step of recognizing a screw insertion motion of the medical staff holding a screw driver through a deep learning model, wherein the screw driver is used to insert the screw, a step of adding information about the recommended motion to the actual image to create an augmented reality image when a difference between the recognized motion and a predefined recommended motion is greater than or equal to a reference value, and a step of providing the augmented reality image to the medical staff.

[0021] According to some embodiments of the present disclosure for solving the above-described technical problem, a spinal surgery guide system includes one or more processors and a memory storing a computer program executed by the one or more processors, wherein the computer program may include instructions for an operation of acquiring an actual image for a spinal surgery of a patient, an operation of recognizing a position and direction of a screw inserted into a spinal region of the patient in the actual image through an object detection model based on an oriented bounding box, and an operation of providing guide information for the spinal surgery to a medical staff performing the spinal surgery based on the recognized position and direction.

[0022] According to some embodiments of the present disclosure for solving the above-described technical problem, a computer program may be stored in a computer-readable recording medium in combination with a processor of a computer to execute the steps of: acquiring an actual image of a patient's spinal surgery; recognizing a position and direction of a screw inserted into a spinal region of the patient in the actual image through an object detection model based on an oriented bounding box; and providing guide information for the spinal surgery to a medical staff performing the spinal surgery based on the recognized position and direction.

[0023] According to some embodiments of the present disclosure, guidance information for spinal surgery can be provided to medical staff based on augmented reality (AR) and artificial intelligence.

[0024] For example, an augmented reality image containing information about the current position and direction of a screw (or cage), recommended direction, target insertion point, etc. can be provided to the medical staff.

[0025] Alternatively, an alarm may be provided to the medical staff if the current orientation of the screw (or cage) does not match the recommended orientation.

[0026] In such cases, the safety and accuracy of spinal surgery can be greatly improved, and the problem of spinal surgery results varying greatly depending on the experience and condition of the medical staff and the patient's condition can be easily resolved.

[0027] In addition, the problem of radiation exposure for patients undergoing spinal surgery can be completely resolved (because no X-ray equipment, etc. is used during spinal surgery).

[0028] Additionally, the direction of the screw (or cage) can be recognized in real images using an object detection model (e.g., YOLO) based on an oriented bounding box (OBB).

[0029] Specifically, a directional bounding box and a horizontal bounding box (HBB) for a screw (or cage) in an actual image are predicted, and the direction of the screw (or cage) can be recognized based on the angular difference between the two bounding boxes.

[0030] In such cases, the direction of the screw (or cage) being inserted into the patient's spine can be accurately recognized, and as a result, the safety and accuracy of spinal surgery can be further improved.

[0031] The effects according to the technical idea of ​​the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0032] FIGS. 1 and 2 are exemplary drawings for explaining the configuration and operation of a spinal surgical guide system according to some embodiments of the present disclosure.

[0033] FIG. 3 is an exemplary flowchart illustrating a spinal surgical guide method according to some embodiments of the present disclosure.

[0034] FIG. 4 is an exemplary flowchart illustrating a screw direction recognition method according to some embodiments of the present disclosure.

[0035] FIG. 5 is an exemplary drawing to further explain a screw direction recognition method according to some embodiments of the present disclosure.

[0036] FIG. 6 is an exemplary flowchart illustrating a method for providing guide information according to some embodiments of the present disclosure.

[0037] FIG. 7 is an exemplary drawing for explaining a method for providing guide information according to some other embodiments of the present disclosure.

[0038] FIG. 8 is an exemplary drawing for explaining a method for providing guide information according to some other embodiments of the present disclosure.

[0039] FIGS. 9 and 10 are exemplary drawings for explaining a method of providing guide information according to some other embodiments of the present disclosure.

[0040] FIG. 11 illustrates an exemplary computing device that can implement a spinal surgical guide system or the like according to some embodiments of the present disclosure.

[0041] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0042] FIG. 1 is an exemplary drawing for explaining the configuration and operation of a spinal surgical guide system according to some embodiments of the present disclosure.

[0043] As illustrated in FIG. 1, a spinal surgery guide system according to embodiments may be configured to include an augmented reality (AR) device (11), a photographing device (12), and a surgery support system (10).

[0044] Below, each component (10 to 12) of the spinal surgical guide system is described.

[0045] The augmented reality device (11) is a device used to provide medical staff with augmented reality images containing guide information for spinal surgery (e.g., spinal fusion surgery).

[0046] For example, the augmented reality device (11) can receive an augmented reality image from the surgical support system (10) and provide (display) it to medical staff.

[0047] Alternatively, the augmented reality device (11) may receive guide information from the surgical support system (10) and generate an augmented reality image based on the information to provide (display) to medical staff.

[0048] In the present disclosure, the term augmented reality may encompass the concepts of mixed reality (MR) and virtual reality (VR), and the term image may encompass the concepts of video and image.

[0049] The guide information may include various information for accurate insertion of a screw (or medical device such as a screw or cage, etc.) (e.g., screw (or cage) position, direction, recommended direction, target insertion point, medical staff posture / motion, recommended posture / motion, etc.).

[0050] However, the scope of the present disclosure is not limited thereto, and the guide information may include, without limitation, various information helpful for spinal surgery.

[0051] The augmented reality device (11) may be, for example, an HMD (Head Mounted Display), a fixed / mobile display (e.g., monitor) installed at a surgical site (or a computing device equipped with a display, such as a laptop or desktop), etc.

[0052] However, the scope of the present disclosure is not limited thereto. Fig. 2 assumes that the augmented reality device (11) is an HMD.

[0053] In a case as illustrated in FIG. 2, medical staff (21) can perform spinal surgery while wearing an augmented reality device (11), and can receive an augmented reality image including guide information for spinal surgery through the augmented reality device (11).

[0054] Next, the photographing device (12) is a device equipped with an image capturing function (or equipped with a camera module), and is a device used by medical staff to photograph the process of spinal surgery (e.g., spinal fusion surgery) performed on a patient.

[0055] For example, as illustrated in FIG. 2, a photographing device (12) is mounted on an augmented reality device (11) to capture a spinal surgery process for a patient (24) in real time, and transmit the captured actual image (i.e., an image captured from the viewpoint of a medical staff member (21)) to a surgical support system (10).

[0056] Then, the surgical support system (10) can generate an augmented reality image including guide information for the corresponding spinal surgery based on the received actual image and transmit it to the augmented reality device (11).

[0057] Figure 2 assumes a case where a medical staff (21) performs a surgery in which a screw (23) is inserted into the spine of a patient (24) using a screw driver (22).

[0058] In some cases, the filming device (12) may be used to film the process of medical staff (21) practicing spinal surgery, which will be described later.

[0059] The photographing device (12) may further include a device for photographing medical staff, surgical sites, etc.

[0060] For example, the shooting device (12) may further include a shooting device installed at the surgical site in addition to the shooting device (e.g., built-in / external camera) mounted on the augmented reality device (11).

[0061] Next, the surgical support system (10) is a computing device / system equipped with various assistance functions for spinal surgery (e.g., spinal fusion surgery).

[0062] For example, the surgical support system (10) can support (assist) the medical staff in performing spinal surgery by providing guide information for spinal surgery in conjunction with an augmented reality device (11) and a photographing device (12).

[0063] Alternatively, the surgical support system (10) may support (assist) the medical staff's spinal surgery in another way.

[0064] For a specific example, as illustrated in FIG. 2, the surgical support system (10) can receive an actual image of a spinal surgical procedure performed by a medical staff (21) from a photographing device (12) and generate an augmented reality image including guide information based on the actual image.

[0065] Next, the surgical support system (10) can provide the augmented reality image to the medical staff (21) by transmitting the augmented reality image to the augmented reality device (11).

[0066] As another example, the surgical support system (10) may provide an alarm to the medical staff when a predefined condition is satisfied (e.g., when spinal surgery is not performed according to the guide information, etc.).

[0067] As a more specific example, if the difference between the direction of the screw (or cage) (i.e., insertion direction) and the recommended direction is greater than a reference value, the surgical assistance system (10) can provide an alarm (i.e., an alarm for direction correction purposes) to the medical staff.

[0068] As another example, if the difference between the medical staff's screw (or cage) insertion posture / motion and the recommended posture / motion is greater than a reference value, the surgical assistance system (10) can provide an alarm (i.e., an alarm for posture correction purposes) to the medical staff.

[0069] As another example, if the difference between the posture / motion of the hand holding the screwdriver and the recommended posture / motion is greater than a reference value, the surgical support system (10) can provide an alarm to the medical staff.

[0070] As another example, the surgical assistance system (10) may provide an alarm to the medical staff to notify them of the start and / or end of the spinal surgery.

[0071] As another example, the surgical assistance system (10) may provide an alarm based on various combinations of the examples described above.

[0072] The method of providing an alarm may be any method. For example, the surgical support system (10) may provide a visual alarm through an augmented reality device (11) or may provide a tactile alarm through a haptic device (not shown) carried by the medical staff (e.g., providing vibration, etc.).

[0073] Alternatively, the surgical assistance system (10) may provide an audible alarm through an output device (not shown) such as a speaker.

[0074] In the above example, the surgical support system (10) may automatically change the alarm provision method depending on the status of the medical staff and / or the surgical situation.

[0075] Here, the status of the medical staff and / or the surgical situation may be determined by analyzing an actual image obtained through, for example, a photographing device (12) (e.g., the posture / motion / status of the medical staff can be determined through a deep learning model), or may be determined using various sensors (e.g., sensors mounted on an augmented reality device (11), a haptic device, etc.).

[0076] However, the scope of the present disclosure is not limited thereto. For example, when a medical professional is inserting a screw (or cage), the surgical assistance system (10) may provide an alarm to the medical professional in a visual manner (or with a smaller sound or weak vibration).

[0077] This can be understood as an attempt to prevent medical accidents caused by medical staff's distraction or mistakes.

[0078] In the opposite case, the surgical assistance system (10) can provide an alarm to the medical staff in a tactile or auditory manner (or with a louder sound or strong vibration).

[0079] As another example, if a medical professional is holding a screwdriver, the surgical assistance system (10) can provide an alarm to the medical professional in a visual manner (or with a smaller sound or weak vibration).

[0080] In the opposite case, the surgical assistance system (10) can provide an alarm to the medical staff in a tactile or auditory manner (or with a louder sound or strong vibration).

[0081] As another example, if the amount of movement of the medical staff exceeds a standard value (e.g., during surgery), the surgical assistance system (10) can provide an alarm to the medical staff in a visual manner (or with a smaller sound or weak vibration).

[0082] As another example, if the medical staff is determined to be in a state of concentration, the surgical assistance system (10) may provide an alarm to the medical staff in a visual manner (or with a smaller sound or weak vibration).

[0083] The surgical support system (10) may be able to determine whether the medical staff is in a state of concentration based on the medical staff's posture / movement, whether or not the medical staff is holding a screwdriver, the amount of movement, etc., but the scope of the present disclosure is not limited thereto.

[0084] As another example, the surgical support system (10) can play preset music through an output device when it is determined that the medical staff is in a tense state.

[0085] Alternatively, the surgical support system (10) may provide the medical staff with a spinal surgical guide image (e.g., an image showing a screw insertion process by a skilled medical staff) through an augmented reality device (11).

[0086] The surgical support system (10) may be able to determine whether the medical staff is in a state of tension based on the medical staff's facial expression / posture / motion, whether or not the medical staff is holding a screwdriver, the amount of movement, the degree of tremor in the body (e.g., hand area, etc.), etc. (e.g., if the degree of tremor is above a standard value, it is determined that the medical staff is in a state of tension), but the scope of the present disclosure is not limited thereto.

[0087] The surgical support system (10) may be able to determine the degree of bodily tremor of a medical staff member based on measurement data, such as a vibration sensor mounted on a screwdriver, a movement sensor (e.g., acceleration sensor, etc.) mounted on an augmented reality device (11), etc., but the scope of the present disclosure is not limited thereto.

[0088] As another example, the surgical support system (10) may provide (support) a simulation (practice) function for spinal surgery to medical staff.

[0089] Specifically, the surgical support system (10) can provide medical staff with an augmented reality image including guide information before performing a spinal surgery, and the medical staff can simulate (practice) the spinal surgery in advance through such augmented reality image.

[0090] At this time, the surgical support system (10) may create an augmented reality image by aligning a virtual spinal structure image of the patient with an actual image taken of a patient model, or may create an augmented reality image by aligning a virtual patient object and spinal structure image with an actual image taken of a surgical site.

[0091] As another example, the surgical support system (10) may support a medical staff's spinal surgery based on various combinations of the examples described above.

[0092] Details of the methods performed by the surgical support system (10) to support spinal surgery will be described in more detail later with reference to drawings below FIG. 3.

[0093] The above-described surgical support system (10) can be implemented by at least one computing device.

[0094] For example, all functions of the surgical support system (10) may be implemented in one computing device, or the first function of the surgical support system (10) may be implemented in the first computing device and the second function may be implemented in the second computing device.

[0095] Alternatively, specific functions of the surgical support system (10) may be implemented on multiple computing devices.

[0096] A computing device may include any device having computing capabilities, and for an example of such a device, see FIG. 11.

[0097] A computing device is a collection of interacting components (e.g., memory, processor, etc.), so it may sometimes be called a 'computing system'.

[0098] Of course, the term computing system can also encompass the concept of a collection of interacting computing devices.

[0099] The components (10 to 12) of the spinal surgical guide system described above can communicate, for example, via a network.

[0100] Here, the network can be implemented as any type of wired / wireless network, such as a local area network (LAN), a wide area network (WAN), a mobile radio communication network, or Wibro (Wireless Broadband Internet).

[0101] Meanwhile, in some embodiments, the spinal surgical guide system may further include an alarm device (not shown).

[0102] The alarm device (not shown) may be, for example, a haptic device that provides a tactile alarm, an output device (e.g., a speaker) that provides an audible alarm, etc., but the scope of the present disclosure is not limited thereto.

[0103] If the alarm is provided visually, the augmented reality device (11) may also function as an alarm device.

[0104] The configuration and operation of a spinal surgical guide system according to some embodiments of the present disclosure have been described with reference to FIGS. 1 and 2.

[0105] Hereinafter, various methods that can be performed in the spinal surgery guide system described above will be described with reference to the drawings below FIG. 3.

[0106] In the following, for the convenience of understanding, the execution process of the methods to be described later will be explained focusing on the operation of the surgical support system (10), and the explanation will continue assuming that all steps / operations of the methods to be described later are performed in the surgical support system (10, e.g., at least one processor).

[0107] Therefore, if the subject of a specific step / action is omitted, it can be understood that the step / action is performed by the surgical support system (10).

[0108] However, in a real environment, some steps / operations of the methods described below may be performed on other computing devices.

[0109] Hereinafter, for convenience of explanation, the surgical support system (10) will be abbreviated as ‘system (10)’.

[0110] FIG. 3 is an exemplary flowchart schematically illustrating a spinal surgical guide method according to some embodiments of the present disclosure.

[0111] However, this is only an exemplary embodiment for achieving the purpose of the present disclosure, and it is obvious that some steps may be added or deleted as needed.

[0112] As illustrated in FIG. 3, the spinal surgery guide method according to the embodiments may begin at step S31 of acquiring an actual image for spinal surgery of a patient.

[0113] For example, the system (10) can receive real-time images of spinal surgery (e.g., images taken from the medical staff's perspective) through a shooting device (e.g., a camera mounted on an augmented reality device (11)).

[0114] In step S32, the position and direction of the screw (or cage) inserted into the patient's spine in the actual image are recognized through the object detection model.

[0115] For example, the system (10) can recognize the location of a screw (or cage) in an actual image through an object detection model trained to detect a screw (or cage) in an image (e.g., image).

[0116] In the following description, the technical details related to screws can also be applied to cages.

[0117] However, the object detection model used for cage recognition may be a different model from the object detection model used for screw recognition.

[0118] The object detection model may be, for example, a deep learning model configured to output object class information (e.g., class-specific confidence scores) and bounding box information (e.g., bounding box vertex coordinates, bounding box width, height, etc.).

[0119] An example of such a model would be YOLO (You Only Look Once), but the scope of the present disclosure is not limited thereto.

[0120] The specific method of recognizing the direction of the screw in this step S32 may vary depending on the embodiment.

[0121] In some embodiments, as illustrated in FIG. 4, the system (10) can recognize the direction of a screw in a real image using an object detection model (e.g., YOLO) based on an oriented bounding box (OBB) (S41, S42).

[0122] Specifically, the system (10) can predict a directional bounding box for a screw in an actual image through an object detection model (S41).

[0123] Here, predicting a directional bounding box for a screw may mean predicting information (e.g., coordinates of the vertices of the bounding box) of a directional bounding box corresponding to the screw class through an object detection model.

[0124] Next, the system (10) can recognize the direction of the screw based on the angular difference between the predicted directional bounding box and the horizontal bounding box (HBB) for the screw (S42).

[0125] For example, as illustrated in FIG. 5, the system (10) can predict the directional bounding box (54) and the horizontal bounding box (55) of a screw (53, i.e., a screw object) in an actual image (52, e.g., a frame of an image) through an object detection model (51).

[0126] And, the system (10) can recognize the direction of the screw (53) based on the angular difference (refer to 'θ') between the directional bounding box (54) and the horizontal bounding box (55).

[0127] By doing so, the direction of the screw (53) can be accurately recognized in the actual image.

[0128] For reference, FIG. 5 depicts an object detection model (51) outputting an image (52) in which bounding boxes (54, 55) are displayed, but this is only for convenience of understanding, and the object detection model (51) may be configured to output information (e.g., coordinates of the vertices of the bounding box, etc.) of a specific bounding box (e.g., 54).

[0129] In addition, the system (10) can also predict a directional bounding box (54) and a horizontal bounding box (55) for the screw (53) using multiple object detection models.

[0130] In detail, the system (10) may predict a directional bounding box (54) and a horizontal bounding box (55) for a screw (53) through one object detection model (51) (i.e., when the object detection model (51) is configured to output information on both the directional bounding box (54) and the horizontal bounding box (55)), or may predict a directional bounding box (54) for a screw (53) using a first object detection model and a horizontal bounding box (55) for a screw (53) using a second object detection model.

[0131] In some other embodiments, the system (10) may recognize the orientation of the screw based on the orientation of the screw driver (i.e., the medical tool used to insert the screw).

[0132] This can be understood as taking advantage of the fact that it is easier to detect a screwdriver in an image than a screw (because the screwdriver is larger) and that the direction of the screw is almost the same as the direction of the screwdriver.

[0133] Specifically, the system (10) can predict a directional bounding box for a screwdriver through an object detection model (i.e., a model trained to detect a screwdriver) and predict the orientation of the screwdriver based on the angular difference between the predicted directional bounding box and the horizontal bounding box for the screwdriver.

[0134] For this, please refer to the description of the previous embodiments. Next, the system (10) can determine the direction of the screw based on the predicted direction (e.g., determine the direction of the screw to be in the same direction as the screw driver).

[0135] In some other embodiments, the system (10) may recognize the orientation of the screw based on the orientation of the hand of the medical professional (i.e., the hand holding the screwdriver).

[0136] This can be understood as taking advantage of the fact that the direction of the hand area and the direction of the screw are closely related to each other.

[0137] Specifically, the system (10) can predict a directional bounding box for a hand region of a medical professional through an object detection model (i.e., a model trained to detect a hand region) and can predict the direction of the hand region based on the angular difference between the predicted directional bounding box and the horizontal bounding box for the corresponding hand region.

[0138] For this, please refer to the description of the previous embodiments. Next, the system (10) can determine the direction of the screw based on the predicted direction (e.g., determine the direction of the screw based on the relationship between two predefined directions).

[0139] In some cases, the system (10) may recognize the direction of the hand of the medical staff in the actual image of the medical staff through a deep learning model (or a deep learning model configured to detect key points of the hand) to predict the posture of the hand (see FIG. 9).

[0140] In some other embodiments, the direction of the screw may be recognized based on various combinations of the embodiments described above.

[0141] For example, the system (10) can recognize the direction of the hand of the medical staff and the direction of the screwdriver through one or more object detection models, and recognize the direction of the screw by comprehensively considering the two recognized directions.

[0142] As another example, the system (10) can recognize the direction of the screw and the direction of the screw driver through one or more object detection models and comprehensively consider the two recognized directions to determine the final direction of the screw (e.g., determine the final direction (value) of the screw by correcting the direction (value) of the screw based on the direction (value) of the screw driver).

[0143] This is explained again with reference to Figure 3.

[0144] In step S33, guidance information for spinal surgery is provided to the medical staff performing the spinal surgery based on the recognized position and direction.

[0145] For example, the system (10) can provide medical staff with real-time guidance information for spinal surgery through an augmented reality device (11, eg, HMD).

[0146] By doing so, the safety and accuracy of spinal surgery can be greatly improved, and the problem of spinal surgery results varying greatly depending on the experience and condition of the medical staff and the patient's condition can be easily resolved.

[0147] In addition, the problem of radiation exposure to the patient can be completely solved (because no X-ray equipment, etc. is used during spinal surgery).

[0148] The specific method of providing guide information in this step S33 may vary depending on the embodiment.

[0149] In some embodiments, as illustrated in FIG. 6, the system (10) may generate an augmented reality image by adding (synthesizing) a first direction indicator (e.g., a visual indicator such as an arrow object, a line object, etc.) indicating a direction of the screw (i.e., a direction recognized through an object detection model) and a second direction indicator indicating a preset recommended direction (e.g., a correct insertion direction) to an actual image (S61), and provide the augmented reality image to a medical professional through an augmented reality device (11) (see further description of FIG. 8).

[0150] Here, the recommended direction may be derived by the medical staff by reading the patient's spinal structure image before surgery (e.g., the system (10) may receive recommended direction information from the medical staff), or may be derived automatically by the system (10).

[0151] For example, the system (10) may automatically derive a recommended direction for inserting the screw from the current location to the target insertion point by comprehensively considering the target insertion point of the screw (e.g., the insertion point of the screw planned by the medical staff by reading the patient's spinal structure image), the current location of the screw, the patient's spinal structure information, etc.

[0152] According to these embodiments, the safety and accuracy of spinal surgery can be significantly improved as the difficulty of screw insertion is reduced.

[0153] In the preceding embodiments, the system (10) may add (synthesize) the first direction indicator and / or the second direction indicator to the actual image in response to satisfaction of a preset condition.

[0154] For example, the system (10) can add a first direction indicator and / or a second direction indicator to the actual image when the difference between the direction of the screw recognized in the actual image and the recommended direction is greater than a reference value (e.g., when the screw is being inserted in the recommended direction, the indicator for the recommended direction is not displayed).

[0155] In these cases, the complexity of the augmented reality images provided to medical staff can be reduced, easily preventing distraction. Furthermore, if the current screw direction deviates from the recommended direction, a direction indicator appears in the augmented reality image, effectively alerting medical staff to screw misalignment.

[0156] The reference value may be a preset fixed value or a value that varies depending on the situation.

[0157] For example, the reference value may be a value that varies based on the distance from the current position of the screw to the target insertion point (e.g., the longer the distance, the smaller the reference value is determined).

[0158] In some other embodiments, the system (10) may provide an alarm to the medical staff if the difference between the direction of the screw recognized in the actual image and the preset recommended direction is greater than a threshold value.

[0159] For example, the system (10) may provide a visual alarm through an augmented reality device (11) or a tactile alarm through a haptic device.

[0160] In some cases, the system (10) may determine an alarm provision method (e.g., alarm intensity, etc.) based on the difference between the direction of the screw recognized in the actual image and the recommended direction.

[0161] For example, the system (10) can increase the alarm intensity as the difference between the two directions increases.

[0162] Any method for increasing the alarm intensity is acceptable (e.g., providing alarms in more diverse ways, increasing the number of alarm devices, increasing the alarm duration, providing alarms with a louder sound, providing alarms with stronger vibration, etc.).

[0163] Alternatively, the system (10) may provide a visual alarm when the difference between the two directions is below a threshold and an audible (or tactile) alarm when the difference is above the threshold.

[0164] In some other embodiments, as illustrated in FIG. 7, the system (10) can generate an augmented reality image (74) by aligning a virtual spinal structure image (72) with an actual image (73) of the patient (i.e., overlaying the spinal structure image (72) on the spinal region of the actual image (73)).

[0165] This alignment may be performed using a screwdriver or an augmented reality marker placed at another location (e.g., detecting an augmented reality marker in the actual image (73) and performing alignment based on the detected marker), but the scope of the present disclosure is not limited thereto.

[0166] The virtual spinal structure image (72) may be, for example, a CT (Computed Tomography) and / or MR (Magnetic Resonance) image (71) of the patient's spinal region taken before surgery, or may be an image of the patient's spinal structure created based on such an image (71).

[0167] For reference, since the medical staff will read the CT and / or MR (Magnetic Resonance) images (71) of the patient's spine to determine (plan) the target insertion point of the screw (or cage), if a virtual spinal structural image (72) is provided to the medical staff performing the spinal surgery, the safety and accuracy of the spinal surgery can be further improved (because the probability of the screw (or cage) being inserted into the target point will increase).

[0168] In some other embodiments, the system (10) can create an augmented reality image by adding (synthesizing) an indicator (e.g., a virtual screw object) indicating the target insertion point of the screw to an actual image, and provide the augmented reality image to the medical staff through an augmented reality device (11).

[0169] In some other embodiments, guide information may be provided based on various combinations of the embodiments described above.

[0170] For example, as illustrated in FIG. 8, the system (10) may create an augmented reality image (81) by aligning a patient's spinal structure image with an actual image and adding (synthesizing) an indicator (85) indicating a target insertion point on the spinal structure image, a first direction indicator (83) indicating a current direction of a screw (82), and a second direction indicator (84) indicating a recommended direction.

[0171] Meanwhile, according to some embodiments of the present disclosure, the system (10) may recognize the posture and / or movement (e.g., full body posture / motion, hand posture / motion, etc.) of a medical staff performing spinal surgery through a deep learning model and provide guide information for the spinal surgery based on the recognized result.

[0172] Hereinafter, these embodiments will be described in detail with reference to FIGS. 9 and 10.

[0173] FIGS. 9 and 10 are exemplary drawings for explaining a method of providing guide information based on the results of posture and / or motion recognition of medical staff.

[0174] The system (10) can recognize the posture of a medical staff member holding a screwdriver in an actual image (i.e., an image of the medical staff member) through a deep learning model (91) as illustrated in FIG. 9.

[0175] For example, the system (10) can receive an actual image of medical staff captured through a shooting device (12) installed at a surgical site and recognize the posture of the medical staff from the received actual image through a deep learning model (91).

[0176] The deep learning model (91) is a model trained to recognize a posture in an image (92), and may be, for example, a model that detects key points for a hand area (or the whole body) in an image (92) and recognizes (classifies) a human posture based on a skeleton (93) composed of the key points.

[0177] Those working in the relevant technical field will likely already be familiar with the structure and operating principles of this skeleton-based pose recognition model, so a detailed explanation will be omitted.

[0178] The system (10) can recognize the posture of the hand part of the medical staff in an actual image or the posture of other parts through a deep learning model (91).

[0179] Alternatively, the system (10) may recognize the posture of the medical staff's entire body. The system (10) may perform this posture recognition using a single deep learning model (91) or may perform this posture recognition using multiple deep learning models (e.g., recognizing the posture of the hand using a first deep learning model and recognizing the posture of the entire body using a second deep learning model).

[0180] Next, the system (10) can compare the posture of the medical staff recognized in the actual image with a predefined recommended posture.

[0181] For example, the system (10) can compare the posture of a hand part recognized in an actual image (e.g., gripping posture of a screwdriver) with a recommended posture of the corresponding hand part.

[0182] Alternatively, the system (10) may compare the full-body posture of the medical staff recognized in the actual image (e.g., screw insertion posture) with the recommended posture.

[0183] The system (10) may perform the posture comparison by comparing skeletons or may perform the posture comparison in another manner.

[0184] As a result of the comparison, if the difference between the recognized posture and the recommended posture is greater than the reference value, the system (10) can provide an alarm (i.e., an alarm for posture correction purposes) to the medical staff.

[0185] Alternatively, as illustrated in FIG. 10, the system (10) may generate an augmented reality image (103) by adding (synthesizing) recommended posture information (101) to an actual image (102, e.g., an image taken from the viewpoint of medical staff) and provide the augmented reality image to medical staff.

[0186] At this time, the recommended posture information (101) can be included in the augmented reality image (103) in various forms such as text, voice, and image.

[0187] In some cases, the system (10) may also generate an augmented reality image (103) including recommended posture information (101) regardless of the comparison results (or when other conditions are satisfied).

[0188] For example, if it is detected that a medical professional has grasped a screwdriver in the actual image (102), the system (10) may add recommended posture information (101) related to grasping the screwdriver (or inserting a screw using a screwdriver) to the actual image (102) to generate an augmented reality image (103).

[0189] Additionally, the system (10) can recognize the movements of medical personnel in an actual video (i.e., a video in which medical personnel appear) through a deep learning model trained to recognize movements in a video.

[0190] For example, the system (10) can recognize a medical staff member's screw insertion motion (e.g., full body motion, hand motion, etc.) and other motions through a deep learning model.

[0191] A deep learning model, for example, can recognize (classify) human motion based on a series of skeletons extracted from video frames. Those skilled in the art will likely already be familiar with the structure and operating principles of such skeleton-based motion recognition models, so a detailed description will be omitted.

[0192] Next, the system (10) can compare the medical staff's movements recognized in the actual image with predefined recommended movements.

[0193] For example, the system (10) can compare the screw insertion motion of the medical staff recognized in the actual image with the recommended motion.

[0194] The system (10) may perform motion comparison by comparing a series of skeletons or may perform motion comparison in another manner.

[0195] As a result of the comparison, if the difference between the recognized motion and the recommended motion is greater than the reference value, the system (10) can provide an alarm (i.e., an alarm for motion correction purposes) to the medical staff.

[0196] Alternatively, the system (10) may create an augmented reality image by adding (synthesizing) recommended motion information to an actual image (e.g., an image taken from the viewpoint of a medical professional) and provide the augmented reality image to the medical professional.

[0197] At this time, recommended action information can be included in the augmented reality video in various forms such as text, voice, and video.

[0198] In some cases, the system (10) may also generate an augmented reality image that includes recommended action information regardless of the comparison results (or if other conditions are met).

[0199] For example, if it is detected in the actual video that a medical professional has grasped a screwdriver, the system (10) may create an augmented reality video by adding recommended action information related to screw insertion to the actual video.

[0200] So far, a spinal surgery guide method according to some embodiments of the present disclosure has been described with reference to FIGS. 3 to 10.

[0201] As described above, augmented reality-based guidance information for spinal surgery can be provided to medical staff.

[0202] For example, an augmented reality image containing information about the current position and direction of the screw (or cage), the recommended direction, and the target insertion point can be provided to the medical staff.

[0203] Alternatively, an alarm may be provided to the medical staff if the current orientation of the screw (or cage) does not match the recommended orientation.

[0204] In these cases, the safety and accuracy of spinal surgery can be significantly improved, and the problem of significant differences in spinal surgery outcomes depending on factors such as the medical team's experience and physical condition, as well as the patient's condition, can be easily resolved. Furthermore, the issue of radiation exposure for patients undergoing spinal surgery can be completely resolved (since no X-ray equipment is used during spinal surgery).

[0205] Additionally, the direction of the screw (or cage) can be recognized in real images using an object detection model based on directional bounding boxes (e.g., YOLO).

[0206] Specifically, a directional bounding box and a horizontal bounding box for a screw (or cage) are predicted in a real image, and the direction of the screw (or cage) can be recognized based on the angular difference between the two bounding boxes.

[0207] In such cases, the direction of the screw (or cage) being inserted into the patient's spine can be accurately recognized, and as a result, the safety and accuracy of spinal surgery can be further improved.

[0208] Hereinafter, an exemplary computing device capable of implementing the spinal surgery guide system described above will be described with reference to FIG. 11.

[0209] Figure 11 is an exemplary hardware configuration diagram showing a computing device (110).

[0210] As illustrated in FIG. 11, a computing device (110) may include one or more processors (111), a bus (113), a communication interface (114), a memory (112) that loads a computer program (116) executed by the processor (111), and a storage (115) that stores the computer program (116).

[0211] However, only components related to the embodiments of the present disclosure are illustrated in FIG. 11. Therefore, those skilled in the art to which the present disclosure pertains will appreciate that other general components may be included in addition to the components (111 to 116) illustrated in FIG. 11.

[0212] That is, the computing device (110) may include various components in addition to the components (111 to 116) illustrated in FIG. 11.

[0213] Additionally, in some cases, the computing device (110) may be configured in a form in which some of the components (111 to 116) illustrated in FIG. 11 are omitted. Hereinafter, each component of the computing device (110) will be described.

[0214] The processor (111) can control the overall operation of each component of the computing device (110).

[0215] The processor (111) may be configured to include at least one of a CPU (Central Processing Unit), an MPU (Micro Processor Unit), an MCU (Micro Controller Unit), a GPU (Graphics Processing Unit), or any other type of processor well known in the art of the present disclosure.

[0216] Additionally, the processor (111) may perform operations on at least one application or computer program to execute specific operations / steps / methods. The computing device (110) may include one or more processors.

[0217] Next, the memory (112) can store various data, commands and / or information.

[0218] The memory (112) can load a computer program (116) from the storage (115) to perform specific operations / steps / methods.

[0219] The memory (112) may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

[0220] Next, the bus (113) can provide communication capabilities between components of the computing device (110).

[0221] The bus (113) can be implemented as various types of buses such as an address bus, a data bus, and a control bus.

[0222] Next, the communication interface (114) can support wired and wireless Internet communication of the computing device (110).

[0223] Additionally, the communication interface (114) may support various communication methods other than Internet communication.

[0224] For this purpose, the communication interface (114) may be configured to include a communication module well known in the technical field of the present disclosure.

[0225] Next, the storage (115) can non-temporarily store one or more computer programs (116).

[0226] Storage (115) may be configured to include non-volatile memory such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present disclosure pertains.

[0227] Next, the computer program (116) may include instructions that cause the processor (111) to perform specific operations / steps / methods when loaded into the memory (112).

[0228] That is, the processor (111) can perform specific operations / steps / methods by executing a computer program (116) loaded into the memory (112).

[0229] For example, the computer program (116) may include instructions for performing an operation of acquiring an actual image of a patient's spinal surgery, an operation of recognizing a position and direction of a screw to be inserted into a spinal region of the patient in the actual image using an object detection model based on a directional bounding box, and an operation of providing guidance information for the spinal surgery to a medical staff performing the spinal surgery based on the recognized position and direction.

[0230] As another example, the computer program (116) may include instructions to perform at least some of the operations / steps / methods described with reference to FIGS. 1 to 10.

[0231] In the case as exemplified, a spinal surgical guide system and / or surgical support system (10) according to some embodiments of the present disclosure may be implemented through a computing device (110).

[0232] Meanwhile, in some embodiments, the computing device (110) illustrated in FIG. 11 may mean a virtual machine implemented based on cloud technology.

[0233] For example, the computing device (110) may be a virtual machine operating on one or more physical servers included in a server farm.

[0234] In this case, at least some of the processor (111), memory (112), and storage (115) illustrated in FIG. 11 may be virtual hardware, and the communication interface (114) may also be implemented as a virtualized networking element such as a virtual switch.

[0235] So far, with reference to FIG. 11, an exemplary computing device (110) capable of implementing a spinal surgical guide system according to some embodiments of the present disclosure has been described.

[0236] Various embodiments of the present disclosure and effects according to the embodiments have been described with reference to FIGS. 1 to 11 so far.

[0237] The effects according to the technical idea of ​​the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0238] In addition, even though it has been described in the above embodiments that a plurality of components are combined or combined to operate as one, the technical idea of ​​the present disclosure is not necessarily limited to these embodiments.

[0239] That is, within the scope of the technical idea of ​​the present disclosure, all of the components may be selectively combined and operated one or more times.

[0240] The technical ideas of the present disclosure described so far can be implemented as computer-readable codes on a computer-readable recording medium.

[0241] A computer program recorded on a computer-readable recording medium can be transmitted to another computing device via a network such as the Internet and installed on the computing device, thereby allowing the computer program to be used on the computing device.

[0242] Although operations are depicted in the drawings in a particular order, it should not be understood that the operations must be performed in the particular order depicted or in any sequential order, or that all depicted operations must be performed to achieve the desired result.

[0243] In certain circumstances, multitasking and parallel processing may be advantageous. While various embodiments of the present disclosure have been described with reference to the attached drawings, those skilled in the art will appreciate that the technical concepts of the present disclosure can be implemented in other specific forms without altering the technical concepts or essential features thereof. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive. The scope of protection of the present disclosure should be interpreted by the claims below, and all technical concepts within the scope equivalent thereto should be interpreted as being included within the scope of the technical concepts defined by the present disclosure.

Claims

1. In a method performed by at least one processor, Step of acquiring actual images of the patient's spinal surgery; A step of recognizing the location and direction of a screw inserted into the patient's spine in the actual image using an object detection model based on an oriented bounding box; and A step of providing guide information for the spinal surgery to a medical staff performing the spinal surgery based on the recognized position and direction, How to guide spine surgery.

2. In paragraph 1, The step of recognizing the position and direction of the above screw is: A step of predicting a directional bounding box for the screw through the object detection model; and A step of recognizing the direction of the screw based on the angular difference between the predicted directional bounding box and the horizontal bounding box for the screw, How to guide spine surgery.

3. In paragraph 1, The step of recognizing the position and direction of the above screw is: A step of predicting a directional bounding box for a screw driver used for inserting the screw through the object detection model; and A step of recognizing the direction of the screw based on the angular difference between the predicted directional bounding box and the horizontal bounding box for the screw driver, How to guide spine surgery.

4. In paragraph 1, The step of recognizing the position and direction of the above screw is: A step of predicting a directional bounding box for a hand part of the medical staff holding a screwdriver using the object detection model, wherein the screwdriver is used to insert the screw; and A step of recognizing the direction of the screw based on the angular difference between the predicted directional bounding box and the horizontal bounding box for the hand part, How to guide spine surgery.

5. In paragraph 1, The step of providing guidance information for the above spinal surgery is: A step of acquiring a virtual spinal structure image of the above patient; A step of creating an augmented reality image by aligning the virtual spinal structure image with the actual image; and Including a step of providing the guide information to the medical staff based on the augmented reality image. How to guide spine surgery.

6. In paragraph 5, The above augmented reality image is provided to the medical staff through HMD (Head Mounted Display). How to guide spine surgery.

7. In paragraph 1, The step of providing guidance information for the above spinal surgery is: Including a step of providing an alarm to the medical staff when the difference between the recognized direction and the preset recommended direction is greater than a reference value. How to guide spine surgery.

8. In paragraph 1, The step of providing guidance information for the above spinal surgery is: A step of generating an augmented reality image by adding a first direction indicator indicating the recognized direction and a second direction indicator indicating a preset recommended direction to the actual image; and Including a step of providing the augmented reality image to the medical staff, How to guide spine surgery.

9. In paragraph 8, The first direction indicator or the second direction indicator is added to the actual image when the difference between the recognized direction and the recommended direction is greater than a reference value. How to guide spine surgery.

10. In paragraph 1, A step of recognizing the posture of the hand of the medical staff holding a screwdriver through a deep learning model, wherein the screwdriver is used to insert the screw; If the difference between the recognized posture and the predefined recommended posture is greater than a reference value, a step of generating an augmented reality image by adding information about the recommended posture to the actual image; and Further comprising a step of providing the augmented reality image to the medical staff, How to guide spine surgery.

11. In paragraph 1, A step of recognizing a screw insertion motion of the medical staff holding a screwdriver through a deep learning model, wherein the screwdriver is used to insert the screw; If the difference between the recognized motion and the predefined recommended motion is greater than a reference value, a step of generating an augmented reality image by adding information about the recommended motion to the actual image; and Further comprising a step of providing the augmented reality image to the medical staff, How to guide spine surgery.

12. One or more processors; and A memory storing a computer program executed by one or more processors, The above computer program: The action of obtaining actual images of a patient's spinal surgery; An operation of recognizing the location and direction of a screw inserted into the patient's spine in the actual image using an object detection model based on an oriented bounding box; and Including instructions for an operation that provides guidance information for the spinal surgery to the medical staff performing the spinal surgery based on the recognized position and direction. Spine surgical guide system.

13. Combined with the computer's processor, Step of acquiring actual images of the patient's spinal surgery; A step of recognizing the location and direction of a screw inserted into the patient's spine in the actual image using an object detection model based on an oriented bounding box; and In order to execute a step of providing guide information for the spinal surgery to the medical staff performing the spinal surgery based on the recognized position and direction, stored in a computer-readable recording medium, Computer program.

Images