Method for generating surgical assistance data and apparatus using same

Abstract

A method for generating surgical assistance data, performed by at least one processor, according to the present invention, may comprise the steps of: acquiring, from a medical staff terminal, patient information and first image data, second image data, and third image data corresponding to a treatment region of a patient; generating a three-dimensional modeling image on the basis of the first image data and the second image data; registering the three-dimensional modeling image with the third image data and placing the registered images in a single spatial coordinate system; generating surgical assistance data on the basis of the registered images and the patient information; and transmitting the surgical assistance data to the medical staff terminal.

Description

Method for generating surgical assistance data and device using the same

[0001] The present invention relates to a method for generating surgical assistance data and an apparatus thereof, and more specifically, to a method for generating surgical assistance data that helps medical staff overcome temporal and spatial constraints during surgery.

[0002]

[0003] Existing surgical systems are performed only at the operating room surrounding the medical team operating on the patient; consequently, these systems face temporal and spatial limitations in surgeries requiring high-level technical skills. Therefore, there is a need for technology that reduces the surgical burden on medical staff.

[0004] Furthermore, conventional medical imaging cannot intuitively represent the location of surgical instruments within a body part. Therefore, determining the precise location of the instrument by comparing multiple cross-sectional images requires not only the expertise of medical staff but can also be time-consuming.

[0005] The present invention provides a method for generating surgical assistance data to solve the aforementioned problems.

[0006]

[0007] The various embodiments described in this specification are intended to provide a method for generating surgical assistance data that helps overcome the time and spatial constraints of medical staff surgery.

[0008] The problems that this disclosure aims to solve are not limited to those described above, and other unmentioned problems will be clearly understood by a person skilled in the art from the description below.

[0009]

[0010] In one embodiment, a method for generating surgical assistance data performed by at least one processor,

[0011] A method for generating surgical assistance data is provided, comprising: a step of acquiring first image data, second image data, and third image data regarding patient information and the patient's treatment area from a medical staff terminal; a step of generating a three-dimensional modeling image based on the first image data and the second image data; a step of aligning the three-dimensional modeling image and the third image data and placing them in a single spatial coordinate system; a step of generating surgical assistance data based on the aligned image and the patient information; and a step of transmitting the surgical assistance data to the medical staff terminal.

[0012] Here, the first image data and the second image data are data from any one of CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound, X-RAY, PET (Positron Emission Tomography), and three-dimensional scan images, and the third image data may be image data of a support device located at the patient's treatment site.

[0013] Here, the step of generating the three-dimensional modeling image may include: providing a first user interface (UI) including first image data and second image data; obtaining user input related to arrangement points for the first image data and second image data through the first user interface; and arranging the first image data and second image data based on the user input.

[0014] Here, the user input includes a first signal corresponding to the selection of a first point in the first image data and a second signal corresponding to the selection of a second point in the second image data, and the step of arranging the first image data and the second image data may be a step of matching the first point and the second point based on the first signal and the second signal.

[0015] Here, the step of generating the three-dimensional modeling image may include: providing a second user interface including the first image data, the second image data, and the first button; and, when user input corresponding to pressing the first button is obtained through the second user interface, matching the first point and the second point.

[0016] Here, the three-dimensional modeling image includes a plurality of distinct components, and the plurality of components may include at least one of bones, nerves, and organs.

[0017] Here, the first image data includes a plurality of detectable first markers, the third image data includes a plurality of second markers within the support device located at the patient's treatment site, and the three-dimensional modeling image based on the first image data and the second image data may include a plurality of first markers.

[0018] Here, the step of arranging in a single spatial coordinate system may include: providing a third user interface including the three-dimensional modeling image and third image data; obtaining a third signal that selects some of the plurality of first markers included in the three-dimensional modeling image as points and a fourth signal that selects some of the plurality of second markers included in the third image data as points through the third user interface; and aligning the three-dimensional modeling image and the third image data based on points using the third signal and the fourth signal.

[0019] Here, the step of placing in the above-mentioned spatial coordinate system further includes a step of calculating the error of the point-based alignment; and a step of displaying the error of the point-based alignment, wherein the error of the point-based alignment may include a reference point designation error and a reference point alignment error.

[0020] Here, the step of placing in the above-mentioned spatial coordinate system may include the step of providing a fourth user interface including a modification button for changing the position and angle of the first image data and the second image data.

[0021] Here, the step of generating the surgical assistance data may include: a step of obtaining information about a medical product for the patient's treatment area; a step of providing a fifth user interface including the matched image and obtaining a signal for selecting an area within the matched image through the fifth user interface; and a step of placing the medical product based on the signal.

[0022] Herein, the method may further include the steps of: setting a target area in the matched image based on the position where the selected medical product is placed in the matched image and the patient's information; generating fourth image data including the target area; obtaining current location information for a fixation device to be placed in the target area from the medical staff terminal and generating matching degree data based on the degree of matching between the location information and the target area; and transmitting the matching degree data and the fourth image data to the medical staff terminal.

[0023] In another embodiment, a computer program stored on a computer-readable storage medium may be provided to execute the surgical assistance data generation method on a computer system.

[0024] In another embodiment, the device includes a communication unit for transmitting and receiving data with a medical staff terminal; a storage unit; a user interface unit; a display unit; and a control unit connected to the communication unit, the storage unit, the interface unit and the display unit.

[0025] The above control unit may provide a surgical assistance data generation device that acquires first image data, second image data, and third image data regarding patient information and the patient's treatment area from a medical staff terminal, generates a three-dimensional modeling image based on the first image data and the second image data, aligns the three-dimensional modeling image and the third image data to place them in a single spatial coordinate system, generates surgical assistance data based on the aligned image and the patient information, and transmits the surgical assistance data to the medical staff terminal.

[0026]

[0027] According to one embodiment of the present invention, a method for generating surgical assistance data and an apparatus thereof may be provided, thereby overcoming temporal and spatial constraints on surgery performed by medical staff.

[0028] The effects according to the present invention are not limited to those described above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0029]

[0030] FIG. 1 is an environment diagram of a surgical assistance data generation method according to one embodiment.

[0031] FIG. 2 is a configuration diagram of a surgical assistance data generation device according to one embodiment.

[0032] FIG. 3 is a flowchart of a method for generating surgical assistance data according to one embodiment.

[0033] FIG. 4 is a flowchart of a method for generating a three-dimensional modeling image including a first user interface according to one embodiment.

[0034] FIG. 5 is an example diagram illustrating the step of acquiring a first signal and a second signal according to one embodiment and matching a first point and a second point.

[0035] FIG. 6 is a diagram illustrating a second user interface including a first button according to one embodiment.

[0036] FIG. 7 is an example diagram of a second user interface according to one embodiment.

[0037] FIG. 8 is a flowchart of the step of matching through a third user interface according to one embodiment.

[0038] FIG. 9 is a diagram illustrating the step of calculating the error of point-based alignment according to one embodiment.

[0039] FIG. 10 is a diagram illustrating a fourth user interface including a modification button according to one embodiment.

[0040] FIG. 11 is an example diagram of a fourth user interface including a modification button according to one embodiment.

[0041] FIG. 12 is a flowchart of the step of generating surgical assistance data by selecting an area within a matched image through a fifth user interface according to one embodiment.

[0042] FIG. 13 is an example diagram of a method for placing a medical product through a fifth user interface according to one embodiment.

[0043] FIG. 14 is a diagram illustrating the step of generating surgical assistance data by generating matching degree data based on location information of a target area and an actual product according to one embodiment.

[0044] FIG. 15 is an example of fourth image data including a target area according to one embodiment.

[0045]

[0046] Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the contents described in the attached drawings. However, the present invention is not limited or restricted by exemplary embodiments. Unless otherwise defined, all terms used in this specification (including technical and scientific terms) shall be used in a meaning that is commonly understood by those skilled in the art to which this disclosure belongs, but this may vary depending on the intent of those skilled in the art, case law, the emergence of new technology, etc.

[0047] Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. In certain cases, terms have been selected at the applicant's discretion, and in such cases, their meanings will be described in detail in the relevant explanatory sections. Accordingly, terms used in this disclosure should be defined not merely by their names, but based on their meanings and the content throughout this disclosure.

[0048] Throughout this specification, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components. Furthermore, the singular form used in this specification includes the plural form unless specifically stated otherwise. Additionally, the expression "at least one of a, b, and / or c" as used throughout this specification may encompass 'a alone', 'b alone', 'c alone', 'a and b', 'a and c', 'b and c', or 'a, b, and c all'.

[0049] Meanwhile, terms such as "first and / or second" used in this specification may be used to describe various components, but they are used solely for the purpose of distinguishing one component from another and are not intended to limit the scope to the components referred to by such terms. For example, without departing from the scope of the present invention, the first component may be named the second component, and the second component may also be named the first component.

[0050] Additionally, terms such as “…part,” “…module,” etc., as described in this specification refer to a unit that processes at least one function or operation, which may be implemented in hardware or software, or a combination of hardware and software. Furthermore, embodiments of this disclosure may be represented in this specification by functional block configurations and various processing steps. These functional blocks may be implemented by various numbers of hardware and / or software configurations that execute specific functions. For example, embodiments of this disclosure may employ integrated circuit configurations such as memory, processing, logic, look-up tables, etc., which can execute various functions under the control of one or more microprocessors or other control devices.

[0051] In an embodiment according to the present disclosure, functions related to artificial intelligence may be implemented through a processor and memory. In this case, the processor may be any one of a general-purpose processor such as a CPU (Center Processing Unit), AP (Application Processor), DSP (Digital Signal Processor), a graphics-dedicated processor such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit), and an artificial intelligence-dedicated processor such as an NPU (Neural Network Processing Unit). The processor may process input data according to predefined operation rules or artificial intelligence models stored in memory. Alternatively, if the processor is an artificial intelligence-dedicated processor, the artificial intelligence-dedicated processor may be designed with a hardware structure specialized for processing a specific artificial intelligence model. In some embodiments according to the present disclosure, functions related to artificial intelligence may be implemented through a plurality of processors.

[0052] In an embodiment according to the present disclosure, a predefined operation rule or artificial intelligence model may be configured to perform machine learning. Here, being configured to perform machine learning means that the predefined operation rule or artificial intelligence model is configured to perform a desired characteristic (or objective) by learning using a plurality of training data based on a learning algorithm. Such learning may be performed on the device itself in which the artificial intelligence according to the present disclosure is implemented, or it may be performed through a separate server and / or system.

[0053] Artificial intelligence models can be implemented as neural networks (or artificial neural networks) and can operate based on statistical learning algorithms that mimic biological neurons in machine learning and cognitive science. A neural network can refer to a model in which artificial neurons (nodes), which form a network through the connection of synapses, change the strength of synaptic connections through learning to possess problem-solving capabilities. A neural network can be composed of multiple neural network layers; for example, a neural network may include an input layer, a hidden layer, and an output layer. Each of the multiple neural network layers may include at least one node and at least one weight, and neural network operations can be performed through operations between the results of operations of the previous (precious) layer and the weights. At least one weight possessed by the multiple neural network layers may be optimized based on the learning results of the artificial intelligence model. For example, at least one weight may be updated so that the loss value or cost value obtained from the artificial intelligence model during the learning process is reduced or minimized. Neural networks can infer a result to be predicted from an arbitrary input.

[0054] The learning methods of artificial intelligence models can be classified according to the learning approach into supervised learning, where input and output data are provided as training data and the correct answer (output data) corresponding to the problem (input data) is predetermined; unsupervised learning, where only input data is provided without output data and the correct answer (output data) corresponding to the problem (input data) is not predetermined; and reinforcement learning, where a reward is granted whenever an action is taken from the current state and learning proceeds in a direction that maximizes this reward. Alternatively, they can be classified according to the architecture, which is the structure of the learning model.

[0055] In the embodiments of the present disclosure, the artificial intelligence model is a Convolutional Neural Network (CNN) such as GoogleNet, AlexNet, VGG Network, Region with Convolutional Neural Network (R-CNN), Region Proposal Network (RPN), Recurrent Neural Network (RNN), Stacking-based Deep Neural Network (S-DNN), State-Space Dynamic Neural Network (S-SDNN), Deconvolution Network, Deep Belief Network (DBN), Restructured Boltzmann Machine (RBM), Fully Convolutional Network, Long Short-Term Memory Network (LSTM), Classification Network, Generative Modeling, eXplainable AI, Continual AI, Representation Learning, AI for Material Design, BERT, SP-BERT, MRC / QA for Natural Language Processing, Text Analysis, Dialog System, GPT-3, GPT-4, Visual Analytics, Visual Understanding, Video Synthesis for Vision Processing, Anomaly Detection, Prediction, Time-Series Forecasting, Optimization, Recommendation for ResNet Data Intelligence, At least one of various artificial intelligence structures and algorithms, such as data creation, may be used. The examples described above are merely examples of artificial intelligence structures and algorithms used according to the embodiments of the present disclosure and do not limit the artificial intelligence structures and algorithms used according to the embodiments of the present disclosure.

[0056] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In describing the embodiments, technical details that are well known in the art to which the present invention pertains and are not directly related to the present invention will be omitted. This is to ensure that the essence of the present invention is conveyed more clearly without obscuring it by omitting unnecessary explanations. For the same reason, some components in the accompanying drawings may be exaggerated, omitted, or schematically depicted. Furthermore, the size of each component does not entirely reflect its actual size. Throughout this specification, the same reference numerals may refer to the same or corresponding components.

[0057] FIG. 1 is an environment diagram of a surgical assistance data generation method according to one embodiment.

[0058] Referring to FIG. 1, a method for generating surgical assistance data according to one embodiment can be performed through a server (1000), a surgical assistance data generating device (100, hereinafter 'device'), and a medical staff terminal (200).

[0059] A server (1000) is a computing device comprising at least one processor and may be the overall entity of a surgical assistance data generation method. The server (1000) may include a controller comprising at least one processor, a communication module, and memory. Additionally, the components included in the server (1000) may be physically contained in a single server or may be distributed servers distributed according to their respective functions. Unless otherwise specifically stated below, the operation of the server (1000) may be interpreted as being performed under the control of at least one processor included in the server (1000).

[0060] The server (1000) can acquire and store data from the device (100). At this time, the device (100) may be a computing device used by the medical staff to generate surgical assistance data necessary when planning surgery. The device (100) may be located in a different location from where the medical staff terminal is located. Therefore, the device (100) can provide surgical assistance data to the medical staff terminal remotely.

[0061] The server (1000) can acquire and store data from the medical staff terminal (200). At this time, the medical staff terminal (200) may be a computing device used by medical staff. The medical staff terminal (200) may be located in a different location from where the device (100) is located.

[0062] The server (1000) receives data from the device (100) and the medical staff terminal (200), and the device (100) can generate surgical assistance data and transmit it to the medical staff terminal (200) through the server (1000).

[0063] FIG. 1 illustrates a device (100) and a medical staff terminal (200) that transmit and receive data through a server (1000), but is not limited thereto, and the device (100) and the medical staff terminal (200) may be able to communicate directly with each other.

[0064]

[0065] FIG. 2 is a configuration diagram of a surgical assistance data generation device according to one embodiment.

[0066] Referring to FIG. 2, a surgical assistance data generation device (100) according to one embodiment may include a control unit (110), a storage unit (120), a communication unit (130), a display unit (140), and an interface unit (150). FIG. 2 illustrates five components included in the device (100), but is not limited thereto, and the device (100) may include fewer or more components. Additionally, each component of the device (100) may be physically included in a single device, or may be configured as individual devices distributed according to each function. For example, the device (100) may include a control device (or module), a storage device, a communication device, a display device, an interface device, etc., but is not limited thereto.

[0067] The control unit (110) can control the overall operations of the device (100). For example, the control unit (110) can control the transmission and reception of data and signals through the communication unit (130). According to one embodiment, the control unit (110) can control the storage unit (120), the communication unit (130), the display unit (140), and the interface unit (150) so that the device (100) performs methods according to various embodiments described below. Unless otherwise specifically mentioned below, the operation of the device (100) may be interpreted as being performed under the control of the control unit (110).

[0068] The storage unit (120) can store various data and programs necessary for the device (100) to operate. The storage unit (120) can store information for performing methods according to various embodiments described below. In one embodiment, the storage unit (120) can store patient information and image data.

[0069] The storage unit (120) can store data temporarily or semi-permanently. Examples of the storage unit (120) may include a hard disk drive (HDD), a solid state drive (SSD), flash memory, ROM (Read-Only Memory), RAM (Random Access Memory), or cloud storage. However, it is not limited to these, and the storage unit (120) may be implemented as various modules for storing data.

[0070] The communication unit (130) performs functions for transmitting and receiving data and signals according to various embodiments described below through a wireless channel. In one embodiment, data or signals provided under the control of the control unit (110) of the device are transmitted to a medical staff terminal (200) through the communication unit (130), and data or signals received from the medical staff terminal (200) are received by the device (100).

[0071] The communication unit (130) may be a communication module that supports at least one of a wired communication method and a wireless communication method. For example, the communication unit (130) may acquire data from an external device using communication methods such as WiFi, Bluetooth, Zigbee, BLE (Bluetooth Low Energy), and RFID, but is not limited thereto.

[0072] The display unit (140) can display information or data according to various embodiments described below through a screen. The display unit (140) can display individually independent data or display one or more data. For example, the display unit (140) may be a display panel such as an LCD, LED, OLED, AMOLED, PMOLED, etc., but is not limited thereto. At this time, the form in which the display unit (140) outputs information may include at least one of a visual form, an auditory form (sound), and a tactile form (vibration).

[0073] The interface unit (150) can generate a user interface to be displayed on the display unit (140). Specifically, the interface unit (150) can generate an interface to be displayed on a device in which functions for input and output are integrated into one, such as a touchscreen. The interface unit (150) can receive user input through the display unit (140) and can receive data to be output through the display unit (140) in response to user input from the control unit (110).

[0074] The medical staff terminal (200) may also include a control unit, a storage unit, a communication unit, a display unit, and an interface unit (not shown).

[0075]

[0076] FIG. 3 is a flowchart of a method for generating surgical assistance data according to one embodiment.

[0077] Referring to FIG. 3, the surgical assistance data generation method may include the steps of: obtaining patient information and image data from a server (S100); generating a three-dimensional modeling image from the image data (S200); aligning the three-dimensional modeling image and placing it in a single spatial coordinate system (S300); generating surgical assistance data based on the aligned image (S400); transmitting the surgical assistance data to a server (S500); obtaining actual surgical data (S600); and generating surgical assistance data by reflecting the actual surgical data (S700).

[0078] FIG. 3 illustrates steps S100 to S700 being performed sequentially, but is not limited thereto and the order of steps may be changed, some steps omitted, or new steps added.

[0079] In step S100, the device (100) can obtain patient information and image data from the server (1000). Specifically, the device (100) can obtain patient information and image data transmitted by the medical staff terminal (200) from the server (1000) through the communication unit (130).

[0080] Patient information may include, but is not limited to, the patient's basic information, medical records, current condition, and surgery date. For example, the patient's basic information may include the patient's name, date of birth, gender, contact information, etc. Medical records may include the type of disease, history of surgery and hospitalization, and results of tests performed prior to surgery. The patient's current condition may include the patient's level of consciousness, blood pressure, heart rate, etc.

[0081] Image data may include first image data, second image data, and third image data. The first image data, second image data, and third image data may refer to images used in digital imaging and communication in medical devices. For example, the data may be in the DICOM (Digital Imaging and Communications in Medicine) format, but is not limited to that, and may be any one of CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound, X-RAY, PET (Positron Emission Tomography), and 3D scan images. The image data may be captured using a camera or various sensors.

[0082] Specifically, the third image data may be image data of a support device fixed to the patient's treatment site. The support device may be a three-dimensional object or frame structure, for example, a plate, fixed to the patient's treatment site. The third image data may be data captured after the patient wears the support device, or data captured of the support device itself.

[0083] In step S200, the device (100) can generate a three-dimensional modeling image based on the first image data and the second image data by providing a user interface.

[0084] Hereinafter, the step of generating a 3D modeling image will be described in detail in FIGS. 4 to 7.

[0085]

[0086] FIG. 4 is a flowchart of a method for generating a three-dimensional modeling image including a first user interface according to one embodiment.

[0087] Referring to FIG. 4, the step of generating a three-dimensional modeling image according to one embodiment may include providing a first user interface (UI) including first image data and second image data (S210), obtaining user input related to an arrangement point through the first user interface (S220), and arranging the first image data and second image data based on the user input (S230).

[0088] In step S210, the server (1000) may provide a first user interface to the device (100) to arrange the first image data and the second image data based on user input.

[0089] In step S220, the server (1000) can obtain user input related to arrangement points for the first image data and the second image data from the device (100) through the first user interface. Specifically, the server (1000) can obtain a first signal corresponding to the selection of a first point within the first image data and a second signal corresponding to the selection of a second point within the second image data. Here, the first point and the second point may be multiple points within the image.

[0090] In step S230, the server (1000) can arrange the first image data and the second image data based on user input related to the arrangement point obtained in step S220. Specifically, the server (1000) can arrange the first image data and the second image data by matching the arrangement point of the first image data with the arrangement point of the second image data. More specifically, the server (1000) can arrange the first image data and the second image data by matching the first point and the second point based on the first signal and the second signal.

[0091] The server (1000) can generate three-dimensional modeling image data by arranging the first image data and the second image data.

[0092] In this specification, the term 'arrange' may mean aligning different image data and may be used interchangeably with the terms 'place' and 'align'.

[0093]

[0094] FIG. 5 is an example diagram illustrating the step of acquiring a first signal and a second signal according to one embodiment and matching a first point and a second point.

[0095] According to the method of FIG. 5, a first signal is obtained by selecting four first points (11, 12, 13, and 14) from the first image data on the left and a second signal is obtained by selecting four second points (21, 22, 23, and 24) from the second image data on the right, and the first image data and the second image data can be arranged by matching the four first points and the four second points based on the first signal and the second signal.

[0096] For example, the first image data and the second image data can be arranged by matching the first-1 point (11) in the first image data with the second-1 point (21) in the second image data, the first-2 point (12) in the first image data with the second-2 point (22) in the second image data, the first-3 point (13) in the first image data with the second-3 point (23) in the second image data, and the first-4 point (14) in the first image data with the second-4 point (24) in the second image data.

[0097]

[0098] FIG. 6 is a diagram illustrating a second user interface including a first button according to one embodiment.

[0099] Referring to FIG. 6, the second user interface may include a first button (2300), first image data (2100), and second image data (2200). When user input corresponding to pressing the first button is obtained through the second user interface, the first image data and the second image data may be arranged to generate a three-dimensional modeling image (2400). Specifically, when user input corresponding to pressing the first button is obtained, a first point in the first image data and a second point in the second image data may be aligned to generate a three-dimensional modeling image.

[0100] A 3D modeling image may include multiple distinct components. The multiple components may include at least one of bones, nerves, and organs. Accordingly, a 3D modeling image can be generated by displaying the multiple components in different colors.

[0101]

[0102] FIG. 7 is an example diagram of a second user interface according to one embodiment.

[0103] According to the method of FIG. 7, when user input corresponding to pressing the first button (2300) in FIG. 7 is obtained through the second user interface, the CT data (2100) on the upper left and the 3D scan image data (2200) on the upper right can be arranged to generate the 3D modeling image (2400) at the bottom.

[0104] In FIG. 7, the locations where the first image (2100), second image (2200), first button (2300), and 3D modeling image (2400) are displayed in the second user interface are exemplary and may be changed. In FIG. 7, the shape of the first button is shown as a box, but is exemplary and is not limited.

[0105]

[0106] In step S300, the server (1000) can align the 3D modeling image and the third image data and place them in a single spatial coordinate system.

[0107] The step of arranging in a single spatial coordinate system below will be described in detail in FIGS. 8 to 11.

[0108] FIG. 8 is a flowchart of the step of matching through a third user interface according to one embodiment.

[0109] Referring to FIG. 8, the step of matching through a third user interface may include providing a third user interface including a three-dimensional modeling image and third image data (S310), acquiring a third signal and a fourth signal for selecting some of a plurality of markers as points (S320), and matching based on points based on the third signal and the fourth signal (S330).

[0110] In step S310, the three-dimensional modeling image may include a plurality of first markers. Specifically, the first image data includes a plurality of detectable first markers, and the three-dimensional modeling image generated based on the first image data may include a plurality of first markers. Additionally, the third image data may include a plurality of second markers within a support device located at the patient's treatment site.

[0111] The support device may include a plurality of markers of a detectable material. For example, the markers of the detectable material may be metal beads. A plurality of first markers and a plurality of second markers may be sensed by a camera as detectable materials.

[0112] In step S320, the server (1000) may obtain from the device (100) a third signal for selecting some of the plurality of first markers included in the three-dimensional modeling image as points through a third user interface, and a fourth signal for selecting some of the plurality of second markers included in the third image data as points. Here, the selection of markers may correspond to clicking a marker or placing a cursor over a marker.

[0113] In step S330, the server (1000) can perform point-based registration of some of the first markers and some of the second markers selected based on the third signal and the fourth signal. Point-based registration may refer to a technique that uses corresponding points in two data sets to move or transform one data to fit the other data to the fixed data using a fixed reference.

[0114]

[0115] FIG. 9 is a diagram illustrating the step of calculating the error of point-based alignment according to one embodiment.

[0116] Referring to FIG. 9, the step of placing in a spatial coordinate system may further include a step of calculating the error of point-based alignment (S340) and a step of displaying the error of point-based alignment (S350).

[0117] In step S340, the point-based alignment may include a reference point designation error and a reference point alignment error.

[0118] Fiducial localization error (FLE) may refer to an error that occurs while designating markers. Specifically, fiducial localization error may refer to an error that occurs based on one of the corresponding points when selecting some of a plurality of first markers as points and some of a plurality of second markers as points through a third user interface.

[0119] Fiducial registration error (FRE) may refer to an error that occurs during registration based on a reference point. Specifically, fiducial registration error may refer to an error that occurs during registration based on a first marker in the 3D modeling image and a second marker in the third image data. Fiducial registration error is an error that occurs during registration based on a reference point and may appear by reflecting the reference point designation error.

[0120] In step S330, if the reference point designation error or the reference point alignment error exceeds a preset threshold, the method may further include a step of outputting a warning signal to allow the user to re-select the third signal or the fourth signal. For example, if the reference point designation error exceeds the threshold and a warning signal is output, the user can re-select the third signal or the fourth signal to correct the reference point designation error. Alternatively, if the reference point alignment error exceeds the threshold, the server (1000) can re-acquire the third signal or the fourth signal through the device (100) and the server (1000) can correct the reference point designation error. Additionally, since the reference point alignment error reflects the reference point designation error, the server (1000) can also correct the reference point designation error.

[0121] In step S340, the server (1000) can display the reference point designation error and the reference point alignment error on the device (100) to allow the user to select the third signal or the fourth signal again.

[0122]

[0123] FIG. 10 is a diagram illustrating a fourth user interface including a modification button according to one embodiment.

[0124] Referring to FIG. 10, the fourth user interface may include a point-based alignment error (3100), a correction button (3200), and a surgical assistance sequence (3300).

[0125] The error (3100) of point-based alignment may be the error calculated in step S340, specifically, the reference point designation error and the reference point alignment error.

[0126] The modification button (3200) may be a button for changing the position and angle of the first image data and the second image data. For example, the modification button may be an icon indicating up, down, left, and right for positional movement. Additionally, it may include an icon indicating clockwise or counterclockwise direction for angle movement, but is not limited thereto. Through this, the degree of arrangement of the three-dimensional image data can be increased. When user input corresponding to pressing the modification button (3200) is obtained through the fourth user interface, the position and angle of the first image data and the second image data can be changed.

[0127] The surgical assistance sequence (3300) may be a part containing characters describing steps S200 to S400.

[0128]

[0129] FIG. 11 is an example diagram of a fourth user interface including a modification button according to one embodiment.

[0130] According to the method of FIG. 11, the device (100) can output a point-based alignment error (3100) including a reference point designation error (FLE) and a reference point alignment error (FRE).

[0131] Additionally, when the server (1000) obtains user input corresponding to pressing an icon indicating up, down, left, and right through the fourth user interface, it moves the positions of the first image data and the second image data based on this user input to enable more accurate arrangement.

[0132] In FIG. 11, the locations where the point-based alignment error (3100), the correction button (3200), and the surgical assistance sequence (3300) are output from the device (100) are exemplary and may be changed.

[0133]

[0134] In step S400, the device (100) can generate surgical assistance data based on the image and patient information matched in step S300.

[0135] The steps for generating surgical assistance data below will be described in detail in FIGS. 12 to 15.

[0136] FIG. 12 is a flowchart of the step of generating surgical assistance data by selecting an area within a matched image through a fifth user interface according to one embodiment.

[0137] Referring to FIG. 12, the step of generating surgical assistance data according to one embodiment may include the step of obtaining a selection for a medical product (S410), the step of obtaining a signal for selecting a region within a matched image through a fifth user interface (S420), and the step of placing the medical product based on the signal (S430).

[0138] In step S410, the server (1000) can obtain information about a medical product to be inserted into the patient's treatment site. Specifically, the device (100) can store options for the medical product in memory and output them, obtain a signal corresponding to the selection of one of the medical products, and transmit it to the server (1000). The signal corresponding to the selection of one medical product can be obtained by clicking the outputted option with a cursor.

[0139] Medical products to be inserted into the patient's treatment area may include, for example, cosmetic and plastic surgery implants including breast implants, orthopedic implants including spinal fixation devices, cardiovascular implants including stents, dental implants including implants, auditory implants including cochlear implants, but are not limited thereto.

[0140] In step S420, the server (1000) provides a fifth user interface including a matched image and can obtain a signal for selecting an area within the matched image through the fifth user interface. The signal for selecting an area within the matched image can be obtained by clicking a cursor.

[0141] In step S430, the server (1000) can place the medical product selected in step S410 based on a signal selecting an area within the matched image.

[0142] In addition, the medical product placed through the fifth user interface can be rotated and its angle adjusted within the aligned image.

[0143]

[0144] FIG. 13 is an example diagram of a method for placing a medical product through a fifth user interface according to one embodiment.

[0145] According to the method of FIG. 13, the server (1000) can acquire information of the implant (type, diameter, length, etc.), acquire a signal (31) for selecting an area within the aligned image through the fifth user interface, and place the implant based on this signal.

[0146]

[0147] FIG. 14 is a diagram illustrating the step of generating surgical assistance data by generating matching degree data based on location information of a target area and an actual product according to one embodiment.

[0148] Referring to FIG. 14, the step of generating surgical assistance data may include the step of setting a target area in a matched image (S440), the step of generating a fourth image data including the target area (S450), and the step of generating a degree of matching data (S460).

[0149] In step S440, the server (1000) may set a target area on the aligned image based on the location where the medical product was placed in step S430 and the patient's information. Step S440 may be a step for inserting a fixation device to insert the medical product into the patient's treatment area. Step S440 may be a step for pre-setting the location of the fixation device to assist the medical staff in surgery. Thus, the target area may be an area where the fixation device is to be inserted. The fixation device may include, for example, a screw, a pin, or a metal plate.

[0150] In step S450, the server (1000) may generate fourth image data including a target area. The fourth image data may be data to be sent to a medical staff terminal (200), including a target area which is an area where a fixation device is to be inserted.

[0151] In step S460, the server (1000) obtains current location information of a fixation device to be inserted into a target area from a medical staff terminal (200) and can generate matching degree data based on the degree of matching between the current location information and the target area. The current location information of the fixation device can be sensed by a camera. The current location information and the target area may overlap, and thus, the matching degree data may be overlap degree data. For example, the matching degree data may be color-coded according to the degree of matching.

[0152] The server (1000) can transmit matching degree data and fourth image data to the medical staff terminal.

[0153]

[0154] FIG. 15 is an example of fourth image data including a target area according to one embodiment.

[0155] According to the method of FIG. 15, the server (1000) can generate fourth image data by pre-setting a target area (41, brown) of a screw, which is a fixing device for inserting an implant. The server (1000) can acquire current location information (42, yellow) for the screw to be inserted into the target area and generate matching degree data (43, red) based on the degree of matching between the current location information and the target area.

[0156]

[0157] Referring again to FIG. 3, in step S600, the device (100) obtains actual surgical data from the server (1000), and in step S700, the device (100) can generate surgical assistance data by reflecting the actual surgical data.

[0158] In step S700, the artificial intelligence model of the server (1000) can learn the difference using the surgical assistance data transmitted by the device (100) and the actual surgical data received from the medical staff terminal (200). The artificial intelligence model of the server (1000) can transmit an optimized result value to the control unit (110) based on the patient's information and image data. For example, the control unit (110) can generate a 3D modeling image using the optimized result value in step S200, or use it in step S300 to align the 3D modeling image and the third image data and place them in a single spatial coordinate system, or use it in step S400 to generate surgical assistance data based on the aligned image and the patient's information.

[0159] As such, according to the embodiments of the present invention, all pre-operative processes can be prepared through the interface of the surgical assistance data generation device, thereby overcoming the temporal and spatial constraints of the medical team's surgery.

[0160] Meanwhile, the embodiments disclosed in this specification may be implemented in the form of a recording medium that stores instructions executable by a computer. The instructions may be stored in the form of program code and, when executed by a processor, may generate a program module to perform the operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium. A computer-readable recording medium may include all types of recording media that store instructions decipherable by a computer. Examples include ROM, RAM, magnetic tape, magnetic disk, flash memory, optical data storage devices, etc.

[0161] The above descriptions are specific embodiments for carrying out the present disclosure. The present disclosure will include not only the embodiments described above, but also embodiments that can be simply modified or easily modified. Furthermore, the present disclosure will include technologies that can be easily modified and implemented using the embodiments described above. Accordingly, the scope of the present disclosure should not be limited to the embodiments described above, but should be defined by the claims set forth below as well as equivalents to the claims of the present disclosure.

Claims

1. A method for generating surgical assistance data performed by at least one processor, A step of obtaining patient information and first image data, second image data, and third image data regarding the patient's treatment area from a medical staff terminal; A step of generating a three-dimensional modeling image based on the first image data and the second image data; A step of aligning the above 3D modeling image and the third image data to place them in a single spatial coordinate system; A step of generating surgical assistance data based on the above-mentioned matched image and the patient's information; and The step of transmitting the above surgical assistance data to the medical staff terminal; Method for generating surgical assistance data.

2. In Paragraph 1, The first image data and the second image data are data from any one of CT (Computed Tomography), MRI (Magnetic Resonance Imaging), ultrasound, X-RAY, PET (Positron Emission Tomography), and 3D scan images, and The above third image data is image data of a support device located at the patient's treatment site. Method for generating surgical assistance data.

3. In Paragraph 1, The step of generating the above 3D modeling image A step of providing a first user interface (UI) including first image data and second image data; A step of obtaining user input related to arrangement points for the first image data and the second image data through the first user interface; and A step of arranging the first image data and the second image data based on the above user input The above method for generating surgical assistance data.

4. In Paragraph 3, The above user input includes a first signal corresponding to the selection of a first point in the first image data and a second signal corresponding to the selection of a second point in the second image data, and The step of arranging the first image data and the second image data is a step of matching the first point and the second point based on the first signal and the second signal. The above method for generating surgical assistance data.

5. In Paragraph 4, The step of generating the above 3D modeling image A step of providing a second user interface including the first image data, the second image data, and the first button; and When user input corresponding to pressing the first button is obtained through the second user interface, the method includes the step of matching the first point and the second point. The above method for generating surgical assistance data.

6. In Paragraph 1, The above 3D modeling image includes a plurality of distinct components, and The above plurality of components includes at least one of bone, nerve, and organ. Method for generating surgical assistance data.

7. In Paragraph 2, The first image data includes a plurality of detectable first markers, and The third image data includes a plurality of second markers within the support device located at the patient's treatment site, and The three-dimensional modeling image based on the first image data and the second image data includes a plurality of first markers. Method for generating surgical assistance data.

8. In Paragraph 7, The step of arranging in the above-mentioned spatial coordinate system A step of providing a third user interface including the above-mentioned three-dimensional modeling image and third image data; A step of obtaining a third signal for selecting some of the plurality of first markers included in the three-dimensional modeling image as points and a fourth signal for selecting some of the plurality of second markers included in the third image data as points through the third user interface; and A step comprising aligning the 3D modeling image and the 3rd image data based on points using the 3rd signal and the 4th signal. Method for generating surgical assistance data.

9. In Paragraph 8, The step of arranging in the above-mentioned spatial coordinate system A step for calculating the error of the above point-based alignment; and It further includes a step of displaying the error of the above point-based alignment. The error of the above point-based alignment includes reference point designation error and reference point alignment error. Method for generating surgical assistance data.

10. In Paragraph 1, The step of arranging in the above-mentioned spatial coordinate system The method includes the step of providing a fourth user interface comprising a modification button for changing the position and angle of the first image data and the second image data. Method for generating surgical assistance data.

11. In Paragraph 1, The step of generating the above surgical assistance data is, A step of obtaining information about a medical product for the treatment area of ​​the above patient; A step of providing a fifth user interface including the above-mentioned aligned image and obtaining a signal to select a region within the above-mentioned aligned image through the above-mentioned user interface; and A step comprising deploying the medical product based on the above signal Method for generating surgical assistance data.

12. In Paragraph 11, A step of setting a target area in the matched image based on the position where the selected medical product is placed in the matched image and the patient's information; A step of generating fourth image data including the above target area; A step of obtaining current location information for a fixation device to be placed in a target area from the medical staff terminal and generating matching degree data based on the location information and the matching degree of the target area; and The method further includes the step of transmitting the above-mentioned matching degree data and the above-mentioned fourth image data to a medical staff terminal. Method for generating surgical assistance data.

13. A computer program stored on a computer-readable non-transient recording medium to execute the surgical assistance data generation method of claim 1 on a computer system.

14. A communication unit that transmits and receives data with a medical staff terminal; Storage section; User interface section; Display unit; and It includes a control unit connected to the above communication unit, storage unit, interface unit and display unit, and The above control unit Acquire patient information and first image data, second image data, and third image data regarding the patient's treatment area from a medical staff terminal, and A 3D modeling image is generated based on the first image data and the second image data, and The above 3D modeling image and the above third image data are aligned and placed in a single spatial coordinate system, and Surgical assistance data is generated based on the above-mentioned aligned image and the above-mentioned patient information, and Transmitting the above surgical assistance data to the medical staff terminal Surgical assistance data generation device.

Images