Remote consultation support system capable of marking real-time operation images
Through the remote consultation support system, the attribution of responsibility in real-time surgery is marked, and the problem of difficult to determine the attribution of responsibility in remote consultation is solved, the responsibility judgment and transaction transparency of images are achieved, and the efficiency of medical accidents and the utilization rate of images are improved.
Patent Information
- Application Number
- CN202280093561.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-17
- Filing Date
- 2022-03-23
- Publication Date
- 2024-11-01
AI Technical Summary
During remote consultation, it is difficult to accurately grasp the specific content and responsibility of the consultation in the surgical image, especially in real-time surgery, it is difficult to clearly distinguish the responsibility of the surgical image, which affects the judgment of medical accidents.
Using the remote consultation support system, images are shot and shared at the real-time surgical site through the first and second head-mounted displays. The remote consultation support server generates tags and marks them in the real-time image based on whether the surgeon is accepted or not. The responsible subject information is stored on the blockchain to realize the discrimination of the responsibility of the image.
Support remote consultation during real-time surgery, clarify the ownership of responsibilities, improve the utilization rate of images and transaction transparency, and ensure the judgment of responsibility for medical accidents and reliable transactions of images.
Smart Images

Figure CN118891682A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a remote consultation support system capable of marking real-time surgical images, and more specifically, to a remote consultation support system capable of marking real-time surgical images, when a consulting doctor's consultation suggestion occurs during real-time surgery, the responsibility attribution based on whether the surgeon accepts or not is marked in the real-time image, thereby being used for determining the responsibility attribution in medical accidents. Background Art
[0002] Recently, as multimedia technology has been rapidly developed and applied to various fields, the medical field is also developing technology that can produce surgical scenes on patients into filmed images so that doctors can present them at relevant medical conferences or use them for other purposes.
[0003] In addition, in order to record the surgical process and prevent potential medical disputes, surgical imaging has been widely used in various medical devices, and the number of surgical imaging is increasing rapidly.
[0004] In some cases, especially in large hospitals or for complex diseases, it is becoming more and more common to perform surgery through consultation with expert doctors. However, due to time and space limitations, remote consultations are conducted through messages and images.
[0005] However, in this case, it is difficult to accurately grasp the specific content of the consultation in the surgical images. Therefore, a solution is urgently needed to clearly distinguish the responsibility in the surgical images during the remote consultation process.
[0006] In addition, during remote consultation, it is necessary to provide the consulting doctor with a real-time reproduction of the atmosphere of the surgical site, and to share it with the main surgeon through precise pointing when advising the consultation. In particular, when pointing to the affected part of the patient, it is difficult to point inside the body. Therefore, a solution that allows pointing in three dimensions in the real-time surgical image shared by the consulting doctor and the main surgeon is needed.
[0007] In addition, recorded surgical images have various utilization values. In particular, if various medical information is added to surgical images, they have their own utilization and asset value, and it is necessary to develop a system that can stably trade and manage these images.
[0008]
Related technical literature
[0009] (Patent Document 1): Korean Patent Publication No. 0-2020-0001266 (published on January 6, 2020) Summary of the invention
[0010] Technical issues
[0011] The technical problem to be solved by the present invention is to provide a remote consultation support system that can mark real-time surgical images. When a consultation suggestion is made by a consulting doctor during real-time surgery, the responsibility attribution based on whether the surgeon accepts or not will be marked on the real-time image, which can be used to determine the responsibility attribution in the event of a medical accident.
[0012] The technical problems of the present invention are not limited to the above-mentioned technical problems, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.
[0013] Technical Solution
[0014] In order to solve the above-mentioned problem, a remote consultation support system according to an embodiment of the present invention includes: a first head-mounted display, which is worn by a surgeon at the actual surgical site and includes a camera for capturing the surgeon's gaze area and surrounding area and a display unit for outputting real-time images captured by the camera; a second head-mounted display, which is worn by a consulting doctor who is remote from the surgeon, receives and outputs real-time images captured by the first head-mounted display, and receives suggestions on the surgical execution sequence or a specific treatment sequence included in the surgery by the consulting doctor; and a remote consultation support server, which, when receiving the input suggestion from the second head-mounted display, provides the first head-mounted display with a user interface for selecting whether to accept the suggestion, generates a label based on whether the surgeon's suggestion is accepted, and marks the real-time image.
[0015] At this time, the tag may include information on the responsible party for the real-time surgery according to the suggestion.
[0016] In addition, the tag also includes tag location information within the real-time image, including information on the tag stored in a blockchain-based distributed ledger.
[0017] In addition, if a preset specific input is received from the first head mounted display, the remote consultation support server generates a timestamp and marks it to the real-time image, and divides the real-time surgery into a plurality of detailed intervals defined by the surgeon based on the marked multiple timestamps.
[0018] At this time, the mark location information includes a start timestamp and an end timestamp of a detailed section of the mark location including the corresponding tag.
[0019] In addition, the preset specific input is at least one of a gesture, an audio command, and a control signal command, and the detailed interval is an interval corresponding to the detailed treatment and surgical sequence of the real-time surgery defined by the surgeon.
[0020] In addition, the remote consultation support server is used to provide the suggestion together with the user interface to the first head-mounted display as a preview, and when the suggestion is accepted through the first head-mounted display, the augmented reality data corresponding to the suggestion is inserted into the real-time image and displayed by the first head-mounted display and the second head-mounted display.
[0021] In addition, when the remote consultation support server accepts the advice from the consulting doctor from the second head-mounted display, it provides the first head-mounted display and the second head-mounted display with an avatar of the consulting doctor who automatically signs the consulting doctor who is responsible for the advice together with the advice, and when the first head-mounted display accepts the advice, it provides the first head-mounted display and the second head-mounted display with an avatar of the surgeon who automatically signs the surgeon who is responsible for receiving the advice.
[0022] In addition, the remote consultation support server automatically sets and receives a responsibility consent for a specific treatment or surgery predefined by surgical classification and patient status from each of the first head-mounted display and the second head-mounted display, and when a suggestion for the predefined specific treatment or surgery occurs, provides the first head-mounted display and the second head-mounted display with an avatar of a preset signature of the corresponding head-mounted display in which the responsibility consent is automatically set.
[0023] In addition, it also includes: a database for storing surgical images marked with at least one label; and a recommendation server based on artificial intelligence (AI), which automatically recommends the label contained in the detailed interval to the first head-mounted display when the detailed interval included in a similar operation similar to the real-time operation in the surgical images pre-stored in the database corresponds to the predefined specific treatment or operation set by the second head-mounted display responsibility consent.
[0024] Furthermore, when the remote consultation support server receives a suggestion for the predefined specific treatment or surgery for which the responsibility consent is automatically set from the first head mounted display, the remote consultation support server automatically marks it on the real-time image.
[0025] In addition, it also includes: an artificial intelligence-based recommendation server, which automatically recommends at least one tag contained in the detailed interval of a similar operation similar to the real-time operation in a pre-stored surgical image to the first head-mounted display when the detailed interval corresponds to the predefined specific treatment or operation set by the first head-mounted display responsibility consent.
[0026] In addition, the remote consultation support server tags the real-time image with at least one tag automatically suggested by the artificial intelligence-based recommendation server.
[0027] In addition, the second head-mounted display selectively outputs at least one of a first screen and a second screen, wherein the first screen outputs a real-time image captured by the first head-mounted display, and the second screen displays at least one suggestion that can be suggested during real-time surgery in a preview window.
[0028] In addition, it also includes: a database, which classifies and stores surgical effects marked with at least one label according to doctors, patient status, treatment and surgical categories; the second head-mounted display sets at least one of the labels marked in the surgical images stored in the database as a focus label, and receives the display time and display interval of the focus label; the remote consultation support server outputs the focus label to the preview window of the second screen according to the setting of the second head-mounted display.
[0029] In addition, it also includes: an artificial intelligence-based recommendation server for recommending tags of similar surgeries that are similar to the real-time surgery from the previous surgical images of the consulting doctor through the preview window.
[0030] In addition, it also includes: an artificial intelligence-based recommendation server, which automatically provides the labels marked during similar operations similar to the real-time operation to the first head-mounted display as suggestions of the consulting doctor when the labels marked during previous surgical images performed by the consulting doctor were not suggested during the real-time operation.
[0031] In addition, the artificial intelligence-based recommendation server determines the marking locations and marking intervals of the labels recommended in the real-time image based on the marking locations and marking intervals of the labels marked during the similar surgery, and automatically recommends them to the first head-mounted display.
[0032] In addition, it also includes: a third head-mounted display, which is worn by multiple assistants during the real-time operation at the actual operation site; and a database for storing surgical images marked with at least one label, based on the role information of each wearer of the head-mounted display in the real-time operation, extracting the labels that must be played from each head-mounted display, and automatically suggesting them to the corresponding head-mounted display.
[0033] Details of other embodiments are included in the detailed description and drawings.
[0034] Effects of the Invention
[0035] The remote consultation support system capable of marking real-time surgical images according to an embodiment of the present invention supports remote consultation during real-time surgery, and marks events such as consultation suggestions occurring during real-time surgery in the implementation images and stores the surgical images so that they can be used.
[0036] In particular, when a consulting doctor makes a consultation suggestion during a real-time surgical procedure, the responsible party information can be marked in the real-time image based on whether the surgeon accepts it or not, which can be used to determine the attribution of responsibility in the event of a medical accident.
[0037] In addition, through the head-mounted display (HMD) of the consulting doctor located remotely, the shooting field of view of each camera benchmark at the actual surgical site is provided as a field of view fixed mode to add augmented reality, and a field of view fixed release mode is provided so that the consulting doctor can independently grasp the current status of the operation.
[0038] In addition, it is recommended that during the consultation, a three-dimensional pointing system that can perform three-dimensional pointing to real-time images is used. The consulting doctor can three-dimensionally point to the patient's affected part required for the consultation, that is, the inside of the body, and share the medical images of the corresponding affected part with the surgeon.
[0039] In addition, the label information is recorded in the blockchain ledger, and NFT (Non-Fungible Token, NFT)-based ownership is granted to the surgical images containing the labels to mediate use and transactions, thereby mediating transparent and reliable transactions and distributing transaction profits.
[0040] In addition, through AI-based recommendations, optimal surgical images and labels can be provided according to the playback purpose of the user (surgeon, consulting physician, assistant one, assistant two, intern), and the utilization rate of surgical images can be improved.
[0041] The effects according to the present invention are not limited to the contents of the above-mentioned examples, and more various effects are also included in this specification. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 2 is a schematic diagram showing a remote consultation support system according to the present invention.
[0043] Figure 2 It is shown Figure 1 A block diagram of the composition of a head-mounted display.
[0044] Figure 3 It is shown Figure 1 A block diagram of the structure of the remote consultation support server.
[0045] Figure 4 and Figure 5 It is shown Figure 1Schematic diagram of a second head mounted display for remote consultation.
[0046] Figure 6 is a block diagram showing the configuration of a three-dimensional pointing system according to an embodiment of the present invention.
[0047] Figure 7 is a flow chart showing a three-dimensional pointing control method according to an embodiment of the present invention.
[0048] Figure 8 : is a schematic diagram showing a control device of a second head mounted display according to the first embodiment of the present invention.
[0049] Fig. 9 : is a schematic diagram showing a control device of a second head mounted display according to a second embodiment of the present invention.
[0050] Fig.10 : is a schematic diagram showing a control device of a second head mounted display according to a third example of the present invention.
[0051] Fig.11 is a schematic diagram showing label interval setting according to an embodiment of the present invention.
[0052] Fig.12 FIG. 4 is a schematic diagram showing a preview window of a consultation suggestion according to an embodiment of the present invention.
[0053] Fig.13 FIG. 4 is a schematic diagram showing a display screen of a first head mounted display for receiving a consultation suggestion according to an embodiment of the present invention.
[0054] Fig.14 2 is a schematic diagram showing authentication based on a suggestion responsibility and a suggestion receiving responsibility according to an embodiment of the present invention.
[0055] Fig.15 It is shown Figure 1 A block diagram of the structure of the image intermediary server.
[0056] Fig.16 : is a schematic diagram showing the blockchain structure of the remote consultation support system. DETAILED DESCRIPTION
[0057] The following content is merely an example of the principles of the present invention. Therefore, those skilled in the art can implement the principles of the invention and can derive various devices included in the concept and scope of the present invention, even though they are not clearly described or shown in this specification. In addition, all conditional terms and embodiments listed in this specification are in principle explicitly used only for the purpose of facilitating the understanding of the concept of the present invention, and should not be understood as being limited to the specifically listed embodiments and conditions.
[0058] Furthermore, in the following description, ordinal numbers such as first, second, etc. are intended to describe equal and independent objects, and should be understood as having no main / sub or master / slave meanings for their order.
[0059] The above-mentioned objects, features and advantages will become more apparent through the following detailed description related to the accompanying drawings, so that those skilled in the art to which the present invention belongs can easily realize the technical idea of the present invention.
[0060] Each feature of the various embodiments of the present invention may be partially or completely combined or combined, and as those skilled in the art can fully understand, various linkages and drives can be performed technically, and each embodiment may be implemented independently of each other, or may be implemented together due to related relationships.
[0061] A remote consultation support system according to an embodiment of the present invention is related to supporting remote consultation of a consulting doctor located remotely when a surgeon is performing real-time surgery at an actual surgical site.
[0062] At this time, it is necessary to determine the responsibility for real-time surgery and consultation. In the present invention, a label for determining the responsibility for consultation suggestions is marked on the real-time surgical image taken in real time, which is used for determining the responsibility in the event of a medical accident.
[0063] In addition, in order to provide the remotely located consulting doctors with the atmosphere of an actual surgical site, the remote consultation support system of the present invention provides augmented reality images corresponding to the actual surgical site based on real-time images, and can also constitute a three-dimensional pointing system for advice and sharing, so as to perform three-dimensional pointing to specific locations in real-time images.
[0064] In addition, the remote consultation support system of the present invention can also be configured as an image mediation system based on the use of surgical images with image marking labels taken during real-time surgery (based on the purpose of use by the surgeon, consulting physician and educator / trainer) and the interest structure of use.
[0065] Will refer to Figures 1 to 14 This is explained in detail.
[0066] Figure 1 2 is a schematic diagram showing a remote consultation support system according to the present invention.
[0067] The remote consultation support system may be implemented by a first head mounted display 100 worn by a primary surgeon 10 and a second head mounted display 200 worn by a remotely located consulting physician 20, and may include a remote consultation support server 1000 for remote consultation support, a database 2000, an artificial intelligence based recommendation server 3000 and an image intermediary server 4000.
[0068] Furthermore, in order to support remote consultation for real-time surgery at the actual surgery site 1, multiple cameras may be provided to film the real-time surgery, thereby sharing the recorded images with the consulting doctors located remotely.
[0069] Specifically, a surveillance camera 2 is installed at the actual operating scene 1, and in order to provide a vivid real-time three-dimensional image of the operating scene 1, it may also include a third head-mounted display 100-1, 100-2 worn by multiple assistants 11, 12, a fourth head-mounted display 300 worn by the patient 30, and a camera 400 installed on the ceiling of the operating scene 1.
[0070] The camera 400 installed on the ceiling specifies the coordinates of each individual in the first real-time image captured and grasps the initial position, and grasps the positions of the first head-mounted display 100 worn by the surgeon 10 and the third head-mounted displays 100-1, 100-2 worn by the assistant 1 11 and the assistant 2 12 during the real-time operation and performs movement tracking. In addition, the movement tracking of all individuals such as surgical tools captured by the ceiling camera 400 can be performed.
[0071] Figure 2 It is shown Figure 1 A block diagram of the composition of a head mounted display. Figure 2 Each head mounted display 100, 100-1, 100-2, 200, 300, 500 may be respectively equipped with a camera, a display unit, an input unit, a control unit and a communication unit, and may also be equipped with a speaker, a microphone and the like.
[0072] At this time, the image provided by each of the configured on / off or display units is applied differently according to the wearer, that is, the wearer mode set as a primary surgeon mode, an assistant mode, a patient mode, an education and training mode, etc., so that necessary functions can be provided to the corresponding wearer. Alternatively, the head-mounted display according to the wearer may also be composed of a separate head-mounted display that provides only functions according to the corresponding mode.
[0073] At this time, the camera of the head-mounted display may include a first camera arranged in front of the corresponding head-mounted display to capture the wearer's gaze area and second cameras arranged on both sides of the corresponding head-mounted display to capture the wearer's surrounding area.
[0074] In addition, the display unit displays the images of the wearer's gaze area captured by the camera in the main surgeon mode and the auxiliary mode, and can receive and display the real-time images of each camera captured at the actual surgical site in the consulting doctor mode. In addition, the patient mode can provide information as needed, and the education and training mode can display selected playback images.
[0075] The input unit may be an interface or a sensing unit that receives control signals for controlling the head-mounted display and controlling the displayed image. The control unit controls the head-mounted display according to the control signals received through the input unit, and may receive various remote consultation supports from each external server 1000 to 4000 through the communication unit.
[0076] Figure 3 1 is a block diagram showing the configuration of the remote consultation support server 1000. The remote consultation support server 1000 may include a consultation support unit 1100 and an image marking unit 1200. In this case, each configuration may also be composed of a separate module, server, or the like.
[0077] Among them, remote consultation support may involve real-time surgery-related consultation suggestions, determination of responsibility based on the suggestions, image marking of images taken in real time when various events occur, three-dimensional pointing control of displayed images, use of labeled surgical images, and distribution of benefits from their use.
[0078] The consultation support unit 1100 collects real-time images output from all cameras captured in the operating room 1 and transmits them to the second head mounted display 200 so that the consulting doctor can participate remotely.
[0079] At this time, in order to allow the consulting doctor to participate remotely, the consulting doctor can independently grasp the situation in the operating room 1 together with the surgeon by wearing the second head mounted display 200 .
[0080] Therefore, according to the surgeon's choice, the second head-mounted display 200 worn by the consulting physician can be provided with real-time images of the shooting fields of view between the surveillance camera 2 installed in the operating room 1, the camera 400, the camera of the first head-mounted display 100 worn by the surgeon, and the cameras of the third head-mounted displays 100-1, 100-2 worn by multiple assistants 11, 12 respectively.
[0081] As an example, when the second head mounted display 200 worn by the consulting doctor selects a specific position of the actual surgical site selected by the consulting doctor through the display unit (for example, Figure 5 The camera of the first head-mounted display 100 worn by the surgeon as shown in the figure) can output the real-time image captured by the field of view of the first camera of the first head-mounted display 100 to the display unit of the second head-mounted display 200.
[0082] At this time, if Figure 4 and Figure 5As shown, the consulting doctor secures the same field of view as the surgeon's perspective, and after securing the field of view, the consulting doctor can grasp the situation in the operating room 1 by shifting his or her sight independently from the surgeon.
[0083] That is to say, Figure 4 In the embodiment, when the consulting doctor turns his sight to the direction of the auxiliary first 11 or the auxiliary second 12, among the real-time images captured by the shooting fields of the second head mounted displays provided on both sides of the first head mounted display 100 worn by the main surgeon 10, the real-time images captured by the specific second camera corresponding to the direction of the consulting doctor's sight can be output through the second head mounted display 200 of the consulting doctor 20. That is, since there is no fixed periphery, the field of view can be shifted independently from the main surgeon (field of view fixation release mode).
[0084] In addition, you can also independently shift the field of view when selecting a specific camera. Alternatively, you can fix the perimeter and only receive live images from the selected camera (fixed field of view mode).
[0085] Thus, in the present invention, the second head mounted display 200 worn by the consulting doctor 20 can provide a visual field fixing mode through peripheral fixing according to the consulting doctor's selection. Figure 6 The three-dimensional pointing system for remote consultation (hereinafter referred to as the three-dimensional pointing system) is used for consultation in a fixed field of view mode to suggest a location that needs to be shared and conduct a consultation.
[0086] Three-dimensional pointing system for remote consultation
[0087] refer to Figure 6 According to an embodiment of the present invention, the three-dimensional pointing system further includes Figure 1 The remote consultation support system controls the control device 210 of the second head mounted display 200, and the control device can be composed of Figures 8 to 10 The control devices 500, 500a, 600 shown are implemented in various ways. Figure 7 The three-dimensional pointing control method is described in the present invention.
[0088] In the actual surgical scene 1, real-time images captured by the first camera and the second camera of the first head-mounted display 100 worn by the surgeon 10 participating in the real-time surgery, the first camera and the second camera of the third head-mounted display 110-1, 110-2 worn by multiple assistants 11, 12, the camera of the fourth head-mounted display 300 worn by the patient 30, the surveillance camera 1 and the ceiling camera 400 can be transmitted to the remote consultation support server 1000.
[0089] At this time, in step S701, the real-time image captured by the first camera of the display unit belonging to the first head mounted display 100 is output and transmitted to the remote consultation support server 1000. In addition, the real-time image captured by the first camera of the display unit belonging to each of the third head mounted displays 110-1, 110-2 may be output and transmitted to the remote consultation support server 1000.
[0090] Next, in step S702 , the consultation support unit 1100 of the remote consultation support server 1000 may collect real-time images from each camera and synchronize the time, and then transmit the images to the second head mounted display 200 .
[0091] Next, in step S703, the second head mounted display 200 may receive and display the real-time image of the first head mounted display 100, and when receiving the real-time images of the multiple head mounted displays 100, 100-1, 100-2, the real-time image corresponding to the camera selected by the consulting doctor or the preset camera may be displayed. At this time, the real-time image of the first head mounted display 100 of the surgeon is preferably output, and may also be changed by the subsequent consulting doctor.
[0092] Next, the second head-mounted display 200 can three-dimensionally point to the specific position for consultation advice according to the real-time image output to the display unit by the control signal received from the second head-mounted display 200 through the input unit. At this time, the specific position can be a specific position in the actual operating room 1, specifically, it can be a specific position of the affected part of the patient for consultation advice. In the present invention, the surgeon 10 and the consulting doctor 20 share the patient's treatment affected part, and it is necessary to set the real-time image (the image generated by the first camera that captures the gaze area of the first head-mounted display 100) in a fixed field of view mode on the second head-mounted display 200 for consultation. In other words, the consulting doctor can confirm and give the surgeon advice on the operation from the surgeon's point of view.
[0093] refer to Figure 4 and Figure 5 , in the case of fixed field of view mode, such as Figure 4 As shown, the second head mounted display 200 outputs only the real-time image output from the first camera of the first head mounted display 100 to the second head mounted display 200, so that the consulting doctor can confirm the real-time surgery within the range corresponding to the shooting field of view α of the first camera. In other words, even if the consulting doctor moves (turns the head in the direction of auxiliary one or auxiliary two), only the real-time image within the shooting field of view α is output.
[0094] On the other hand, if the peripheral fixation is released (field of view fixation release mode), a real-time image of the shooting field of view of the second camera corresponding to the moving direction is provided when the second head mounted display 200 moves, thereby independently shifting the field of view to confirm the periphery with the surgeon.
[0095] For example, in Figure 4 and Figure 5 In the process of looking at the first real-time image within the shooting field of view α (the real-time image corresponding to the shooting field of view α of the first camera of the first head-mounted display 100) and turning his head to the direction of auxiliary - 11 to divert his line of sight, the second real-time image captured by the shooting field of view of the second camera arranged on the side of the auxiliary - 11 of the first head-mounted display 100 worn by the surgeon 10 is continuously provided, thereby diverting the field of view independently of the surgeon.
[0096] The three-dimensional pointing control system of the present invention can set the real-time image displayed on the second head mounted display 200 to a field of view fixed mode, and then perform three-dimensional pointing for consultation suggestions.
[0097] The control device 210 for three-dimensional pointing may include a pointing device 211 and a direction selection device 212. In step S704, the control device 210 may output a control signal for generating a virtual line and a three-dimensional pointer in a three-dimensional real-time image to the second head mounted display 200 under the control of the surgeon, and the movement of the three-dimensional pointer may be controlled by using the pointing device 211 and the direction selection device 212.
[0098] The pointing device 211 moves in the depth direction ( Figures 8 to 10 The direction selection device 212 moves the three-dimensional pointer in the z direction, and the direction selection device 212 is perpendicular to the depth direction ( Figures 8 to 10 Move the 3D pointer in the x or y direction.
[0099] The pointing device 211 and the direction selection device 212 can be Figures 8 to 10 The control devices 500, 500a, 600 etc. may be implemented in various ways, and the direction selection device 212 may also be utilized in the form of a foot pedal.
[0100] In addition, the control signal in step S704 may be inputted through a preset gesture, eye line processing, eye movement (blinking, etc.) of the consulting doctor sensed by the input unit of the second head mounted display 200, or by Figures 8 to 10 An input is input into one of the control devices.
[0101] First, when using the preset gestures, sight lines, sight line movement, etc. of the consulting doctor sensed by the input unit of the second head mounted display 200, as an example, the visual field fixed mode is set by the first gesture (the action of opening both arms at a certain angle), the pointer P1 and the virtual line 3 are formed on the affected part by sight line processing (staring at the affected part for pointing for more than a preset time), and the pointer P2 is moved along the depth direction on the virtual line 3 by the second gesture (hand movement). Through this, three-dimensional pointing can be performed to the affected part inside the patient's body.
[0102] At this time, the depth can be set to match the depth and adjust the depth according to the patient's internal organs and blood vessels. More specifically, the surgeon can use the control device 210 to move the pointer in the depth direction z and the directions x and y perpendicular to the depth direction to select a specific affected segment area.
[0103] Specifically, by learning the image history of the affected part of other patients with the same medical history as the patient, the affected part segment area of the patient can be statistically determined. In particular, the affected part segment area of the patient can be reflected and distinguished by indocyanine green (ICG) staining.
[0104] Indocyanine green is a medical agent that emits fluorescence at an infrared wavelength of 805nm as a green pigment and has the property of binding to albumin, a water-soluble protein present in the cytoplasm and tissues in the body. After indocyanine green is injected into the patient's blood vessels by intravenous injection and stained with indocyanine green, infrared rays are irradiated and the filtered reflected light is captured by a camera equipped with a filter to obtain an image of the affected area stained with fluorescence.
[0105] The remote consultation support server 1000 collects images of the affected part and learns the stained area for each disease name of each patient, thereby statistically dividing the affected part segment area.
[0106] In addition, the affected area may be divided into a plurality of lower segments based on the arrangement structure of blood vessels of the corresponding affected area, the arrangement structure of other surrounding organs, and the depth difference of the affected area observed from the respective viewpoints of the surgeon and the consulting physician.
[0107] Among them, the depth difference of the affected area observed from the respective viewpoints of the surgeon and the consulting doctor can be the difference in the distance (depth stage) in the depth direction z between the detailed affected area gazed by the surgeon and the detailed affected area gazed by the consulting doctor for one affected segment area.
[0108] At the same time, when an image of an affected part stained with indocyanine green is used in real-time surgery, the image of the affected part stained with indocyanine green taken by a camera using a filter can be received from the first head-mounted display and the second head-mounted display and integrated into the real-time image for display.
[0109] Through this, the surgeon and the consulting doctor can select the affected segment area through three-dimensional pointing to perform real-time surgery. As an example, in step S705, when the surgeon uses the pointing device 211 to generate a virtual line and a three-dimensional pointer located on the virtual line in the three-dimensional real-time image, the server 1000 determines that the affected segment area of the patient at the shortest vertical distance from the three-dimensional (x, y, z) of the three-dimensional pointer is selected, so that the selection representation can be provided to the corresponding affected segment area. In this regard, the surgeon and the consulting doctor can share the selection area through the first head-mounted display 100 and the second head-mounted display 200.
[0110] In addition, the surgeon selects a specific direction signal of the direction (x, y) perpendicular to the depth direction z using the direction selection device 212, so that the three-dimensional pointer can be moved in the vertical direction. That is, in step S706, the server 1000 can move the selection representation so as to select other adjacent affected segment regions in the specific direction selected by the direction selection device 212 based on the affected segment region of the pre-selected representation.
[0111] In addition, the present invention can Figures 8 to 10 Various embodiments of the control device and the like provide three-dimensional pointing.
[0112] (First Embodiment)-Control Device 500
[0113] refer to Figure 8 , the control device 500 may include a first input unit 501, a second input unit 502 and a control unit 503. At this time, the first input unit 501 and the second input unit 502 may be provided on the main body in the form of buttons or the like.
[0114] When receiving an input signal by pressing a button, the first input unit 501 outputs a first control signal to set the real-time image output from the second head mounted display 200 to a field of view fixed mode or a field of view fixed release mode. The second head mounted display 200 can set the mode by receiving the first control signal.
[0115] In addition, when the field of view fixed mode is set by the first input unit 501, the first input unit 501 receives an input signal by pressing a button, and the second control signal for generating the virtual line 3 and the pointer p1 can be output to the real-time image output by the second head-mounted display 200.
[0116] Next, the control unit 503 detects the movement of the main body of the control device in the depth direction or the opposite direction of the depth of the virtual line, and outputs a third control signal to move the pointer p1→p2 on the virtual line 3, so that three-dimensional pointing can be performed on the real-time image displayed in the second head-mounted display 200.
[0117] At this time, the layer L2 selected from the layers (L1, L2) indicating the depth phase matching at the pointed position may be selected. The layer may be an affected part image that divides a three-dimensional affected part image into affected part segment regions.
[0118] That is, the affected part segment area at the shortest vertical distance in the depth direction z can be selected at the position of the three-dimensional pointer P2. In addition, the main body of the control device 500 is moved in a direction (x, y) perpendicular to the depth direction z, that is, leftward or rightward or up and down with the user's hand as the center to move the three-dimensional pointer, so that other affected part segment areas can be selected using the direction selection device.
[0119] (Second Embodiment)-Control Device 500a
[0120] In other embodiments, Fig. 9 The control device 500a is in the form of a gun, which can improve the user's grip and convenience. Specifically, the control device 500a may include a head unit 530a, a handle unit 510a equipped with an input button 511 and a latch 520, and a control unit 540.
[0121] When the input button 511 is pressed, the head unit 530a may generate a virtual line in the real-time image and guide the direction so that the direction of the virtual line points to a specific position.
[0122] The input button 511 of the handle unit 510a can set the field of view fixed mode when receiving an input signal, and use the head unit 530a to generate a virtual line 3 and a pointer p1. In addition, when the field of view fixed mode is set, if the input button 511 is pressed again (the input signal is received again), the field of view fixed release mode can be set.
[0123] The latch 520 of the handle unit 510 a may output a control signal to move the pointer from the virtual line 3 to the depth direction p1 → p2 of the virtual line 3 whenever the latch 520 is operated.
[0124] At this time, the affected part segment area having a number matching the preset depth on the virtual line 3 may be provided cyclically in sequence according to the depth each time the latch is operated.
[0125] In addition, the main body of the control device 500a is moved in the direction (x, y) perpendicular to the depth direction z, that is, left or right or up and down with the user's hand as the center, so as to move the three-dimensional pointer for the direction selection device, and other affected segment areas can be selected. At the same time, the control device 500, 500a can also be used only as the pointing device 211. In this case, the separate foot pedal is used as the direction selection device 212 to control the three-dimensional pointing.
[0126] (Third Embodiment)-Control Device 600
[0127] In another embodiment, except Fig. 9 In addition to the control device 500a, Fig.10 The control device 600 further includes a control device 500b. That is, the control device 600 includes a first control device 500b and a second control device 500a. At this time, the shape of the second control device 500a is similar to that of the first control device 500b. Fig. 9 The same as in the example above, but some functions may be different depending on the settings.
[0128] The first control device 500b includes a handle unit 510b having a first button for setting a field of view fixing mode and a second button for setting a field of view fixing releasing mode, and a guide head unit 530b forming a guide ring.
[0129] The head unit 530a of the second control device 500a is focused by guiding the guide ring of the head unit 530b so that the virtual line generation direction passing through the head unit 530a can be guided.
[0130] The input button 511 of the handle unit 510a can set the field of view fixed mode when receiving an input signal, and use the head unit 530a to generate a virtual line 3 and a pointer p1. In addition, when the field of view fixed mode is set, if the input button 511 is pressed again (the input signal is received again), the field of view fixed release mode can be set.
[0131] The latch 520 of the handle unit 510 a may output a control signal to move the pointer from the virtual line 3 to the depth direction p1 → p2 of the virtual line 3 whenever the latch 520 is operated.
[0132] At this time, the affected part segment area having a number matching the preset depth on the virtual line 3 may be provided cyclically in sequence according to the depth each time the latch is operated.
[0133] At this time, every time the latch 520 of the handle unit 520 is operated, a control signal for moving the pointer in the depth direction of the virtual line 3 from p1 to p2 may be output.
[0134] At this time, when the consultation support unit 1100 of the remote consultation support server 1000 receives a specific pointer position (a position in the pointers such as p1, p2, etc.) on the virtual line 3 from the second head mounted display 200, the affected segment region of the organ or blood vessel corresponding to the pointer position can be inserted and provided in the real-time image output from the first head mounted display to the third head mounted display. In addition, the selected affected segment region L1 can be divided into lower segment regions and provided.
[0135] Meanwhile, the explanation will focus on the three-dimensional pointing of the surgeon, but the surgeon can also perform three-dimensional pointing on the affected area by the same method.
[0136] Image Marking in Remote Consultation Support System
[0137] In addition, when the remote consultation support system performs real-time surgery, consultation suggestions made during remote consultation, recommendations made by the artificial intelligence-based recommendation server 3000, and events directly set by the surgeon, etc., are marked in the real-time images taken by each camera, so that they can be used for subsequent surgical procedures, consultations, education / training, and responsibility determination.
[0138] To this end, the remote consultation support system is composed of a remote consultation support server 1000, a database 2000, an artificial intelligence-based recommendation server 3000, and an image intermediary transaction server 4000. Figure 3 The image marking unit 1200 of the remote consultation support server 1000 may include a user interface providing unit 1110 , a marking unit 1120 and a responsibility setting unit 1130 .
[0139] refer to Figure 3 , remote participation of consulting doctors and real-time image marking according to an embodiment of the present invention will be described.
[0140] At this time, the remote consultation support system presets the priority order of the cameras and images used to shoot the real-time surgery, and displays the images shot by the corresponding cameras in turn on the display unit of the second head mounted display 200 according to the preset priority order, so that the consulting doctor can monitor and decide on the consultation suggestion. As an example, the consulting doctor can be set in the order of the first head mounted display 100 of the main surgeon, the monitoring camera, the auxiliary camera, etc., or can be preset in the consultation server 1100.
[0141] In addition, according to the needs of the consulting doctor, a specific camera can be selected to display the corresponding captured images.
[0142] refer to Figures 1 to 3 The consulting doctor 20 monitors the real-time surgical image of the operating doctor 10 through the second head mounted display 200 and can send suggestions when consultation is needed.
[0143] The consultation support unit 1100 of the remote consultation support server 1000 may collect real-time images from each camera photographing a real-time operation and transmit the collected images to the second head mounted display 200 .
[0144] At this time, when receiving a consultation suggestion from the second head mounted display 200 or recommending a suggestion from the artificial intelligence based recommendation server 3000, the user interface providing unit 1110 may provide a user interface for accepting and selecting the suggestion to the first head mounted display 100. The user interface for accepting the suggestion may be provided in the form of text or audio, such as Fig.13 shown.
[0145] When a label based on a suggestion and an event occurs during a real-time operation, the labeling unit 1120 can label the real-time image according to the surgeon's acceptance or not. At this time, the consulting doctor's suggestion can be a suggestion on the order of performing the operation or a specific treatment order included in the operation. When the user interface providing unit 1110 receives the acceptance or not of the suggestion, the labeling unit 1120 can generate a label of the surgeon's acceptance or not from the first head mounted display 100 and label the collected real-time image.
[0146] At this time, the tag may include information about the responsible subject for the real-time surgery according to the suggestion and the marking location information within the real-time image, and the information included in the tag may be stored in a distributed ledger based on blockchain.
[0147] Among them, tags can serve as metadata about the consulting doctor's suggestions during real-time surgery, recommendations made by the artificial intelligence-based recommendation server 3000, and events directly set by the surgeon, and can become marking information inserted into real-time images.
[0148] As an example, tags may be an augmented reality surgical demonstration of the consulting physician, the consulting physician's opinions, previous surgical images similar or identical to the real-time surgery, specific surgical tools, audio or text information, vital signs, etc., all information that may be an issue for the real-time surgery.
[0149] At this time, the remote consultation support system of the present invention can insert the responsible subject information for determining the responsibility when marking the real-time image label, and can distinguish the label part marked in the real-time surgery according to the detailed interval unit of the detailed treatment / surgery distinction of the real-time surgery.
[0150] Specifically, when receiving a preset specific input from the first head mounted display 100, the marking unit 1120 can generate a timestamp and mark it on the real-time image, and can distinguish the real-time surgery into multiple detailed intervals defined by the surgeon based on the marked multiple timestamps.
[0151] The preset specific input is at least one of a gesture, an audio command, and a control signal command, and the detailed interval may be an interval corresponding to a detailed treatment and surgical sequence of a real-time surgery defined by a surgeon.
[0152] Fig.11 2 is a schematic diagram showing label interval settings according to an embodiment of the present invention. The surgeon 10 may provide the consulting doctor 20 with information about the real-time surgery in advance.
[0153] At this time, the information provided in advance may be surgical information of the real-time surgery, a rough detailed treatment or a surgical sequence, etc. At this time, each detailed treatment is remotely participated in by a different consulting doctor, and the consultation support unit 1100 may also limit the second head mounted display 200 that provides real-time images to the corresponding consulting doctor 20 during the detailed participation interval.
[0154] In addition, for each detailed interval in the real-time surgery, the surgeon 10 and the consulting doctor 20 can display a flag indicating that the detailed interval of the currently ongoing surgery and treatment is their own responsible interval. At this time, when either party displays the flag, the label information of the corresponding detailed interval can be marked as the responsible interval of either party.
[0155] At this time, if both parties display the mark or withdraw the mark, the person who disagrees with the responsibility interval can display an objection message to the objection application, and the consultation support unit 110 can support communication by audio after a certain period of time after displaying the objection message.
[0156] In addition, when determining who is responsible for a detailed interval, the augmented reality providing unit 1130 may provide a preset avatar of the corresponding responsible person to the real-time surgical image and confirm the responsible interval.
[0157] In addition, if the surgeon 10 lacks relevant surgical experience, the relevant specialist can become the consulting doctor 20 and conduct a remote consultation. At this time, if the consulting doctor 20 has more specialist experience than the surgeon 10 or the evaluation numerical data of the surgical experience is above the predetermined value, the consultation support unit 1100 displays the consulting doctor's responsibility interval in the real-time image, and can be marked by the history tag information (metadata) linked to the recorded image interval as a priority.
[0158] refer to Fig.11As an example, in the case of a coronary stent surgery, the surgery name (coronary stent), patient information, detailed treatment or surgery (anesthesia, stent, etc.) are provided to the consulting physician participating remotely, and the detailed intervals can be distinguished according to the specific input of the surgeon 10 during the real-time surgery.
[0159] At this time, the marking unit 1120 may insert a time stamp at a time point when a specific input is input through the first head mounted display 100 , and distinguish the start and end of the detailed interval.
[0160] At this time, since the surgeon is the main surgeon and both hands are not idle, the specific input is preferably a preset gesture or audio input, and the detailed interval setting is distinguished by label insertion or artificial intelligence analysis after real-time image capture. In addition, the input can also be made by audio or the like of an assistant who is not the main surgeon 10 but participates in the actual operation scene 1.
[0161] When the user interface providing unit 1110 receives a suggestion from the second head mounted display 200 or the artificial intelligence based recommendation server 3000, as shown in FIG. Fig.12 As shown, the consultation suggestion is provided to the first head mounted display 100 together with the user interface in a preview.
[0162] Fig.12 FIG. 4 is a schematic diagram showing a display screen of a first head mounted display for receiving consultation suggestions according to an embodiment of the present invention.
[0163] refer to Fig.12 , a is a screen output on the first head mounted display 100 of the surgeon, which shows a real-time image 71 captured by the first head mounted display 100 in real time, a preview of suggested contents during the consultation, and a user interface 72 asking whether to accept the suggestion.
[0164] In addition, b is the screen output by the third head mounted display 100-1 of auxiliary one when receiving the advice of the consulting doctor, and c is the screen output by the third head mounted display 100-1 of auxiliary two when receiving the advice of the consulting doctor. At this time, the advice received in auxiliary one and auxiliary two can be information received according to the acceptance degree of the advice of the surgeon. That is to say, the head mounted display 100 of the surgeon can also display the advice for b and c, and auxiliary one and auxiliary two can perform assistance according to the advice accepted by the surgeon. In other words, the surgeon can decide whether to accept the occurrence of the event.
[0165] In other words, Fig.12 As shown in b, a consulting doctor at a remote location sets up "virtual surgical tools" required for surgery and displays them on the head-mounted displays of surgical participants 10, 11, 12, and by viewing this setting, the surgical participants can prepare "actual surgical tools" in advance.
[0166] In addition, the consulting doctor can map the virtual surgical tool to the specific affected part of the real-time image, thereby suggesting the surgical tool to be used to the main surgeon, and using the surgical tool, the timer or treatment sequence required for treatment can be suggested. At this time, the suggested surgical tool, treatment timer, treatment sequence, etc. can be provided to the real-time image through the head-mounted display by the augmented reality data.
[0167] If the suggestion is accepted on the first head-mounted display 100, after performing authentication based on the received suggestion, the augmented reality data (text, video, image, audio, etc.) corresponding to the suggestion is actually marked and displayed to the first head-mounted display 100 and the second head-mounted display 200.
[0168] The authentication may be performed simultaneously on the second head mounted display 200 providing the suggestion and the first head mounted display receiving the suggestion. Fig.13 , the responsibility setting unit 1130 can receive signatures from the first head-mounted display 100 and the second head-mounted display 200 and pre-store them, and display them on the first head-mounted display 100 and the second head-mounted display 200 so as to accept the shared signature authentication screen according to the suggestion and the suggestion.
[0169] like Fig.13 As shown in a, when receiving advice from the consulting doctor 20 from the second head-mounted display 200, the avatar of the consulting doctor who automatically executes the signature of the consulting doctor preset for the advice responsibility together with the advice can be provided to the first head-mounted display 100 and the second head-mounted display 200.
[0170] In addition, if Fig.13 As shown in b, when the first head mounted display 100 accepts the suggestion, the avatar of the surgeon who automatically executes the preset signature of the surgeon accepting the responsibility for the suggestion can be provided to the first head mounted display and the second head mounted display.
[0171] In addition, when the surgeon 10 and the consultant 20 set the conditions for automatic signature, the corresponding avatars can also perform automatic signature. Specifically, the responsibility setting unit 1130 collects the results of a questionnaire survey conducted in advance on the consultant 20 and the surgeon 10 or the data on the surgical style of the surgical image recommended by the consultant / surgeon, and provides the avatar attributes (automatic signature conditions) belonging to the consultant 20 and the surgeon 10 respectively to each terminal (head-mounted display or user terminal, etc.) of the consultant and the surgeon, so that the avatar attributes can be specified.
[0172] As an example, the consulting doctor can recommend other recorded surgical images to the surgeon and the participants (Assistant 1 and Assistant 2) before or during the operation. The remote consultation support server 1000 can statistically extract and grasp the surgical methods preferred by the consulting doctor using the metadata of these recommended surgical images.
[0173] Through this, the surgeon can accept the suggestion and agree to undertake it without having to deal with each suggestion acceptance or rejection that occurs during real-time surgery.
[0174] At this time, if the advice is the advice of the consulting physician received from the second head mounted display 200, the advice acceptance responsibility of the operating physician 10 wearing the first head mounted display 100 and the advice responsibility of the consulting physician 20 wearing the second head mounted display 200 are recorded in the tag as responsibility attribution information.
[0175] At this time, according to the characteristics of the advice, based on the role of the assistant one or the assistant two, the advice is provided not only to the first head mounted display 100 but also to the corresponding third head mounted display 300 .
[0176] In addition, the second head mounted display 200 may directly set advice and transmit it to the first head mounted display 100 , and receive advice information to be suggested to the first head mounted display 100 from the artificial intelligence based recommendation server 3000 and provide it to the first head mounted display 100 .
[0177] In addition, the first head mounted display 100 may directly receive advice of recommendation (automatic suggestion) from the artificial intelligence-based recommendation server 3000 .
[0178] At this time, recommendations can be made based on the label interval information of the label interval, that is, recommendations can be made according to the treatment, surgery, and minimally invasive treatment corresponding to the detailed interval. When similar / identical recommended labels occur, the artificial intelligence-based recommendation server 3000 considers the preset label importance, treatment / surgery difficulty, etc., and can recommend the best label information based on the preset priority sequence.
[0179] (First embodiment) Suggestion - The second head mounted display directly suggests to the first head mounted display
[0180] The database 2000 stores surgical images marked with at least one tag, and can be classified and stored according to classification criteria such as hospital / doctor, patient status, treatment and surgical category.
[0181] refer to Fig.14 The second head mounted display 200 may selectively output a first screen (not shown) outputting a real-time image captured by the first head mounted display 100 and a second screen (not shown) displaying at least one suggestion that may be suggested when performing a real-time surgery. Fig.14 ) At this time, the first screen and the second screen can be provided by the consultation supporting unit 1100.
[0182] The second head mounted display 200 can load at least one tag marked in the surgical image stored in the database 2000 into the preview window as a tag of interest, and can receive settings for the display time and display interval of the tag of interest. The consultation support unit 1100 can output the tag of interest to the preview window of the second screen according to the settings of the second head mounted display 200.
[0183] The second head mounted display 200 may selectively or automatically suggest tags of interest according to the setting of the first head mounted display 100 .
[0184] (Second embodiment) Suggestion - The second head mounted display suggests AI-based recommendations to the first head mounted display
[0185] The AI-based recommendation server 3000 can extract tags of similar surgeries marked on the real-time surgeries from the consulting doctor's previous surgical images, and recommend them through the preview window.
[0186] The previous surgical images may be images of surgeries in which the consulting physician served as the lead surgeon or participated as a consulting physician, surgical images played by the consulting physician, and the like.
[0187] The consulting doctor may selectively or automatically suggest a label of a recommended preview window to the first head mounted display 100 according to settings.
[0188] (Third Embodiment) Recommendation-Automatically recommending AI-based recommendations from the consulting doctor to the first head mounted display
[0189] If, from previous surgical images in which consulting doctors in database 2000 participated, labels marked when performing similar surgeries similar to the real-time surgery are not suggested to the current real-time surgery, the artificial intelligence-based recommendation unit 3000 can automatically suggest to the first head-mounted display the labels marked when performing similar surgeries as the consulting doctor's suggestions.
[0190] In this case, the AI-based recommendation server 3000 determines the marking locations and marking intervals of the tags recommended in the real-time image based on the marking locations and marking intervals of the tags marked during similar surgeries, and automatically suggests them to the first head-mounted display.
[0191] (Fourth Embodiment) Suggestions - Recommendations based on artificial intelligence corresponding to automatic setting of responsibility synonyms
[0192] The first head mounted display 100 and the second head mounted display 200 may automatically set responsibility synonyms for specific treatments or surgeries predefined by surgery category and patient status.
[0193] The responsibility setting unit 1130 may request the artificial intelligence-based recommendation server 3000 for recommendation of a specific treatment or surgery automatically set by the responsibility synonym of the head mounted display.
[0194] In addition, when a suggestion for a predefined specific treatment or surgery occurs per head mounted display, the responsibility setting unit 1130 provides an avatar of a preset signature of the corresponding head mounted display for which responsibility consent is automatically set to the corresponding head mounted display and performs authentication.
[0195] If a detailed interval of a similar operation similar to the real-time operation is included in the surgical image pre-stored in the database 2000 and corresponds to a predefined specific treatment or operation for which responsibility consent is set through the second head-mounted display 200, the artificial intelligence-based recommendation server 3000 can automatically suggest tags included in the detailed interval to the first head-mounted display 100.
[0196] At this time, when receiving a suggestion for a predefined specific treatment or surgery for automatically setting a responsibility synonym, the marking unit 1120 may automatically mark the real-time image.
[0197] In addition, when a recommendation for a specific treatment or surgery predefined for each head mounted display occurs, the responsibility setting unit 1130 provides an avatar of a preset signature of the corresponding head mounted display for automatically executing responsibility consent to be automatically set to the first head mounted display and the second head mounted display.
[0198] (Fifth Embodiment) Suggestion - Automatically suggest necessary tags for head mounted display during surgery
[0199] The AI-based recommendation server 3000 extracts necessary tags that must be played on the head-mounted display based on the role information of the real-time surgery (consultant, assistant one, assistant two) in the wearer information of each head-mounted display, and automatically recommends them to the corresponding head-mounted display.
[0200] Meanwhile, the above explains that in real-time surgery, the surgeon accepts the advice of the consultant and marks the responsibility for the detailed interval. Different from this, if the surgeon and the consultant agree on the surgical responsibility interval before the operation begins, the consultation support unit 1100 divides the entire surgical interval into known treatments and distinguishes them into: Fig.11The detailed interval shown in a is provided to the real-time images output by each head-mounted display, and the responsible subject of the pre-agreement is matched and marked in the detailed interval corresponding to the known treatment.
[0201] At this point, it is necessary to determine whether the treatment currently being performed by the surgeon is a known treatment, and this can be determined by machine learning including a model generated by data in existing surgical images.
[0202] As an example, when the entire pre-agreement operation is performed in the order of skin incision, vascular clamping and bilateral vascular connection, the remote consultation support server 1000 divides each treatment / operation into detailed intervals and provides them to the real-time image, and judges that the surgeon performs the processing specified in the "order pre-specified by the consulting doctor", the preset consulting doctor's avatar can provide the augmented reality data signed in the responsible interval to the real-time image.
[0203] In this way, the remote support consultation system of the present invention can extract tags that may become issues in real-time surgery and mark them on real-time images through the methods of the above-mentioned various embodiments. At this time, each embodiment can be separate or combined.
[0204] In particular, when labeling the surgical procedure or specific treatment sequence recommended by the consulting physician, the responsible subject information based on whether the surgeon accepts or not is recorded as label information, which can be conveniently used for determining responsibility attribution.
[0205] Meanwhile, when marking the real-time image, if the patient requires a review request for the protection of personal information, the marking unit 1120 may block the processing or replace it with another real-time image portion of the same time period.
[0206] The remote consultation surgery support server 1000 can store the surgical images marked with multiple tags in the real-time surgery in the database 2000. At this time, the stored surgical images can be used according to the purpose of the surgeon, the consulting doctor, and the educator / trainer. The remote consultation support system of the present invention can use and mediate the surgical images by influencing the mediation server 4000, and can also be implemented as a surgical image mediation system that distributes revenue according to use.
[0207] A brokerage system for trading and using surgical images including labels
[0208] Fig.15 It is shown Figure 1 A block diagram of the structure of the image intermediary server. Fig.16 : is a schematic diagram showing the blockchain structure of the remote consultation support system.
[0209] refer to Fig.15The image intermediary server 4000 may include an NFT management unit 4100 , an image transaction unit 4200 , a search unit 4300 and a user interface providing unit 4400 .
[0210] In addition, reference Fig.16 , the remote consultation support system can form a blockchain network with the hospital servers (a to n) of the member hospitals, and collect surgical images generated during real-time surgery through each hospital server (a to n) and provide the surgical images for use.
[0211] When surgical images are stored and collected in the database 2000 from each hospital server (a to n), the NFT management unit 4100 assigns an inherent ID to the collected surgical images, so that ownership based on non-fungible tokens can be issued to the corresponding hospital.
[0212] When the sale or use of the surgical image or the tag included in the surgical image occurs from the third-party user terminal, the image intermediary server 4000 can settle and manage the usage fee according to the sale or use using the cryptocurrency.
[0213] In addition, when a request is made to split and sell a surgical image owned by a specific hospital server, the image intermediary server 4000 assigns a unique ID to each tag interval included in the surgical image and can issue NFT ownership per tag interval. Therefore, split and selling of surgical or treatment images per tag interval can be provided.
[0214] If the preset fee is settled when the surgical images are traded on the hospital server, the NFT management unit 4100 can transfer the management to the hospital that sells the NFT ownership.
[0215] In addition, the NTF management unit 4100 compares the hospital to which the user terminal requesting the use of surgical images belongs with the owner of the surgical images requested through the user terminal. If they are different, it is judged to be a third-party user terminal that does not have ownership, and the image transaction unit 4200 can be requested to calculate the usage fee.
[0216] The image transaction unit 4200 may calculate a usage fee based on the number of tags included in the surgical image requested for use by the user terminal and the preset importance of each tag.
[0217] Among them, the label may include at least one of label information, responsible party information, label interval in the corresponding surgical image, label interval information, label interval information inputter, NFT-based ownership information, sales and usage history, and token settlement information.
[0218] The label interval in the surgical image may be a detailed treatment included in the surgical image or an interval divided by a detailed surgery. In addition, the label interval information may be detailed treatment or detailed surgery information, and the label interval information inputter may be a surgeon or a consulting physician who performs the detailed treatment or detailed surgery.
[0219] According to the use of user terminals and sales by hospital servers, if the information included in the tag changes or new information occurs, the blockchain based on the updated information can be attached and updated to the distributed ledger through the consensus of the nodes in the blockchain.
[0220] Furthermore, when tag section information is used at a specific user terminal, the image transaction unit 4200 may accumulate cryptocurrency to the corresponding tag section information inputter based on the usage fee collected from the corresponding user terminal.
[0221] The search unit 4300 provides a search service according to the purpose of image playback based on the tag information and tag interval information of the tags stored in the database 2000, so that the surgical images and tags required by the user terminal can be used.
[0222] The user terminal may be a head mounted display 100, 100-1, 100-2, 200, 500 worn by at least one of a surgeon performing a real-time surgery, a consulting surgeon who consults on the real-time surgery, and at least one assistant and educator who participates in the real-time surgery.
[0223] When receiving the participating surgery information from the user terminal, the AI-based recommendation server 3000 can recommend and provide surgical images and tags for similar surgeries.
[0224] In addition, when the artificial intelligence-based recommendation server 3000 receives user role information participating in an operation from a user terminal, tags included in similar operations may be recommended as necessary play tags based on the user role information.
[0225] The user interface providing unit 4400 may provide a directory for selecting to use or delete a plurality of tags included in similar surgeries recommended to the user terminal by the artificial intelligence-based recommendation server 300 to the corresponding user terminal.
[0226] The user interface providing unit 4400 may preset must-play tag information per user terminal based on role information of the user terminal, and tags corresponding to the preset must-play tag information may be excluded from the catalog and provided.
[0227] At this time, the image transaction unit 4200 can reflect the number of tags used or deleted by the corresponding user terminal and calculate and settle the usage fee.
[0228] In this way, the intermediary system of the present invention grants NFT-based ownership to surgical images including tags and intermediaries use and transactions, thereby intermediating transparent and reliable transactions and distributing transaction proceeds.
[0229] In addition, through AI-based recommendations, optimal surgical images and labels can be provided according to the playback purpose of the user (surgeon, co-surgeon, assistant one, assistant two, educator and trainer), and the utilization rate of surgery can be improved.
[0230] The embodiments of the present invention may be implemented in the form of program instructions executed by various computer tools and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., alone or in combination.
[0231] The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known to be used by those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magneto-optical media such as floppy disks, and specially configured hardware devices that store program instructions of read-only memory (ROM), random access memory (RAM), flash memory, etc. and execute. Examples of program instructions include machine language codes, such as codes generated by a compiler, and high-level language codes that can be executed by a computer using an interpreter, etc.
[0232] The hardware devices may be configured to operate as one or more software modules for performing the operations of the present invention, and vice versa.
[0233] Although the embodiments of the present invention have been described in more detail with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments, and various modifications may be made without departing from the technical spirit of the present invention.
[0234] Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but are for illustrative purposes, and the scope of the technical idea of the present invention is not limited to these embodiments.
[0235] Therefore, the above embodiments should be understood as illustrative rather than restrictive in all aspects. The protection scope of the present invention should be interpreted according to the appended claims, and all technical ideas within the equivalent scope thereof should be interpreted as included in the right scope of the present invention.
Claims
1. A remote consultation support system, characterized in that: include: A first head mounted display is worn by a surgeon at an actual surgical site and includes a camera that captures the surgeon's gaze area and surrounding areas and a display unit that outputs a real-time image captured by the camera; a second head mounted display, worn by a consulting physician located remotely from the primary surgeon, receiving and outputting real-time images captured by the first head mounted display, and receiving a suggestion of a sequence of surgical execution or a specific sequence of treatments included in the surgery by the consulting physician; as well as The remote consultation support server, when receiving the input suggestion from the second head mounted display, provides the first head mounted display with a user interface for selecting whether to accept the suggestion, generates a label according to whether the surgeon accepts the suggestion, and marks the real-time image. The tag includes information about a responsible party for real-time surgery based on the recommendation.
2. The remote consultation support system according to claim 1, characterized in that: in, The tag also includes tag location information within the real-time image, including information on the tag stored in a blockchain-based distributed ledger.
3. The remote consultation support system according to claim 2, characterized in that: in, If a preset specific input is received from the first head mounted display, the remote consultation support server generates a timestamp and marks it to the real-time image, and divides the real-time surgery into a plurality of detailed intervals defined by the surgeon based on the marked multiple timestamps. The marked location information includes a start timestamp and an end timestamp of a detailed interval of the marked location including the corresponding tag.
4. The remote consultation support system according to claim 3, characterized in that: in, The preset specific input is at least one of a gesture, an audio command, and a control signal command. The detailed section is a section corresponding to the detailed treatment and surgical sequence of the real-time surgery defined by the operating surgeon.
5. The remote consultation support system according to claim 1, characterized in that: in, The remote consultation support server, for providing the suggestion together with the user interface as a preview to the first head mounted display, When the suggestion is accepted through the first head mounted display, augmented reality data corresponding to the suggestion is inserted into the real-time image and displayed by the first head mounted display and the second head mounted display.
6. The remote consultation support system according to claim 1, characterized in that: in, The remote consultation support server, When the second head mounted display receives the advice from the consultant, the first head mounted display and the second head mounted display are provided with an avatar of the consultant who automatically signs the advice together with the advice, When the first head mounted display accepts the suggestion, an avatar of the surgeon who automatically signs the surgeon's responsibility preset for accepting the suggestion is provided to the first head mounted display and the second head mounted display.
7. The remote consultation support system according to claim 1, characterized in that: in, The remote consultation support server, automatically setting and receiving from each of the first head mounted display and the second head mounted display responsible consent for a specific procedure or surgery predefined by surgical classification and patient status, When a suggestion for the predefined specific treatment or surgery occurs, an avatar that automatically executes a preset signature of the corresponding head mounted display to which the responsibility consent is automatically set is provided to the first head mounted display and the second head mounted display.
8. The remote consultation support system according to claim 7, characterized in that: Also includes: A database for storing surgical images marked with at least one tag; as well as The artificial intelligence-based recommendation server automatically suggests tags contained in the detailed interval to the first head-mounted display when the detailed interval included in a similar operation similar to the real-time operation in the surgical images pre-stored in the database corresponds to the predefined specific treatment or operation set by the second head-mounted display responsibility consent.
9. The remote consultation support system according to claim 7, characterized in that: in, The remote consultation support server, When a suggestion for the predefined specific treatment or surgery in which the responsibility consent is automatically set is received from the first head mounted display, it is automatically marked to the real-time image.
10. The remote consultation support system according to claim 7, characterized in that: Also includes: The artificial intelligence-based recommendation server automatically recommends at least one tag included in the detailed interval of a similar surgery similar to the real-time surgery in a pre-stored surgical image to the first head-mounted display when the detailed interval corresponds to the predefined specific treatment or surgery set by the first head-mounted display responsibility consent, The remote consultation support server tags the real-time image with at least one tag automatically suggested by the artificial intelligence-based recommendation server.
11. The remote consultation support system according to claim 1, characterized in that: in, The second head mounted display selectively outputs at least one of a first screen and a second screen, wherein the first screen outputs a real-time image captured by the first head mounted display, and the second screen displays at least one suggestion that can be suggested during real-time surgery in a preview window.
12. The remote consultation support system according to claim 11, characterized in that: Also includes: A database that classifies and stores the surgical impact marked by at least one tag by physician, patient status, treatment, and surgical category; The second head mounted display sets at least one of the tags marked in the surgical image stored in the database as a focus tag, and receives a display time and a display interval of the focus tag. The remote consultation support server outputs the attention tag to the preview window of the second screen according to the setting of the second head mounted display.
13. The remote consultation support system according to claim 11, characterized in that: Also includes: The artificial intelligence-based recommendation server is used to recommend tags of similar surgeries marked on the previous surgical images of the consulting doctor that are similar to the real-time surgery through the preview window.
14. The remote consultation support system according to claim 1, characterized in that: Also includes: The artificial intelligence-based recommendation server automatically provides the labels marked during similar surgeries in previous surgical images performed by the consulting physician to the first head-mounted display as suggestions of the consulting physician when the labels marked during similar surgeries that were similar to the real-time surgery were not suggested during the real-time surgery.
15. The remote consultation support system according to claim 14, characterized in that: in, The artificial intelligence-based recommendation server, Based on the marking positions and marking intervals of the labels marked during the similar surgery, the marking positions and marking intervals of the labels suggested in the real-time image are determined and automatically suggested to the first head mounted display.
16. The remote consultation support system according to claim 1, characterized in that: Also includes: a third head mounted display, respectively worn by a plurality of assistants during real-time surgery at the actual surgical site; as well as A database for storing surgical images marked with at least one tag, Based on the role information of each head mounted display wearer of the real-time surgery, tags that must be played are extracted from each head mounted display and automatically suggested to the corresponding head mounted display.
