Remote medical collaboration support system capable of tagging real-time surgical video

Abstract

The present invention relates to a remote medical cooperation support system capable of tagging real-time surgical video. More specifically, the remote medical cooperation support system of the present invention includes a first HMD worn by a surgeon at an actual surgical site, including a camera for capturing the gaze area and surrounding area of ​​the wearing surgeon and a display unit for outputting real-time video captured by the camera, a second HMD worn by a medical assistant located remotely from the surgeon, receiving and outputting real-time video captured by the first HMD, and inputting a suggestion by the medical assistant for a surgical procedure sequence or a specific procedure sequence included in the surgery, and a remote medical cooperation support server that, when receiving a suggestion input from the second HMD, provides a UI for the first HMD to select whether or not to accept the suggestion, generates a tag according to the acceptance of the surgeon, and tags the collected real-time video, the tag including information of a responsible entity for the real-time surgery according to the suggestion.

Description

[Technical field]

[0001] The present invention relates to a remote medical cooperation support system capable of tagging real-time surgical video. More specifically, the present invention relates to a remote medical cooperation support system capable of tagging real-time surgical video that can be used to determine responsibility in the event of a medical accident by tagging the real-time video with responsibility based on whether or not the surgeon accepts a medical cooperation proposal made by a medical cooperation doctor during real-time surgery. [Background technology]

[0002] Recently, as multimedia technology has rapidly developed and is being used in various fields, technology has been developed in the medical field as well, allowing doctors to record surgical procedures on patients so that they can present them at medical conferences or use them for other purposes.

[0003] In addition, surgical footage is being filmed to document the procedure and to prevent any medical disputes that may arise. As most medical equipment is now equipped with video recording capabilities, the amount of surgical footage being filmed is rapidly increasing.

[0004] In addition, surgeries are often performed with the cooperation of various specialists depending on the size of the hospital and the illness. However, due to time and space constraints, remote medical cooperation is being carried out using messages, images, etc.

[0005] When surgery is performed through such remote medical cooperation, it is difficult to understand the details of the medical cooperation from the surgical video, so a method is needed to determine who is responsible for the surgical video during remote medical cooperation.

[0006] In addition, during remote medical cooperation, it is necessary to reproduce the atmosphere of the surgery site in real time and provide it to the collaborating doctor, and when proposing medical cooperation, it must be shared with the surgeon through accurate pointing. In particular, it is difficult to point to the affected area inside the body when indicating the patient's affected area. Therefore, there is a demand for a method that allows the collaborating doctor to point in 3D on the real-time surgery video shared with the surgeon.

[0007] In addition, the recorded surgery video can be used in various ways. In particular, when various medical information is added to the surgery video, it has a great potential for use and asset value, so a system that can mediate stable transactions is required. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Korean Patent Publication No. 10-2020-0001266 (Published on January 6, 2020) Summary of the Invention [Problem to be solved by the invention]

[0009] The technical problem that the present invention aims to solve is to provide a remote medical cooperation support system that enables real-time surgical video tagging that can be used to determine responsibility in the event of a medical accident by tagging the real-time video with the responsibility depending on whether the surgeon accepts a medical cooperation proposal made by a medical cooperating doctor during real-time surgery.

[0010] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the following description. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems, a remote medical cooperation support system according to one embodiment of the present invention may include a first HMD worn by a surgeon at an actual surgical site, the first HMD including a camera that captures the gaze area and surrounding areas of the worn surgeon and a display unit that outputs real-time images captured by the camera, a second HMD worn by a clinical assistant located remotely from the surgeon, the second HMD receiving and outputting real-time images captured by the first HMD and receiving suggestions from the clinical assistant for a surgical procedure order or a specific procedure order included in the surgery, and a remote medical cooperation support server that, when receiving the suggestions input from the second HMD, provides a UI to the first HMD for receiving a selection of whether to accept the suggestions, and generates a tag based on the surgeon's acceptance and tags the collected real-time images.

[0012] In this case, the tag may include information about the responsible party for the real-time operation according to the proposal.

[0013] In addition, the tag may further include tagged location information within the real-time video, and the information included in the tag may be stored in a blockchain-based distributed ledger.

[0014] In addition, when a predetermined specific input is received from the first HMD, the remote medical cooperation support server can generate a timestamp and tag the real-time video, and divide the real-time surgery into a number of subsections defined by the surgeon based on the tagged timestamps.

[0015] In this case, the tagged position information may include a start timestamp and an end timestamp of a detailed section including a tagged point of the corresponding tag.

[0016] In addition, the predefined specific input may be at least one of a gesture, a voice command, and a control signal command, and the detailed section may be a section corresponding to a detailed procedure and surgical sequence of the real-time surgery defined by the surgeon.

[0017] In addition, the remote medical cooperation support server may provide a preview of the proposal together with the UI to the first HMD, and when the proposal is accepted via the first HMD, AR data corresponding to the proposal may be inserted into the real-time image and displayed on the first HMD and the second HMD.

[0018] In addition, when the remote medical cooperation support server receives a proposal from the medical cooperating doctor from the second HMD, it provides the first HMD and the second HMD with a medical cooperating doctor avatar that automatically executes the default medical cooperating doctor's signature for the proposal responsibility along with the proposal, and when the proposal is accepted by the first HMD, it can provide the first HMD and the second HMD with a surgeon avatar that automatically executes the default surgeon's signature for the proposal acceptance responsibility.

[0019] In addition, the remote medical cooperation support server can automatically set a responsibility consent for a specific procedure or surgery predefined by surgery category and patient condition from each of the first HMD and the second HMD, and when a proposal for the predefined specific procedure or surgery occurs, provide the first HMD and the second HMD with an avatar that automatically performs the default signature of the corresponding HMD for which the responsibility consent was automatically set.

[0020] The present invention may further include a DB for storing surgical videos tagged with at least one tag, and an AI-based recommendation server for automatically suggesting to the first HMD, when a detailed section included in a similar surgery similar to the real-time surgery among the surgical videos pre-stored in the DB corresponds to the pre-defined specific procedure or surgery for which a responsibility agreement is set by the second HMD, the detailed section.

[0021] In addition, the remote medical cooperation support server can automatically tag the real-time video when it receives from the first HMD a proposal for the predefined specific procedure or surgery for which the responsibility agreement is automatically set.

[0022] In addition, the system may further include an AI-based recommendation server that automatically suggests at least one tag included in a detailed section of a similar surgery similar to the real-time surgery among pre-stored surgical videos to the first HMD when the detailed section of the similar surgery corresponds to the pre-defined specific procedure or surgery for which a liability agreement is set by the first HMD.

[0023] In addition, the remote medical cooperation support server can tag the real-time video with at least one tag automatically suggested by the AI-based recommendation server.

[0024] In addition, the second HMD can selectively output at least one of a first screen that outputs real-time video captured by the first HMD and a second screen that displays at least one suggestion that can be proposed during real-time surgery in a preview window.

[0025] The system further includes a DB that classifies and stores surgical images tagged with at least one tag by doctor, patient condition, procedure, and surgery category, and the second HMD sets at least one of the tags tagged to the surgical images stored in the DB as a tag of interest, and a display time and a display interval of the tag of interest are set, and the remote medical cooperation support server outputs the tag of interest to a preview window of the second screen according to the setting of the second HMD.

[0026] The system may further include an AI-based recommendation server that recommends tags that are tagged to similar surgeries that are similar to the real-time surgery from previous surgery videos of the cooperating physician in the preview window.

[0027] In addition, the system may further include an AI-based recommendation server that automatically provides the first HMD with tags tagged during a similar surgery similar to the real-time surgery among previous surgical videos performed by the clinical assistant physician as a suggestion from the clinical assistant physician if the tags tagged during the similar surgery are not suggested during the real-time surgery.

[0028] In addition, the AI-based recommendation server can determine the tagging points and tagging intervals of the tags to be proposed in the real-time video based on the tagging points and tagging intervals of the tags tagged during the similar surgery, and automatically suggest them to the first HMD.

[0029] In addition, based on the third HMD worn by multiple assistants during real-time surgery at the actual surgical site, a DB storing surgical videos tagged with at least one tag, and role information of the wearer of each HMD for the real-time surgery, tags that must be played on each HMD can be extracted and automatically suggested to the corresponding HMD.

[0030] Further details of the embodiments are included in the detailed description and the drawings. Effect of the Invention

[0031] The remote medical cooperation support system capable of tagging real-time surgical videos according to an embodiment of the present invention supports remote medical cooperation during real-time surgery, and can tag events that occur during real-time surgery, such as medical cooperation suggestions, to the real-time video to store and utilize the surgical video.

[0032] In particular, when a collaborating physician proposes medical cooperation during real-time surgery, information about the responsible party can be tagged to the real-time video depending on whether the surgeon accepts it, and this can be used to determine who is responsible in the event of a medical accident.

[0033] In addition, the camera can provide a fixed-view mode to capture the field of view of each camera at the actual surgery site through the HMD of a remote collaborating physician, adding a sense of augmented reality. The camera can also provide a fixed-view mode to allow the collaborating physician to independently grasp the current state of surgery.

[0034] In addition, the 3D pointing system allows for 3D pointing on real-time video when proposing collaborative treatment, allowing the collaborating physician to point in 3D to the affected area of ​​the patient, i.e., the inside of the body, which is necessary when collaborating on medical treatment, and allows the medical image of the affected area to be shared with the surgeon.

[0035] In addition, by recording tag information on the blockchain ledger and granting NFT-based ownership to surgery videos containing tags, it is possible to mediate use and transactions, allowing for transparent and reliable transactions and share transaction revenues.

[0036] In addition, through AI-based recommendations, it is possible to provide optimal surgical videos and tags according to the playback purpose of the user (surgeon, collaborating physician, assistant 1, assistant 2, trainee), thereby increasing the utilization of surgical videos.

[0037] The effects of the present invention are not limited to the above-mentioned examples, and various other effects are included in the present specification. [Brief description of the drawings]

[0038] [Figure 1] FIG. 1 is a diagram for explaining a remote medical cooperation support system according to the present invention. [Diagram 2] FIG. 2 is a block diagram for explaining the configuration of the HMD in FIG. [Diagram 3] 2 is a configuration block diagram for explaining the configuration of the remote medical consultation cooperation support server of FIG. 1. [Figure 4] FIG. 2 is a diagram for explaining remote medical cooperation using the second HMD of FIG. 1. [Diagram 5] FIG. 2 is a diagram for explaining remote medical cooperation using the second HMD of FIG. 1. [Figure 6] 1 is a block diagram illustrating a configuration of a three-dimensional pointing system according to an embodiment of the present invention. [Figure 7] 4 is a flowchart illustrating a three-dimensional pointing control method according to an embodiment of the present invention. [Figure 8] FIG. 2 is a diagram for explaining a control device of a second HMD in the first embodiment of the present invention. [Figure 9] FIG. 11 is a diagram for explaining a control device of a second HMD in accordance with a second embodiment of the present invention. [Figure 10] FIG. 11 is a diagram for explaining a control device of a second HMD in a third embodiment of the present invention. [Figure 11] FIG. 13 is a diagram illustrating a tag section setting according to an embodiment of the present invention. [Figure 12] FIG. 13 is a diagram illustrating a preview window for a medical cooperation proposal according to an embodiment of the present invention. [Figure 13] 13 is a diagram illustrating a display screen of a first HMD that has received a medical cooperation proposal in accordance with an embodiment of the present invention. FIG. [Figure 14] FIG. 13 is a diagram for explaining authentication related to proposal responsibility and proposal acceptance responsibility in one embodiment of the present invention. [Figure 15] 2 is a block diagram for explaining the configuration of the video mediation server of FIG. 1. [Figure 16] FIG. 11 is a diagram to explain the blockchain structure of the remote medical consultation cooperation support system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0039] The following merely illustrates the principles of the invention. Thus, those skilled in the art can devise various devices that embody the principles of the invention and fall within the concept and scope of the invention, even if not expressly described or illustrated herein. Furthermore, any conditional terms and examples recited herein are expressly intended only for the purpose of allowing the concept of the invention to be understood in principle, and should be understood as being not limiting to the examples and conditions so specifically recited.

[0040] In addition, in the following description, ordinal expressions such as "1", "2", etc. are intended to describe objects which are equivalent and independent of each other, and the order should be understood as having no meaning of main / sub or master / slave.

[0041] The above-mentioned objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, which will enable those skilled in the art to which the invention pertains to easily implement the technical ideas of the invention.

[0042] The features of the various embodiments of the present invention may be partially or fully combined or combined with each other, and may be technically interlocked and driven in various ways, as will be fully understood by those skilled in the art, and each embodiment may be implemented independently of each other, or may be implemented together in a related relationship.

[0043] A remote medical cooperation support system according to one embodiment of the present invention relates to a system that can support remote medical cooperation by a medical cooperation doctor located remotely when a real-time surgery is being performed by a surgeon at an actual surgical site.

[0044] At this time, it is necessary to determine who is responsible for the real-time surgery and medical cooperation. In the present invention, a tag for determining who is responsible for the medical cooperation proposal is attached to the real-time surgical video taken in real time, which can be used to determine who is responsible in the event of a medical accident.

[0045] In addition, the remote medical cooperation support system of the present invention can provide AR images corresponding to an actual surgical site based on real-time video in order to provide the atmosphere of an actual surgical site to a medical cooperation doctor located remotely, and can also configure a 3D pointing system that can point in 3D to a specific location in the real-time video for suggestion and sharing.

[0046] In addition, the remote medical cooperation support system of the present invention can also be configured as a video intermediation system for utilizing surgical video tagged with images taken during real-time surgery (utilization according to the purposes of the surgeon, medical cooperating physician, and education / trainees) and for generating revenue from such utilization.

[0047] This will be described in detail with reference to FIGS.

[0048] FIG. 1 is a diagram for explaining the remote medical cooperation support system of the present invention.

[0049] The remote medical cooperation support system can be implemented using a first HMD 100 worn by a surgeon 10 and a second HMD 200 worn by a medical cooperation doctor 20 located remotely, and can include a remote medical cooperation support server 1000, a DB 2000, an AI-based recommendation server 3000, and a video intermediation server 4000 for remote medical cooperation support.

[0050] In addition, in order to support remote medical cooperation for real-time surgery at the actual surgery site 1, multiple cameras can be installed to record the real-time surgery and the recorded footage can be shared with a medical cooperation doctor located in a remote location.

[0051] Specifically, a CCTV 2 is installed at the actual surgical site 1, and the surgical site 1 may further include a third HMD 100-1, 100-2 worn by multiple assistants 11, 12, a fourth HMD 300 worn by the patient 30, and a camera 400 installed on the ceiling of the surgical site 1 to provide a clear 3D image of the actual surgical site 1.

[0052] The camera 400 installed on the ceiling can identify the initial position by specifying object-specific coordinates in the first real-time image captured, and can identify the positions of the first HMD 100 worn by the surgeon 10, and the third HMDs 100-1 and 100-2 worn by assistant 1 (11) and assistant 2 (12) during real-time surgery and track their movements. Furthermore, the movements of all objects such as surgical tools captured by the ceiling camera 400 can be tracked.

[0053] Fig. 2 is a block diagram for explaining the configuration of the HMD in Fig. 1. Referring to Fig. 2, each of the HMDs 100, 100-1, 100-2, 200, 300, and 500 may include a camera, a display unit, an input unit, a control unit, and a communication unit, and may further include a speaker, a microphone, and the like.

[0054] At this time, the on / off of each component and the image provided by the display unit can be applied differently depending on the wearer, i.e., the wearer mode set to, for example, a surgeon mode, an assistant mode, a patient mode, an education and training mode, to provide the necessary functions to the corresponding wearer. Alternatively, the HMD for each wearer can be configured as a separate HMD that provides only the functions corresponding to the corresponding mode.

[0055] In this case, the camera of the HMD may include a first camera provided in front of the HMD to capture the gaze area of ​​the wearer, and second cameras provided on both sides of the HMD to capture the surrounding area of ​​the wearer.

[0056] In addition, the display unit displays an image of the wearer's gaze area captured by a camera in the surgeon mode and assistant mode, and receives and displays real-time images captured by each camera at an actual surgical site in the medical assistant mode. Furthermore, the patient mode can provide information as needed, and the education and training mode can display a selected playback image.

[0057] The input unit may be an interface or a sensing unit to which control signals for controlling the HMD and the displayed image are input, and the control unit controls the HMD according to the control signals received through the input unit and can receive various remote medical cooperation support from each external server 1000-4000 through the communication unit.

[0058] 3 is a block diagram showing the configuration of the remote medical treatment cooperation support server 1000. The remote medical treatment cooperation support server 1000 may include a medical treatment cooperation support unit 1100 and a video tagging unit 1200. In this case, each component may be composed of a separate module, server, etc.

[0059] Here, remote medical cooperation support may involve real-time surgery-related medical cooperation proposals, determining responsibility for the proposals, video tagging to tag videos taken in real time when various events occur, 3D pointing control for displayed videos, utilization of tagged surgical videos, and revenue distribution for utilization.

[0060] The medical cooperation support unit 1100 collects real-time images output from all cameras in the operating room 1 and transmits them to the second HMD 200 so that a medical cooperation doctor can participate remotely.

[0061] At this time, in order for the cooperating physician to remotely participate, the cooperating physician must be able to grasp the situation in the operating room 1 independently of the surgeon via the second HMD 200 that he or she is wearing.

[0062] Here, the second HMD 200 worn by the collaborating physician can provide the surgeon with real-time images captured by the CCTV 2 and camera 400 installed in the operating room 1, the camera of the first HMD 100 worn by the surgeon, and the cameras of the third HMDs 100-1 and 100-2 worn by multiple assistants 11 and 12, at the surgeon's option.

[0063] As an example, when a specific position of the actual surgical site selected by the collaborating physician (for example, the camera of the first HMD 100 worn by the surgeon as shown in FIG. 5 ) is selected via the display unit of the second HMD 200 worn by the collaborating physician, real-time images captured within the field of view of the first camera of the first HMD 100 can be output to the display unit of the second HMD 200.

[0064] At this time, as shown in Figures 4 and 5, the collaborating physician will be able to secure the same field of view as the surgeon, and after securing the field of view, he or she can also look independently of the surgeon and grasp the situation inside the operating room 1.

[0065] 4, when the assistant doctor looks in the direction of assistant 1 (11) or assistant 2 (12), among the real-time images captured in the field of view of the second cameras provided on both sides of the first HMD 100 worn by the surgeon 10, the real-time image captured by the specific second camera corresponding to the line of sight of the assistant doctor can be output via the second HMD 200 of the assistant doctor 20. In other words, the field of view can be moved independently of the surgeon without fixing the surroundings (field of view fixed release mode).

[0066] In addition, when a specific camera is selected, the view can be moved independently in the same way, or the surroundings can be fixed and only real-time images captured by the selected camera can be received (view fixed mode).

[0067] In this way, the present invention can provide a fixed field of view mode to the second HMD 200 worn by the medical cooperating doctor 20 according to the selection of the medical cooperating doctor by peripheral fixation. In the fixed field of view mode, the medical cooperating doctor can cooperate in medical treatment by three-dimensionally pointing to the position that needs to be shared for medical cooperation proposal using the three-dimensional pointing system for remote medical cooperation (hereinafter, three-dimensional pointing system) shown in Fig. 6.

[0068] 3D pointing system for remote medical collaboration 6, the 3D pointing system according to an embodiment of the present invention further includes a control device 210 for controlling the second HMD 200 in the remote medical cooperation support system of FIG 1, and the control device can be variously implemented as control devices 500, 500a, 600 as shown in FIG 8 to FIG 10. The 3D pointing control method will be described together with FIG 7.

[0069] Real-time images captured by the first and second cameras of the first HMD 100 worn by the surgeon 10 participating in the real-time surgery at the actual surgical site 1, the first and second cameras of the third HMDs 110-1, 110-2 worn by multiple assistants 11, 12, the camera of the fourth HMD 300 worn by the patient 30, CCTV 1, and ceiling camera 400 can be transmitted to the remote medical cooperation support server 1000.

[0070] At this time, the real-time image captured by the corresponding first camera can be output to the display unit of the first HMD 100 and transmitted to the remote medical consultation cooperation support server 1000 (S701). Also, the real-time image captured by the corresponding first camera can be output to the display unit of each of the third HMDs 110-1 and 110-2 and transmitted to the remote medical consultation cooperation support server 1000.

[0071] Next, the medical cooperation support unit 1100 of the remote medical cooperation support server 1000 can collect real-time images from each camera, synchronize the time sync, and transmit the images to the second HMD 200 (S702).

[0072] Next, the second HMD 200 can receive and display the real-time image of the first HMD 100 (S703), and when receiving real-time images of multiple HMDs 100, 100-1, 100-2, it can display the real-time image corresponding to the camera selected by the cooperating physician or a default camera. At this time, it is preferable that the real-time image of the first HMD 100 of the surgeon is output, and it can be changed by the cooperating physician later.

[0073] Next, the second HMD 200 can three-dimensionally point a specific position for a medical cooperation proposal on the real-time image output to the display unit according to a control signal received by the second HMD 200 via the input unit. At this time, the specific position can be a specific position in the actual operating room 1, and in particular, can be a specific position of the patient's affected area for a medical cooperation proposal. In the present invention, the real-time image (image generated by the first camera that captures the gaze area of ​​the first HMD 100) needs to be set to a fixed field of view mode in the second HMD 200 so that the surgeon 10 and the medical cooperation doctor 20 can share the patient's treated affected area and cooperate in medical treatment. In other words, the medical cooperation doctor needs to be able to check from the surgeon's viewpoint and make a proposal for the surgery to the surgeon.

[0074] 4 and 5, in the fixed field of view mode, as shown in Fig. 4, the second HMD 200 outputs only the real-time image output from the first camera of the first HMD 100 to the second HMD 200, and the medical assistant doctor can check the real-time surgery within the range corresponding to the imaging field of view α of the first camera. That is, even if the medical assistant doctor moves (turning his head toward the auxiliary 1 or auxiliary 2 direction), only the real-time image within the imaging field of view α is output.

[0075] On the other hand, when the peripheral fixation release (field of view fixation release mode) is set, when the second HMD 200 moves, a real-time image corresponding to the field of view of the second camera corresponding to the direction of movement is provided, and the surgeon can also shift his or her field of view independently to check the surroundings.

[0076] As an example, in Figures 4 and 5, when the assistant physician gazes at the first real-time image (the real-time image corresponding to the field of view α of the first camera of the first HMD 100) within the field of view α and then turns his head to look in the direction of Auxiliary 1 (11), the second real-time image captured in the field of view of the second camera provided on the Auxiliary 1 (11) side of the first HMD 100 worn by the surgeon 10 is continuously provided, allowing the surgeon to shift his field of view independently of the surgeon.

[0077] The 3D pointing control system of the present invention can perform 3D pointing for suggesting medical cooperation after setting the real-time image displayed on the second HMD 200 to a fixed view mode.

[0078] The control device 210 for 3D pointing may include a pointing means 211 and a direction selection means 212. The control device 210 may output a control signal for generating a virtual line and a 3D pointer in a 3D real-time image to the second HMD 200 according to the control of the surgeon, and may control the movement of the 3D pointer using the pointing means 211 and the direction selection means 212 (S704).

[0079] The pointing means 211 moves a three-dimensional pointer in the depth direction (z direction in Figures 8 to 10) along a virtual line generated in the three-dimensional real-time image, and the direction selection means 212 moves the three-dimensional pointer in a direction perpendicular to the depth direction (x or y direction in Figures 8 to 10).

[0080] The pointing means 211 and the direction selection means 212 can be implemented in various ways in the control devices 500, 500a, 600 as shown in Figs. 8 to 10, and the direction selection means 212 can be used in the form of a foot pedal.

[0081] In addition, the control signal at step S704 can be input by a predetermined gesture, gaze processing, or eye movement (such as blinking) of the medical assistant physician sensed by the input unit of the second HMD 200, or can be a signal input by one of the control devices of Figures 8 to 10.

[0082] First, when using the predetermined gestures, gaze, gaze movement, etc. of the medical assistant physician sensed by the input unit of the second HMD 200, as an example, a fixed field of view mode can be set by a first gesture (movement of spreading both arms at a certain angle), a pointer P1 and a virtual line 3 can be formed on the affected area by gaze processing (staring at the affected area for pointing for a predetermined period of time or more), and a pointer P2 can be moved in the depth direction on the virtual line 3 by a second gesture (hand movement). This allows three-dimensional pointing to the affected area inside the patient's body.

[0083] At this time, the depth can be adjusted by matching the depth and the organs and blood vessels inside the patient's body according to the depth. More specifically, using the control device 210, the surgeon can select a specific affected area segment region by moving a pointer in the depth direction (z) and in the direction perpendicular to the depth direction (x, y).

[0084] Specifically, the patient's affected area segment area may be statistically determined by learning the history of images of the affected areas of other patients who have the same medical history as the patient's affected area. In particular, the patient's affected area segment area may be divided to reflect the affected area confirmed by indocyanine green (ICG) staining.

[0085] ICG is a medical reagent that is a green pigment that fluoresces at an infrared wavelength of 805 nm and binds to albumin, a water-soluble protein present in the body's cytoplasm and tissues. After ICG is injected into the patient's blood vessels via intravenous injection to stain them with ICG, infrared light is irradiated to excite the blood vessels, and the reflected light is captured with a camera equipped with a filter, allowing the fluorescently stained image of the affected area to be obtained.

[0086] The remote medical treatment cooperation support server 1000 can collect images of affected areas, learn stained areas for each patient and each disease, and statistically classify and divide affected area segments.

[0087] In addition, the affected area can be divided into multiple sub-segments based on the arrangement of blood vessels relative to the affected area, the arrangement of other surrounding organs, and the depth difference between the affected area as observed from the perspectives of the surgeon and the collaborating physician.

[0088] Here, the depth difference of the affected area observed from the viewpoints of the surgeon and the cooperating physician can be the difference in distance (depth step) in the depth direction (z) between the detailed affected area gazed upon by the surgeon and the detailed affected area gazed upon by the cooperating physician for one affected area segment area.

[0089] Meanwhile, when using an ICG-stained image of the affected area in a real-time surgery, the ICG-stained image of the affected area captured by a camera using a filter can be received by the first and second HMDs and displayed in conjunction with the real-time image.

[0090] As a result, the surgeon and the medical assistant can proceed with the real-time surgery by pointing and selecting the affected area segment area in three dimensions. As an example, when the surgeon uses the pointing means 211 to generate a virtual line and a three-dimensional pointer located on the virtual line in the three-dimensional real-time image, the server 1000 determines that the affected area segment area of ​​the patient located at the shortest vertical distance from the three-dimensional position (x, y, z) of the three-dimensional pointer has been selected, and can provide a selection display for the affected area segment area (S705). As a result, the surgeon and the medical assistant can share the selected area via the first HMD 100 and the second HMD 200.

[0091] The surgeon can also move the three-dimensional pointer vertically by selecting a specific direction signal that is perpendicular (x, y) to the depth direction (z) using the direction selection means 212. That is, the server 1000 can move the selection display so that another diseased area segment area adjacent to the specific direction side selected by the direction selection means 212 is selected based on the diseased area segment area already selected and displayed (S706).

[0092] Three-dimensional pointing can also be provided by various embodiments of the present invention, such as the control device shown in Figures 8-10.

[0093] (First embodiment) - Control device 500 8, the control device 500 may include a first input unit 501, a second input unit 502, and a control unit 503. In this case, the first input unit 501 and the second input unit 502 may be provided on the main body in the form of a button, etc.

[0094] When an input signal is received by pushing a button, the first input unit 501 may output a first control signal for setting a view fixed mode or a view non-fixed mode for a real-time image output from the second HMD 200. The second HMD 200 may receive the first control signal and set the mode.

[0095] In addition, when the field of view fixed mode is set by the first input unit 501, the first input unit 502 can output a second control signal that generates a virtual line 3 and a pointer p1 in the real-time image output to the second HMD 200 when it receives an input signal by pushing a button.

[0096] Next, the control unit 503 detects the movement of the main body of the control device in the depth direction of the virtual line or the opposite direction to the depth direction, and outputs a third control signal to move a pointer (p1 → p2) on the virtual line 3, thereby enabling 3D pointing to the real-time image displayed on the second HMD 200.

[0097] At this time, a selected layer L2 from among layers L1 and L2 matched to the depth level of the pointed position can be selected and displayed. Here, the layer can be an affected area image obtained by dividing the 3D affected area image into affected area segment areas.

[0098] That is, the affected segment area that is the shortest vertical distance in the depth direction (z) from the position of the 3D pointer P2 can be selected. Also, the body of the control device 500 can be moved in a direction (x, y) perpendicular to the depth direction (z), i.e., left and right or up and down around the user's hand, to move the 3D pointer and use it as a direction selection means to select another affected segment area.

[0099] Second embodiment: Control device 500a 9 may have a gun-like shape to improve the user's grip and convenience. Specifically, the control device 500a may include a header 530a, a handle 510a having an input button 511 and a trigger 520, and a control unit 540.

[0100] When the input button 511 is pressed, the header unit 530a can generate a virtual line in the real-time image and guide the direction of the virtual line to a specific position.

[0101] The input button 511 of the handle section 510a can set the view fixing mode when an input signal is received, and generate the virtual line 3 and the pointer p1 using the header section 530a. In addition, when the input button 511 is pushed again (receives the input signal again) in the state in which the view fixing mode is set, it can be set to the view fixing cancellation mode.

[0102] The trigger 520 of the handle portion 510a can output a control signal for moving the pointer on the imaginary line 3 in the depth direction of the imaginary line 3 (p1→p2) every time the trigger is operated.

[0103] At this time, the diseased area segment areas, which are numbered according to a predetermined depth on the virtual line 3, can be provided in a cyclical manner according to the depth every time the trigger is operated.

[0104] Also, the body of the control device 500a can be moved in a direction (x, y) perpendicular to the depth direction (z), i.e., left and right or up and down around the user's hand, to move the 3D pointer and select another affected segment area. Meanwhile, the control device 500, 500a can be used only as the pointing means 211, and in this case, a separate foot pedal can be used as the direction selecting means 212 to control the 3D pointing.

[0105] Third embodiment: Control device 600 As another embodiment, a control device 600 shown in Fig. 10 further includes a controller 500b in addition to the control device 500a shown in Fig. 9. That is, the control device 600 includes a first control device 500b and a second control device 500a. In this case, the form of the second control device 500a is the same as that of Fig. 9, but some functions may be changed according to settings.

[0106] The first control device 500b may include a handle 510b having a first button for setting a view fixation mode and a second button for setting a view fixation release mode, and a guide header portion 530b having a guide ring formed thereon.

[0107] The head portion 530a of the second control device 500a can be focused via the guide ring of the guide header portion 530b, and the direction in which the virtual line is generated by the head portion 530a can be guided.

[0108] The input button 511 of the handle section 510a can set the view fixing mode when an input signal is received, and generate the virtual line 3 and the pointer p1 using the header section 530a. In addition, when the input button 511 is pushed again (receives the input signal again) in the state in which the view fixing mode is set, it can be set to the view fixing cancellation mode.

[0109] The trigger 520 of the handle portion 510a can output a control signal for moving the pointer on the imaginary line 3 in the depth direction of the imaginary line 3 (p1→p2) every time the trigger is operated.

[0110] At this time, the diseased area segment areas, which are numbered according to a predetermined depth on the virtual line 3, can be provided in a cyclical manner according to the depth every time the trigger is operated.

[0111] At this time, a control signal for moving the pointer in the depth direction of the virtual line 3 (p1→p2) can be output every time the trigger 520 of the handle portion 520 is operated.

[0112] In this case, when the medical cooperation support unit 1100 of the remote medical cooperation support server 1000 receives a specific pointer position (one of the pointer positions p1, p2, etc.) on the virtual line 3 from the second HMD 200, it can provide an affected area segment area for an organ or blood vessel corresponding to the pointer position by inserting it into the real-time images output from the first to third HMDs. Also, it can provide the selected affected area segment area L1 by dividing it into sub-segment areas.

[0113] Although the above explanation focuses on the surgeon's 3D pointing, the surgeon can also 3D point to the affected area using the same method.

[0114] Video tagging for remote medical collaboration support system In addition, the remote medical cooperation support system tags real-time images captured by each camera when a real-time surgery is in progress, medical cooperation suggestions made during remote medical cooperation, recommendations made by the AI-based recommendation server 3000, and events set directly by the surgeon occur, and can be used for subsequent surgery, medical cooperation, education / training, and determining who is responsible.

[0115] To this end, the remote medical consultation cooperation support system comprises a remote medical consultation cooperation support server 1000, a DB 2000, an AI-based recommendation server 3000, and a video intermediation transaction server 4000. Referring to FIG. 3, the video tagging unit 1200 of the remote medical consultation cooperation support server 1000 may include a UI providing unit 1110, a tagging unit 1120, and a responsibility setting unit 1130.

[0116] Referring to FIG. 3, a description will be given of remote participation of a collaborating physician and real-time video tagging according to one embodiment of the present invention.

[0117] In this case, the remote medical cooperation support system sets priorities for the cameras and images that capture the real-time surgery, and the images captured by the corresponding cameras are displayed in order according to the set priorities on the display unit of the second HMD 200, allowing the medical cooperation doctor to decide on a medical cooperation proposal while monitoring. For example, the order of the camera of the first HMD 100 of the surgeon, CCTV, assistant 1, etc. can be set by the medical cooperation doctor, or it can be set by the medical cooperation server 1100.

[0118] In addition, a specific camera can be selected and the corresponding footage can be displayed according to the needs of the cooperating physician.

[0119] 1 to 3, the medical assistant 20 monitors the real-time surgery video of the surgeon 10 via the second HMD 200, and can transmit a suggestion when medical assistance is required.

[0120] The medical cooperation support unit 1100 of the remote medical cooperation support server 1000 can collect real-time images from each camera capturing real-time surgery and transmit them to the second HMD 200.

[0121] In this case, when a medical cooperation proposal is received from the second HMD 200 or a proposal is recommended from the AI-based recommendation server 3000, the UI providing unit 1110 may provide a UI for selecting whether to accept the proposal to the first HMD 100. The UI for selecting whether to accept the proposal may be provided as text or voice as shown in FIG.

[0122] The tagging unit 1120 may tag the real-time video according to the surgeon's acceptance or non-acceptance when a tag is generated according to a suggestion or event during real-time surgery. In this case, the suggestion of the medical assistant may be a suggestion for the order of surgery progress or a specific procedure order included in the surgery. When the UI providing unit 1110 receives the acceptance or non-acceptance of the suggestion, the tagging unit 1120 may generate a tag for the surgeon's acceptance or non-acceptance from the first HMD 100 and tag the collected real-time video.

[0123] In this case, the tag may include information on the responsible party for the proposed real-time surgery and tagged location information within the real-time video, and the information included in the tag may be stored in a blockchain-based distributed ledger.

[0124] Here, the tags can be marking information inserted into real-time video as metadata regarding suggestions from collaborating physicians that occur during real-time surgery, recommendation suggestions generated by the AI-based recommendation server 3000, and events directly set by the surgeon.

[0125] As an example, tags could be any information that can be an issue in a real-time surgery, such as an AR surgery demonstration by a collaborating physician, comments from the collaborating physician, footage of a previous surgery that is similar or identical to the real-time surgery, specific surgical tools, audio or text information, vital signs, etc.

[0126] In this case, the remote medical cooperation support system of the present invention can insert responsible party information to determine responsibility when tagging real-time video, and can divide the tagged sections of the real-time surgery into detailed sections that are divided into detailed procedures / surgeries of the real-time surgery.

[0127] Specifically, when a predetermined specific input is received from the first HMD 100, the tagging unit 1120 generates a timestamp and tags the real-time image, and can divide the real-time surgery into multiple subsections defined by the surgeon based on the multiple tagged timestamps.

[0128] Here, the predefined specific input may be at least one of a gesture, a voice command, and a control signal command, and the detailed section may be a section corresponding to a detailed procedure and surgical sequence of a real-time surgery defined by a surgeon.

[0129] 11 is a diagram for explaining tag section setting according to an embodiment of the present invention. The surgeon 10 can provide information regarding real-time surgery to the medical assistant 20 in advance.

[0130] At this time, the information provided in advance may be real-time surgery information, outline of detailed procedures, surgery sequence, etc. At this time, different medical assistant doctors may remotely participate in each detailed procedure, and the medical assistant support unit 1100 may limit the real-time video to be provided to the second HMD 200 of the corresponding medical assistant doctor 20 only during the detailed participation period.

[0131] In addition, for each detailed section during real-time surgery, the surgeon 10 and the clinical assistant 20 can set a flag indicating that the detailed section of the ongoing surgery or procedure is their own responsibility. In this case, when either party sets a flag, the tag information of the corresponding detailed section can be marked as the section that either party is responsible for.

[0132] In this case, if both flags are set or all flags are set, a person who does not agree with the responsibility section can display an objection message to the objection application, and the medical cooperation support unit 110 can support voice communication after a certain time has elapsed after the objection message is displayed.

[0133] In addition, when it is determined who is responsible for a detailed section, the AR providing unit 1130 can provide a predefined avatar of the responsible person to the real-time surgical video to confirm the real-time responsible section.

[0134] In addition, if the surgeon 10 does not have enough experience in the relevant surgery, the relevant specialist can cooperate in remote medical treatment as the medical cooperation doctor 20. In this case, if the medical cooperation support unit 1100 receives evaluation numerical data from the surgeon 10 regarding the specialist's experience and the current surgical experience that is equal to or greater than a predetermined value, the medical cooperation support unit 1100 can prioritize displaying the section of responsibility of the medical cooperation doctor on the real-time video and tagging it in the history tag information (metadata) linked to the recorded video section.

[0135] Referring to FIG. 11, for example, in the case of a coronary stent procedure, the procedure name (coronary stent), patient information, detailed procedures or sequence (anesthesia, stent, etc.) can be provided to a remote collaborating physician, and detailed sections can be divided based on specific input by the surgeon 10 during real-time surgery.

[0136] At this time, the tagging unit 1120 can insert a time stamp at the time when a specific input is input through the first HMD 100, and distinguish the start and end of each detailed section.

[0137] At this time, since the surgeon cannot use both hands due to the surgical operation, it is preferable that the specific input is a predetermined gesture or voice input, and the detailed section can be set by inserting tags after shooting the real-time video or by classifying it by AI analysis, etc. Also, it can be input by the voice of an assistant who participates in the actual surgical scene 1, instead of the surgeon 10.

[0138] When the UI providing unit 1110 receives a proposal from the second HMD 200 or the AI-based recommendation server 3000, it can provide a preview of the medical cooperation proposal together with the UI on the first HMD 100 as shown in FIG. 12(a).

[0139] FIG. 12 is a diagram illustrating a display screen of the first HMD that has received a medical cooperation proposal according to an embodiment of the present invention.

[0140] Referring to FIG. 12, (a) is a screen output to the surgeon's first HMD 100, which displays a real-time image 71 captured in real time by the first HMD 100, a preview proposal for medical cooperation, and a UI 72 for inquiring whether or not to accept the proposal.

[0141] Also, (b) can be a screen output to the third HMD 100-1 of assistant 1 when a proposal from the clinical assistant doctor is received, and (c) can be a screen output to the third HMD 100-1 of assistant 2 when a proposal from the clinical assistant doctor is received. At this time, the proposal received by assistants 1 and 2 can become information received by the surgeon's acceptance of the proposal. That is, the proposals for (b) and (c) can also be displayed on the surgeon's HMD 100, and assistants 1 and 2 can perform assistance based on the proposal accepted by the surgeon. That is, the surgeon can decide whether to accept the occurrence of an event.

[0142] That is, as shown in FIG. 12(b), a collaborating doctor in a remote location can set up the “virtual surgical tools” required for the surgery and display them on the HMD of the surgical participants 10, 11, and 12. By viewing this setting, the surgical participants can prepare the “actual surgical tools” in advance.

[0143] In addition, the cooperating doctor can map virtual surgical tools to a specific affected area in the real-time video and suggest surgical tools to be used by the surgeon, and can also suggest a time timer or treatment sequence that requires treatment using the surgical tools. In this case, the suggested surgical tools, treatment time timer, treatment sequence, etc. can be provided to the real-time video via the HMD as AR data.

[0144] When the proposal is accepted by the first HMD 100, authentication in response to receiving the proposal can be performed, and then the real-time video can be tagged with AR data (text, video, image, audio, etc.) corresponding to the proposal and displayed on the first HMD 100 and the second HMD 200.

[0145] Here, the authentication may be performed simultaneously by the second HMD 200 that provided the proposal and the first HMD that received the proposal. Referring to Fig. 13, the responsibility setting unit 1130 may receive and pre-store the signatures from the first HMD 100 and the second HMD 200, and may display the signature authentication screen on the first HMD 100 and the second HMD 200 so that the signature authentication screen can be shared according to the proposal and the proposal acceptance.

[0146] As shown in FIG. 13(a), when a proposal from the medical assistant doctor 20 is received from the second HMD 200, a medical assistant doctor avatar that automatically executes the default medical assistant doctor's signature for the proposal responsibility along with the proposal can be provided to the first HMD 100 and the second HMD 200.

[0147] Also, as shown in FIG. 13(b), a surgeon avatar that automatically executes a default surgeon's signature for the proposal acceptance responsibility when the first HMD 100 accepts the proposal can be provided to the first HMD and the second HMD.

[0148] In addition, when the surgeon 10 and the medical assistant 20 set conditions for automatic signature, the corresponding avatar can perform automatic signature. Specifically, the responsibility setting unit 1130 collects the results of a questionnaire survey previously given to the medical assistant 20 and the surgeon 10 or data on the surgical style of the surgical video recommended by the medical assistant / surgeon, and provides the avatar attributes (automatic signature conditions) corresponding to the medical assistant 20 and the surgeon 10 to the respective terminals (HMD or user terminal, etc.) of the medical assistant and the surgeon, and can specify the avatar attributes.

[0149] For example, the medical assistant doctor can recommend other pre-recorded surgical videos to the surgeon and participants (assistant 1 and assistant 2) before or during the surgery. The remote medical assistance support server 1000 can statistically extract and grasp the preferred surgical style of the medical assistant doctor by using the metadata of such recommended surgical videos.

[0150] This allows the surgeon to accept suggestions and agree to responsibility without having to respond to each suggestion that arises during real-time surgery.

[0151] In this case, if the proposal is a proposal from a medical assistant physician received from the second HMD 200, the tag can record the responsibility of the surgeon 10 wearing the first HMD 100 for accepting the proposal and the responsibility of the medical assistant physician 20 wearing the second HMD 200 for making the proposal as responsibility information.

[0152] In this case, the suggestion may be provided not only to the first HMD 100 but also to the corresponding third HMD 300 based on the role of Auxiliary 1 or Auxiliary 2 according to the characteristics of the suggestion.

[0153] In addition, the second HMD 200 can directly set suggestions and transmit them to the first HMD 100, or can receive suggestions of suggested information to be suggested to the first HMD 100 from the AI-based recommendation server 3000 and provide the same to the first HMD 100.

[0154] In addition, the first HMD 100 can also receive direct recommendations (automatic suggestions) from the AI-based recommendation server 3000.

[0155] In this case, suggestions may be made based on tag section information of the tag section, i.e., procedures, surgeries, and treatments corresponding to detailed sections. When similar / identical recommended tags occur, the AI-based recommendation server 3000 may consider the importance of the default tag, the difficulty of the procedure / surgery, etc., and suggest optimal tag information according to the default priority.

[0156] (First Example) Proposal - The second HMD makes a direct proposal to the first HMD DB2000 stores surgical images tagged with at least one tag, and can store the images by classification criteria such as by hospital / doctor, by patient condition, by procedure and surgery category.

[0157] 14, the second HMD 200 can selectively output at least one of a first screen (not shown) for outputting a real-time image captured by the first HMD 100 and a second screen (FIG. 14) for displaying at least one suggestion that can be proposed during real-time surgery in a preview window. In this case, the first screen and the second screen can be provided by the medical cooperation support unit 1100.

[0158] The second HMD 200 can call up at least one of the tags tagged to the surgical video stored in the DB 2000 as a tag of interest in a preview window and can set a display time and a display interval of the tag of interest. The medical cooperation support unit 1100 can output the tag of interest in the preview window of the second screen according to the setting of the second HMD 200.

[0159] The second HMD 200 may selectively or automatically suggest interest tags according to the settings of the first HMD 100 .

[0160] (Second Example) Proposal - The second HMD proposes AI-based recommended proposals to the first HMD The AI-based recommendation server 3000 can extract tags that are tagged to similar surgeries that are similar to the real-time surgery from the previous surgery video of the collaborating doctor and recommend them in the preview window.

[0161] Here, the previous surgery video may be a video of a surgery in which the cooperating physician participated as a surgeon or a cooperating physician, or a video of a surgery played by the cooperating physician.

[0162] The collaborating physician can selectively or automatically suggest recommended preview window tags to the first HMD 100 according to settings.

[0163] (Third Example) Proposal - Automatically propose AI-based recommendations to the first HMD as suggestions from a medical cooperating doctor If tags tagged during a similar surgery similar to the real-time surgery among previous surgery videos in which a medical cooperating physician participated from DB2000 are not proposed during the current real-time surgery, the AI-based recommendation unit 3000 can automatically suggest to the first HMD the tags tagged during the similar surgery as a suggestion from the medical cooperating physician.

[0164] In this case, the AI-based recommendation server 3000 can determine the tagging points and tagging intervals of the tags to be proposed in the real-time video based on the tagging points and tagging intervals of the tags tagged during a similar surgery, and automatically suggest them to the first HMD.

[0165] (Example 4) Proposal - AI-based recommendation proposal that supports automatic setting of responsibility consent The first HMD 100 and the second HMD 200 can automatically set liability agreements for specific procedures or surgeries that are predefined by surgical category and patient condition.

[0166] The responsibility setting unit 1130 may request a recommendation from the AI-based recommendation server 3000 for a specific procedure or surgery for which an HMD-specific responsibility agreement has been automatically set.

[0167] In addition, when a proposal for a specific procedure or surgery predefined for each HMD occurs, the responsibility setting unit 1130 can perform authentication by providing an avatar to the corresponding HMD that automatically executes a default signature of the corresponding HMD for which the responsibility consent is automatically set.

[0168] When a detailed section included in a similar surgery similar to the real-time surgery among the surgical videos pre-stored in DB2000 corresponds to a pre-defined specific procedure or surgery for which responsibility consent has been set by the second HMD2000, the AI-based recommendation server 3000 can automatically suggest tags included in the detailed section to the first HMD100.

[0169] In this case, the tagging unit 1120 can automatically tag the real-time video when receiving a proposal for a predefined specific procedure or surgery for which a liability agreement is automatically set.

[0170] In addition, when a proposal for a specific procedure or surgery predefined for each HMD occurs, the responsibility setting unit 1130 can provide the first HMD and the second HMD with an avatar that automatically performs a default signature of the corresponding HMD for which the responsibility consent is automatically set.

[0171] (Fifth Example) Proposal - Automatic suggestion of required tags for each HMD during surgery The AI-based recommendation server 3000 can extract tags that must be played on each HMD based on the role information in the current real-time surgery (assistant physician, assistant 1, assistant 2) from the information on each HMD wearer for the real-time surgery, and automatically suggest them to the corresponding HMD.

[0172] Meanwhile, in the above, the responsibility for the detailed sections is tagged depending on whether the surgeon accepts the proposal of the medical assistant doctor during the real-time surgery. In contrast, when the surgeon and the medical assistant doctor discuss the surgical responsibility section before the start of the surgery, the medical assistant support unit 1100 can divide the entire surgical section into known procedures and provide them as detailed sections to be output to each HMD in real-time, and can match and tag the previously discussed responsible entity to the detailed section corresponding to the known procedure.

[0173] At this point, it is necessary to determine whether the procedure currently being performed by the surgeon corresponds to a known procedure. This can be determined using a model generated by machine learning of data contained in existing surgical footage.

[0174] As an example, when a previously agreed overall surgery is performed in the order of skin incision, vascular clamping, bilateral vascular ligation, etc., the remote medical cooperation support server 1000 can divide each procedure / surgery into detailed sections and provide them in real-time video. When it is determined that the surgeon has performed the procedures specified in the "order previously specified by the medical cooperation physician," the remote medical cooperation support server 1000 can provide AR data in which the avatar of the predetermined medical cooperation physician signs the section of responsibility in the real-time video.

[0175] In this manner, the remote support and medical cooperation system of the present invention can extract tags that may become issues in real-time surgery and tag real-time video by using the methods of the various embodiments as described above. At this time, each embodiment can be used individually or in combination.

[0176] In particular, when tagging based on the surgical procedure order or a specific procedure order proposed by a medical assistant, the responsible party information based on whether the surgeon accepts it or not can be recorded as tag information, which can be easily used to determine who is responsible.

[0177] Meanwhile, when tagging the real-time video, the tagging unit 1120 may mask and process any part requested by the patient to be censored for the purpose of protecting personal information, or may replace it with another part of the real-time video from the same time period.

[0178] The remote medical cooperation support server 1000 can store surgical videos tagged with multiple tags during real-time surgery in the DB 2000. At this time, the stored surgical videos can be provided to the surgeon, medical assistants, and educators / trainees so that they can be used according to their purposes. The remote medical cooperation support system of the present invention can also be embodied as a surgical video intermediation system that mediates the use of surgical videos through the video intermediation server 4000 and distributes profits according to the use.

[0179] Mediation system for trading and using surgical images including tags Fig. 15 is a block diagram for explaining the configuration of the video intermediation server in Fig. 1. Fig. 16 is a diagram for explaining the block chain structure of the remote medical consultation cooperation support system.

[0180] Referring to FIG. 15, the video brokerage server 4000 may include an NFT management unit 4100, a video trading unit 4200, a search unit 4300, and a UI providing unit 4400.

[0181] Also, referring to FIG. 16, the remote medical cooperation support system can form a blockchain network with the hospital servers (a-n) of member hospitals, collect surgical videos generated during real-time surgery through each hospital server (a-n), and provide the surgical videos for utilization.

[0182] When surgical videos are collected and stored in DB2000 from each hospital server (a to n), the NFT management unit 4100 can assign a unique ID to the collected surgical videos and issue NFT (Non-Fungible Token)-based ownership to the corresponding hospital company.

[0183] The video intermediation server 4000 can settle and manage the usage fee for the sale or use using cryptocurrency when the sale or use occurs from a third party's user terminal for the surgical video or the tag included in the surgical video.

[0184] In addition, when a specific hospital server requests installment sales of the surgery video that the video brokerage server 4000 owns, the video brokerage server 4000 can assign a unique ID to each tag section included in the surgery video and issue NFT ownership rights for each tag section. This allows installment sales of surgery or treatment videos for each tag section to be provided.

[0185] When the prescribed fee is settled during the transaction of the surgical video by the hospital server, the NFT management unit 4100 can transfer and manage the ownership of the NFT to the selling hospital company.

[0186] In addition, the NFT management unit 4100 compares the hospital company to which the user terminal that requested the use of the surgical video belongs with the owner of the surgical video requested by the user terminal, and if they are not the same, it determines that the user terminal is a third-party user without the ownership, and can request the video trading unit 4200 to calculate the usage fee.

[0187] The image transaction unit 4200 can calculate the usage fee based on the number of tags included in the surgical image requested for use from the user terminal and the predetermined importance of each tag.

[0188] Here, the tag may include at least one of tag information, responsible party information, tag section within the corresponding surgical video, tag section information, person who input information for the tag section, NFT-based ownership information, buying and selling and usage history, and token settlement information.

[0189] Here, the tag section in the surgical video may be a section divided according to a minor procedure or minor surgery included in the surgical video, and the tag section information may be the minor procedure or minor surgery information, and the person who inputs the tag section information may be the surgeon who performed the minor procedure or minor surgery or a medical assistant.

[0190] When the information contained in the tag changes or new information is generated due to the use of the user's device or trading by the hospital server, the blockchain with the updated information can be added to the distributed ledger and updated through the agreement of the nodes in the blockchain network.

[0191] In addition, when tag section information is used in a specific user terminal, the video transaction unit 4200 can provide cryptocurrency to a person who inputs the information of the corresponding tag section based on the usage fee collected from the corresponding user terminal.

[0192] The search unit 4300 provides a search service for video playback based on tag information and tag section information of tags stored in the DB 2000, and allows a user to use a required surgical video and tag for each user terminal.

[0193] Here, the user terminal may be an HMD 100, 100-1, 100-2, 200, 500 worn by at least one of the surgeon performing the real-time surgery, the cooperating physician providing medical assistance to the real-time surgery, and at least one assistant and trainee participating in the real-time surgery.

[0194] The AI-based recommendation server 3000 can recommend and provide surgery images and tags for similar surgeries when receiving participating surgery information from a user terminal.

[0195] In addition, when the user's role information for the participating surgery is input from the user terminal, the AI-based recommendation server 3000 can recommend tags included in similar surgeries as required playback tags based on the user's role information.

[0196] The UI providing unit 4400 may provide a list to the user terminal for selecting to use or remove multiple tags included in similar procedures recommended to the user terminal by the AI-based recommendation server 3000 according to the purpose of use.

[0197] The UI providing unit 4400 may predetermine required playback tag information for each user terminal based on role information of the user terminal, and may provide tags corresponding to the predetermine required playback tag information by excluding them from the list.

[0198] At this time, the image transaction unit 4200 can calculate and settle the usage fee reflecting the number of tags used or removed by the corresponding user terminal.

[0199] In this way, the intermediation system of the present invention can mediate transparent and reliable transactions by granting NFT-based ownership to surgical videos including tags and mediating their use and transactions, and can distribute transaction revenues.

[0200] In addition, AI-based recommendations can provide optimal surgical videos and tags according to the playback purpose of each user (surgeon, clinical associate, assistant 1, assistant 2, trainee), thereby increasing the utilization of surgical videos.

[0201] Exemplary embodiments of the present invention may be embodied in the form of program instructions that can be executed by various computer means and stored on a computer-readable medium, which may include, alone or in combination with the program instructions, data files, data structures, and the like.

[0202] The program instructions recorded on the medium may be those specially designed and constructed for the present invention, or may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD_ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially constructed to store and execute program instructions, such as ROMs, RAMs, flash memories, etc. Examples of program instructions include not only machine code, such as that generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc.

[0203] The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

[0204] Although the embodiments of the present invention have been described in more detail above with reference to the accompanying drawings, the present invention is not necessarily limited to these embodiments and can be modified in various ways without departing from the technical spirit of the present invention.

[0205] Therefore, the embodiments disclosed herein are intended to illustrate, but not to limit, the technical idea of ​​the present invention, and the scope of the technical idea of ​​the present invention is not limited by such embodiments.

[0206] Therefore, it should be understood that the above-described embodiments are illustrative in all respects and are not limiting. The scope of protection of the present invention should be interpreted according to the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

Claims

1. A first HMD is worn by a surgeon at an actual surgical site and includes a camera for capturing the surgeon's gaze area and a surrounding area, and a display unit for outputting real-time images captured by the camera; A second HMD is worn by a medical assistant located remotely from the surgeon, receives and outputs real-time images captured by the first HMD, and receives suggestions from the medical assistant regarding the order of surgery or the order of specific procedures included in the surgery; and a remote medical cooperation support server that, when receiving the input proposal from the second HMD, provides a UI for the first HMD to select whether or not to accept the proposal, generates a tag according to the acceptance or non-acceptance of the surgeon, and tags the real-time video; The tag includes information about a responsible party for the proposed real-time surgery.

2. The remote medical cooperation support system of claim 1 , wherein the tag further includes tagged location information within the real-time video, and the information included in the tag is stored in a blockchain-based distributed ledger.

3. The remote medical consultation cooperation support server includes: When a predetermined specific input is received from the first HMD, a timestamp is generated and tagged to the real-time image, and the real-time surgery is divided into a plurality of subsections defined by the surgeon based on the tagged timestamps; The remote medical cooperation support system according to claim 2 , wherein the tagged position information includes a start timestamp and an end timestamp of a detailed section including a tagged point of the corresponding tag.

4. the predetermined specific input is at least one of a gesture, a voice command, and a control signal command; The remote medical cooperation support system according to claim 3 , wherein the detailed section corresponds to a detailed procedure and a surgical sequence of the real-time surgery defined by the surgeon.

5. The remote medical consultation cooperation support server includes: providing a preview of the proposal together with the UI on the first HMD; The remote medical cooperation support system according to claim 1 , wherein, when the suggestion is accepted via the first HMD, AR data corresponding to the suggestion is inserted into the real-time image and displayed on the first HMD and the second HMD.

6. The remote medical consultation cooperation support server includes: When the suggestion by the medical assistant doctor is received from the second HMD, a medical assistant doctor avatar that automatically executes the suggestion together with a signature of a default medical assistant doctor for the suggestion responsibility is provided to the first HMD and the second HMD; The remote medical cooperation support system according to claim 1 , further comprising: providing, to the first HMD and the second HMD, a surgeon avatar that automatically executes a default surgeon's signature for the proposal acceptance responsibility when the proposal is accepted by the first HMD.

7. The remote medical consultation cooperation support server includes: A responsibility agreement is automatically set for a specific procedure or surgery predefined by surgery category and patient condition from each of the first HMD and the second HMD; The remote medical cooperation support system of claim 1, wherein when a proposal for the predefined specific procedure or surgery occurs, an avatar that automatically executes a default signature of the corresponding HMD in which the liability agreement is automatically set is provided to the first HMD and the second HMD.

8. A database for storing surgical images tagged with at least one tag; and 8. The remote medical cooperation support system of claim 7, further comprising an AI-based recommendation server that automatically suggests tags included in a detailed section to the first HMD when the detailed section included in a similar surgery similar to the real-time surgery among the surgical videos pre-stored in the DB corresponds to the pre-defined specific procedure or surgery for which responsibility consent has been set by the second HMD.

9. The remote medical consultation cooperation support server includes: The remote medical cooperation support system of claim 7, wherein when a proposal for the predefined specific procedure or surgery for which the responsibility agreement is automatically set is received from the first HMD, the real-time video is automatically tagged.

10. The system further includes an AI-based recommendation server that automatically recommends at least one tag included in a detailed section of a similar surgery similar to the real-time surgery among pre-stored surgical videos to the first HMD when the detailed section corresponds to the pre-defined specific procedure or surgery for which a responsibility agreement is set by the first HMD, The remote medical cooperation support system according to claim 7 , wherein the remote medical cooperation support server tags the real-time video with at least one tag automatically suggested by the AI-based recommendation server.

11. The remote medical cooperation support system of claim 1, wherein the second HMD selectively outputs at least one of a first screen that outputs real-time video captured by the first HMD and a second screen that displays at least one suggestion that can be proposed during real-time surgery in a preview window.

12. The system further includes a database for storing the surgical images tagged with at least one tag by doctor, patient condition, procedure, and surgical category; The second HMD sets at least one of the tags tagged to the surgical video stored in the DB as a tag of interest, and a display time and a display interval of the tag of interest are set; The remote medical consultation cooperation support system according to claim 11 , wherein the remote medical consultation cooperation support server outputs the interest tag to a preview window of the second screen in accordance with a setting of the second HMD.

13. The remote medical cooperation support system of claim 11, further comprising an AI-based recommendation server that recommends tags tagged with similar surgeries similar to the real-time surgery from previous surgery videos of the medical cooperation doctor in the preview window.

14. 2. The remote medical cooperation support system of claim 1, further comprising an AI-based recommendation server that automatically provides the first HMD with tags tagged during a similar surgery similar to the real-time surgery among previous surgical videos performed by the medical cooperation doctor as a suggestion from the medical cooperation doctor if the tags tagged during the similar surgery are not suggested during the real-time surgery.

15. The AI-based recommendation server is The remote medical cooperation support system of claim 14, wherein the tagging points and tagging intervals of the tags proposed in the real-time video are determined based on the tagging points and tagging intervals of the tags tagged during the similar surgery, and automatically proposed to the first HMD.

16. a third HMD worn by each of a plurality of assistants during real-time surgery at the actual surgical site; A database for storing surgical images tagged with at least one tag; and The remote medical cooperation support system according to claim 1 , further comprising: extracting tags that require mandatory reproduction in each HMD based on role information of a wearer of each HMD for the real-time surgery; and automatically suggesting the tags to the corresponding HMD.

Images