Mixed reality auxiliary system and method for eye socket operation visual field sharing

Through mixed reality technology, the surgical field of surgery between the surgeon and the assistant is shared in ophthalmic surgery, which solves the problem that the assistant has difficulty in obtaining a good surgical field of surgery, improves surgical efficiency and teaching value, and achieves a more efficient ophthalmic operation and learning experience.

CN118975861APending Publication Date: 2024-11-19XVISIO TECH CORP +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411065963.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-05
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

During ophthalmic surgery, assistants find it difficult to obtain a good surgical field, resulting in low surgical efficiency, high risk and poor learning experience, and they cannot effectively cooperate with the surgeon and observe and learn.

Method used

Using mixed reality technology, the main MR end worn by the attending doctor and the sub-MR end worn by the assistant is constructed and the video streaming data is synchronized to realize real-time sharing of the surgeon's vision. The assistant can obtain a clear surgical field under the same coordinate.

Benefits of technology

It improves surgical efficiency, reduces surgical risks, improves the assistant's operating experience and observation and learning experience, promotes the development of ophthalmic surgery teaching level, and fills the gap in surgical field assistive equipment.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118975861A_ABST
    Figure CN118975861A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of mixed reality display, in particular to a mixed reality auxiliary system and method for eye socket operation visual field sharing, the system comprises a main MR terminal worn by an attending doctor, N sub MR terminals worn by assistants and a server, and N is greater than or equal to 1; the main MR end or the sub MR ends construct a space map, and all the MR ends are placed under the same coordinate; the server is used for receiving the 6Dof information of the N sub-MR ends, sending the 6Dof information to the main MR end and synchronizing video stream data formed by the main MR end to the N sub-MR ends; through the intelligent device based on the MR technology, operation view sharing of a surgeon and an assistant is achieved, so that the assistant is promoted to cooperate with the surgeon to work, the operation efficiency is improved, the operation risk is reduced, and the observation and learning experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of mixed reality display technology, and in particular to a mixed reality auxiliary system and method for sharing orbital surgery field of view. Background Art

[0002] Orbital diseases are generally divided into four categories: inflammatory, neoplastic, traumatic, and congenital. The main clinical manifestations are visual impairment and abnormal eye tissue structure, which can be life-threatening in severe cases. Most patients with orbital diseases require surgical treatment. However, ophthalmic surgery is limited by the narrow operating space of the orbit, the deeper surgical site, and the important structural tissues, which are very difficult and risky. Current orbital surgery can only meet the surgical field requirements of the surgeon as much as possible, but cannot take into account the assistant's satisfactory surgical field. This undoubtedly increases the difficulty for the assistant to cooperate with the surgeon, limits the efficiency of the operation, and increases the risk of the operation. At the same time, the lack of surgical field also hinders the observation and learning of assistants and other learners. Summary of the invention

[0003] (I) Purpose of the invention

[0004] In order to solve the technical problems existing in the background technology, the present invention proposes a mixed reality auxiliary system and method for sharing orbital surgical field of view, which can convert the surgical field image from the perspective of the surgeon into the surgical field image from the corresponding perspective of the assistant's own position in real time according to the relative spatial position of the wearer, which is helpful for the assistant to carry out work and observe and learn.

[0005] (II) Technical solution

[0006] In order to solve the above problems, the first aspect of the present invention provides a mixed reality auxiliary system for orbital surgery field sharing, comprising:

[0007] A main MR terminal worn by the attending physician, N sub-MR terminals worn by assistants, and a server, N ≥ 1;

[0008] The main MR terminal or the sub-MR terminal constructs a spatial map to place all MR terminals under the same coordinates;

[0009] The server receives the 6Dof information from the N sub-MRs and sends it to the master MR, and

[0010] The video stream data generated by the main MR end is synchronized to N sub-MR ends.

[0011] As a technical solution of the present invention, the MR end includes:

[0012] Vs l am module, used to provide tracking and positioning services to the target MR end;

[0013] The map construction module is used to construct virtual space data according to the target MR terminal tracking and positioning information obtained by the Vs l am module.

[0014] As a technical solution of the present invention, the main MR end includes:

[0015] The camera module is used to take pictures of the surgical site of the patient to obtain the image data of the corresponding surgical site during the operation by the attending physician to form a three-dimensional surgical image;

[0016] As a technical solution of the present invention, the camera module includes but is not limited to an RGB camera.

[0017] As a technical solution of the present invention, the MR end further includes:

[0018] The three-dimensional rendering module is used to mark the three-dimensional image in the virtual space constructed by the map construction module, and to fuse and render the three-dimensional surgical image and the virtual space to generate video stream data.

[0019] As a technical solution of the present invention, the RGB camera coordinates are set to capture MR video stream data.

[0020] As a technical solution of the present invention, the map construction module constructs virtual space information including the following steps:

[0021] S100, the camera scans the environment structure to generate an image with environment texture data;

[0022] S200, gridding the image, and extracting some high-response FAST corner points in each grid;

[0023] S300, selecting FAST corner features and performing texture analysis on the image to obtain key points and descriptors;

[0024] S400, constructing virtual space map information based on the acquired feature points, key points and descriptors and the point cloud data scanned by the camera and using the MR end as the coordinate origin in the initial state according to an estimation algorithm.

[0025] As a technical solution of the present invention, the virtual space data and video stream data are uploaded and stored in a server for subsequent MR users to download and join.

[0026] As a technical solution of the present invention, the MR end can be fixed to the user's head, including but not limited to a headband structure and a glasses structure.

[0027] A second aspect of the present invention provides a mixed reality method for sharing orbital surgical field of view, comprising the steps of:

[0028] Step S1, the assistant scans the space map through the sub-MR terminal Vs l am;

[0029] Step S2, transmitting the space to the main MR terminal of the attending physician, so that the attending physician can enter the same space coordinates;

[0030] Step S3, the attending doctor obtains the assistant's 6Dof information;

[0031] Step S4, setting the camera coordinates;

[0032] Step S5, capturing the MR video stream, and sending the video stream to the assistant sub-MR terminal, so as to realize the sharing of the surgical field of view between the attending physician and the assistant.

[0033] The above technical solution of the present invention has the following beneficial technical effects:

[0034] The present invention uses intelligent devices based on MR technology to achieve the sharing of surgical fields between the surgeon and the assistant, thereby promoting the assistant to cooperate with the surgeon, improving surgical efficiency, reducing surgical risks, and improving the observation and learning experience. The present invention is different from previous surgical auxiliary equipment. It not only pays attention to the surgeon, but also focuses on the assistant's intraoperative work, strives to improve the work efficiency and mutual cooperation of each surgical participant, improves orbital surgery operations from multiple angles and with higher efficiency, and converts the perspective of relative spatial position to provide assistants with a more comfortable and reasonable visual perception, which conforms to operating habits, and can take into account the observation and learning experience of more users. It has high teaching significance and maximizes the value of each orbital surgery.

[0035] The present invention fills the gap of the lack of assistant surgical field auxiliary equipment in ophthalmic surgery, greatly improves the operating experience of assistants in ophthalmic surgery, and enriches the market of surgical auxiliary equipment for assistants in ophthalmic surgery in my country. The present invention provides new ideas for the research and development of ophthalmic surgical auxiliary equipment in the future, helps to promote the diversified development of surgical auxiliary equipment, and enables every surgical participant to enjoy the convenience brought by technological changes. It has great practical value in clinical applications, medical education and other scenarios, and has great social benefits for surgical treatment of orbital diseases and protecting the health of patients. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A system flow chart in one embodiment of the present invention;

[0037] Figure 2 This is a method flow in one embodiment of the present invention;

[0038] Figure 3 This is a diagram of the map construction steps in the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the present invention.

[0040] The present invention overcomes the problem that it is difficult for assistants to obtain a good surgical field of view during orbital surgery. A sufficiently clear surgical field of view is crucial for the cooperation of assistants during ophthalmic surgery, but there is currently a lack of relevant equipment to assist assistants in obtaining the surgical field of view, which limits the cooperation between assistants and surgeons to a certain extent, limits surgical efficiency, and increases surgical risks. Compared with traditional surgery, the surgery using the present invention is conducive to the accurate cooperation of assistants with surgeons under a sufficient, clear, and full surgical field of view, avoiding the limitations and high risks caused by deep surgical sites, narrow surgical spaces, and complex anatomical structures, promoting the development of surgical treatment of orbital diseases, and improving the surgical efficacy and prognosis of a large number of orbital patients.

[0041] The present invention provides learners with a clear and complete perspective for observation and learning, facilitates clinical surgical teaching, and contributes to the innovation of orbital surgery teaching methods and the development of teaching levels. In traditional orbital surgery, learners often find it difficult to obtain a good perspective for observation and learning, cannot follow the surgeon's operations, and find it difficult to understand the surgeon's instructions in the absence of a surgical field of view, which greatly limits the teaching value of traditional surgery. The present invention shares the field of view through MR glasses, providing learners with the surgeon's real-time field of view, facilitating observation or communication, promoting the development of ophthalmic surgery teaching levels, and providing new possibilities for the development of ophthalmic surgery teaching methods in the future.

[0042] refer to Figure 1-3 To solve the above problems, this application adopts the following technical solutions:

[0043] refer to Figure 1 The first aspect of the present invention provides a mixed reality auxiliary system for sharing orbital surgical field of view, which consists of a main MR terminal worn by an attending physician, N sub-MR terminals worn by assistants, and a server, N≥1; the main MR terminal or the sub-MR terminal constructs a spatial map to place all MR terminals under the same coordinates; the server is used to receive 6Dof information of the N sub-MR terminals and send it to the main MR terminal, and synchronize the video stream data formed by the main MR terminal to the N sub-MR terminals.

[0044] Specifically, the MR end includes: a Vs l am module, which is used to provide tracking and positioning services to the target MR end;

[0045] A map construction module is used to construct virtual space data according to the target MR end tracking and positioning information obtained by the Vs l am module. A three-dimensional rendering module is used to calibrate the three-dimensional image in the virtual space constructed by the map construction module, and to fuse and render the three-dimensional surgical image and the virtual space to generate video stream data.

[0046] In one embodiment, the main MR terminal includes: a camera module for photographing the surgical site of the patient to obtain the image data of the corresponding surgical site of the attending doctor during the operation to form a three-dimensional surgical image. The camera module includes but is not limited to an RGB camera. In this embodiment, an RGB camera is used, and the RGB camera coordinates are set to capture MR video stream data.

[0047] refer to Figure 3 The map construction module constructs virtual space information including the following steps: S100, scanning the environment structure by a camera to generate an image with environment texture data; S200, gridding the image and extracting some high-response FAST corner points in each grid; S300, selecting FAST corner point features and performing texture analysis on the image to obtain key points and descriptors; S400, constructing virtual space map information based on the acquired feature points, key points and descriptors and the point cloud data scanned by the camera according to the estimation algorithm with the MR end as the coordinate origin in the initial state.

[0048] Specifically, a feature point pair that meets the normal distribution characteristics may be selected, and an area is randomly selected in the vicinity of the feature point pair, and the area is divided into sub-areas, and the response of each sub-area is calculated to describe the sub-area, such as the Haar Wave L et response. The purpose is to obtain three-dimensional point cloud information data.

[0049] In some embodiments, the virtual space data and video stream data can also be uploaded and stored in the server for subsequent MR users to download and use, so that the present invention can not only be carried out in real time on site, but also provide pseudo-real-time teaching materials for subsequent teaching demonstrations.

[0050] It should be noted that the MR end described in the present invention can be fixed on the user's head, including but not limited to a headband structure and a glasses structure. In this embodiment, smart glasses based on MR technology are taken as an example.

[0051] The second aspect of the present invention proposes a mixed reality method for sharing orbital surgical field of view, comprising the following steps: step S1, the assistant scans the space map through the sub-MR terminal Vs l am; step S2, transmits the space to the main MR terminal of the attending physician, so that the attending physician can enter the same space coordinates; step S3, the attending physician obtains the assistant's 6Dof information; step S4, sets the camera coordinates; step S5, captures the MR video stream, and sends the video stream to the assistant's sub-MR terminal to achieve the sharing of the surgical field of view between the attending physician and the assistant.

[0052] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and the above embodiments and descriptions are only preferred examples of the present invention, and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements all fall within the scope of the present invention to be protected.

Claims

1. A mixed reality auxiliary system for orbital surgery field sharing, characterized in that: include: A main MR terminal worn by the attending physician, N sub-MR terminals worn by assistants, and a server, N ≥ 1; The main MR terminal or the sub-MR terminal constructs a spatial map to place all MR terminals under the same coordinates; The server receives the 6Dof information from the N sub-MRs and sends it to the master MR, and The video stream data generated by the main MR end is synchronized to N sub-MR ends.

2. A mixed reality auxiliary system for orbital surgery field sharing according to claim 1, characterized in that: The MR end includes: Vslam module, used to provide tracking and positioning services to the target MR end; The map construction module is used to construct virtual space data according to the target MR end tracking and positioning information obtained by the Vslam module.

3. A mixed reality auxiliary system for orbital surgery field sharing according to claim 2, characterized in that: The main MR terminal includes: The camera module is used to take pictures of the surgical site of the patient to obtain the image data of the corresponding surgical site during the operation by the attending physician to form a three-dimensional surgical image.

4. A mixed reality auxiliary system for orbital surgery field sharing according to claim 3, characterized in that: The camera module includes but is not limited to an RGB camera.

5. A mixed reality auxiliary system for orbital surgery field sharing according to claim 4, characterized in that: The MR end also includes: The three-dimensional rendering module is used to mark the three-dimensional image in the virtual space constructed by the map construction module, and to fuse and render the three-dimensional surgical image and the virtual space to generate video stream data.

6. A mixed reality auxiliary system for orbital surgery field sharing according to claim 4, characterized in that: Set the RGB camera coordinates to capture MR video stream data.

7. A mixed reality auxiliary system for orbital surgery field sharing according to claim 2, characterized in that: The map construction module constructs virtual space information including the following steps: S100, the camera scans the environment structure to generate an image with environment texture data; S200, gridding the image, and extracting some high-response FAST corner points in each grid; S300, selecting FAST corner features and performing texture analysis on the image to obtain key points and descriptors; S400, constructing virtual space map information based on the acquired feature points, key points and descriptors and the point cloud data scanned by the camera and using the MR end as the coordinate origin in the initial state according to an estimation algorithm.

8. The mixed reality auxiliary system for orbital surgery field sharing according to claim 5, characterized in that: The virtual space data and video stream data are uploaded and stored in the server for subsequent MR users to download and use.

9. A mixed reality auxiliary system for orbital surgery field sharing according to claim 1, characterized in that: The MR end can be fixed to the user's head, including but not limited to a headband structure and a glasses structure.

10. A mixed reality method for orbital surgery field of view sharing, characterized in that: Includes steps: Step S1, the assistant scans the space map through the sub-MR terminal Vslam; Step S2, transmitting the space to the main MR terminal of the attending physician, so that the attending physician can enter the same space coordinates; Step S3, the attending doctor obtains the assistant's 6Dof information; Step S4, setting the camera coordinates; Step S5, capturing the MR video stream, and sending the video stream to the assistant sub-MR terminal, so as to realize the sharing of the surgical field of view between the attending physician and the assistant.