AR-based intelligent vision use method applied to surgical operation

Abstract

The invention relates to the technical field of surgical operations, in particular to an AR-based intelligent vision use method applied to surgical operations. The method comprises the following steps: S1, acquiring an anatomical structure image of a patient in real time through AR glasses equipment, and performing multi-modal fusion registration on the anatomical structure image and preoperative CT, MRI and 3D ultrasonic image data to generate a dynamic three-dimensional surgical navigation model; according to the method, multi-mode image fusion is combined with dynamic three-dimensional navigation, submillimeter intraoperative positioning is achieved, the misoperation risk is remarkably reduced, the intraoperative adjustment frequency is reduced through eye-controlled amplification and real-time early warning, and the operation time is shortened; meanwhile, the virtual screen and microscope linkage technology relieves the burden of the cervical vertebra and reduces the neck load of medical staff, and compared with a traditional teaching mode through holographic labeling and operation tracks, the training period of doctors can be obviously shortened, idle equipment can be automatically recognized, and the surgical accuracy, efficiency and medical resource popularity can be comprehensively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surgical operations, and particularly to a method for using AR intelligent vision applied to surgical operations. Background Art

[0002] In the medical field, surgical operations have always been highly specialized and refined procedures. Traditional surgical methods rely on doctors' experience and skills. Although significant progress has been made through years of development and accumulation, there are still certain limitations and challenges.

[0003] Currently, the complexity and risk during the surgical process cannot be ignored. Especially when facing complex anatomical structures and delicate operation requirements, even experienced doctors may encounter difficulties. In addition, uncertain factors during the surgical process, such as individual differences among patients and changes in the lesion site, may have an adverse impact on the surgical outcome.

[0004] Moreover, with the continuous development of medical technology, the requirements of patients and doctors for surgical outcomes and safety are also increasing. Traditional surgical methods often struggle to fully meet these needs, especially in surgeries that require high precision and real-time guidance. Therefore, it is particularly important to develop a new technology that can assist doctors in performing surgeries and improve surgical precision and safety.

[0005] In recent years, augmented reality (AR) technology has been widely applied in the medical field due to its unique advantages. AR technology combines virtual images with the real environment, providing doctors with a new way of surgical guidance. However, the existing AR technology still has some limitations in surgical applications: there are problems such as unclear display and inaccurate positioning in the existing technology, which make it difficult for doctors to obtain accurate information and guidance during the surgical process. At the same time, there is a phenomenon of blocking the display of the real surgical field, making it easy for medical staff to miss important visual cues in the real environment when observing virtual information, and unable to fully focus on the surgery and the patient's condition. To solve the above problems, we propose a method for using AR intelligent vision applied to surgical operations. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for using AR intelligent vision applied to surgical operations to solve the problems raised in the above background art.

[0007] To solve the above technical problems, the present invention adopts the following technical solutions:

[0008] A method for using AR intelligent vision applied to surgical operations includes the following steps:

[0009] S1: Obtain the anatomical structure images of the patient in real time through the AR glasses device, and perform multi-modal fusion registration with the preoperative CT, MRI and 3D ultrasound image data to generate a dynamic three-dimensional surgical navigation model;

[0010] S2: Superimpose the virtual surgical navigation information and the real surgical field on the perspective module of the AR glasses, and support the doctor to display physiological parameters, imaging materials and surgical path layout in split screen on the AR projection interface through gesture sliding or voice commands;

[0011] S3: Based on eye movement tracking to capture the doctor's line of sight focus area, automatically magnify the target anatomical structure in combination with deep learning algorithms, and the dynamic adjustment range of the magnification factor is 5-77 times, with an error ≤ 1.8 mm;

[0012] S4: Integrate the teaching assistance module, and display patient image annotations, anesthesia depth indicators and surgical step annotations in real time on the AR projection interface, and support hierarchical permission management to avoid information interference with the operation of the surgeon.

[0013] Preferably, the multi-modal fusion registration uses SLAM to compensate for the intraoperative head displacement in real time through the IMU sensor, and realizes the sub-millimeter-level spatial registration of the virtual image and the real surgical field.

[0014] Preferably, the eye movement tracking uses an infrared pupil capture system, dynamically adjusts the boundary of the magnification area in combination with the position data of the surgical instrument, and sets an anti-mis-touch mechanism to avoid unintentional operations.

[0015] Preferably, the teaching assistance module includes a case database and a surgical video playback function, supports the display of operation key points in the form of holographic annotation, and is used for the skill training of young doctors.

[0016] Preferably, it further includes the following steps:

[0017] S21: In the surgical scene under the microscope, the AR glasses are linked and calibrated with the microscope optical system, allowing the doctor to freely adjust the head posture on the premise of maintaining a stable surgical field, increasing the cervical spine activity range by 60%, and reducing the neck load during a single operation by 40%;

[0018] S22: Identify the usage status of the operating room equipment through computer vision, automatically turn off the idle physical display screen, and optimize the wind speed distribution model of the laminar flow purification system, so that the space utilization rate of the operating room is increased by 50% and the purification energy consumption is reduced by 30%;

[0019] S23: Adopt a split-type AR glasses design, equipped with diopter adjustment lenses that can be quickly disinfected and an optical waveguide module with a light transmittance ≥ 85%, and support continuous use for 4 hours without visual fatigue.

[0020] Preferably, the three-dimensional surgical navigation model realizes real-time rendering of holographic images through edge computing nodes, supports multi-terminal collaborative operations including surgical robots and anesthesia monitors, and controls the latency within 50 ms.

[0021] Preferably, the teaching assistance module is implemented through the following steps:

[0022] S31: Integrate preoperative images, intraoperative endoscopic videos, vital sign data, and postoperative pathology reports to construct a standardized case database;

[0023] Use deep learning algorithms to automatically segment anatomical structures and generate an interactive 3D annotation model;

[0024] Extract voice annotations in the surgical video through natural language processing technology and synchronize them with the time axis of the operation screen;

[0025] S32: Based on eye-tracking data, highlight teaching key points in the surgical field in real time on the AR projection interface;

[0026] Overlay the difference analysis between the preoperative planning path and the intraoperative actual operation trajectory to assist young doctors in understanding operation deviations; embed expert operation demonstration videos.

[0027] Preferably, it further includes: The surgeon dynamically controls the display priority of teaching content through gesture swiping or voice commands;

[0028] The AR glasses on the trainee side restrict annotation interference during key surgical stages and only allow the teaching playback function to be called in the paused state;

[0029] The surgical video is automatically encrypted and stored in the edge server, and blockchain technology is used to record data access logs to ensure compliance with HIPAA / GDPR privacy regulations.

[0030] It can be seen without doubt that through the above technical solutions of this application, the technical problems to be solved by this application can surely be solved.

[0031] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0032] 1. The present invention accurately integrates virtual images with the real surgical field through AR technology, providing intuitive and clear surgical guidance for doctors. This not only greatly improves the accuracy of surgery but also effectively reduces the surgical risks caused by improper operation or positioning errors, significantly increasing the success rate of surgery.

[0033] 2. The present invention combines multi-modal image fusion with dynamic three-dimensional navigation to achieve sub-millimeter intraoperative positioning, significantly reducing the risk of misoperation. Moreover, eye-controlled magnification and real-time warning reduce the frequency of intraoperative adjustments, shortening the operation time. At the same time, the linkage technology between the virtual screen and the microscope reduces the cervical spine burden, increases the head movement range of medical staff, and reduces the neck load of medical staff.

[0034] 3. By comparing the holographic annotation and operation trajectory with the traditional teaching method, the present invention can significantly shorten the doctor training cycle, automatically identify idle equipment, improve space utilization rate, reduce equipment maintenance costs, and comprehensively improve the surgical accuracy, efficiency and universality of medical resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0036] Figure 1 It is a schematic diagram of the general steps of the present invention;

[0037] Figure 2 It is a schematic diagram of the sub-steps of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0038] In order to make the objectives, technical solutions and advantages of the present invention clearer, the following further details the present invention in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0039] Embodiment 1

[0040] Refer to Figure 1-2 As shown, a method for using AR intelligent vision applied to surgical operations includes the following steps:

[0041] S1: Real-time obtain the anatomical structure image of the patient through the AR glasses device, and perform multi-modal fusion registration with the preoperative CT, MRI and 3D ultrasound image data to generate a dynamic three-dimensional surgical navigation model;

[0042] S2: Superimpose and display the virtual surgical navigation information and the real surgical field on the perspective module of the AR glasses, and support the doctor to display physiological parameters, image data and surgical path layout in a split screen on the AR projection interface through gesture sliding or voice commands, replacing the traditional decentralized observation mode of multiple physical display screens;

[0043] S3: Based on eye tracking to capture the focus area of the doctor's line of sight, combined with deep learning algorithms to automatically magnify the target anatomical structure, the magnification dynamic adjustment range is 5 - 77 times, and the error ≤ 1.8 mm;

[0044] S4: Integrate a teaching assistance module to display patient image annotations, anesthesia depth indicators, and surgical step notes in real time on the AR projection interface, support permission - level management to avoid information interference with the surgeon's operation. By integrating CT, MRI, and 3D ultrasound image data, generate a dynamic three - dimensional navigation model, improving the intraoperative anatomical structure recognition accuracy to the sub - millimeter level, reducing the error caused by brain shift in traditional navigation (error ≤ 1.8 mm).

[0045] By replacing traditional multi - physical display screens, eliminate the cervical spine burden caused by frequent switching of the doctor's line of sight. At the same time, local magnification based on pupil tracking combined with deep learning assists doctors in accurately identifying key structures such as microvessels and nerves, reducing the rate of misoperation.

[0046] Multi - modal fusion registration uses SLAM to compensate for intraoperative head displacement in real time through an IMU sensor, achieving sub - millimeter - level spatial registration between the virtual image and the real surgical field. Through the fusion of SLAM + IMU sensors, ensure that the spatial registration error between the virtual image and the real surgical field ≤ 2 mm, solving the problem of image drift caused by head movement in traditional AR.

[0047] Eye tracking uses an infrared pupil capture system, dynamically adjusts the boundary of the magnified area in combination with the position data of surgical instruments, and sets an anti - accidental touch mechanism to avoid unintentional operations. The infrared pupil capture system achieves a focus positioning accuracy of ±0.5°, dynamically adjusts the magnification boundary in combination with the position data of surgical instruments, and avoids blind spots in the surgical field. Determine unintentional operations (such as blinking or brief line - of - sight deviation) through thresholds, reducing the false trigger rate to < 5%.

[0048] The teaching assistance module includes a case database and a surgical video playback function, supports the display of operation key points in the form of holographic annotations, and is used for the skill training of young doctors. Through a 3D interactive annotation model (such as the annotation of blood vessel ligation angle), shorten the skill - mastering cycle of young doctors.

[0049] It also includes the following steps:

[0050] S21: In the surgical scene under the microscope, the AR glasses are linked and calibrated with the microscope optical system, allowing doctors to freely adjust their head postures on the premise of maintaining a stable surgical field. The cervical spine activity range is increased by 60%, and the neck load during a single surgery is reduced by 40%;

[0051] S22: Identify the usage status of operating room equipment through computer vision, automatically turn off idle physical display screens, optimize the wind speed distribution model of the laminar flow purification system, increasing the operating room space utilization rate by 50% and reducing the purification energy consumption by 30%;

[0052] S23: Adopt a split - type AR glasses design, equipped with diopter - adjustable lenses that can be quickly disinfected and an optical waveguide module with a light transmittance of ≥85%, support continuous use for 4 hours without visual fatigue. The optical waveguide module cooperates with the quickly disinfected lenses to support seamless connection of consecutive surgeries, and the equipment turnover rate is increased by 2 times.

[0053] The three - dimensional surgical navigation model realizes real - time rendering of holographic images through edge computing nodes, supports multi - terminal collaborative operations including surgical robots and anesthesia monitors, with the latency controlled within 50 ms, supports collaborative operations with da Vinci surgical robots, and the intraoperative decision - making response speed is increased by 20%.

[0054] Example 2

[0055] Further optimize Example 1. Specifically, the teaching assistance module is realized through the following steps:

[0056] S31: Integrate preoperative images, intraoperative endoscope videos, vital sign data, and postoperative pathology reports to construct a standardized case database;

[0057] Adopt a deep - learning algorithm to automatically segment anatomical structures and generate an interactive 3D annotation model;

[0058] Extract voice annotations in the surgical video through natural language processing technology and synchronize them with the operation screen timeline;

[0059] S32: Based on eye - movement tracking data, highlight teaching key points in the AR projection interface in real - time; The standardized case database integrates preoperative images, intraoperative vital signs, and postoperative pathology data to provide a training basis for AI - recommended surgical plans. The difference analysis module significantly reduces the cognitive deviation of young doctors for key steps.

[0060] It also includes: The surgeon dynamically controls the display priority of teaching content through gesture swiping or voice commands;

[0061] The AR glasses at the trainee end restrict annotation interference during key surgical stages and only allow the teaching playback function to be called in the paused state;

[0062] The surgical video is automatically encrypted and stored in the edge server, and blockchain technology is used to record data access logs to ensure compliance with HIPAA / GDPR privacy specifications. Blockchain encrypted storage and access log auditing meet HIPAA / GDPR requirements, and the risk of data leakage is reduced by 90%. It can achieve teaching interference control. During key stages, such as vascular anastomosis, teaching annotations are automatically blocked, and the operation error rate of the surgeon is reduced by 25%.

[0063] From the above, it can be seen that:

[0064] Technical problems addressed by the present invention: However, the existing AR technology still has some limitations in surgical applications: the existing technology has problems such as unclear display and inaccurate positioning, which make it difficult for doctors to obtain accurate information and guidance during the operation. At the same time, there is a phenomenon of blocking the display of the real surgical field of view, making it easy for medical staff to miss important visual cues in the real environment when observing virtual information, and unable to pay full attention to the operation and the patient's condition. By adopting the technical solutions of the above embodiments, through the above settings, this application will surely solve the above technical problems. At the same time, the following technical effects are achieved:

[0065] 1. The present invention precisely integrates virtual images with the real surgical field of view through AR technology, providing intuitive and clear surgical guidance for doctors. This not only greatly improves the accuracy of the operation but also effectively reduces the surgical risks caused by improper operation or positioning errors, significantly increasing the success rate of the operation.

[0066] 2. The present invention realizes sub-millimeter intraoperative positioning through multimodal image fusion combined with dynamic three-dimensional navigation, significantly reducing the risk of misoperation. And eye-controlled magnification and real-time warning reduce the frequency of intraoperative adjustments, shortening the operation time. At the same time, the virtual screen and microscope linkage technology reduces the cervical spine burden, increases the head movement range of medical staff, and reduces the neck load of medical staff.

[0067] 3. By comparing holographic annotation and operation trajectories with traditional teaching methods, the present invention can significantly shorten the doctor training cycle, can automatically identify idle equipment, improve space utilization rate, reduce equipment maintenance costs, and can comprehensively improve surgical accuracy, efficiency and the universality of medical resources.

[0068] In the present invention, unless otherwise clearly specified and limited, terms such as "installation", "connection", "connection", "fixation" and other terms should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or integrated; it can be a mechanical connection, an electrical connection or communicable with each other; it can be directly connected, or indirectly connected through an intermediate medium, and can be the communication inside two components or the interaction relationship between two components, unless otherwise clearly limited. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific situations.

[0069] Obviously, the embodiments described above are only a part of the embodiments of the present invention, rather than all of them. The preferred embodiments of the present invention are shown in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosed content of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. Any equivalent structure that makes use of the content of the specification and drawings of the present invention, directly or indirectly applied in other related technical fields, is equally within the scope of the patent protection of the present invention.

Claims

1. A method for using AR intelligent vision in surgical operations, characterized in that: The following steps are involved: S1: The patient's anatomical structure image is acquired in real time through AR glasses, and multi-modal fusion registration is performed with preoperative CT, MRI and 3D ultrasound image data to generate a dynamic 3D surgical navigation model; S2: The virtual surgical navigation information is superimposed on the real surgical field and displayed in the perspective module of the AR glasses, allowing doctors to display physiological parameters, imaging data and surgical path layout in a split-screen manner on the AR projection interface through gesture sliding or voice commands; S3: Based on eye tracking to capture the doctor's focus area, combined with deep learning algorithms, the target anatomical structure is automatically magnified. The magnification is dynamically adjusted from 5 to 77 times, with an error of ≤1.8mm. S4: Integrated teaching assistance module, real-time display of patient image annotations, anesthesia depth indicators and surgical step annotations on the AR projection interface, supporting hierarchical authority management to prevent information from interfering with the surgeon's operation.

2. The method for using AR intelligent vision in surgical operations according to claim 1, characterized in that: The multimodal fusion registration adopts SLAM to compensate for the intraoperative head displacement in real time through the IMU sensor, thereby achieving sub-millimeter spatial registration of the virtual image and the real surgical field.

3. The method for using AR intelligent vision in surgical operations according to claim 1, characterized in that: The eye tracking system uses an infrared pupil capture system, combined with surgical instrument position data to dynamically adjust the boundaries of the magnified area, and sets an anti-mistouch mechanism to avoid unintentional operations.

4. The method for using AR intelligent vision in surgical operations according to claim 1, characterized in that: The teaching auxiliary module includes a case database and surgical video playback function, and supports the display of key operation points in the form of holographic annotations for skill training of young doctors.

5. The method for using AR intelligent vision in surgical operations according to claim 1, characterized in that: The following steps are also included: S21: In the surgical scene under the microscope, AR glasses are calibrated in conjunction with the microscope optical system, allowing doctors to freely adjust their head posture while maintaining a stable surgical field of view. The range of motion of the cervical spine is increased by 60%, and the neck load in a single operation is reduced by 40%. S22: Identify the use status of operating room equipment through computer vision, automatically turn off idle physical display screens, and optimize the wind speed distribution model of the laminar flow purification system, so that the space utilization rate of the operating room is increased by 50% and the purification energy consumption is reduced by 30%; S23: It adopts a split AR glasses design, equipped with a diopter adjustment lens that can be quickly disinfected and an optical waveguide module with a transmittance of ≥ 85%, supporting 4 hours of continuous use without visual fatigue.

6. The method for using AR intelligent vision in surgical operations according to claim 1, characterized in that: The three-dimensional surgical navigation model realizes real-time rendering of holographic images through edge computing nodes, supports multi-terminal collaborative operations including surgical robots and anesthesia monitors, and controls the delay within 50ms.

7. The method for using AR intelligent vision in surgical operations according to claim 4, characterized in that: The teaching auxiliary module is implemented by the following steps: S31: Integrate preoperative images, intraoperative endoscopic videos, vital signs data, and postoperative pathology reports to build a standardized case database; Use deep learning algorithms to automatically segment anatomical structures and generate interactive 3D annotated models; The voice annotations in the surgical video are extracted through natural language processing technology and synchronized with the timeline of the operation screen; S32: Based on eye tracking data, the key points of teaching in the surgical field are highlighted in real time on the AR projection interface; Superimpose the difference analysis between the preoperative planning path and the actual operation trajectory during the operation to help young doctors understand the operation deviation; Embed expert operation demonstration video.

8. The method for using AR intelligent vision in surgical operations according to claim 7, characterized in that: Also includes: The surgeon can dynamically control the display priority of teaching content through gesture sliding or voice commands; The trainee's AR glasses limit the interference of annotations in the key stages of the surgery, and only allow the teaching playback function to be called in the pause state; Surgical videos are automatically encrypted and stored on edge servers, and blockchain technology is used to record data access logs to ensure compliance with HIPAA / GDPR privacy regulations.

Images